Title: Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
Authors: Sergey Zagoruyko
Nikos Komodakis
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Supervised Learning

Abstract: Attention plays a critical role in human visual experience. Furthermore, it has recently been demonstrated that attention can also play an important role in the context of applying artificial neural networks to a variety of tasks from fields such as computer vision and NLP. In this work we show that, by properly defining attention for convolutional neural networks, we can actually use this type of information in order to significantly improve the performance of a student CNN network by forcing it to mimic the attention maps of a powerful teacher network. To that end, we propose several novel methods of transferring attention, showing consistent improvement across a variety of datasets and convolutional neural network architectures.


Review discussion:
Here are a few short pre-review questions
1) Why don't you use the same settings for the activation-based and gradient-based attention transfer on the CIFAR-10, it would be nice to compare those two approaches as it not clear what are their relative advantages/drawbacks.
2) Intermediate hints in Fitnet have been introduced to ease the optimization of the student network. It would be nice to see if the attention transfer have similar effect, or if it provide more benefits in term of generalization.
Also, you mention that you have experimented with Fitnet hint. It would be nice to report the obtained results, and do a comparison with your activation-based attention transfer as the two approaches are close.
3) Gradient attention transfer does not seems to improve upon knowledge distillation in Table 3. Do you have an intuition explaining this result?
4) What value are you using for the hyperparameter \beta (controlling the attention transfer cost). Did you try decaying it during training?
Thanks for the questions, they were indeed very helpful and we updated the paper with new experiments and better explanations. Below are our answers:
1) It is unclear how batch normalization should work with double backpropagation.
So far we tried to fix it on second bprop, and couldn’t reproduce the behaviour of a network without batch normalization. Maybe a better way would be do a fprop+bprop+double bprop with fixed statistics after a normal fprop+bprop, but we haven’t tried that yet. We mention this in the updated version.
2) Right, they have a similar effect, both AT and full-activation transfer (similar to FitNets) speed up training significantly (see figure 7b in the updated paper), but AT gives better final accuracy.
Since the setup is quite different from original FitNets paper, we tried to do full-activation transfer with 1x1 convolutional layers to match tensor shapes, and l_2 normalization. We added these results to the paper. For ResNets the improvements from such approach are minimal (see table 1 in the updated paper, F-ActT column), and for NIN the improvement is slightly better, which aligns with FitNet results since the architecture is more similar.
3) It is difficult to compare the two, because teacher information is propagated from an opposite side of the network in gradient attention transfer, still it’s able to improve student as much. We think that gradient attention transfer could be improved by making teacher gradient information less noisy and more stable. We actually tried to aggregate gradients from several teacher evaluations, and this seems to improve gradient attention transfer already, but we didn’t include this due to lack of space.
4) We set it to ~0.1, 10^3 divided by the number of elements in attention map and batch size. We tried to decay it, it doesn’t improve results when only AT or full-activation transfer (similar to FitNet) used, but is needed to combine well with KD. We mention this in the updated version.
F is a mapping from a 3 tensor to a matrix defined in Eq. (1).
So I assume I assume the Lp norm in Eq. (2) is in fact matrix norm.
So the question is do you solve for the largest eigenvalue of F(A)^T F(A) in order to normalize the attention map F(A) ?
Also are the absolute value notations in section 3.1 element-wise or matrix-wise?
Sorry for the ambiguous notation, the operations are in fact element-wise, so it’s not a matrix norm. We updated the paper to reflect this.
This paper proposes to investigate attention transfers between a teacher and a student network.
Attention transfer is performed by minimising the l2 distance between the teacher/student attention maps at different layers, in addition to minimising the classification loss and optionally a knowledge distillation term.
Authors define several activation based attentions (sum of absolute feature values raise at the power p or max of values raised at the power p). They also propose a gradient based attention (derivative of the Loss w.r.t. inputs).
They evaluate their approaches on several datasets (CIFAR, Cub/Scene, Imagenet) showing that attention transfers  does help improving the student network test performance.
However, the student networks performs worst than the teacher, even with attention.
Few remarks/questions:
- in section 3 authors  claim that networks with higher accuracy have a higher spatial correlation between the object and the attention map.
While Figure 4 is compelling, it would be nice to have quantitative results showing that as well.
- how did you choose the hyperparameter values, it would be nice to see what is the impact of $\beta$.
- it would be nice to report teacher train and validation loss in Figure 7 b)
- from the experiments, it is not clear what at the pros/cons of the different attention maps
- AT does not lead to better result than the teacher.
However, the student networks have less parameters. It would be interesting to characterise the corresponding speed-up. If you keep the same architecture between the student and the teacher, is there any benefit to the attention transfer?
In summary:
Pros:
- Clearly written and well motivated.
- Consistent improvement of the student with attention compared to the student alone.
Cons:
- Students have worst performances than the teacher models.
- It is not clear which attention to use in which case?
- Somewhat incremental novelty relatively to Fitnet
Dear Reviewer,
Below are our answers:
Q: in section 3 authors claim that networks with higher accuracy have a higher spatial correlation between the object and the attention map.
While Figure 4 is compelling, it would be nice to have quantitative results showing that as well.
A: Thanks for the suggestion. We are going to try to look into how to do that for the VOC or COCO dataset.
However, it should be noted that this is in fact a not so trivial task (e.g., due to the need of bounding box or segmentation mask annotations, due to the need to deal with overlapping objects etc.)
Q: how did you choose the hyperparameter values, it would be nice to see what is the impact of $\beta$.
A: We found our method to be quite stable with respect to different values of beta, i.e. in all our experiments we observed that there’s typically a wide range of values which give improvements close to optimal;
Q: it would be nice to report teacher train and validation loss in Figure 7 b)
A: Thanks, we are going to add this;
Q: from the experiments, it is not clear what at the pros/cons of the different attention maps
A: We are currently running attention transfer experiments using grad-based and activation-based attention maps under the same conditions, and going to update the paper as soon as we have the results;
Q: AT does not lead to better result than the teacher.
However, the student networks have less parameters. It would be interesting to characterise the corresponding speed-up. If you keep the same architecture between the student and the teacher, is there any benefit to the attention transfer?
A: The baselines we choose have very different numbers of parameters, thus making it very difficult for student to have better or even the same accuracy as teacher, as network performance largely depends on this number. This was shown in a number of recent works, including our Wide ResNet paper. It can be shown though that AT leads to drastic improvements in speed and number of parameters needed to achieve the same accuracy.
For example, to achieve 7.5% on CIFAR-10 one would need a ResNet with 300k (e.g. WRN-16-1.3) parameters, whereas AT+KD achieves it with only 160k parameters, resulting in about 2x less parameters and a much more efficient network;
Q: Somewhat incremental novelty relatively to Fitnet
A: We would like to note that one main goal/contribution of this work is to convey the idea that the use of attention transfer during training can be an important factor for improving a network’s performance.  In a case like this, the student is forced to mimic only a small “summary” of the teacher’s data, as opposed to FitNets that try to mimic the full activation maps.
Yet we show that our method achieves much better results, which we consider as a very interesting finding.
Thanks.
The paper presented a modified knowledge distillation framework that minimizes the difference of the sum of statistics across the a feature map between the teacher and the student network. The authors empirically demonstrated the proposed methods outperform the fitnet style distillation baseline.
Pros:
+ The author evaluated the proposed methods on various computer vision dataset
+ The paper is in general well-written
Cons:
- The method seems to be limited to the convolutional architecture
- The attention terminology is misleading in the paper. The proposed method really just try to distill the summed squared(or other statistics e.g. summed lp norm) of  activations in a hidden feature map.
- The gradient-based attention transfer seems out-of-place. The proposed gradient-based methods are never compared directly to nor are used jointly with the "attention-based" transfer. It seems like a parallel idea added to the paper that does not seem to add much value.
- It is also not clear how the induced 2-norms in eq.(2) is computed. Q is a matrix \in \mathbb{R}^{H \times W}  whose induced 2-norm is its largest singular value. It seems computationally expensive to compute such cost function. Is it possible the authors really mean the Frobenius norm?
Overall, the proposed distillation method works well in practice but the paper has some organization issues and unclear notation.
Dear Reviewer,
Below are our answers:
Q: The method seems to be limited to the convolutional architecture
A: Fist, we would like to note that our method can be directly applied even to non convolutional  architectures (such as MLPs) given that our definitions of attention are also valid for such architectures. We’re also currently looking into applying attention transfer for recurrent neural networks in NLP tasks. Independently of the above, we consider that the convolutional architecture is one of the most important and widely used type of network architectures (with a really huge impact on several fields).
Q: The attention terminology is misleading in the paper. The proposed method really just try to distill the summed squared(or other statistics e.g. summed lp norm) of  activations in a hidden feature map.
A: We respectfully disagree with this statement. In our view a spatial attention map is supposed to indicate how important each spatial location of the input layer (or of an intermediate layer) is w.r.t. the output computed by the network (in other words, roughly how much focus the network puts per spatial location). Here we provide two simple ways of defining such an attention map, an activation based one and a gradient based one.
Q: The gradient-based attention transfer seems out-of-place. The proposed gradient-based methods are never compared directly to nor are used jointly with the "attention-based" transfer. It seems like a parallel idea added to the paper that does not seem to add much value.
A: A main contribution of this work is the idea that attention transfer can be very useful for improving the performance of a network. We therefore wanted to show that this is indeed true even when using different ways for defining attention, which is why we included results for both activation based and gradient based attention.
Nevertheless we agree that it will be useful to compare the  two methods under the same conditions, and, to that end, we are going to update the paper with new experiments.
Q: It is also not clear how the induced 2-norms in eq.(2) is computed. Q is a matrix \in \mathbb{R}^{H \times W}  whose induced 2-norm is its largest singular value. It seems computationally expensive to compute such cost function. Is it possible the authors really mean the Frobenius norm?
A: Sorry for the confusion, it is indeed a Frobenius norm. Essentially, in eq. (2)  we consider that all attention maps are in vectorized form, in which case the standard norm notation for vectors applies. We have updated the paper respectively.
Thanks.
Regarding "Q: The gradient-based attention transfer seems out-of-place. The proposed gradient-based methods are never compared directly to nor are used jointly with the "attention-based" transfer. It seems like a parallel idea added to the paper that does not seem to add much value.":
We trained thin NIN with activation based AT in the same conditions as grad-based experiments, and, interestingly, it converges to better accuracy than KD or grad-based methods, 11.2% error on CIFAR-10 (table 3 in the paper).
I like the additional experiments author have done to compare act-AT and grad-AT in Table 3. It pretty much shows the act-AT is a much superior method comparing to the computationally expensive grad-AT that does not provide any benefit over KD baseline.
Therefore, the grad-AT does not add much value to the paper and require an additional adhoc horizontal flip trick. The story will be much cleaner if the paper just focuses on summed square attention map and show it is a much simpler and more effective way to achieve fitnet like transfer. The authors have addressed my concerns from the previous review. I have changed my review score accordingly.
The paper proposes a new way of transferring knowledge.
I like the idea of transferring attention maps instead of activations.
However, the experiments don’t show a big improvement compared with knowledge distillation alone and I think more experiments are required in IMAGENET section.
I would consider updating the score if the authors extend the last section 4.2.2.
Dear Reviewer,
Below is our answer:
Q: However, the experiments don’t show a big improvement compared with knowledge distillation alone and I think more experiments are required in IMAGENET section.
A: We’ve updated the paper with an ImageNet experiment where we used two attention losses and trained  from scratch, and got 1.1%/0.8% top1/top5 accuracy improvement over baseline ResNet-18, ~30% reduction in error difference between teacher and student. We think that this is a remarkable result, considering our very limited computational resources (8 GPUs), and are sure it could be easily further improved by using more AT losses, tuning hyperparameters, or using more powerful teachers.
We are currently running experiments with:
1. 	All four AT losses, one for each group of residual blocks;
2. 	KD baseline;
and will update the paper as soon as we get the results.
Thanks.
Although we tried several things, we were not able to achieve positive results with KD on ImageNet. With ResNet-18/ResNet-34 student/teacher pair it actually hurts convergence with the same hyperparameters as on CIFAR. As it was reported that KD struggles to work if teacher and student have different architecture/depth (we observed the same on CIFAR), we tried using the same architecture and depth for attention transfer:
ResNet-18 as student and trained ResNet-18 as teacher
ResNet-34 as student and trained ResNet-34 as teacher
On CIFAR both AT and KD work well in this case and improve convergence and final accuracy, on ImageNet though KD converges significantly slower (we didn’t train it until the end, because we can do only 1-2 experiments at a time). We also couldn’t find in the literature any reported results of FitNets, KD or similar methods on ImageNet, and we also contacted the authors of FitNets, as well as a few people who had similar experience with KD on ImageNet. Given the above, we can assume that our proposed activation-based attention transfer approach is the first knowledge transfer method to be successfully applied on ImageNet.
We would like to thank the reviewers for their comments and for providing us with valuable feedback. We already updated the paper with new results on ImageNet, and are also going to further update the paper based on the reviewer comments as explained in our responses below.
We also plan to release the code for our experiments next week.
We updated the paper with the following new experiments:
- Activation based AT in the same conditions as grad-based AT: activation-based AT gives better results than KD or grad-based AT;
- KD on ImageNet: we were unable to get any improvement over baseline ResNets.
We elaborate on both experiments in our corresponding responses below.
Also, the code for CIFAR and ImageNet experiments is publicly available: https://github.com/szagoruyko/attention-transfer
We plan to add more experiments in the future.

Meta-review:
role: PC
data: comment: Important task (attention models), interesting distillation application, well-written paper. The authors have been responsive in updating the paper, adding new experiments, and being balanced in presenting their findings. I support accepting this paper.

Decision: Accept (Poster)