On the Surprising Effectiveness of Attention Transfer for Vision Transformers
Authors: Alexander Cong Li
Yuandong Tian
Beidi Chen
Deepak Pathak
Xinlei Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Transformers
self-attention
attention
transfer
pre-training
fine-tuning
understanding
analysis
Primary area: machine_vision

Abstract: Conventional wisdom suggests that pre-training Vision Transformers (ViT) improves downstream performance by learning useful representations. Is this actually true? We investigate this question and find that the features and representations learned during pre-training are not essential.  Surprisingly, using only the attention patterns from pre-training (i.e., guiding how information flows between tokens) is sufficient for models to learn high quality features from scratch and achieve comparable downstream performance. We show this by introducing a simple method called attention transfer, where only the attention patterns from a pre-trained teacher ViT are transferred to a student, either by copying or distilling the attention maps. Since attention transfer lets the student learn its own features, ensembling it with a fine-tuned teacher also further improves accuracy on ImageNet. We systematically study various aspects of our findings on the sufficiency of attention maps, including distribution shift settings where they underperform fine-tuning. We hope our exploration provides a better understanding of what pre-training accomplishes and leads to a useful alternative to the standard practice of fine-tuning.

On the Surprising Effectiveness of Attention Transfer for Vision Transformers
Alexander C. Li Carnegie Mellon University
Yuandong Tian FAIR
Beidi Chen Carnegie Mellon University
Deepak Pathak
Xinlei Chen
Carnegie Mellon University
FAIR
Abstract
Conventional wisdom suggests that pre-training Vision Transformers (ViT) im- proves downstream performance by learning useful representations, Is this actually true? We investigate this question and find that the features and representations learned during pre-training are not essential. Surprisingly, using only the attention patterns from pre-training (i.e., guiding how information flows between tokens) is sufficient for models to learn high quality features from scratch and achieve com- parable downstream performance. We show this by introducing a simple method called attention transfer, where only the attention patterns from pre-trained teacher ViT are transferred to a student, either by copying or distilling the attention maps. Since attention transfer lets the student learn its own features, ensembling it with a fine-tuned teacher also further improves accuracy on ImageNet. We systemat- ically study various aspects of our findings on the sufficiency of attention maps, including distribution shift settings where they underperform fine-tuning. We hope exploration provides better understanding of what pre-training accomplishes and leads to useful alternative to the standard practice of fine-tuning, Code to reproduce our results is at http:///hhhhcoooo//llllaaaa/tlte
Introduction
Pre-training has emerged as a dominant paradigm in machine learning and has significantly improved performance variety of tasks (27
IT
2 222 In computer vision in particular, self-supervised
22 and weakly supervised methods [40] have enabled learning from large amounts of images. It is widely accepted that these methods work because they teach models useful features that are relevant for downstream tasks. But is this story actually true? Perhaps there another capability learned during pre-training that is sufficient explain its benefits.
representation learning methods [21] 6
4
In this paper, we present an alternative explanation: pre-training teaches the model how information should be routed between tokens. We specifically focus on Vision Transformers (ViT) [12, not only because they are the most popular architecture for scaling, but also because Transformers explicitly decouple this information flow. Inter-token communication is solely fulfilled by attention, while the remaining bulk of computation are intra-token operations that are applied to each token independently. In contrast, other architectures such as ConvNets 33 20 simultancously expand the receptive fields and extract the features, making it difficult to isolate the effect of information flow. We hypothesize that the features computed by the intra-token operations pre-training, and that the pre-trained attention maps are typically sufficient for downstream tasks.
not essential to explain the benefits of
We test our hypothesis by introducing new set of methods called attention transfer. Concretely, we treat a pre-trained ViT as the teacher and train student model for downstream tasks while
Work done during an internship at FAIR.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

transferring only the attention patterns from the teacher. In contrast to the common fine-tuning paradigm of transferring all the weights (which mixes the effect of features and attention maps), only the inter-token flow is transferred. In this way, the student must learn features from scratch, while isolating the benefits of the attention maps learned during pre-training.
We study two types of attention transfer. The transfer (seratch)
full transfer (fne--une)
first Attention Copy, which directly "copy- and-pastes" the attention maps. The learning fully decoupled, as inter-token computation is entirely from the teacher, and the student only learns intra-token patterns routed by the teacher's attention maps. This is well-suited as scientific probe, but is less practical since both networks need to be forwarded during the inference. The second is Attention Distillation, where the student simply distills attention pat- terns from the teacher, whose attention maps are no longer used after training. This is prac- tical, but also helps identify the importance of the teacher's 's inter-token information flow.
77.8%
Atteation Copy
85.1
1080%
Atteation Distillaiin
85.0
857
Figure 1: Using only attention is sufficient for full performance. By copying the attention maps (top) from MAE 22] pre-trained ViT-L 12), VÍT-L can reach a top-1 accuracy of 85.1 on ImageNet-IK IK [I0] recovering 77.8% of the gap between no transfer (training from scratch, 83.0) and full transfer (fine- tuning all the weights, 85.7). Distilling attention maps (bottom) can even fully match MAE weight tuning while only transferring the inter-token flow.
While both attention transfer variants are straightforward, we find them highly effective. (illutttilssttttes this with a ViT-L. [12] pre-
trained using Masked Autoencoding (MAE) (22 Compared to no transfer (training from scratch) and full transfer (fine-tuning all the MAE weights), Attention Copy can close most of the gap in performance, whereas Attention Distillation can match the fine-tuning accuracy on ImageNet- I classification [TO. This is achieved by only transferring the inter-token flow from the same model. Furthermore, since attention transfer requires the student to learn features from scratch, those features are significantly different from the teachers' (Figure 5) and improve ImageNet- IK accuracy score to 86.3 (+0.6) when ensembled with the teacher (Figure 6).
To summarize, we make the following contributions:
Detailed analysis on the sufficiency of attention maps. We find that solely using the pre- trained attention patterns is typically sufficient to achieve the same downstream accuracy as fine-tuning on ImageNet-IK: IK. Furthermore, we observe practical benefits, as ensembling with attention transfer significantly improves ImageNet performance. 'his calls into question the commonly -ennoney-bellived story that pre-training is only about feature learning. While our main observation is robust w.r.t. different models and pre-training methods, we do find settings where pre-trained features are indeed necessary to realize the full gains from pre-training. Our bare-minimum solution for attention transfer is more affected by data distribution shifts compared to weight tuning. Section 4 presents extensive analyses to better understand the behaviors of attention transfer. They are i) partial transfer with subset of layers or heads; ii) variants of our method that transfer other antention-relatee activations; and importantly, iii) various ways to verify that the student is not just re-learning the teacher model. Section [ systematically tests how well our findings apply across a variety of pre-training and fine-tuning datasets, pre-training methods, model sizes, and tasks.
A Attention transfer methods. We introduce Attention Copy and Attention Distillation, which are methods to train a ViT on a downstream task while utilizing only the attention maps of pre-trained teacher ViT. These methods help us understand the role of the features versus the attention patterns learned during pre-training. With further research, attention transfer could offer potential alternative to the decade-long practice of fine-tuning pre- trained vision models 16 12 22. Nearly all aspects of the fine-tuning pipeline have been thoroughly examined, suggesting probable saturation of recipes. Weight sharing can also face security risks (e.g., white-box attacks (17). We hope our systematic examination of attention transfer sheds new light how to leverage pre-trained ViTs, and will help establish this approach as an effective alternative when weight transfer is less applicable.

---

student
teacher
student
tencher
b
MLP block
MLP block
MLP block
MLP block
MSA
dietill
sélcation maup
allestion teep
atlictiion maap
X
XL
X.
XL
Attention Copy
Attention Distillation
Figure 2: Two types of Attention transfer for Vision Transformers. Attention Copy (left): We simply icopy-and-paste the attention maps from pre-trained teacher model to randomly initialized student one. Other weights of the student are then trained via supervised learning. This fully decouples inter-token learning (from the teacher) and intra-token learning (in the student); but is less practical. Attention Distillation (right): The student computes its own attention maps, with an additional cross-entropy loss to distill patterns from the teacher during training. The teacher is no longer used during inference. number of heads; L: number Transformer layers.
2
Attention Transfer
2.1 Attention Preliminaries
To work with Vision Transformer (ViT) [12. image is first "patchified" into N tokens. Their intermediate activations are represented as a sequence X= 22, ,xx] ER, where C is the embedding dimension. The self-attention (57) mechanism mainly introduces three learnable parameters W Ws, WoERCxC/n (H is the number of heads). Q=XW, is often referred to as the queries, K=X We as the keys, and V=XW, XW, the values. Then the attention function is defined as:
Jan softmax (QK
attention
where the softmax function computed per query for the attention map. Attention maps determine how the values from other tokens are aggregated, and with multiple heads, each token uses multiple attention distributions within the same Multi-headed Self-Attention (MSA) block.
For an L-layer Transformer, MSA blocks are interleaved with MLP blocks, and each Transformer layer contains one of each block type. Most operations intra-token computations, which applied independently to each token: value and projection matrices, normalization layers M. and MLPs. The only inter-token computation is applying the attention map softmax (QK which the only way for information to flow between tokens. Transformers are unique because their inter- and intra-token computations are decoupled; however, the relative importance each type of operation is not well understood, and Transformers are typically trained by jointly fine-tuning all the weights.
Deviating from the common practice of joint weight tuning, we propose two attention transfer methods with the goal of exploring decoupled training for ViTs, described next.
2.2 Attention Copy
In this setup, utilize two separate networks: pre-trained teacher network that only does a forward pass to compute its attention maps, and a student network that directly copies the attention maps from the teacher but computes all of the other activations. The student's weights and trained via back- propagation, while the teacher's 's weights are kept frozen. This setting fully isolates the attention maps from the features that they applied to, and thus is ideal for measuring the utility of pre-trained attention patterns when the student network learns to perform other tasks (e.g., image classification).
randomly initialized
We call this method Attention Copy, as we "copy-and-passe the attention maps from teacher to student. Figure 2 (left) shows diagram of this approach. Note that it requires forward passes

---

through both the teacher and student networks during the inference time. Given the extra computation, Attention Copy is not meant to be an entirely practical method. We mitigate this issue next.
2.3 Attention Distillation
In Attention Distillation, the teacher network is only utilized at the training time. Given each training example, we forward both networks in parallel, with the student also computing its own attention maps. But besides the task-driven loss, we also enforce a distillation loss between student's 's attention maps and the teacher's counterparts as (soft) targets. Formally, using Q.K. for the student and QrKy for the teacher, the loss is then defined as:
dist = H. softmax(Q. softmax(Q,Kk)))
where H. computes the cross entropy. As there can be multiple heads and layers in a Transformer, we simply sum up all the losses from wherever attention distillation is applied. Again, the student is trained via back mack-propagation. Figure 2 (right) shows the diagram of Attention Distillation.
Compared Attention Copy, Attention Distillation is much more practical. After training. the teacher is no longer needed, and the student can be used as a standalone model. Compared to training ViTs from scratch, the only addition is the distillation loss, meaning most of the optimization (e.g., learning rate, momentum) and regularization (e.g.. weight decay, dropout rate (50)) hyperparameters hyperparameter X,
follow the scratch recipe with minimal adjustments. It does introduce which weights the distillation loss and balances it with the task loss.
Attention Distillation can be viewed as a form of generalized knowledge distillation, but it has several key differences from the design proposed by Hinton et al. [26]. Attention Distillation trains the student match the teacher's 's intermediate attention maps, not the final teacher output. This gives the flexibility of distilling from models trained on any task, not just models trained on the same final task. This property is well-suited for today's "pre-train and transfer" paradigm, where the pre- pre-training task (e.g., reconstruction) and the downstream task (e.g., classification) are usually different. However, Attention Distillation does add the constraint that the architecture needs to compute attention maps. We leave experimenting on this idea for other architectures as future work.
Overall, while fancier designs can be used for both Attention Copy and Attention Distillation, choose to keep them simple for cleaner assessments of their effectiveness.
2.4 Connection Transformer Training Dynamics
Our investigation is also linked to recent attempts to theoretically understand the training dynamics of Transformers. Specifically, the inter-token flow encoded in the pre-trained attention maps can be regarded as discovered latent hierarchy from the dataset. Self-attention can quickly capture frequently co-occurring token pairs 31 52 However, more occasional co-occurrences need to be explained by the top-level hierarchy, rather than directly learned in the lower levels (53). This due to many potential spurious correlations [30], especially the over-parameterized setting. Transferring attention maps from a trained teacher reduces these spurious inter-token correlations, so the student can focus on intra-token learning (i.e., computing useful features).
Main Results
As featured in Figure I attention transfer is highly effective despite its simplicity. Specifically, we demonstrate this with a ViT-L [12 pre-trained with Masked Autoencoding (MAE) [22 for ImageNet- IK classification (10). Note that this is the signature result that established MAE's ' effectiveness for pre-training: compared to a ViT-L trained from scratch (with an accuracy score of 83.0), fine-tuning the MAE pre-trained on the same dataset results in significant improvement to 85.7
For attention transfer, we use the same pre-trained MAE model as our teacher, and since scratch training can be viewed as no transfer, and fine-tuning weights transfers all the weights, the above two results serve as natural lower and upper bounds for the effectiveness of our attention transfer methods. We make two observations (from Table D and Figure
If not otherwise specified, our results are based faithful reimplementation of the official code in JAX.

---

method
acc.
scratch
83.0
transfer target
acc.
fine-tune
85.7
Q
85.6
attn. copy
85.1
K
84.4
copy from fine-tuned
85.6
V
84.4
attn. distill
85.7
Q. K
85.1
ensemble attn. distill fine-tune
86.3
Table 2: Transfer other attention activations.
Table I: Main results. We show that the pre- trained attention patterns are sufficient to match fine-tuning accuracy on ImageNet. Attention Copy closes most of the gap, and Attention Dis- tillation achieves the same accuracy.
We test copying alternative attention activations other than the attention map softmax All alternatives do better than training from scratch, and transferring queries Q actually does better than transferring the attention map.
Attention Copy can largely close the gap between scratch training and weight fine-tuning
We report an accuracy of 85.1 with Attention Copy. This has largely closed the gap between scratch and full weight tuning (to be precise, 77.8% of the 2.7 percentage point gap). This is surprising, since teacher's s attention maps are frozen after pre-training for a different task (image reconstruction), and the student must learn everything else (the intra-token operations) completely from scratch.
As another upper-bound, also experimented with Attention Copy from the fine-tuned model. This reaches an accuracy score of 85.6 almost matching the teacher's performance (85.7), suggesting that adapting attention maps the specific task still helpful, but not crucial, especially as MAE pre-training is performed on the same data distribution.
Attention Distillation match fine-tuning performance
Even more surprisingly, we find Attention Distillation can achieve 85.7 on par with fine-tuning the VIT-L weights from MAE. Since Attention Distillation and weight tuning both result in the same- sized model, which requires the same compute budget for inference, this result suggests Attention Distillation be effective drop-in replacement for weight tuning when the latter is less applicable (e.g., if weight sharing poses security risks, we can instead send the teacher's attention maps).
We hypothesize that distillation is better than copy because the student choose how closely it matches the teacher attention maps, to better suit the task objective. This is also supported by the 85.6 accuracy of copying from fine-tuned MAE, which has the correct task-specific attention maps.
Analysis
Next, we provide extensive analyses to better understand the effectiveness of attention transfer. Broadly speaking, the explorations are driven by the following two questions:
How important are different activations, layers, and heads? (Section 4.1
iii) Is attention transfer re-learning everything from the teacher? (Section 4.2
4.1 Variants of Attention Transfer
We study four variants of attention transfer. We use Attention Copy within this section, since it is dully-decoupled setting well-suited for scientific analysis.
Transfer a subset of Q, K and V. A natural alternative to transferring attention maps to transfer different activations that come with self-attention (Eq. TD, namely queries Q, keys K, values V. Without loss of generality, if we transfer the teacher's 's Q, the student will compute its own K and V and use them normally. Note that transferring both Q and K is equivalent transferring the map softmax QKTT) Table shows that transferring Q works surprisingly well, and is actually better than transferring the attention map.
We suggest that copying Q gives the model the flexibility to deviate from the teacher attention maps and use attention patterns that are better suited for the downstream task. This is supported by the fact
5

---

8500
64
848
84.0
85.1
85.0
85.J
82.5
$1.9
82.5
812
62.1
80.0
80.3
81.2
80.0
from bottom
27.5
76,
from
77.5
76
774
12
16
number of layers transferred
number of heads transferred
Figure 3: Copy subset of layers. By default, all 24 ViT-I. layers are transferred. Here we only transfer a subset, and find: more layers always helps; and attention maps from top layers are more beneficial than those from bottom layers.
Figure 4: Copy subset of heads. The pre- trained ViT-I. has 16 heads in each MSA block. By default, all of them are transferred. Here we only transfer a subset, and find more heads helps in general, but performance saturates 12 heads.
that copying Q and Attention Copy from the fine-tuned model both achieve the same accuracy of 85.6. Appendix [B3 dives deeper into this hypothesis and finds that the attention maps for copying Q are similar to the teacher's but less constrained than they are in other transfer methods. While more investigation could be done in future work, our findings suggests that the queries Q are more important than the keys, which is consistent with previous findings text sequence modeling where the number of keys and values per layer can be significantly reduced [49].
Finally, we test whether distilling Q could outperform full Attention Distillation. However, Table 3 shows that Q distillation does signif- icantly worse. We hypothesize that this is because it is harder for the student to learn useful keys K while Q is still being learned, and because Attention Distillation already has the flexibility adjust its attention maps to suit the downstream task.
method
acc.
attn. distill
85.7
distill
81.3
Table 3: Attention Distillation outperforms Q distillation.
Partial transfer: layers. We next change the number of Transformer layers transferred, aiming to identify which layers are more valuable from the teacher. The baseline transfers all the layers. In Figure we try transferring attention maps only from the first or last layers. For the remaining layers, the student learns to compute its own attention maps.
We make several observations: i) We find transferring more layers always more helpful. This is bit surprising, as one may expect attention patterns optimized for MAE's patch-wise reconstruction task to be sub-optimal for a high-level recognition task like classification. ii) We find transferring the later attention maps is generally better. In fact, performance roughly saturates when transferring the last 15 attention maps low-level representations, as long as they are told how to combine these features into higher-level ones; but not vice versa. This reinforces the theory from Tian al. [53] that guidance on top-level hierarchy is more important, as there are many more possibilities, and attention transfer can reduce possible spurious correlations in the lower levels.
of total of 24. This indicates that ViTs are capable of learning good
Partial transfer: heads. Finally, we switch back to transferring all the layers, but change the number of heads copied from each MSA block. Specifically, instead of copying the attention map from every head, we can selectively choose to use subset of the teacher's 's heads. The student can then compute its own attention patterns for the unused heads. Figure 4 shows the effect of transferring fewer heads for each layer. Performance improves as we do attention transfer with more heads, though performance almost saturates at 12 out of 16 heads. Note that we simply follow a naïve selection strategy and use the first set of heads; more advanced strategies based on diversity or activation magnitude can potentially improve the robustness as we reduce the number heads.
4.2
Are We Re-Learning the Teacher?
Since attention transfer provides significant amount of information from the teacher (ViT-L attention maps have about 10M activations total per image, see Appendix A. for detailed calculations), a natu- ral question is whether the student performs well because it simply relearns the same representations as the teacher. We thoroughly test this hypothesis different aspects of the student model.

---

1.0
pre-train
altn, copy
CKA with fine-tuned
0.8
scratch
sitn. distill.
Figure 5: CKA representation similarity to the fine-tuned model. We CKA [32 to measure the layer-wise similarity between representations learned in different models against the fine-tuned MAE model. Higher means more similar. We find that attention transfer methods quite dis- simlar to the fine-tuned model, with roughly the same CKA an independent scratch model.
0.6
0.4
0.2
10
15
20
layer index
Figure 6: Ensemble accuracy with the fine-tuned model. We plot the accuracy of ensembling our at- tention transfer models and fine-tuned MAE. This measures model prediction similarity with the fine- tuned model. The ensemble yields notable accuracy gains over the base model, reaching up to 86.3. This is even higher than ensembling two separate fine- tuned MAE models² (86.2) and indicates that the attention transfer models are less correlated with the fine-tuned model.
fine tune
0.5 86.2
individual
X ees. with fine-tuned
attn. copy
x 86.2
ottn. distill-
86.3
scratch-
2.7
85.7
accuracy
Representation similarity. One way that the student can reproduce the teacher is by learning the same intermediate features. We measure this using Centered Kernel Alignment (CKA) [32),
similarity measure for representations that has been shown to be effective even for networks trained with different initializations architectures. CKA is a layer-wise comparison that ranges from 0 (completely dissimilar) to (identical) and is invariant to orthogonal linear transformations and isotropic scaling of the features. Figure shows the CKA between our finettune model and our attention transfer methods. We also show the pre-trained model and a VIT-L trained from scratch for reference. We compute CKA with respect to the fine-tuned model, even though copy distill from the pre-trained MAE model, since the features change significantly during fine-tuning to become more task-specific. Overall, Attention Copy and Attention Distillation fine-tuned MAE model, following the same similarity trend as the scratch model. Our sanity check passes, CKA shows that pre-trained and fine-tuned MAE have very similar representations in the early layers. This is expected since fine-tuning with layer-wise learning rate decay [ means the carliest layers change very little during fine-tuning.
both quite dissimilar the
Prediction similarity and ensembling, Our CKA analysis may not catch some similarity of the intermediate representations, as CKA does detect all forms of the same information (e.g., invertible nonlinear transforms) [32]. Since intermediate representations may not tell the full story, we also examine the similarity of the network outputs. We measure this using network ensembling: given softmax predictions Pn from the fine-tuned model and Poter from another model, we test the accuracy of their average (Pn + Pottee) /2. The more independent the model predictions are, the higher their ensemble accuracy is. Figure 6 compares accuracy before and after ensembling with the fine-tuned model. Attention transfer is dissimilar enough achieve high ensemble accuracy, and ensembling Attention Distillation with fine-tuned MAE achieves 86.3, +0.6 over the fine-tuned MAE model.
Finally, Appendix B.4 visualizes the attention maps learned by Attention Distillation and shows that they match for distilled attention blocks but are drastically different for layers that are not distilled.
5 Generalization and Limitations
In this section, we test how well our findings on attention transfer apply across a variety pre-training and fine-tuning datasets, pre-training methods, model sizes, and tasks.
NNew result: we tine-tune two MAE models (85.8, 86.0) from shared pre-trained trained MAE initialization, then ensemble them to achieve 86.2. These use a new pre-trained MAE model, as the original MAE checkpoint used in this paper was accidentally deleted. Note that these fine-tuned models are stronger than the original fine-tune (85.7 7), yet underperform the ensemble of the original fine-tuned MAE and the Attention Distillation model.
7

---

pre-training data
tune
copy
eval. data
scratch
tune
copy
distill
ImageNet
85.7
85.1
iNat 2017
49.7
75.9
69.1
69.3
ImageNet-22K
85.5
84.4
iNat 2018
64.3
79.9
71.8
74.1
COCO
85.2
83.1
iNat 2019
66.2
83.8
77.9
80.0
Table 4: Different pre-training datasets. We pre-train MAE on more datasets, and then either fine-tune or copy for ImageNet- IK classifica- tion. Attention transfer works well when the data distribution stays stable, but its effectiveness is more negatively affected by distribution shifts.
Table Long-tail recognition on iNaturalist, with ImageNet-1K pre-trained MAE. We tune weights or do attention transfer on iNaturalist, and we again find attention transfer is worse than tuning weights when the pre-training dataset different from the downstream dataset.
out-of-distribution evaluation
scratch
tune
copy
distill
ImageNet-A 24
32.0
56.5
48.9
54.3
ImageNet-R R
23
51.9
59.6
57.5
56.8
ImageNet-S
660
38.0
45.2
43.1
42.9
ImageNet-V2 47
72.4
76.4
75.5
75.9
robustness. We take two models that achieve the same accuracy on ImageNet-IK (fine-tuned and distilled), and evaluate them on suite of distribution shifts. Attention Distillation does well when the distribution is close (e.g., on ImageNet-Vas V2), but loses the mild "effective robustness" that fine-tuned MAE has been found to have [4].
Table 6:
5.1 Pre-training and fine-tuning datasets
So far, we have focused on a MAE model pre-trained and evaluated on ImageNet-IK: What happens if we pre-train or evaluate on different datasets? We first test this by pre-training MAE ViT-L models on two new datasets: ImageNet-229 [IO] and COCO (37]. These have substantially different dataset bias (54) from ImageNet, across axes like appearance, class balance, and diversity. shols that resulting MAE models maintain relatively good performance when fine-tuning on ImageNet- K, with maximum drop of at most 0.5. However, Attention Copy accuracy drops more, losing as much as 2.1. We see a similar phenomenon in Table where we use a MAE pre-trained on ImageNet and transfer to the iNaturalist datasets [56]. Again, when the pre-training dataset does not match the transfer dataset, Attention Copy accuracy drops significantly. We hypothesize that the frozen teacher's s attention maps are ill-suited for the fine-tuning dataset, which limits the performance.
5.2 Out-of-distribution robustness
One notable aspect of standard fine-tuned MAE model is that it shows slight "effective robustness," i.e., it achieves slightly better dittr--distriuution (OOD) accuracy than expected based on its in- distribution (ID) accuracy [14]. We test whether Attention Distillation, which achieves the same ID accuracy, has the same benefits OOD. Table shows that Attention Distillation still does quite well, but has lower accuracy than fine-tuned MAE on all 4 distribution shifts tried. These results indicate that the attention maps do not account for the full robustness benefits, and that the features learned by MAE during pre-training are helpful OOD even if they are not ID.
5.3 Pre-training methods
We have so far focused on MAE, reconstruction-basee pre-training method. We now check whether attention transfer still works if the teacher has been pre-trained with different algorithm. Specifically. we test MoCo-v3 77 self-super contrastive learning approach, and FLIP (36]. which does image-text contrastive learning. Table [ shows that Attention Copy still achieves most of the performance benefits for each pre-training method. Impressively, VIT-L is even able to reuch 86.6 by just transferring atlention mups from FLIP. This confirms that learning the proper attention patterns indeed a significant bottleneck during learning. Note that the FL.IP model we used is pre-trained on AB [48), yet its effectiveness is less affected by distribution shifts to ImageNet-IK
8

---

pre-training method
tune
copy
distill
MAE [2
85.7
85.1
85.7
model
scratch
tune
copy
distill
MoCo-v3
84.0
ViT-B
82.5
83.6
82.0
83.4
FLIP 36
87.4
86.6
86.1
VIT-L
83.0
85.7
85.1
85.7
DINO
83.2
82.3
82.8
VIT-H 83.0
86.9
86.1
86.3
83.0
72.7
76.3
Table 8: Different model size with MAE pre- trained on ImageNet- IK. Similar to weight tun- ing, the classification accuracy of attention trans- fer scales well as we vary the model size, while scratch training saturates.
Table 7: Different pre-training methods. Atten- tion transfer works for all pre-training methods, reaching 86.6 with FLIP teacher. As sanity check, transferring from a randomly ini- tialized VIT significantly hurts. *INO is ViT-B.
metric
scratch
tunc
distill
pbox
39.1
46.3
(+7.2)
43.6 (+4.5)
Apmask
34.6
40.6 (+6.0)
38.7 (+4.1)
Table 9: Object detection results COCO with MAE ViT-B pre-trained on COCO. Attention
transfer achieves majority of the gains of pre-training in this setting as well.
5.4 Model size
We test whether attention transfer works across model sizes. For all experiments so far, we have used ViT-L:; here, we try Attention Copy from a smaller (VTT-B) and larger (ViT-H) model, both pre-trained with MAE. Table 8 shows that Attention Copy continues improve with scale, even reaching 86. I % accuracy with ViT-H. It can do this even though scratch model performance already saturates at the ViT-L size. Again, this indicates that models need proper inter-token routing in order to learn good features that generalize. Otherwise, they cannot properly utilize increased model capacity.
5.5 Object Detection
Finally, we examine the performance of attention transfer in the standard ViTDet pipeline [35] for COCO object detection. We compare training from scratch against fine-tuning and attention transfer from MAE ViT-B pre-trained on COCO, which is done to mitigate the effect of distribution shift. For fair comparisons, we use 448x445 input remove the effect from window attention and positional embedding interpolation, and remove the effect of relative positional embeddings. Table D shows that Attention Distillation recovers majority of the gains from pre-training in this dense prediction setting as well. Based on Table 8 we anticipate that the gap between fine-tuning and attention transfer will decrease with ViT-L., but we are limited by computational resources.
6 Related Work
Structure in attention patterns. Previous works have studied the attention patterns of pre-trained vi- sion transformers [59 63 43). These works present these differences only as qualitative observations, whereas we are able to isolate the attention patterns and show that they are causally responsible for most of the differences in fine-tuning performance. Other methods, such as Lora (28) Prompt-to- Prompt (25), do rely on the importance of high quality attention patterns within pre-trained networks, but they also utilize pre-trained features and do not provide our insight that these features are typically unmecessary for the tasks we examine. Trockman and Kolter [55] observe diagonal structure within the product of attention layer weights weights with this structure moderately improves accuracy for small models early in training. Zhang al. (68) the language domain, find that pre-trained BERT models improve length generalization on particular synthetic tasks. They attribute this to the attention patterns of few, specific heads and show that hardcoding these patterns into the network achieves the same benefit. Our work is complementary and emphasizes the importance of attention maps over features.
trained supervised network. They show that initializing the
Decoupling inter- and intra-token operations. GLoMo 64 also attempts to decouple features from the way they should be combined. They use unsupervised pre-training to train network to

---

output a graph, which is later used to combine task-specific features. We find that there is no need to develop a specialized architecture to achieve this Vision Transformers already do this naturally.
Knowledge Distillation Knowledge distillation is a popular framework for training small, high- performing student networks [26]. Knowledge distillation methods typically add a loss to encourage the student network to match the teacher network's 's logits, but variants often use other feature [3 intermediate feature similarities [1)] or intermediate feature magnitudes [66 [4)] This last approach has also previously been called "attention transfer," but their method is quite different and actually refers to distilling spatial activation magnitudes in ConvNets. These knowledge distillation approaches all assume that students need to be explicitly taught the right features. In contrast, our analysis with attention transfer shows that attention maps are sufficient to recover all of the gains from pre-training. Some papers have used attention distillation as an auxiliary loss to help a smaller model learn the teacher outputs more effectively [62 [1]. However, these only consider transferring the same function across model sizes, instead of transferring knowledge from a pre- rreetrained model to a different downstream task.
statistics as targets, such as the final representations
Connection to the lottery ticket hypothesis The lottery ticket hypothesis [15 suggests that large, dense neural networks contain small, sparse subnetworks ("winning tickets") that, when trained from scratch, can match even outperform the performance of the original dense network. This is particularly interesting because these sparse subnetworks maintain their performance only with their original initialization values; the strength of their connections between neurons is special in some way. Our findings draw a parallel, indicating that the connections between patches, controlled solely by the attention patterss, are similarly special within pretrained ViT'. Frankle and Carbin [15 further conjecture that overparameteriiztioo improves performance because larger models contain exponentially more sparse subnetworks in superposition and are thus more likely to contain "winning ticket" - hypothesis supported by subsequent empirical and theoretical work [46 42
[ However. this phenomenon does not arise our setting with ViT attention patterns, since there are only a handful of attention maps per layer (rather than thousands of neurons). Consequently, good attention patterns are unlikely to appear by chance and must instead be learned during mre-training. We hope that new model architecture that efficiently combines many more attention maps per layer can address this limitation and learn better from scratch than existing ViTs.
7 Conclusion
Even as Transformers have surged in popularity, the way we use them has remained stagnant: pre- train, then fine-tune the weights. In this work, we present attention transfer, simple alternative to ViT fine-tuning that decouples intra-token operations (how to extract more usable features for each token) from inter-token operations (how those features should be combined). Our key finding is that the attention patterns (inter-token operations) are the key factor behind much of the effectiveness of pre-training - our Attention Distillation method completely matches fine-tuning on ImageNet- IK. We do find some limitations: attention transfer does not work well if the pre-training and transfer datasets are different, and it loses a bit of OOD robustness. Nevertheless, our findings provide insights into the role of attention in pre- tre-trained ViTs, and we hope future work fixes attention transfer's 's shortcomings and explores the advantages of this new transfer method.
Some directions for future work are particularly interesting. First, a deeper investigation of the queries Q could help us better understand their importance and potentially yield better transfer strategies. Second, attention transfer eliminates the need for tricks that fine-tuning requires, such as layerwise learning rate decay. Layerwise learning rate decay adds the prior that early layers should change less compared to later layers. However, this prior may be overly restrictive for next-generation models, since it prevents early features from changing, and getting rid of could open up new opportunities. Finally, attention maps could potentially be transferred more easily across model sizes. Pre-training smaller model and transferring its attention patterns to a larger downstream model could be more practical than the current practice of fine-tuning.
Acknowledgments and Disclosure of Funding
AL. supported by the NSF GRFP DGE 745576 and DGF22400339 and performed the work during an internship at FAIR.
10

---

References
[1] J. L., Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv: 067,00450, 2016.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan,
Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child,
A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray,
B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[3] R. Burkholz. Most activation functions can win the lottery without excessive depth. Advances in Neural Information Processing Systems, 35:18707-18720, 2022.
[4] M. Caron, H. Touvron, I. Mísra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[5] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In ICML, 2020.
[6] T. Chen, S. Korbblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[7] X. Chen, S. Xie, and K. He. An empirical study of training self supervisdd Vision Transformers. In ICCV, 2021.
[8] K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning. ELECTRA: Pre-training text encoders as
discriminators rather than generators. In ICLR, 2020.
[9] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, 2020.
[10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. CVPR, 2009.
[11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional
transformers for language understanding. In NAACL, 2019.
[12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani,
M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[13] Q. Duval, I. Misra, and N. Ballas. A simple recipe for competitive low-compute self supervised vision models. arXiv preprint arXiv:2301 2.0.043... 2023.
[14] A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In ICML, 2022.
[15] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint /003:0/80000330 03635, 2018.
[16] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
[17] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1412.657, 2014.
[18] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and
K. He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv:/706.02667, 2017.
[19] Z. Hao, Guo, D. Jia, K. Han, Y. Tang, C. Zhang, H. Hu, and Y. Wang. Learning efficient vision transformers via fine- fine-graine grained manifold distillation. In NeurIPS, 2022.
[20] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, 2016.
[21] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[22] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[23] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Para- juli, M. Guo, et al. The many faces of robustness: critical analysis of disti--iittionttiio generalization. In ICCV, 2021.

---

[24] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In CVPR, 2021.
[25] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 2022.
[26] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.022 003.02231, 2015.
[27] J. Howard and Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv: 180iv.8801.00 1/801106146 2018.
(28) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 106.0685, 2021.
[29] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
[30] Izmailov, P. Kirichenko, N. Gruver, and A. G. Wilson. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35:38516 6-35332, 2022.
[31] Jelassi, M. Sander, and Y. Li. Vision transformers provably learn spatial structure. Advances in Neural Information Processing Systems, 35:37822-37836, 2022.
[32] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In ICMI., 2019.
[33] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1989.
[34] Y. Li, X. Lyu, N. Koren, L. Lyu, B. Li, and X. Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. arXiv preprint wxxiv2201.009300 2021.
[35] Y. Li, H. Mao, R. Girshick, and K. He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022.
[36] Y. Li. H. Fan, R. Hu, Feichtenhofer, and K. He. Scaling language-image pre-training via masking. In CVPR, 2023.
[37] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[38]
Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR,
2017.
[39] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[40] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
[41] K. Navaneet, S. A. Koohpayegani, A. Tejankar, and H. Pirsiavash. Simreg: Regression simple yet effecttve tool for sule-super knowledge distillation. arXiv preprint wrXiv:2201.05133 2022.
[42] L. Orseau, M. Hutter, and O. Rivasplata. Logarithmic pruning is all you need. Advances in Neural Information Processing Systems, 33:2925-2933 2020.
[43] N. Park, W. Kim, B. Heo, T. Kim, and S. Yun. What do self-supervised vision transformers 2023.
learn? arXiv preprint arXiv:2305.007
[44] A. Pensia, S. Rajput, A. Nagle, H. Vishwakarma, and D. Papailiopoulos. Optimal lottery tickets via subset sum: Logarithmic pverpparamttriization is sufficient. Advances in neural information processing systems, 33:2599-2610, 2020.
[45] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell,
Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision.
In ICML, 2021.
[46] V. Ramanujan, M. Wortsman, A. Kembhavi, A. Farhadi, and M. Rastegari. What's hidden in randomly weighted neural network? In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1 11893-11902, 2020.
12

---

[47] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
[48] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes,
A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurIPS, 2022.
[49] N. Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.021500 2019.
[50] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitring. The Journal of Machine Learning Research, pages 1929-1958, 2014.
[51] C. Szegedy, V. Vanhoucke, S. Ioffe, Shlens, and Z. Wojna. Rethinking the inception architec- ture for computer vision. In CVPR, 2016.
[52] Y. Tian, Y. Wang, B. Chen, and S. Du. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. NeurlPS, 2023.
[53] Y. Tian, Y. Wang, Z. Zhang, B. Chen, and S. Du. Joma: Demystifying multilayer transformers via joint dynamics of mlp and attention. ICLR, 2024.
[54] A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR, 2011.
[55] A. Trockman and J. Z. Kolter. Mimetic initialization of self-attentio layers. arXiv preprint
arXiv:2305.09828 2023.
[56] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and
S. Belongie. The iNaturalist species classification and detection dataset. In CVPR, 2018.
[57] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need, In NeurlPS, 2017.
[58] S. Venkataramanan, A. Ghodrati, Y. M. Asano, F. Porikli, and A. Habibian. Skip-attention: Improving vision transformers by paying less attention. arXiv preprint arXiv:2301.022 00..02000002...... 2023.
[59] M. Walmer, S. Suri, K. Gupta, and A. Shrivastava. Teaching matters: Investigating the role of supervision in vision transformers. In CVPR, 2023.
[60] H. Wang, S. Ge. Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019.
[61] K. Wang. F. Yang. and J. van de Weijer. Attention distillation: self-supervised vision transformer students need more guidance. arXiv preprint arXiv:2210.0044 2022.
[62] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776-5788, 2020.
[63] Z. Xie, Z. Geng, J. Hu, Z. Zhang, H. Hu, and Cao. Revealing the dark secrets of masked image modeling. In CVPR. 2023.
[64] Z. Yang, J. Zhao, B. Dhingra, K. He, W. W. Cohen, R. Salakhutdinov, and Y. LeCun. Glomo: Unsupervisedly learned relational graphs as transferable representations. arXiv preprint 2018.
[65] Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[66] S. Zagoruyko and N. Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
[67] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. In ICLR, 2018.
[68] Y. Zhang, A. Backurs, Bubeck, Eldan, S. Gunasekar, and T. Wagner. Unveiling transform- with lego: synthetic reasoning task. arXiv preprint arXiv:2206.04300 2022.
13

---

Appendix
A
Key Numbers
A.1
Information in Attention Transfer
How much information is transferred during attention transfer? Table 10 shows two ways of doing this accounting. If considering the map 24 layers x 16 heads 197 query tokens 197 key tokens, there are about 15 million activations transferred per example. However, QK is low rank since Q and K very "tall," so the attention map softmax can be considered 24 layers x 16 heads x 197 tokens x 64 head dim x 2 matrices, which is about 9.7 million activations.
Accounting Method
Parameters
Count sizes of Q, K Count size QK"
24 x x 197 64 x 9.7M
24 16 197 197 15M
Table 10: Number of parameters activations transferred per example for a ViT-L teacher.
A.2 Computational Cost Attention Transfer
Attention transfer has the same computational and memory cost any other knowledge distillation method that does a forward pass through teacher. We compare fine-tuning vs attention distillation 16GB NVIDIA GP100 with ViT-L and batch size of 16:
Method
Memory (GB)
Time per iteration (s)
Weight Fine-tuning
9.4
0.93
Knowledge Distillation
11.1
1.23
Attention Copy/Distillation
11.1
1.23
Table 11: Training cost of Attention Transfer.
Training these large models on ImageNet-IK is quite computationally expensive. 100 epochs of regular fine-tuning is about 2070 GPU-hours per 100 epochs, and 100 epochs of attention transfer is about 2735 GPU-hours. In total, we estimate about 150k GPU-hours are required to reproduce all experiments.
B Additional Analysis
B.1
Aggregated Attention Transfer
we conducted thorough analysis of which aspects of attention matter the most. Another way to identify key properties of the teacher's attention maps is to average them across some axis during the transfer. For example, one average the attention maps over all layers of the teacher network, so that the student uses the same map at every layer. Table shows the results with aggregations over several natural axes. Averaging over examples (i.e., using the same attention map, independent of the input) or averaging over query tokens (i.e., each attention distribution is the same, regardless of the query token given an image) does quite poorly. This indicates, unsurprisingly, that these are key elements of self-attention. This also shows that prior work that focuses on aggregate statisties of the attention maps (e.g., averaged over examples) [59] fail capture the per-example nature of the attention maps that are actually responsible for full fine- fine-tuning performance. Attention copy performance is more reasonable when averaging over heads or layers. This partially corroborates previous findings that attention maps can largely be shared across all layers (58). However, while the results can potentially improved with more recipe search, the performance is far short of the full fine-tuning accuracy (85.7).
In Section 4.1
14

---

average over
acc.
examples
79.7
layers
82.7
heads
82.2
query tokens
79.9
none
85.1
Table 12: Aggregated attention transfer. The full set of attention maps from the teacher has shape (examples, layers, heads, query tokens, key tokens). Here we try to average over each of these axes before the transfer. Performance drops the most when averaging over examples query tokens, indicating that these are the most important aspects of the attention maps.
method
attn. distill
85.7
feature distill
81.3
fine-tune
85.7
scratch
83.0
Table 13: Comparison with knowledge distillation. We try knowledge distillation from the pre- trained teacher by adding an auxiliary MSE loss on the residual stream output. This encourages the student model to match the representations of the teacher ("feature distill")). We find that this does much worse than Attention Distillation.
B.2 Comparison Knowledge Distillation
Our central hypothesis has been that the pre-trained attention maps are sufficient, and the pre-trained features are not necessary. Since our attention transfer methods are special instances of knowledge distillation [26]. we additionally compare to a baseline of distilling the residual stream features from pre-trained MAE VYT-L. In Table we obtain downstream accuracy of 81.3 on ImageNet-1k k.. This is significantly lower than the 85.7 that can be achieved through fine-tuning or attention distillation. This makes sense: the features learned during super-spe pre-training are not directly well-suited for classification, so trying to match them can hurt performance. CKA analysis of the features (Figure 5) supports this hypothesis - the fine-tuned MAE does well by significantly changing features in the latter half of the network. Overall, transferring attention appears to do much better than distilling the features.
B.3 Attention Map Analysis for Transferring Q
In Section we found that copying the queries Q does surprisingly well, almost matching Attention Distillation or fine-tuning the pre-trained weights. Here, we compare the attention maps learned by the copy Q model to those of other models, in hopes of understanding why copy Q does so well.
Each of the 24 layers within VIT-L has 16 attention heads, which each compute L L attention map for an image with L patches. We would like to determine the similarity between the attention heads in models using some divergence measure; we use the Sensen-Shanoon divergence (JSD) because it is symmetric in its arguments. However, there is one caveat. Because the output of the attention layer is invariant to the ordering of its heads, is insufficient to compare the ith head of one model against the ith head of another. We need to properly match heads across models. We explored four ways of doing
1. Direct pair: this is the naive approach of computing the JSD between the ith head of the first model and the ith head of the second model. This can fail since similar heads may not be in the same order across models.
2. Bipartite matching: for each layer, we compute the JSD between each of the 16 heads in the first model and the 16 heads one-to-one pairing between the heads that minimizes the cumulative JSD. This solves the previous problem, but can still thrown off, such as if one of the models has heads that applics weight to (V 0 W proj 0 or W proj is orthogonal to the values).
second model. We then use bipartite matching create a
15

---

Direct pair
Bipartite matching
Min
Averaged maps
from PTtenheer
0.4
0.3
0.2
Layer
Layer
Layer
Direct pair
Bipartite matching
Min
Averaged maps
0.1
pre-train distill
copy
fine-ruse
JSD from
0.3
copy
0.0
10
20
10
20
10
20
0
10
2.0
Layer
Layer
Layer
Layer
Figure 7: Attention map similarity across methods. Each column corresponds to a different way of matching up attention heads between two models. The top row shows the Sensen-Shannon divergence (ISD) with respect to the MAE pre-trained teacher, whereas the bottom row shows the JSD with respect the fine-tuned MAE model. These plots match our intuition distillation or fine-tuning methods, but copy Q is consistently dissimilar from the PT and FT models. Note that this may be limitation of this particular analysis, since there are many settings of Q, K, and V that lead the same layer output.
3. Minimum: instead of creating a one-to-one matching, we allow many-to-on matching between heads. We call this Minimum because each head in the first model is paired with the head from the second model with the smallest JSD. This allows our metric to potentially ignore extraneous heads in the second model, but is still susceptible to extraneous heads in the first model.
4. Averaged maps: we average the attention maps of all heads in a layer and compare the averaged maps across models. This can still be thrown off by extraneous heads.
Figure 7 shows the results of comparing models against the pre-trained teacher (top row) fine-tuned model (bottom row) as the second model in the JSD. Most our findings align with our intuition. In the top row, when comparing against the pre-trained teacher, attention distillation matches the teacher maps closely until layer 18, the last layer whose attention maps it is trained to approximate. The fine-tuned model's s attention maps diverge more in later layers, since layerwise learning rate decay ensures that the earlier layers don't I change much. However, copy Q is only somewhat similar to the pre-trained teacher the fine-tuned model, across all of our ways measure attention map similarity. Furthermore, is less similar than copy K is, even though copy K has much lower downstream performance than copy Q.
Note that these plots have major limitations what kinds of similarity they capture. With enough attention heads per layer, the same exact attention map be partitioned differently across the heads between two models. Hypothetically, let's say that an attention layer wants to attend uniformly across all locations (i.e., perform average pooling), and that have 3 models, each with 2 attention heads:
1. Head attends uniformly over all locations, head 2 attends arbitrarily over locations, and the second head's values set to 0.
2. Head attends uniformly over the top half of the image, head 2 attends uniformly over the bottom half of the image, and both use values V/2.
3. Head attends uniformly over the left half of the image, head 2 attends uniformly over the right half of the image, and both use values V/2.
16

---

input
init
scratch
pre-train/ / copy
fine-tune
distill.
Layc 1
Layer
Figure 8: Visualization of attention maps for different methods. We show what the [CLS] token attends to at various layers within the network. Darker patches indicate more attention weight. Notably, the pre-trained MAE model's atfention maps provide significant prior over what the model should use, separating the object from potentially spurious cues like the branch or the background. In contrast, models right at random initialization ("init") start off attending uniformly over the image, which leads the scratch model to use more of the spurious patches. We show more attention map visualizations in Appendix
All 3 heads compute the same exact attention operation, yet would register as highly dissimilar in the setup from Figure 7 Overall, this experiment shows that copy Q's behavior is highly complex, and its strong downstream performance still not fully understand.
B.4 Attention Map Visualizations
provided several results to show that attention transfer does not simply "relearn" the teacher. Here, examine final piece of evidence. We show the attention maps of different networks in Figure 8. We focus on Attention Distillation, since Attention Copy's maps are identical those in the teacher. Attention Distillation's maps generally match the teacher (pre-trained MAE), but are not completely identical for layers that are distilled (e.g., layer 13). For layer 24, which is not distilled, Attention Distillation looks very different from pre-trained model, instead resembling the attention map of a model trained from scratch. These visualizations also highlight the fact that these attention maps are very strong prior on what the model should use. While the randomly initialized model attends completely uniformly over all tokens, the pre-trained teacher attention maps already separate the relevant object from potentially spurious correlations (like the branch or background in the example). We show additional attention map visualizations in Figure 11 and Figure 12
Section 4.2
B.5 Attention Distillation Hyperparameter Sensitivity
We show the sensitivity of Attention Distillation its hyperparameters.
Distillation loss weight We first consider the distillation loss weight which is used to compute the overall loss for the student:
= Cunk + ALdist
Table 9 shows that larger weight, À = 3, does best. This may be because it encourages the student to learn useful attention maps more quickly, letting guide feature learning earlier in training. We use this value of À for our main result, where we match the 85.7 accuracy of fine-tuning. However, all other results in this paper use À = 1 for simplicity.
Partial distillation: layers Just as we tried for Attention Copy, we also tried distilling various numbers layers from the MAE teacher network, starting from the bottom the network. Table TO) shows that there a "sweet spot" when distilling the first 21 out of 24 layers. Distilling all layers may
17

---

Distillation loss weight
Accuracy
Layers distilled
Accuracy
0.3
84.7
85.3
85.3
85.5
3.0
85.7
85.1
Figure 9: Weight distillation loss
Figure 10: Number of layers distilled
Teacher
Student
Accuracy
Random
Random
72.7
MAE
Random
85.1
MoCo-v3
Random
82.5
FLIP
Random
86.6
FLIP
MAE
84.2
MAE
MAE
85.4
MoCo-v3
MAE
82.9
MAE
FLJIP
83.2
Table 14: Decoupling attention maps from features. We try Attention Distillation with various
pre-trained students.
hurt performance by forcing the student to use attention maps that are more suited for reconstruction than classification. Note that all other distillation results in this paper use the first 18 layers by default.
B.6 Mix and Match, Student and Teacher
In the main paper, we focused on transferring attention maps from a pre-trained teacher to a randomly initialized student. However, the fact that Transformers have decoupled inter- and intra-token computation means that we can actually initialize the student with a pre-trained network as well. This entails testing whether the attention patterns from one network can improve the features of an already-pre-tree pre-trnned student model. We try Attention Distillation for various combinations of MAE, MoCo-v3, FL.I,, and a randomly initialized network. Table [4 shows that this "mix-and match" training does better than training from scratch (83.0) but does not match the performance in Table 7 where the students are randomly initialized. These are preliminary results, as the overall training recipe may need to be changed to accommodate the different learning dynamics of different student model. Further hyperparameter tuning may significantly improve these results.
C Implementation Details
We present the training recipe for Attention Copy in Table 15 and the recipe for Attention Distillation in Table [6 For our partial layer transfer experiments in Figure we set /3 0.95 as it helps avoid training instabilities.
18

---

Config
Value
optimizer
AdamW W [39
base learning rate minimum absolute Ir weight decay
le-3
2e-3
0.05
optimizer momentum
31 = 0.9, 82 = 0.999
layerwise decay batch size
0.75
2048
learning rate schedule warmup epochs [8]
cosine decay 338
5
training epochs
100
augmentation
RandAug (9, 0.5) 9
label smoothing 51
0.1
mixup [67
0.8
cutmix (65
1.0
drop path (29
0
exp. moving average (EMA)
0.9999
layers copied
24
Table 15: Training recipe for Attention Copy on VIT-L.
Config
Value
optimizer
Adam W
base learning rate
lc-4
weight decay
0.3
optimizer momentum
81 0.9, 32 0.95 2048
batch size
learning rate schedule
cosine decay
warmup epochs
training epochs
200
augmentation
RandAug (9, 0.5)
label smoothing
0.1
mixup
0.8
cutmix
1.0
drop path
0.2
exp. moving average (EMA)
0.9999
layers copied
18
distillation weight À
Table 16: Training recipe for Attention Distillation on VIT-L.
19

---

D Additional Attention Map Visualizations
input
init
scratch
pre-train/ copy
fine-tune
attn, distill.
Layer
Layer
Layer 23
imput
init
scratch
pre-train/ copy
fine-tune
attn. distill.
Layer
Layer 13
Layer 23
Figure 11: Attention map visualizations on more examples
20

---

imput
scratch
pre-train/ copy
fine-tune
attn. distill.
Layer
Layer 13
Layer
input
init
scratch
pre-truin/ / copy
fine-tune
attn. distill.
Layer
Layer 13
Layer 23
Figure 12: Attention map visualizations on more examples
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We make clear in the abstract and introduction the limits to claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Section 5 thoroughly discusses the settings where our analysis holds, and Appendix A.2 discusses the computational cost of attention transfer.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
We do not provide any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide thorough experimental details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[Yes]
Justification:
reproducible, they We release our code at https//gghhu...oommddccccooolltt -ransttt
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guides/ wwtp///oaaaasiiiiioooooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips httpcc/cpplllgggiuueee hetpgddde//o/eubbbriimmmmm
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide thorough experimental details in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not typically reported in our settings (classification, detection) because it would be too computationally expensive. Furthermore, the relative difference we have observed is much bigger than the standard deviation (e.g.. < 0.2), so we follow common practice and do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this in Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with NeurlPS Code of Ethics http://///ee.epp.cpppppp/t//ttaa
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and believe that we conform with the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The goal of our work is to advance the field of machine learn- ing. There are potential societal consequences of our work, but believe none of them are significant and immediate.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper focuses image classification and object detection, and do not plan to release checkpoints or data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite all datasets and pre-trained checkpoints that we use.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not use human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do use human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "aVCC2AZHAy", "reviewer": "Reviewer_qaog", "summary": "This paper investigates how transferring attention patterns from a pre-trained ViT to a student affect the student's downstream performance. By applying the attention copy strategy, the paper shows that when the pre-trained dataset and downstream dataset are the same, the trained student may achieve performance superior than the student training from scratch. Moreover, it largely recovers the performance of a pretrained-and-finetuned model. Further, the authors propose an attention transfer (distillation) scheme. With attention distillation scheme, the student may achieve performance comparable or even better than the pretrained-and-finetuned model. The authors provide extensive experiments to verify the effectiveness of attention transfer and show when attention transfer works.", "questions": "Please see the weakness part.", "limitations": "The authors adequately addressed the limitations.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow. \n\n- The findings is meaningful and interesting to an extent. \n\n- Extensive experiments are conducted.", "weaknesses": "- The findings is somewhat similar to previous works which apply attention transfer in ConvNets. The only different things is in ViT, attention can be explicitly represented, which eases the operation of attention transfer. \n\n\n- The authors seem to over-claim something, e.g., \"offer a potential alternative to the decade-long practice of fine-tuning pre-trained vision models\". The empirical results in the paper cannot sufficiently support this claim. For example, we only see comparable results on one setting where the model is pretrained on ImageNet (unsupervised/self-supervised pretraining) and finetuned on ImageNet. For other settings, like out-of-distribution tasks, detection tasks, etc., the attention transfer does not work as well as fine-tuning. \n\n- The experiments which study the effect of transferring a subset of Q, K, V seems not to support the main claim of this paper. The results show that transferring Q is the best of all. So why do we choose to transferring Q and K (or attention)? It implies that transferring the attention pattern may not be the key to the superior performance on downstream task, although transferring attention patterns sounds reasonable and interpretable. \n\n- For Table 2-6, it is strange to see only partial results in one table. It is better to show the results for \"training from scratch\", \"fine-tuned\",  \"copy\" and \"distill\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’re glad that you found our work well-written, with extensive experiments and interesting findings. Below, we respond to your questions and comments:\n\n> The findings is somewhat similar to previous works which apply attention transfer in ConvNets. The only different things is in ViT, attention can be explicitly represented, which eases the operation of attention transfer.\n\nPrevious works on ConvNets have mainly looked at transferring different properties of the features, like spatial feature magnitudes. They have also mainly been conducted in the knowledge distillation paradigm, where a task-specific (not pre-trained) downstream teacher is distilled into a smaller student. Our work differs from prior work in a few ways. First, as you point out, ViTs allow us to explicitly decouple the attention patterns from the features that they combine. Second, we extensively investigate the pre-training/fine-tuning paradigm, and we surprisingly find that the attention maps, by themselves, are often sufficient to achieve the same downstream performance. This result is completely new and calls into question the “feature learning” story that typically motivates pre-training in vision.\n\n> The authors seem to over-claim something, e.g., \"offer a potential alternative to the decade-long practice of fine-tuning pre-trained vision models\"\n\nYou’re right – we intended to say that attention transfer achieves surprisingly good results, and we didn’t mean to imply that attention transfer was already sufficiently developed to be a practical alternative to fine-tuning. We tried to make this clear in L327-330 in the conclusion. We will clarify our wording in the introduction and write “with further research, attention transfer could be a potential alternative to fine-tuning pre-trained vision models.”\n\n> The results show that transferring Q is the best of all. So why do we choose to transferring Q and K (or attention)? It implies that transferring the attention pattern may not be the key to the superior performance on downstream task, although transferring attention patterns sounds reasonable and interpretable.\n\nThis is a great question! See [a] in the main rebuttal above. \n\n> For Table 2-6, it is strange to see only partial results in one table. It is better to show the results for \"training from scratch\", \"fine-tuned\", \"copy\" and \"distill\".\n\nFor some of our tables, we had run either “copy” or “distill” due to compute limitations. During the rebuttal period so far, we have been running more experiments to ensure that Tables 2-6 are comprehensive – see [b] in the main rebuttal above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zUufdngTRK", "reviewer": "Reviewer_7AbG", "summary": "The authors propose a novel perspective on the utility of pretraining vision transformers by demonstrating that the actual features and representations learned during pre-training are not crucial. Instead, they find that simply re-using the self attention from pre-training specifically, the way information flows between tokens—is sufficient for models to learn high-quality features from scratch and achieve performance comparable to pre-trained models.\n\nTo support their claim, the authors introduce a method called attention copy and attention distill. This method involves transferring the attention from a pre-trained teacher ViT to a student ViT, either by copying or distilling the attention maps. This approach allows the student model to learn its own features while still benefiting from the pre-trained attention patterns.  The authors also highlight several drawbacks of previous works that heavily rely on pre-trained features and representations. They point out that the conventional fine-tuning approach may not be as effective under distribution shift settings, where the pre-trained features might not generalize well. In contrast, their attention transfer method provides a more robust alternative that maintains high performance even when the distribution shifts.\n\nThrough systematic experiments, the authors examine various aspects of their findings, particularly focusing on the sufficiency of attention maps. They provide evidence that attention patterns alone can guide the learning process effectively, thus questioning the necessity of the entire pretraining paradigm.", "questions": "There have been works such as [51], that illustrate the performance of copying attention across intermediate layers of the network. However, I would urge the authors to also include the following works for completeness, which have shown that copying attention works well in the domain of NLP\n\n[3*] Xiao et al.,  Sharing attention weights for fast transformer, IJCAI 2019\n[4*] Wang et al.,  Evolving attention with residual convolutions, ICML 2021\n[5*] Ying et al., Lazyformer: Self attention with lazy update, arXiv 2021", "limitations": "Yes the authors discuss the limitations", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The following are some strengths of this work\n- The problem is well motivated and also also been discussed in previous works. Not specifically the distillation approach, but reusing attention maps has been interesting not only in the vision community, but widely studied in NLP.\n- I found that the paper is beautifully written, it basically shows the efforts that the authors took in trying to explain each aspect of their work. They keep the language simple and easy to understand, with concise explanations together with proper visualizations and plots where necessary.\n- The analysis of different components such as transferring attention from different layers, different heads, CKA analysis etc. is very well thought and it was a joy to read through the findings. So I thank the authors for this and encourage them to do these kind of analysis in all their future works as well.\n- The experiments on different tasks such as image classification, model robustness etc. show the effectiveness of the approach.", "weaknesses": "The following are some queries:\n- In Figure 5, it is surprising to see that attn-copy has least correlation with respect to the fine-tuned model as compared to attn-distill. In attn-distill from layer 20-24, the correlation increases much more than the pretrained model, which is not the case with attn-copy. Do the authors have any intuition on why this is the case?\n- In general, with the CKA computation. I think it would be more interesting to understand the correlation across the features at different layers of the model. I would refer the authors to the work in [51], where the authors show correlation plot of features across every layer before and after their method is applied. This would help understand how attn-copy and attn-distill affect representations learned by the model. The authors can take a look at Figure 2 and Figure 4 in [2*] for reference.\n- Continuing from above, it would also be interesting to see this correlation across different heads after the model is pretrained with attn-copy and attn-distill. In [1*], the author shows that there exists high correlation across attention heads in ViTs, so it would be interesting to see if attn-copy or attn-distill mitigates this.\n\n[1*] https://github.com/sayakpaul/probing-vits?tab=readme-ov-file#visualizing-mean-attention-distances  \n[2*] Zhou et al., Refiner: Refining Self-attention for Vision Transformers, arxiv 2021\n\n- Do the authors have the same observation for attn-copy and attn-distill when using ViT-L pretrained in a supervised setting. Also, im curious to know if the same observation holds for methods pretrained using self-distillation approaches such as DINO, BYOL, iBOT or distillation approaches like SimSiam, etc. \n- I think I might be missing something, but the visualization of attention from the [CLS] token, in Figure 7 seems that attn-distill has worse attention than attn-copy at the deeper and intermediate layers. Also the localization of the object seems to be bad as well. Can the authors please comment on this.\n- I would also like to see the comparison with different state-of-the-art methods that use ViT-L as their backbone is tasks like classification, object detection and robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’re really glad that you liked the motivation, analysis, and writing of our paper! We respond to your comments below:\n\n> Fig. 5: In attn-distill from layer 20-24, the correlation increases much more than the pretrained model, which is not the case with attn-copy. Do the authors have any intuition on why this is the case?\n\nOur main hypothesis is that this is because we transferred all 24 layers for Attention Copy but only distilled 18 layers for Attention Distillation. This means that Attention Distillation is more flexible in how it combines the features for the last 6 layers, so it can find a strategy similar to the fine-tuned MAE. In contrast, Attention Copy is constrained to be more similar to the pre-trained MAE.\n\n> Do the authors have the same observation for attn-copy and attn-distill when using ViT-L pretrained in a supervised setting. Also, im curious to know if the same observation holds for methods pretrained using self-distillation approaches such as DINO\n\nWe are running this right now! We hoped to have the results by now but had an issue with our cluster. We will post an update below with the results as soon as possible. \n\n> in Figure 7 seems that attn-distill has worse attention than attn-copy at the deeper and intermediate layers. Also the localization of the object seems to be bad as well.\n\nAttention Distillation only really deviates from the pre-trained attention maps at the later layers of the network. This makes sense, since we don’t distill the last 6 layers’ attention maps. Furthermore, the “noisy” attention pattern can sometimes be quite useful in Vision Transformers, since they often tend to store information in low-entropy regions of the background (see “ViTs Need Registers” [6*]). The fine-tuned MAE shows similar “noisy” patterns in its later layers, which we can see after fixing a small issue with our attention map visualizations: since we follow the standard practice of using global average pooling, which averages the representations at all spatial locations, the CLS token representation is not used after the 24th layer. This means that its attention map has no signal to improve. We fix this by now showing the pattern after the 23rd layer instead. The examples in the 1 page PDF show that the fine-tuned MAE also exhibits the same patterns in the later layers as Attention Distillation. \n\n> I would also like to see the comparison with different state-of-the-art methods that use ViT-L as their backbone is tasks like classification, object detection and robustness.\n\nTo the best of our knowledge, fine-tuned MAE is the SOTA ViT-L model without using extra data on ImageNet classification and the OOD robustness benchmarks. For object detection, the SOTA ViT-B-based model using ImageNet-1k is ViTDet [32] with 56.0 $AP^{box}$ and 48.0 $AP^{mask}$. Note that the detection results cannot be directly compared with ours due to further architectural modifications within ViTDet on top of the ViT backbone. \n\n> I would urge the authors to also include the following works for completeness, which have shown that copying attention works well in the domain of NLP\n\nThank you for suggesting these NLP papers – we will definitely add them in our related works section!\n\n[6*] Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2023). Vision transformers need registers. arXiv preprint arXiv:2309.16588.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lJJxYkqWlk", "reviewer": "Reviewer_p7aT", "summary": "The paper introduces attention transfer as an alternative to fine-tuning in Vision Transformers (ViT), separating intra-token and inter-token operations to enhance feature extraction and combination. Attention transfer contains attention copy and attention distillation. Attention distillation matches the performance of fine-tuning on ImageNet-1K while learning different representations, facilitating the performance of feature ensemble. The method is verified across various model scales and datasets.", "questions": "In Table 1, transfer Q is better than transfer Q,K (which is equivalent to the attention map), does it mean distillation on features would achieve better performance? If so, is attention really important and sufficient for transfer tasks?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper first points out the sufficiency of attention maps in pretrainings and provides extensive analyses on attention transfer.\n2. The proposed attention distillation is verified across various model scales and datasets.", "weaknesses": "1. Compared with full fine-tuning,  the proposed attention transfer method introduces additional computation costs on the forward process of the teacher model. Therefore, to make the article more comprehensive, authors should consider including a baseline of distillation on features.\n2. Transfer tasks (e.g., Table 2 and 3) should contain the results of training from scratch.\n3. There is an important result in L169, the result of attention copy from a fine-tuned model (85.6), not appearing in any Tables or Figures. There should be an extra Table or Figure containing the result with other similar settings (e.g., attention copy from only pre-trained model) for better comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback on our work! Below, we respond to your questions and comments:\n\n> In Table 1, transfer Q is better than transfer Q,K (which is equivalent to the attention map), does it mean distillation on features would achieve better performance?\n\nSee the main rebuttal above for a detailed discussion of the Q-copying result. For feature distillation, we followed your suggestion and tried distilling the residual stream features from a pre-trained MAE ViT-L. In our preliminary results, we obtain a downstream accuracy of 81.3 on ImageNet-1k. This is significantly lower than the 85.7 that can be achieved through fine-tuning or attention distillation.  This makes sense: the features learned during self-supervised pre-training are not directly well-suited for classification, so trying to match them can hurt performance. CKA analysis of the features (Fig. 5) supports this hypothesis – the fine-tuned MAE does well by significantly changing the features in the latter half of the network. Overall, transferring attention appears to do much better than distilling the features. \n\n\n> Transfer tasks (e.g., Table 2 and 3) should contain the results of training from scratch.\n\nIn our main rebuttal above, we have updated Table 2-3 with the results of training from scratch. \n\n\n> There should be an extra Table or Figure containing the result with other similar settings (e.g., attention copy from only pre-trained model) for better comparison.\n\nThanks for the suggestion! We will add a new subsection and table that shows the effect of the teacher (pre-trained vs fine-tuned) on attention copy and distillation. We hope this will be more organized for readers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fttlar8K5T", "reviewer": "Reviewer_9FPV", "summary": "In this work, the authors demonstrate that a large part of the benefits of pre-training in ViT models actually comes not from the pre-trained features but the pre-trained knowledge of attention maps. Specifically, the authors propose an alternative to fine-tuning call Attention Transfer, and they use this method to transfer attention maps from pre-trained ViTs to teach from-scratch ViTs. This method achieves results comparable to results from finetuning. The authors present two variations of Attention Transfer, the first of which directly copies the attention maps from the teacher network, while the second teaches the student network to make its own attention maps with a distillation objective derived from the teacher attention maps. The authors demonstrate impressive and surprising results on ImageNet-1k, and present further analysis of this approach in a variety of configurations. While the proposed Attention Transfer method does have some clear limitations, specifically for domain gaps, overall, the work has very interesting implications for ViT pretraining and finetuning.", "questions": "In the Attention Copy setting, are the unnecessary Q and K layers removed from the student network?\n\nThere is an important conclusion in the work that seems somewhat under-acknowledged. In lines 210-217 and in Table 1, it is found that transferring the teacher’s Q is more effective than transferring the attention map. Why was more attention and testing not performed on Q transfer?", "limitations": "The authors discuss the limitations of Attention Transfer and present results for it with many configurations. Overall, they are quite transparent about acknowledging the situations where Attention Transfer underperforms full Fine Tuning.\n\nI do not see any risks for negative societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The main result of the work, which is to demonstrate the key importance of pre-trained attention maps over pre-trained features in ViT finetuning/transfer learning, is very interesting, and has important implications for the use of ViTs. While I think it is unlikely that Attention Transfer (in its current form) will replace standard finetuning (see notes in the following section), I think the implications of the results and analysis are still very important.\n\nThe potential to use Attention Transfer with ensembles is also interesting, and the authors show that it has the potential to boost the performance of self-supervised ViTs further.\n\nThis work also helps to explain some of the properties of MAE. In particular, prior works have found that MAE lends itself well to fine-tuning, but not as well to direct linear probes. This work shows a possible explanation: that the strength of MAE is not in its pretrained features but instead in its pretrained attention maps.\n\nThe work is clearly presented and has wide coverage of many model and training configurations in the main work and appendix. They also include additional analysis on the impact of partial transfer, and assess if the student is or is not re-learning the same features as the teacher. Overall, their analysis is quite comprehensive.", "weaknesses": "While the proposed Attention Transfer method has very interesting implications for ViTs, I’m not sure if it will make its way into practical use, for either the Attention Copy or Attention Distillation variants. The main issue is that the proposed method does not always match the performance of the regular fine-tuning approach, particularly in cases with a domain gap between the pretraining data and the downstream task. In addition, Attention Transfer is more expensive to train than standard Fine Tuning, as acknowledged in Section A.2.\n\nFor some of the analysis results, it would be very helpful to see how Attention Distillation performs as compared to Attention Copy. In particular, in any analysis where there is a domain gap (Table 2 for example), it would be interesting to see if Attention Distillation performs better, as it is suggested that Attention Distillation allows more flexibility in the student and thus may perform better in such cases.\n\nOn a less important note, I find that the visualizations in Figures 7, 10, and 11 are somewhat difficult to see due to the combination of the heat map colors and the background images. I would suggest revising these figures to try and make them clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’re glad that you found our results interesting and our analysis comprehensive. We address your questions below: \n\n> I’m not sure if it will make its way into practical use, for either the Attention Copy or Attention Distillation variants.\n\nWe agree that the attention transfer methods are not currently ready for practical use. One of our main objectives was to use attention transfer to *understand* the role of features vs attention in pre-trained models. But beyond scientific utility, attention transfer may have some advantages over fine-tuning in the future: \n- An attention map (of size LxL, L is the sequence length) does not depend on the model’s dimension, which means the map from a model can be directly transferred to a different-sized model.\n- Attention transfer gets rid of layer-wise learning rate decay. This is a crucial hyper-parameter almost used everywhere for tuning pre-trained vision models (beyond ViT). The fundamental prior here is that early layers should change less compared to later layers. But such a prior can be a restriction for next-generation models and getting rid of it opens up new opportunities.\n- Sharing weights can incur security risks (e.g., white-box attacks). In such settings, we need an effective way to transfer knowledge from the pre-trained model, and attention transfer offers such a possibility.\n\nWe will add more detail on avenues for future research in the paper, and we hope other researchers find more practical ways to use attention transfer!\n\n> For some of the analysis results, it would be very helpful to see how Attention Distillation performs as compared to Attention Copy.\n\nIn our main rebuttal above, we have updated tables so that we have both Attention Copy and Attention Distillation. In general, Attention Distillation performs better than Attention Copy, which follows our existing findings.\n\n> Revising attention map visualizations\n\nWe have modified the figures to be easier to see. We have included samples in the 1 page PDF. Let us  know if you have any further suggestions!\n\n> In the Attention Copy setting, are the unnecessary Q and K layers removed from the student network?\n\nYes, we remove them from the student network. \n\n> Why was more attention and testing not performed on Q transfer?\n\nThank you for pointing this out! See [a] in the main rebuttal above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank reviewers for their time, effort, and feedback for our paper. To recap, reviewers appreciated various strengths of our work: 

- **Significance**: “well motivated” (7AbG), “very interesting implications for ViT pretraining and finetuning” (9FPV), “potential to boost the performance of self-supervised ViTs further” (9FPV), “helps to explain some of the properties of MAE” (9FPV), “findings is meaningful and interesting” (qaog)
- **Experiments**: “extensive experiments” (qaog), “joy to read through the findings” (7AbG), “their analysis is quite comprehensive” (9FPV), “extensive analyses” (p7aT)
- **Clarity**:  “beautifully written” 97AbG), “well-written and easy to follow” (qaog), “ clearly presented” (9FPV)

Next, we provide general comments for some shared topics of discussion. We will address individual concerns and questions by responding to each review.

**[a] High performance from copying Q (Table 1) [qaog, p7aT, 9FPV]**

We do observe higher performance (85.6) from copying the self-attention queries $Q$ than Attention Copy (85.1), but lower performance than Attention Distillation (85.7)

First, we note that copying $Q$ does not transfer any features – the student network only “sees” the result of using $Q$ to compute the attention map softmax($QK^\top$). Thus, copying $Q$ solely provides structure to the student attention maps. This is consistent with our story that the attention patterns are *sufficient* to guide networks to high downstream performance. 

Our hypothesis is that copying $Q$ does well because it gives the model flexibility to change the attention patterns to be more suitable for the downstream task. Directly copying the entire attention map $QK^\top$ is too inflexible, which is why attention distillation works better, as it can deviate from the teacher’s attention maps. 

Overall, we mainly focus on transferring the entire attention map since it’s a clean way to split the network’s computation (it contains all inter-token communication). To be thorough, we are now training models with $Q$-distillation, which encourages the student self-attention queries $Q$ to be close to those of the teacher. We have had some cluster problems but should have the results within a few days.

**[b] Updated Tables 2-6 [qaog, p7aT, 9FPV]**

We update Tables 2-6 so that they all contain these models: trained from scratch, MAE fine-tuned, Attention Copy, Attention Distillation. We will update the main paper with these results as well. 

Table 2:
|Pre-training data | tune | copy | distill | scratch |
|--------------------|-------|---------|--------|-------|
| ImageNet-1k | 85.7 | 85.1 | 85.7 | 83.0 | 
| COCO | 85.2 | 83.1 | 84.6 | 83.0 |

Table 3:
|evaluation data| tune | copy | distill | scratch |
|---------|-------|---------|--------|---------|
| iNat 2018 | 79.9 | 71.8 | 74.1 | 64.3 | 
| iNat 2019 | 83.8 | 77.9 | 80.0 | 66.2 | 

Table 4:
|out-of-distribution evaluation| tune | copy | distill | scratch |
|---------|-------|---------|--------|---------|
| ImageNet-A | 56.5 | 48.9 | 54.3 | 32.0 |
| ImageNet-R | 59.6 | 57.5 | 56.8 | 51.9 |
| ImageNet-S | 45.2 | 43.1 | 42.9 | 38.0 | 
| ImageNet-V2 | 76.4 | 75.5 | 75.9 | 72.4 | 

Table 5:
|pre-training method | tune | copy | distill |
|---------|-------|---------|--------|
| MAE | 85.7 | 85.1 | 85.7 | 
| MoCo-v3 | 84.0 | 82.5 | 83.3 | 
| FLIP | 87.4 | 86.6 | 86.1 | 
| none | 83.0 | 72.7 | 76.3 | 

Table 6:
|model| tune | scratch | copy | distill |
|---------|-------|---------|--------|---------|
| ViT-B | 82.5 | 83.6 | 82.0 | 83.4 | 
| ViT-L | 83.0 | 85.7 | 85.1 | 85.7 |
|ViT-H | 83.0 | 86.9 | 86.1 | 86.3 |

Meta-review: This paper proposes a transformer attention distillation scheme to transfer knowledge to a student model trained from scratch, as a alternative to fine-tuning the teacher model itself.

The paper is found well written and easy to follow, the idea interesting and well-motivated, the analysis well-thought and the experiments extensive, showing the effectiveness of the approach.

There are a number of weaknesses identified:

1. it is not clear if it is the attention or the query features that is the primary factor
2. the approach is overclaimed as an alternative of fine-tuning, especially given its increased cost
3. attention distillation is similar to previous work

The authors addressed most concerns successfully in the rebuttal. Only one review remains negative on the basis of point 1 above, but the AC finds this point adequately addressed in "Author Rebuttal by Authors". The other three reviews are positive.

It is recommended to accept the paper as poster. However, some additional feedback from the AC follows on points 2 and 3 above, which is recommended to follow in revising the paper:

On point 2, section A.2 shows memory and time per iteration of the proposed approach, which is higher than fine-tuning. Total epochs are given for the proposed approach in Tables 12 and 13, but not for fine-tuning. It is necessary to know the total cost of proposed vs. fine-tuning for all experiments and tasks. Since fine-tuning is cheaper, the authors specify some alternative advantages of the proposed approach in their response to Reviewer 9FPV. These should be used to update the motivation of the paper. The third advantage ("Sharing weights can incur security risks") is not clear because to compute teacher attention maps, one still needs to share weights. Is it also suggested to remove "Who Needs Features?" from the title, since the authors admit that the proposed approach will not be a direct replacement of fine-tuning.

On point 3, the authors should consider the following existing work for example:

1. Wang et al. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. NeurIPS 2020.
2. Li et al. Neural attention distillation: Erasing backdoor triggers from deep neural networks. ICLR 2021.
3. Wang et al. Attention distillation: self-supervised vision transformer students need more guidance. BMVC 2022.
4. Dhar et al. Learning without memorizing. CVPR 2019.

It is clear that attention distillation is not new as a method and has been applied to both ConvNets and Transformers for different tasks and purposes. This leaves as main contribution the use of attention distillation as an alternative to fine-tuning. The paper should thus be repositioned, including a paragraph on attention distillation in the related work. In addition, these papers include more variants, e.g. to apply distillation per head or average attention maps first. It is recommended to add such variants in the ablations.

One final point on the quality of attention maps. It is discussed in “ViTs Need Registers” (which is mentioned by the authors in the discussion) that the quality of attention maps in transformers is low in general. Registers are a way to improve it. Other examples are [51], which argues that quality is better in the intermediate layers, and SimPool (ICCV 2023), which discards the CLS token. It is recommended to consider such improved attention maps in additional experiments.
Decision: Accept (poster)