Adaptive Attention Link-based Regularization for Vision Transformers
Authors: Heegon Jin
Jongwon Choi
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Vision transformers
Knowledge transfer
Knowledge distillation

Abstract: Although transformer networks are recently employed in the various vision tasks with the outperforming performance, large training data and a lengthy training time are required to train a model to disregard an inductive bias. Using trainable links between the channel-wise spatial attention of a pre-trained Convolutional Neural Network (CNN) and the attention head of Vision Transformers (ViT), we present a regularization technique to improve the training efficiency of Vision Transformers (ViT). The trainable links are referred to as the attention augmentation module, which is trained simultaneously with ViT, boosting the training of ViT and allowing it to avoid the overfitting issue caused by a lack of data. From the trained attention augmentation module, we can extract the relevant relationship between each CNN activation map and each ViT attention head, and based on this, we also propose an advanced attention augmentation module. Consequently, even with a small amount of data, the suggested method considerably improves the performance of ViT while achieving faster convergence during training.

Adaptive Attention Link-based Regularization for Vision Transformers
Anonymous Author(s)
Affiliation
Address
email
Abstract
Although transformer networks recently employed in the various vision tasks with the outperforming performance, large training data and a lengthy training time required train model to disregard inductive bias. Using trainable links between the channel-wise spatial attention of a pre-trained Convolutional Neural Network (CNN) and the attention head of Vision Transformers (ViT), we present a regularization technique to improve the training efficiency of VIT. The trainable links are referred as the attention augmentation module, which is trained simultaneously with VIT, boosting the training of VIT and allowing it to avoid the overfitting issue caused by lack of data. From the trained attention augmentation module, we can extract the relevant relationship between each CNN activation map and each ViT attention head, and based on this, we also propose an advanced attention augmentation module. Consequently, even with small amount of data, the suggested method considerably improves the performance of ViT while achieving faster convergence during training.
11
13
Introduction
15
Convolutional Neural Networks (CNN) have become standard for solving image-related tasks using deep neural networks since the advent of large publicly available datasets 212) Recently, models with attention mechanisms mainly adopted the area of natural language processing are becoming take part in solving image-based based tasks, which is called the Vision Transformer (ViT) 3 4 20 a transformer- based neural network fed by the patches of images with class-token for classification, replacing its input of the embedded words in natural language processing. Although VIT shows 22 impressive accuracy compared to modern CNNs by ignoring the inductive bias of locality, a significant amount of data required to train the model achieve satisfactory performance without overfitting issues. Furthermore, most researchers with the limited computing hardware are not affordable to train the ViT due its lengthy training time.
16
18
VIT is
23
24
25
26 The overfitting and lengthy training issues must be solved to broaden the usability of ViT, so many recent studies have tried to solve the problem [5 6 7
g 9 [0]. We can divide the studies by three the parameter compression- method B [14], and the knowledge distillation-ban based method 6 [5]. The advanced architecture-
28
categories: the advanced architecture-base based method
30 based methods manipulate the architecture of ViT to achieve improved training efficiency and
31 generalized prediction even with the small dataset. On the other hand, the parameter compression-
32 based methods focus on a low-rank approximation of the transformer encoder in ViT, which results
33 in the boosted training speed and the suppressed overfitting issue. The knowledge distillation
34 methods utilize the prediction of additional CNN models to avoid the overfitting problem and achieve
35 rapid training convergence. The previous studies have shown the meaningful development of ViT for
38 the small dataset and the reduced training time.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

37 However, the previous studies have the remaining limitations where the training datasets must be
38 equivalent for both the student and teacher models. The architectural manipulation of initial ViT []
39 cannot be applied to different versions of ViTs, hence limiting the handling of new ViT models. Even methods can be employed with the small manipulation of a
40 though the knowledge distillation-base model such as the knowledge distillation token, it should be assured to have the initial models trained
41
42 by a target dataset, which takes the additional costs for the acquisition of initial models before the training of main ViT model.
43
44
In this paper, we propose a novel regularization method to reduce the convergence time and avoid the overfitting problem on small dataset, simultaneously. The proposed method utilizes an attention augmentation module containing multiple trainable weights that estimate the affinity between the channel- -wiee activation map of CNN and the head-wise attention map of ViT. Since the attention augmentation module is located to regularize the attention of ViT heads, the architecture of ViT can be perfectly preserved, which lets us enable to employ the proposed algorithm in ViT variants based on the attention maps. In addition, since the attention map can be obtained even when the task of the pre-trained CNN is not equivalent to the target task, we can employ our method without the pre-trained CNN model with the same target dataset. We validate our regularization method by using ImageNet and CIFARI 10 datasets with various scenarios, which show the outperforming accuracy and the reduction of epochs required for its training convergence. Furthermore, we investigate the important factors for ViT to avoid the overfitting issue by analyzing the trained weights of the attention augmentation module, and through the investigation, we present the dissimilarity of the deep layers' roles between CNN and ViT.
45
46
47
48
49
50
51
52
53
54
56
58
We can summarize our contributions as following:
59
We propose a novel regularization method resolve the issues of overfitting and lengthy training time of ViT through the trainable attention links between the ViT attention maps and CNN activation maps.
60
61
The proposed scheme preserves the original architecture of ViT, which results in its general employment regardless of the architecture of ViT.
63
64
Through the proposed algorithm, the performance of ViT can be dramatically improved with the limited size of dataset, and the training time is reduced without the loss of performance in various scenarios.
66
67
The relationship between ViT and CNN is analyzed in terms of attentional regions, which validates the analysis from the previous studies.
2
Related Work
69
70
2.1
Transformers in Vision
71 Transformer models introduced by [16] are neural networks that purely utilize the attention mech-
72 anism. While they have been used broadly in the field of natural language processing, Vision
73 Transformer (ViT) (3] adapted them in the domain of computer vision with minimal modification to
74 its architecture. ViT showed comparable performance to CNN in the condition of large pre-training,
75 For the advanced optimization of VÝT, CaiT 4 used layer normalization in ViT layers and changed
76
the input location of class token to prevent saturation of performance in deep layers. Swin Transform- ers [2] adopted a hierarchical transformer that computes shifted windows to make it suitable for the vision domain. PiT [10] introduced the concept of pooling in VIT from CNN, improving the gener- alization of ViT. T2T-VIT & enhanced sample efficiency by reshaping input tokens and changing the backbone of networks motivated by several CNN architectures. Raghu et al. [17] measured the similarity of representations between specific layers of CNN and ViT using centered kernel alignment. With additional relative positional encoding, Cordonnier et al. [18] proved attention mechanisms in ViTs can perform as convolution layers in CNNs and showed their functional similarity. From the investigation, Con 7 was motivated to use relative positional encoding to give locality the inductive bias of CNNs to VIT.
77
78
79
80
81
83
84
85
86
The research was extended to 19) reparameterizing pre-trained convolutional layers as a format of Con VTT. Refiner [1] tackled the over-smoothing problem between tokens in deep layers of VIT, and relieve it by projecting attention heads into the higher dimensions and applying convolution directly
87
88
89 to attention maps to learn local relationship among the tokens. Those variants of ViT improved
the optimization and data efficiency of the initial ViT model by modifying the architecture itself.
2

---

Teacher CNN
Student VIT
Chisnel-wise
L2 regularization
Augmented
Adaptive
Activation Maps
Attention
Attention Link
Attention
Maps
Maps
Figure 1:
Our distillation procedure. ViT attention maps are augmented through adaptive attention links, which build linear combinations of different original maps. Then, those augmented attention maps mimic CNN activation maps at one-on-one correspondence.
91
However, our method does not touch any part of ViT modules, but only connects attention between ViT and CNN, transferring attention through links to give ViT a learning signal from the teacher.
92
93
2.2
Knowledge Distillation:
94 In knowledge distillation, student model leverages pre-trained teacher model's soft prediction divided by the same temperature values The softened predictions can be regarded as label- smoothing, and by using them, the student model can achieve the data augmentation effect. Distillation between different types of neural architectures has also been proposed, DistilBERT [21 showed the effectiveness of distilled knowledge from BERT [22] into LSTM 23 DeiT 6 distilled knowledge from CNN to VIT, which seems similar our work. The knowledge distillation has been also employed into the transformer- based model of natural language processing, which results in the performance improvement by using the teacher model [24 25 However, in contrast the previous study using the prediction for the knowledge distillation, our framework transfers the knowledge based the similarity of the latent feature maps. As result,
95
96
96
100
102
extend the range of teacher models to cover the models in which the prediction vectors differ from the prediction of the task.
104
105
On the other hand, we can transfer latent representations of teacher models to those of student models. FitNets [26) improved the stability of deep network training by guiding the latent layers to the teacher's well-trained latent representation. Zagoruyko al. [27 considered attention as projected activation maps of CNN into a spatial dimension, which could regarded as spatial attention. They showed that spatial attention contains valuable information that is useful to improve the performance
106
108
110
of the student network. Kim et al. (28 used a paraphraser to extract and pass the teacher's 's knowledge
to the student's 's translator to learn its representation. Meanwhile, Heo et al. [5] demonstrated that
112 the knowledge transfer based on the neurons' activation is a more classification frienndlrllyyyy approach
than the direct transfer using output values. Attention-bart feature distillation (29) measured the
114
similarities between teacher and student features through attention, which determines the importance
of knowledge to transfer.
3 Attention Link-based ViT Regularization
116
117
In this section, we first explain the backgrounds of the self-attention mechanism and VIT. Then, we explain the method to extract the attention maps from ViT, followed by the description of the architecture and the training method of the augmented attention module is described. The overall framework is depicted in Fig.
118
119
120
121
3.1
Background of VIT
122
first explain the self-attention mechanism and the original ViT model referred to by B] The self-attention mechanism mimics the human cognition system making the attention to the external

---

124
stimulus, which is designed by a transformer- based model with the attention matrix estimated by pairs of key and query.
125
Self-attention Mechanism: We define the input sequence by X E R.L.Din where L. is the length of the sequence and Din means the dimension of one sequential element in the sequence. Then, we can estimate the elements of the attention mechanism composed of key, query, and value vectors by linearly projecting the input sequence by the corresponding embedding weights Ws, &, W,, and W respectively. Thus, when we define the key, query, and value vectors by Q, K, and V, respectively, we obtain the vectors as following:
126
128
129
130
131
K = XWW,, &. Q = XW 9 V = -XW.,
(1)
where W E R...n x Dread from W E (W&, Wa, Wo} and Dhead is the dimension of the head embedding.
132
133
134
Then, the self-attention of the head can be estimated by:
f(x) = (QK" //Deaad V E R... Dhead
(2)
135
where (Z) is a function to transfer each row vector of input matrix Z by softmax. According to the derivation, self-attention can consider the semantic dependency among sequential inputs.
136
137
Many transformer-base models are based on the architecture stacked by the Multi-Head Self- Attention layers (MHSA) containing multiple self-attention heads with independent embedding weights. For the given input X, we define the output m-tth self-attention head at re-hh level depth
139
by m) (X). Then, we denote the corresponding key, query, and value vectors by K (m,n)) (m,,)) and (m.n): respectively.
141
142
The original ViT model directly employed the conventional model built for the natural language processing of the visual classification task. We can summarize the inference process of the original ViT as following. At first, we divide an input image by p2 patches with the same size and sequentially order the patches after their vectorization. Since the transformer network is invariant to the order of the sequential data, ViT concatenates positional embedding vectors to the input patches to represent the original position of the patch.
VIT Framework:
seaeformer-basse
143
144
145
146
147
We define the sequential data obtained from one image by Xn E RP*DD.m where Dim is the size of the vector linearly projected from the vectorized image patch and the positional embedding vector. Before we feed Xo into the transformer modules, the trainable class token sized by R Rom is
148
149
150
sequentially connected ahead of X which results in Xo E R(P2+1)xDD Di..
151
152
The transformer-basee encoders of ViT are the modules containing the series of a layer normalization, a self-attention multi-head module, a fully connected layer, and a layer normalization, where every normalization layer has residual connection. We define the serial process by function of X
153
(X,,) +1)x where D is the size of latent vectors. When N modules are stacked in the ceassfoomrbbbbset encoder, the class-wise score is estimated by linearly projecting the final output of the class token as following: p(cls X)- = so softmax(FCC (FC(NN)) For detailed explanation of ViT.
155
156
157
158
you can be referred to 33.
159
3.2
Attention Map Extraction
160
We need compare the ViT attention map and the CNN activation map for our regularization-bass algorithm. Instead of the relative positional embedding 77 or the attention bias 11. we preserve the
161
162
original architecture of VIT to generalize the usability of our framework to cover the ViT variants.
163
To obtain the attention map from the original ViT, we utilize the attention value between the class token and the image patch. The class token takes key role to determine the final prediction, so we can assume that the attention to the class token may represent the importance of image patches for the classification result. Thus, when feeding the class token into the transformer module as its query vector, we obtain the attention value by estimating the dot product between the embedding vectors of the class token and the corresponding image patch. Then, for m-th head in n-th multi-head self-attention layer, we can estimate the attention value as:
165
167
189
(m, = Rec(s (Q(m.n) O (m.n) :17) E PxP,
(3)

---

170
where Rec is a function to reconstruct the rectangular matrix of P x P from its input vector of pe according the order of the sequential patches, and Q (m..n]00 [0] and (m.n) [1 represent the first query vector of the class token and the key vectors of the image patches, respectively.
171
172
173
In the case of the CNN activation map, we extract the activation maps after the normalization of every convolution block. Instead of integrating the channel-wise activation maps, we consider the separated improve the degree of freedom of our attention augmentation module. In contrast to the constant resolution of VIT attention maps, the resolution of the CNN activation maps decreases with deep layers by pooling layers and strides of convolution layers. Thus, to preserve the resolution of every activation map, we resize all the CNN activation maps to have the same size with the ViT attention maps by using bicubic interpolation. We define the c-th resized CNN activation map by B., where e E {l.... C} and C' is the number of entire CNN activation maps.
174
175
activation maps independently
178
177
178
179
180
181
3.3
Attention Augmentation Module
3.3.1
Module Architecture
Even though both the CNN activation and ViT attention maps represent the key parts of the target object for the prediction, their distribution such as a center point and a variance would be different from each other due to the dissimilarities of their operations. For example, while the VIT attention map is distributed between and because of the softmax estimation, the values in the CNN activation map are normalized by batch normalization, which contain negative values. Furthermore, in general, the number of CNN activation maps is much larger than the number of ViT attention maps due to the large channel-win -wie depth of CNNs. As result, it is impossible to directly compare each of the CNN activation maps with the ViT attention maps.
184
186
188
180
The attention augmentation module is designed to solve the problems of different distributions and a varying number of maps. We design the attention augmentation module to contain multiple attention links which are the trainable weight parameters to scale the ViT attention maps. By estimating the weighted summation of VIT attention maps with the attention links, we can obtain the augmented attention maps where the number is equivalent to the number of CNN activation maps. Thus, we can estimate the augmented attention maps as following:
192
194
196
M.N
A
Σ
w (m. (m,,) + be
(4)
M,,=1
where we w! [Im..)) and be are the attention link and a trainable bias for e-th augmented attention map (c E (1,..., C}), respectively. M is the number of self-attention heads in one level depth and N
197
198
(m,n) is used to analyze presents the maximum level depth. Note that the weight of attention link we w.
199
200
the strength of connectivity for each CNN/ViT layer in section 4.1.
201
We implement the augmented attention module by a 1 x 1 convolution layer generating C' 'augmented attention maps from tensor of R R²² XMN where the ViT attention maps are stacked. Because we
202
203
only use the augmented attention maps only for the training loss, the attention augmentation module has no role in the inference, which can be removed after the training of VIT.
204
3.3.2
Module Training
By using the augmented attention module, we can obtain the same number of augmented attention maps A with the CNN attention maps B To ignore the remaining scale gap between the two maps, we first apply the 12 normalization, and then the mean squared error is estimated to build the attention-basee regularization loss as:
207
209
Latt
| / la
Bellale
(5)
210
Then, we integrate the iidentiio-bbsed regularization loss Catt with the cross-entropy loss Lee of original VIT as:
211
f == + XLatt;
(6)
212
where X a scaling factor to control the effect of our regularization. Since the regularization loss can work as an obstacle to ignoring the inductive bias, referred by 77, we suppress the value of À at the specified epochs to increase the effectiveness of the cross-entropy loss. We exponentially decay the
214
value of À by multiplying decay constant between and 0 at every epoch.
5

---

Waggheet
Imagetee
032
330
007
004
002
(a) ImageNet
(b) ImageNet-55%
(c) Cifar-
Figure 2: Relations between CNN Activation Maps and VIT attention maps. The x-axis and y-axis indicates the depth of ViT layer and CNN block, respectively. We obtain the heatmap by averaging the magnitudes of attention links.
CNN Block Level
2
3
a-link 3-link
1 ~ 3
I~5
3~5
4
4~6
6
4~6
6 6~
11
7 ~ 12
ViT Layer Level
1 2~2
I~5
3~5
4~
6
4~7
6
10
10
Table I: Selective Link Configuration.
4
Link Selection for Advanced Regularization
217
In this section, we build the advanced architecture of the attention augmentation module based on the analysis of the fully-trained attention links. After showing the resultant attention links, we explain the advanced link designed by considering the relation between CNN activation and VIT attention maps.
219
220
4.1
Analysis of Resultant Attention Links
We visualize heat maps to show the scale distribution of the attention links after their training. To compare the relationship between the VIT attention and the CNN activation maps, we only consider the magnitude of the weight parameters in the attention links. As shown in Fig. 2 we obtain multiple heat maps by using three datasets including ImageNet 30 5% subset ImageNet, and
222
224
Cifar-10 10 3
226
the ViT attention maps are highly related to the CNN activation maps located at similar level. The results validate that the self-attention multi-heads of ViT can train the hierarchical information by the stacked architecture, which is similar to the training mechanism of CNN. Thus, the CNN activation maps would be helpful to regularize the VIT attention maps if we can find their level similarity.
As analyzed in many previous studies [8 17
227
228
230
231
In addition to the well-known hypothesis, the heat maps from the attention links show the interesting characteristic where the high-level heads are not regularized when the large dataset is given. When
232
233
large dataset is provided, we can observe the suppressed magnitudes of the attention links at high-level heads in comparison small datasets. The discovery exploits the high-level heads should not be trained by the regularization of the high-level CNN layers, which verifies the high-level heads can represent more complicated semantic information than CNN layers. other words, the representation
234
235
236
237
can be seen out of the inductive bias of CNNs, so we can show semantic information that overwhelms the inductive bias is hard be trained without large dataset.
238
239
DeiT 6 and Swin Transformers 12 showed that the employment of the inductive bias of CNN is effective for the training of VIT. At the end of the training, we observed that high-level heads are disconnected from augmented attention maps, which means they are no more regularized by CNN activation maps. This indicates that high-level heads would escape the inductive bias of locality, and
240
241
242
243
they are trained by a long-range dependency that cannot be acquired by CNNs.
We summarize the two hypotheses from the analysis following:
245
The ViT attention and the CNN activation maps are similar to each other similar level.
246
The high-level VIT heads can present the semantic information that cannot be represented by the CNN layers, but training the semantic information requires large dataset.

---

Train
Top-1
Top-S
Size
DeiT
ConViT
AAL (Ours)
Gap
Deit
Von
AAL(Ours)
Gap
5%
34.8%
47.8%
51.7%
49%/8%
57.8%
70.7%
75.9%
31%/7%
10%
48.0%
59.6%
64.7%
35%/99
71.5%
80.3%
85.8% %
20%/77%
30%
66. 66.1%
73.7%
76.1% %
15%/4%%
86.0%
90.7%
93.0%
8%/3%%
50%
74.6%
78.2%
78.9%
6%/1%%
91.8%
93.8%
94.5%
9%/1%
100%
79.9%
81.4%
81.0%
1%/00%
95.0%
95.8%
95.5%
1%/0%%
Table 2:
ImageNet test accuracy with different sampling ratios. The Gap columns represent the relative performance improvement of the AAL over DeiT and ConViT, respectively.
248
4.2
Selective Attention Link
249
Based on the analysis, we additionally propose the selective attention link improve the training efficiency of the attention augmentation module. Instead of the full link in the original attention augmentation module, only a part of the attention links are utilized to obtain the augmented attention maps. Based on the two hypothesis from our analysis, the augmented attention map is generated only by the ViT attention maps with the similar levels, and no link connected to the high-level ViT attention maps when the training dataset is sufficiently large.
250
251
253
255
Accordingly, we build two types of selective attention link, which are denoted by a-link and B-link. a link connects the VIT attention maps to only the augmented attention maps at a similar level. 3-link similar the o-link but the links to the high-level heads are entirely disconnected. The detailed connections are given in Table I
258
257
Experiments
259
In experiments, showed that transferring attention from pre-trained CNN models ViTs inject 260
281
CNN's inductive bias (i.e locality) naturally in standard self-attention layers, without the necessity of additional modules extending the self-atiention network. Thus, we examine how efficiently the method helps ViT to be converged for optimal performance. especially showing large gap in a small
262
283
284
data regime.
285
5.1
Experimental Settings
286
The computing re- source used in our experiments is Nvidia A100. If not mentioned otherwise, the student ViT
Implementation Details:
Models
DeïT-B
CiT
AAL
287
Top-1
97.5%
95.4%
97.5%
268
289
model used for experiments is DeiT-S (distilled version) and used EfficientNet-B3 32 as the
Table 3: CIFAR10 Top-1 test accuracy
270
teacher CNN model. We set À to 2000 and the decay constant for À is set to 0.99 for the first 200
272
epochs and 0.98 for the last 100 epochs. For fair comparison, we preserve the values of the remaining hyperparameters and the training strategies from our baseline model of DeiT 6
274
Comparisons and Dataset: For comparison, we consider two previous studies, which include DeiT and ConVil DeiT utilizes the knowledge distillation method to improve the ViT-based models, and Con VIT shows the state-of-the-e performance when the training data is given sufficiently even without using the knowledge distillation methods. To show the generality of our algorithm, we utilize four classification datasets: ImageNet, CIFARIO, Caltech-UCSD Birds-200-20 201-2011 (CUB-200), and Oxford 102 Flowers (Flower-100 102). In the case of ImageNet, we extract the subsets randomly sampled
275
276
277
278
279
280
with the various ratios (5%, 10%, 30%, 50%), maintaining class balance, to show the validity of the proposed algorithm when the insufficient data is given for the training.
282
5.2
Quantitative Results
283
We first perform the comparisons with the various subsets of ImageNet. As shown in Table 2 the
284
proposed algorithm shows the state-of-the-art performance when the ImageNet subsets are used
to train the model. The performance of our framework is similar to that of Con VIT when the
286
entire dataset considered for training. However, the performance gap between our framework and Con ViT becomes enlarged with the insufficient training data. Furthermore, we should notice that
7

---

Train Size
Top-1
Top-5
Full-link
Selective-link
Gap
Full-link
Selective-link
Gap
5%
48 48.9%
51.7%
5.7%
73.6%
75,9%
3.1%
10%
63.0%
64.7% %
2.6%
84.6%
85.8% %
1.4%
30%
75.2%
76.1 %
1.2 %
92.4%
93.0%
0.6%
50%
78.5%
78.9% %
0.5%
94.3%
95.0%
0.7%
100%
81.0%
80.9%
-0.1%
95.5%
95.5%
0.0%
Table 4: ImageNet test accuracy with different sampling ratios. The Gap columns represent the
relative performance improvement of Selective-link.
Teacher Model
Student Model
Teacher Model
Student Model
Top-1
Top-1
ResNet34
DeiT-S w/ distill
73.3%
79.4%
EfficientNet-B3
DeïT-B w/ distill
81.1%
82.8% %
Table 5: ImageNet test accuracy with various teacher and student models
288
EfficientNet-B3 that is our teacher model needs only 12.2M parameters, which is much smaller
than 86.6M parameters of RegNetY-16GF 16GF (33 used in DeiT 6 Thus,
validate that our proposed framework can overwhelm DeiT-B even by using the light teacher model. In addition, while
290
Con ViT-S needs 5M more parameters than ours or DeiT, our method outperforms both of DeiT and
Con ViT-S, which validates the efficiency of our framework. The quantitative results for CUB and Flower datasets are represented in the supplementary material.
292
293
294
Table shows the experimental results with CIFARIO dataset. The results verifies that the proposed algorithm can increase the robustness to the insufficient size of training data. In addition, in the DeiT paper. 7200 training epochs were needed to achieve 97.5% top-1 test accuracy when training from scratch using the DeiT-B model which has more attention heads than DeīT-S. On the other hand, our method only needed to train 300 training epochs to reach the same test accuracy while using the DeîT-S model, which validates its training efficiency.
296
298
300
5.3
Analysis
In addition to the following analysis, we present the additional experiments show the validity of our framework in the supplementary material. The additional experiments include the performance of weakly supervised object localization, the qualitative results for attention maps, the learning curve, and epoch-wise qualitative changes of attention links.
302
303
304
To show that our selective attention link-based transfer efficiently matches ViT attention maps with CNN activation maps, we compared two different settings on the attention augmentation module. Full-link fully connects each VIT attention map to produce augmented attention maps that match CNN activation maps as one-to-one channel-wise correspondence. In the used for the selective link, while we utilized a-link for
Effectiveness of Selective Links:
306
308
case of the full ImageNet dataset, B-link the other small datasets. As shown in Table 4 the attention transfer with a fully connected attention
310
link shows superior performance to the accuracy of DeiT and ViT (Table 2 in low data regime, and the selective attention links show further improvement from its results.
312
Robustness to Variety of Models: We add results with the variants composing of different teacher and student models to show the generality of our method upon various environments. As shown in Table 5 the proposed framework successfully improves the performance of its teacher model even with the different teacher and student models. Interestingly, when we use light teacher model, we can achieve the large performance gap between the teacher and student models.
314
316
318
Data and Model Efficiency: Our additional trainable module, which is the attention augmentation module, includes only a single 1x1 Conv layer which augments the attention maps of the student VIT.
319
320
In our default settings, the number of the parameter 0.068M, which quite small compared to DeïT-S of 22M parameters. As we mentioned, the lengthy training time of ViT is a critical drawback especially when the computational resources are limited. The reduced training time of our method
322
can be validated through the learning curve represented in the supplementary material.
8

---

Strong Data Aug. (Default)
Weak Data Aug.
Methods
Top-1
Top-5
Methods
Top-1
Top-5
Cross Entropy (CE)
91.3%
99.6%
Cross Entropy (CE)
84.2%
98.7%
CE + AAL
97.4%
99.9%
CE + AAL
92.5%
99.7%
CE + Soft Distillation
91.0%
99.6%
CE + Soft Distillation
84.0%
98.9%
CE + Hard Distillation
92.0%
99.8%
CE + Hard Distillation
85.1%
99.0%
CE AAL Hard Dist.
96.5%
99.9%
CE AAL Hard Dist.
94.1%
99.7%
Table 6: Ablation Test with different settings of data augmentation and distillation methods
324
Ablation studies: For additional verification of our knowledge transfer method, we trained on CIFARIO with different scenarios. To sternly check the performance difference of knowledge dis- tillation effect from each method, we used much weaker data augmentation than the setting used in other experiments with only simple techniques such as random crops and horizontal flips. This allows us to confirm the data efficiency in a low data regime. In addition, we compared our method
325
328
327
328
329
to other knowledge distillation methods introduced by DeiT with teacher model pre-trained on
330
the CIFARIO dataset. As shown in Tāble for both soft label distillation and hard label distillation, distillation method, From this result, we could infer that directly transferring attention gives a better learning signal than giving the teacher model's s output predictions. Furthermore, we could confirm that knowledge earned from a large dataset can give good learning direction. Compared to the result of Table where the teacher model pre-trained by ImageNet was used, the performance drops due to the lack of information in the teacher model trained by CIFARIO. This could be another advantage since teacher models in DeiT are restricted to be trained on the target dataset to give proper output prediction.
331
our method outperforms the class prediction-base
333
334
335
337
338
Various Baseline: In Table 7 we show that applying our method is not only limited to standard ViT. In the experiments, we employ our method to Pooling based ViT (PÝT-S) [IO], and we observed the sample efficiency of the model increased by large margin using our method.
339
340
Robustness to random initialization: We per- form several trials with different random seeds as shown in Table g Our algorithm shows con- sistency even with the various initial parameters.
342
Pit-S
Pit-S + AAL
343
344
Top-1 1 Top-5 25.2%
12.2%
44.0%
345
Due our limited computation, we run 240 epochs of training in contrast to 300 epochs of training the default setting.
67.3%
346
Table 7: Our method with PiT (ImageNet 5%)
347
348
Prediction-based distillation and Fine-tuning: Our method can be jointly applied with class prediction-based distillation. Thus, both Table and Table verify that our method can show synergy with the distillation method proposed in 6 In addition, we perform the additional comparison to
348
350
fine-tuning algorithms with self self-supervised learning (SSL. for ViT 34, SSL with linear classifier, and SSL with k-NN classifier respectively show 77% and 74.5% for ImageNet top-1 test accuracy.
352
353
Every accuracy is lower than our performance of 81.0% with the same ViT model, which validates the synergy of our method with the self-supervised fine-tuning mechanisms.
354
Conclusion
355
356
In this paper, we have introduced a novel method of transferring knowledge from CNN to ViT. By accessing attention of CNNs and adaptively adopting them, student VIT was able to earn high quality of learning signal with CNN's inductive bias. By applying our method, we could train ViT in less training epochs without overfitting
357
Trial I
Trial II
Trial III
358
Top-1 1 Top-5
47.3%
46.5%
47.2%
360
71.9%
71.3%
72.0%
361
362
Table 8: Repeated trials (ImageNet 5%)
363
even with the small dataset or limited labeled
data. Also, we revealed relations between inter-
385
mediate representations from those different types of neural networks, which varied due to the training
386
dataset. Furthermore, by analyzing those relationship with trained attention links, we could take
387
advantage of more efficient connection between networks. We leave wider application of our methods
to new ViT architectures in future works.

---

References
389
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In N/PS, 2012.
371
372
[2]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
373
374
[3] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
378
377
378
[4] Hugo Touvron, Matthicu Cord, Alexandre Sablayrolles, Gabriel Synnacve, and Hervé Jégou. Going deeper with image transformers. ArXiv, abs/2103. 17239, 2021.
379
380
[5] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In AAAI, 2019.
381
6 Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In /CML, 2021.
383
[7] Stéphanc d' Ascoli, Hugo Touvron, Matthew L. Leavitt, Ari S. Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In ICML, 2021.
385
386
[8] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay. Jiashi Feng. and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV. 2021.
388
[9] Andreas Steiner, Alexander Kolesnikov. Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization vision transformers, 2021.
390
[10] Bycongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Scong Joon Oh. Rethinking spatial dimensions of vision transformers. ArXiv, 0ts/2103.16302. 2021.
382
383
Daquan Zhou, Yujun Shi, Bingyi Kang, Weihao Yu, Zihang Jiang, Yuan Li, Xiaojie Jin, Qibin Hou, and Jiashi Feng. Refiner: Refining self-attention for vision transfommers. ArXiv, abs/2106.037/ 2021.
385
[12] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9992- 10002. IEEF, 2021.
397
388
[13] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In ICLR, 2021.
400
401
402
[14] Sinong Wang, Belinda Z. Li. Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with lincar complexity, ArXiv, abs/2006.0444 04768,477828 2020.
405
[15] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, ArXiv, abs/2104. 14294, 2021.
405
[16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
407
[17] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? ArXiv, abs/2108.08810, 2021.
409
[18] Jean- Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. In ICI.R, 2020.
412
[19] Stéphane d' Ascoli, Levent Sagun, Giulio Biroli, and Ari Morcos. Transformed cnns: recasting pre-trained convolutional layers with self-attention. ArXiv, hbs//106.05795.. 2021.
413
414
[20] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv. abs/1503.02531, 2015.
418
[21] Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task task-specific knowledge from BERT into simple neural networks. ArXiv, abs/1903.12213,, 2019.
10

---

418
[22] Jacob Devlin, Ming-Wo Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
420
[23]
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 988):135--1881 1997.
422
[24] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-
distillation for task-agnostic compression of pre-trained transformers. In NeurlPS, 2020.
424
[25] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In EMNLP, 2020.
426
[26]
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
427
428
[27]
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
429
430
[28] Jangho Kim, Sconguk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In NIPS, 2018.
432
[29] Mingi Ji, Byeongho Heo, and Sungrae Park. Show, attend and distill: Knowledge distillation via attention- based feature matching, AAAI, 2021.
433
434
[30] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
435
437
[31] Alex Krizhevsky, Vinod Nair. and Geoffrey Hinton. Cifar-10 10 (canadian institute for advanced research).
438
[32]
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
439
440
[33] Ilia Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. CoRR, abs/2003. 13678, 2020.
441
442
[34] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEF/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 9630-
443
444
9640. IEEE, 2021.
Checklist
448
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A]. You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
448
449
450
451
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data proprietary.
453
454
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/amswwes below.
456
458
1. For all authors...
459
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section
461
(b) Did you describe the limitations of your work? [Yes] See Section 6
462
(c) Did you discuss any potential negative societal impacts of your work? [No] Our research do not have any potential negative societal impacts
11

---

484
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
465
466
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3
(b) Did you include complete proofs of all theoretical results? (Yes] See Section
3. If you ran experiments..
470
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section
471
472
4
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5
474
475
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Section 5
478
477
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5
479
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
480
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 5
481
(b) Did you mention the license of the assets? [Yes] See Section 5
482
(c) Did you include any new assets either in the supplemental material or as a URL? [No] There are no new assets
485
484
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] only open sources
486
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
487
488
5. If you used crowdsourcing or conducted research with human subjects...
489
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [No]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [No]
492
493
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [No]
494
12


Reviews:
[{"review_id": "pTJg_iFaJTM", "reviewer": "Reviewer_Fq3N", "summary": "The paper introduces a novel regularisation method to train ViT. Specifically, they rely on the activation features from a convolutional neural network to transfer the knowledge and augment the training of the vision transformer. Furthermore, they discuss how to select which links to keep during training in order to make the learning more efficient. Finally, they ablate and evaluate the approach on image datasets such as ImageNet and CIFAR.", "questions": "My concerns are listed in the weaknesses section. I would expect authors to discuss:\n\n- Include additional metrics in the table for a fair comparison with the baselines. What is the effect of the cost of running the CNN? \n\n- Have the authors considered other task beyond classification? \n\n- Have the authors considered using other datasets?\n\n- Have the authors considered including more qualitative analysis of the learned model compared to the baseline?", "limitations": "Although in the form authors claim that Section 6 discussed limitations, I do think it would be necessary to extend this discussion for the paper to be accepted. I see Section 6 as a conclusion but no specific mention to the limitation is done.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- S1. The approach is simple and effective. Authors join the advantages of CNNs and ViT with their augmentation procedure in order to make learning more efficient. I really like the simplicity of the approach, which also would make it more likely to be used by the community.\n\n- S2. Analysing the most important links is also one extra step in terms of efficiency. The paper does a good job introducing the overall intuition and the general method and then going into link selection.\n\n- S3. The results show how the approach can be specially useful for low data regimes.\n\n- S4. Most of previous works using transformers specially rely on very large datasets while this paper tackles the problem of training in a low data regime.\n\n- S5. Authors provide sufficient experimental evidence to answer the main questions in the paper. The ablation study is complete and useful for the reader.\n\n**Weaknesses**\n- W1. In my opinion the paper avoid the issue of the additional overhead of running the CNN alongside the transformer training. I think the comparison table should take that into account and report FLOPS or runtime for a complete picture.\n\n- W2. The methodology has only been used for classification. However, it seems like other task which require for awarness of the space would be a better fit for such a paper. I wonder if authors have considered evaluating on additional task.\n\n- W3. Authors evaluate on CIFAR and ImageNet, which are very saturated dataset at this point. Although I think the point of the paper is valid, I would appreciate additional evaluations.\n\n- W4. I am missing some qualitative comparison on *why* the techniques improve data efficiency. What is different in the learning curves? How do the attention weights change? \n\n- W5. Have the authors played a bit with the CNN architecture? I wonder if they could safe computation by running a very basic model. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bhz8LzRV8q0", "reviewer": "Reviewer_xcGt", "summary": "This paper proposes a loss regularization method for training Vision Transformer (ViT) through matching the attention maps of ViT with the activation map of a pre-trained CNN. The authors aggregate ViT's attention maps from different layers into the number of CNN's activation maps and then use their L2 norm as the additional loss regularization of training ViT. It is shown that the proposed method can outperform the selected baseline on ImageNet when the number of training samples is small.", "questions": "See weaknesses above.", "limitations": "I didn't check but I don't think this work will lead to a potential negative societal impact", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe idea of matching attention maps of ViT with the activation maps looks interesting, and the motivation behind this seems reasonable.\n\nWeakness\n1. How do the authors learn the w_c in Eq. 4, is it end-to-end trainable with Eq. 6 or learned separately? Besides, do the attention map (A_c) and activation map (B_c) in Eq. 5 have the same shape? If not, how do you address it?\n2. The computation complexity of Eq. 4 is also a concern. Since w_c is posed for each pixel in A_(m,n), the complexity of w_c will grow exponentially as P or the number of layers in ViT grows. How will it perform when facing larger or deeper ViT models?\n3. I have checked the manuscript, but I cannot find the initialization value of λ in Eq. 6. Please include it and justify the value choice as it is an important hyper-parameter.\n4. The current description of the experimental setting and results, in my opinion, is difficult for the readers to understand. Please re-organize it - first describe the dataset and different experiment settings, and then explain the results one by one. \n5. I don't think Figure 2 is sufficient to support the two hypotheses in 4.1. Besides, the selective attention link should be discussed in more detail, e.g. why it can achieve better results than the full link.\n6. Please clarify whether the baseline model in Table 4 has been trained with a teacher model or not. If not, the comparison may not be fair. Besides, it seems the performance cannot outperform the ConViT with the full train set, and therefore the generality to large-scale data is in doubt.\n\nOverall, although the idea of this paper is interesting, the methodology is not explained or designed properly, the descriptions of the current experimental settings and results are difficult to read, and also the achieved performance is not convincing. I don't think this paper can be accepted, and I tend to vote for rejection.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Swgcg0vQvD0", "reviewer": "Reviewer_DJ1i", "summary": "This paper proposes a new knowledge distillation method to improve the training efficiency of ViT. More concretely, the weighted average of the attention maps in the student ViT model is regularized to approach the activation maps of the CNN teacher model, which is the so-called attention link. Based on the empirical observations, the paper further makes the link selective by only building relations between CNN activation maps and ViT attention maps at similar levels and excluding the attention maps at the high levels. Empirically, the proposed method is compared with DeiT, ConViT and other knowledge distillation methods on ImageNet or CIFAR10. In the setting of small-sized data, the attention link is shown to improve the test accuracy with shorter training time and less computation cost.", "questions": "1. In Natural Language Processing, knowledge distillation has also been widely adopted in transformer-based models to enhance training efficiency. In particular, MiniLM (Wang et al., 2020) and TinyBert (Jiao et al., 2019) also focus on the regularization of attention maps. I think it’s worth including this line of literature in Related Work.\n2. In Table 1, each CNN block level is associated with different numbers of ViT layer level. And beside the ViT layers at high level, the way how others are assigned to CNN block levels is also different between the alpha-link and the beta-link. I would appreciate it if there is any ablation study or verbal explanation on such a “non-intuitive” design. \n3. In Table 2, could you please explain what the two numbers in each cell stand for? (as it’s not clearly written in the caption or the main text.)\n4. In Figure 2, if my guess about the x-axis corresponding to the ViT layers is correct, I’m wondering how the weights of different heads within each layer are aggregated to produce the plot. Besides, as it’s hard to tell that the block diagonal is generally darker than the side in Figure 2(b) and (c), would you mind elaborating more on how to reach the conclusion that “the ViT attention maps are highly related to the CNN activation maps located at a similar level”? \n5. I would appreciate it if the authors could explain more about the argument in line 239 - “the inductive bias is hard to be trained without a large dataset”.\n6. As stated in Question 1, there are other knowledge distillations methods. For example, they may use a transformer as the teacher model instead of a CNN and also utilize the attention map regularization. I’m wondering whether it would be better to add the empirical comparison with these methods. ", "limitations": "Please refer to the Weakness and Questions section for details.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The writing of Sections 1-3 is clear and easy to follow.\n2. The idea of regularizing the global token’s attention maps with the CNN activation maps is novel.\n3. In the setting of small-sized data, the improvements in accuracy and training time are impressive.\n\nWeaknesses:\n(Please refer to the Questions section for more details.)\n1. Some of the tables and figures lack elaboration in the caption.\n2. More empirical evidence is needed to support the superiority of the attention link.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "56HfaXRneGe", "reviewer": "Reviewer_StQc", "summary": "This paper introduces a new method to increase data efficiency for Vision Transformers (VTs). The idea of the method is that Adaptive Attention Links (AAL) between the convolutional filters of a CNN and between the attention heads of the VT are trainable parameters which connect these feature maps between the teacher CNN and the student VT, with only a small parameter increase. This way, the locality inductive bias can be learned from the teacher model, decreasing training time. The paper confirms previous results that higher layer attention maps in a VT contain more high-level semantic information than a CNN, but that other layers roughly correspond to the same type of hierarchical information. Extensive experiments show that the method improves on previous distillation approaches between a CNN and a VT. ", "questions": "1. How was lambda = 0.99 and 0.98 chosen? (L288)\n2. In lines 37-43, do you mean source dataset instead of target dataset?\n3. L175-176, did you find an improvement when using channel-wise attention maps compared to some aggregate version of attention maps?\n4. What is your interpretation of the augmented attention links, what do they represent? Is it mainly about their increased flexibility of representing the CNN maps?\n5. Are the X/Y results in the tables always teacher/student?\n6. What does attention maps “at a similar level” mean? At a similar layer depth?\n7. L266, when you say remarkable results, which results are you referring to then? If you say a strong word such as remarkable it should be clear what you mean, apart from that the result should be strong too, of course. \n8. L327-332: do you compare self-supervised approaches to your classically supervised approach here? Or do you make some kind of SS version of your approach? This is not clear\n", "limitations": "The choice of the hyper parameter lambda could be more accounted for (did it include tuning, etc). The authors do not state any potential limitations of their work, which seems a little worrying, perhaps, in terms of transparency. There are always limitations. However, if this can be added the work is sound.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: The idea of the paper is novel, to the best of my knowledge.\nS2: The idea is clear and conceptually appealing.\nS3: The paper presents extensive experimentation to support its conclusions (a number of different datasets and variants of models). \nS4: The paper promotes better data efficiency for VTs, which is a well-known issue.\nS5: The paper presents ablations and one example of repeated runs which show stable results (this seems reasonable in order to save computation).\nS6: the method is flexible with regards to the VT student architecture, any model containing hierarchical attention maps can be used, as well as wrt the teacher model: any CNN containing hierarchical feature maps can be used. The down-stream task of the teacher model does not matter.\n\nW1 The main weakness of the paper, in my view, is its clarity. This can be straightforwardly fixed by doing a more thorough language check, for example by using Grammarly or similar. Many sentences are hard to parse, eg lines 320-321, 234-239. Furthermore, the captions to tables and figures should be more informative which would help the reading a lot, right now they are not possible to get the gist from without finding their reference in the text. Also, captions should not be boldfaced, but that is a detail of course. \n\nW2, W3: Other weaknesses are lacking mention of how the hyper parameters lambda were chosen, and a lacking discussion of the potential weaknesses of the method— this should be easily amendable.\n\nW4: No code is released.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject