Transformed CNNs: recasting pre-trained convolutional layers with self-attention
Authors: Stéphane d'Ascoli
Levent Sagun
Giulio Biroli
Ari S. Morcos
Venue: ICLR 2022 Submitted (2022)
Keywords: convolutional networks
transformers
hybrid
fine-tuning

Abstract: Vision Transformers (ViT) have recently emerged as a powerful alternative to convolutional networks (CNNs).  Although hybrid models attempt to bridge the gap between these two architectures, the self-attention layers they rely on induce a strong computational bottleneck, especially at large spatial resolutions. In this work, we explore the idea of reducing the time spent training these layers by initializing them from pre-trained convolutional layers. This enables us to transition smoothly from any pre-trained CNN to its functionally identical hybrid model, called Transformed CNN (T-CNN). With only 50 epochs of fine-tuning, the resulting T-CNNs demonstrate significant performance gains over the CNN as well as substantially improved robustness. We analyze the representations learnt by theT-CNN, providing deeper insights into the fruitful interplay between convolutions and self-attention.

Under review as a conference paper at ICLR 2021
TRANSFORMED CNNs:
RECASTING PRE-TRAINED
CONVOLUTIONAL LAYERS WITH SELF-
Anonymous authors Paper under double-blind review
ABSTRACT
Vision Transformers (ViT) have recently emerged as a powerful alternative to convolutional networks (CNNs). Although hybrid models attempt bridge the gap between these two architectures, the self-attention layers they rely on induce a strong computational bottleneck, especially large spatial resolutions. In this work, we explore the idea of reducing the time spent training these layers by ini- tializing them from pre-trained convolutional layers. This enables us transition smoothly from any pre-trained CNN to its functionally identical hybrid model, called Transformed CNN (T-CNN). With only 50 epochs of fine-tuning, the result- ing T-CNNs demonstrate significant performance gains over the CNN well substantially improved robustness. We analyze the representations learnt by the T- T-CNN, providing deeper insights into the fruitful interplay between convolutions and self-attention.
INTRODUCTION
Since the success of AlexNet in 2012 (Krizhevsky et al., 2017), the field of Computer Vision has been dominated by Convolutional Neural Networks (CNNs) (LeCun et al., 1998; 1989). Their local receptive fields give them a strong inductive bias to exploit the spatial structure of natural images (Scherer et al., 2010; Schmidhuber, 2015; Goodfellow et al., 2016), while allowing them to scale to large resolutions seamlessly, Yet, this inductive bias limits their ability capture long-range interactions.
In this regard, self-attention (SA) layers, originally introduced in language models (Bahdanau et al., 2014; Vaswani et al., 2017; Devlin et al., 2018), have gained interest as a building block for vision Ramachandran et al. (2019); Zhao ct al. (2020). Recently, they gave rise a plethora of Vision Transformer (ViT) models, able to compete with otate-of-the-art CNNs in various tasks Dosovitskiy al. (2020); Touvron et al. (2020); Wu et al. (2020): Touvron al. (2021): 1); Liu al. (2021); Heo et al. (2021) while demonstrating better robustness (Bhojanapalli et al., 2021; Mao et al., 2021). However, capturing long-range dependencies necessarily comes at the cost of quadratic complexity in input size, computational burden which many recent directions have tried to alleviate (Bello, 2021; Wang et al., 2020; Choromanski et al., 2020; Katharopoulos et al., 2020). Additionally, ViTs generally harder to train (Zhang et al., 2019; Liu et al., 2020), and require vast amounts of pre-training (Dosovitskiy et al., 2020) or distillation from a convolutional teacher (Hinton et al., 2015; Jiang et al., 2021; Graham al., 2021) match the performance of CNNs.
Faced with the dilemma between efficient CNNs and powerful ViTs, several approaches have aimed to bridge the gap between these architectures. On one side, hybrid models append SA layers onto convolutional backbones (Chen et al., 2018; Bello et al., 2019; Graham et al., 2021; Chen et al., 2021; Srinivas et al., 2021). and have already fueled successful results in variety of tasks (Carion et al., 2020; Hu et al., 2018; Chen et al., 2020; Locatello et al.. 2020; Sun et al., 2019). Conversely, a line of research has studied the benefit of introducing convolutional biases in Transformer architectures to ease learning (d' Ascoli et al., 2021; Wu et al., 2021; Yuan et al., 2021). Despite these interesting compromises, modelling long-range dependencies at low computational cost remains challenge for practitioners.
Contributions At a time when pre-training on vast datasets has become common practice, we ask the following question: does need to train the SA layers during the whole learning process? Could

---

Under review as a conference paper at ICLR 2021
70
Imageh et cccurrsy
InmspatNe -C accurasty
ImageNeet.l accuracy
10
TaagsNNett accurary
50
acourccy
50
60
Vir
30
40
40
DefT
ConVir
50
20
30
30
VIT-21k
ResNet
30
10
20
ResNet- RS
T-ResNet-
10a
10a
10a
Model size (M)
Model size (M)
Model sice (M)
Model size (M)
Model Size (M)
60
anrgraarcrruaaey accuracy
84
ImageNee accurescy
ImageNett A accurscy
40
ImageNei-R eccurssy
-50
accurazy
30
82
60
30
40
40
VIT
DelT
Imagehet
ConVil
50
20
30
30
VIT-21k
10
20
20
ResNet-RS
T-ResNet-RS
10a
Flogs (G)
Flops (G)
Flops (G)
Flops (G)
Flops (G)
Figure 1: Transformed ResNets strike strong accuracy-robustness balance. Our models (red) significantly outperform the original ResNet-RS models (dark blue) they were initialized from when evaluated on ImageNet-Ik. k. They also exhibit solid performances on various robustness benchmarks (ImageNet-C. A and R, and FGSM adversarial attacks from left to right).
one instead learn cheap components such as convolutions first, leaving the SA layers to be learnt the end? this paper, we take a step in this direction by presenting a method to fully reparameterize any pre-trained convolutional layer et al., 2021). The latter is initialized to reproduce the mapping of the convolutional layer, but is then encouraged to learn more general mappings which are not accessible to the CNN by adjusting positional gating parameters.
Gated Positional Self-Attention (GPSA) layer (d' Ascoli
We leverage this method reparametrize pre-trained CNNs functionally equivalent hybrid models. After only 50 epochs of fine-tuning, the resulting Transformed CNNs (T-CNNs) boast significant performance and robustness improvements as shown Fig. 1, demonstrating the practical relevance of our method. We analyze the inner workings of the T-CNNs, and show that they learn more robust representations by combining convolutional heads and attentional heads in a complementary way.
Related work Our work mainly builds two pillars. First, the idea that SA layers can express any convolution, introduced by Cordonnier et al. (2019). This idea was recently leveraged by d' Ascoli al. (2021), which initialize the SA layers of end-to-end Transformers random convolutions to imbue them with local inductive bias and improve their sample efficiency. Our approach leverages the opposite idea: giving an end-to-end CNN the freedom escape locality by learning self-attention at late times.
Second, exploit the following learning paradigm: train simple and fast model, then reparame- terize it as more complex model for the final stages of learning. This approach was studied from scientific point of view in d' Ascoli al. (2019), which shows that reparameterizing a CNN fully-commn fully-connected network (FCN) halfway through training can lead the FCN to outperform the CNN. Yet, the practical relevance of this method limited by the vast increase in number of parameters required by the FCN to functionally represent the CNN. In contrast, our reparameterization hardly in- creases the parameter count of the CNN, making it easily applicable any state-of-the-ard CNN. Note that these reparameterization methods can be viewed an informed version of dynamic architecture growing algorithms such as AutoGrow (Wen et al., 2020).
In the context of hybrid models, various works have studied the performance gains obtained by introducing MHSA layers ResNets with minimal architectural changes (Srinivas et al., 2021; Graham al., 2021; Chen et al., 2021). However, the MHSA layers used in these works initialized randomly and need to be trained from scratch. Our approach is different, as it makes use of GPSA layers, which can be initialized to represent the same function the convolutional layer
2

---

Under review as a conference paper at ICLR 2021
it replaces. We emphasize that the novelty in our work is not in the architectures used, but in the unusual way they are blended together.
BACKGROUND
Multi-head self-attention The SA mechanism is based on trainable associative memory with (key, query) vector pairs. To extract the semantic interpendencies between the L elements of sequence X E Lx2x..,, sequence of "query" embeddings Q = Wayy E RLxDx is matched against another sequence of "key" embeddings result is an attention matrix whose entry (ij) quantifies how semantically relevant Q; is to K,:
Weey X RRLDx using inner products. The
OK
A = softmax
E RLXL
(1)
Drn
Multi-head SA layers use several SA heads in parallel to allow the learning of different kinds of dependencies:
MSA(X) Σ (X)) W out?
SAA(X) A XW vol>
where Wh val R.. and W R.Dxx D are two learnable projections.
To incorporate positional information, ViTs usually add absolute position information to the input at embedding time, before propagating it through the SA layers. Another possibility is to replace the vanilla SA with positional SA (PSA), by including a position-dependen term in the softmax (Ra- machandran et al., 2019; Shaw al., 2018). Although there several ways to parametrize the positional attention, we use encodings rij of the relative position of pixels i and j as in (Cordonnier al., 2019):
A softmax (Q + oopotri))
Each attention head learns an embedding eo E Pyos, and the relative positional encodings Ty E R Dpos only depend on the distance between pixels and j, denoted denoted as a two- dimensional vector dzy.
Self-attention as a generalized convolution Cordonnier al. (2019) shows that a multi-head PSA layer (Eq. 3) with N₁, heads and dimension Dpoe 2 3 can express any convolutional layer of filter size /N with Dan input channels and min( Dow)) output channels, by setting the following:
h upss == aa (1. 1 -24 0,...
{
(118112 62, ......)
W wry W Whey ==
In the above, the center attention A R 2 is the position to which head h pays most attention to, relative to the query pixel, whereas the locality strength a > 0 determines how focused the attention around its center A When a large, the attention is focused only the pixel located at A h: when a is small, the attention is spread out into a larger area. Thus, the PSA layer can achieve convolutional attention map by setting the centers of attention A f each of the possible positional offsets of a VN, x VN, convolutional kernel, and sending the locality strengths a to some large value.
2
APPROACH
In this section, we introduce method for mapping convolutional layer functionally equivalent PSA layer with minimal increase in parameter count. To do this, we leverage the GPSA layers introduced in d' Ascoli al. (2021).

---

Under review as a conference paper at ICLR 2021
Loading the filters We want each head h of the PSA layer to functionally mimic the pixel h of convolutional filter Witerr E R Na x Pra Dout, where we typically have Dout 2 Din. Rewriting the action of the MHSA operator in a more explicit form, we have
NA MHSA(X) - Σ A h=1
W W
(5)
WhERPii Dout
In the convolutional configuration of Eq. 4, A selects pixel h of X. Hence, we need to set Wh W liter- However, as product of matrices, the rank of Wh h is bottlenecked by D.. To avoid
this being a limitation, we need D, 2 Din (since Dont 2 Din). To achieve this with minimal
number parameters, we choose D, = Dins and simply set the following initialization:
W val I,
W out = W
(6)
filter
Note that this differs from the usual choice made in SA layers, where D. = /N. However, to keep the parameter count the same, we share the same Wh. eat across different heads h, since it plays symmetric role at initialization.
Note that this reparameterization introduces three additional matrices compared to the convolu- tional filter: Wary, W Wkey: Wea,, each containing Din x Din parameters. However, since the convolutional filter contains N₁, Dan x Dout parameters, where we typically have N₂. = 9 and Dous E (Di 2D these additional matrices are much smaller than the filters and hardly increase the parameter count. This can be seen from the model sizes in Tab.
Gated Positional self-attention Recent work (d' Ascoli al., 2021) has highlighted an issue with standard PSA: the fact that the content and positional terms in Eq. 3 are potentially of very different magnitudes, in which case the softmax ignores the smallest of the two. This can typically lead the PSA to adopt greedy attitude: choosing the form of attention (content or positional) which is casiest given time then sticking to it.
To avoid this, the authors suggest to sum the content and positional terms after the softmax, with their relative importances governed by a learnable gating parameter ' (one for each attention head). The resulting Gated Positional Self- Self-Attenioon (GPSA) layers are parametrized as follows:
A (1 o (Ar)) softmax (Q) T) (An) softmax ( ri))
where a d > 1/(1+=-)) is the sigmoid function. the positional part, the encodings rsy are fixed rather than learnt (see Eq. 4), which makes changing input resolution straightforward (see SM. C) and leaves only 3 learnable parameters per head: A A and a..
How convolutional should the initialization be? The convolutional initialization of GPSA layers involves two parameters, determining how strictly convolutional the behavior is: the initial value of the locality strength a, which determines how focused each attention head on its dedicated pixel, and the initial value of the gating parameters X, which determines the importance of the positional information versus content. If A > and a >> 1, the T-CNN will perfectly reproduce the input-output function of the CNN, but may want to greedily stay in the convolutional configuration. Conversely, if >o, < and a < 1, the T-CNN will forget about the input-output function of the CNN. Hence, we choose a 1 and À 1 to lie in between these two extremes, encouraging the T-CNNs to escape locality throughout training.
Architectural details To make our setup as canonical possible, we focus on ResNet architec- tures (He et al., 2016), which contain 5 stages, with spatial resolution halfed and number of channels doubled each stage. Our method involves reparameterizing 3 3 convolutions as GPSA layers with 9 attention heads. However, global SA is too costly in the first layers, where the spatial resolution is large. We therefore only reparameterize the last stage of the architecture, while replacing the first stride-2 convolution by a stride-1 I convolution, exactly as in (Srinivas et al., 2021). We also add explicit padding layers to account for the padding of the original convolutions.
Since represents the temperature of the softmax, its value must stay positive all times. To ensure this, instead learn rectified parameter using the softplus function: = log(1 e -"*), with / 5.
4

---

Under review as a conference paper at ICLR 2021
Imageheet accuracy
84
entegeNeet sccurscy
125
Image-Nott acmuray
40
ImsigeNat accarary
PGSM anmarny
50
300
2.0
40
80
ResNet
T-BosNot-RS
30
10
10P
10
10"
10"
10
10a
10'
training time
training time
Total training time
Total training time
Total training time
Figure 2: T-CNNS reach much higher performance and robustness at equal training time. Total training time (original training + finetuning) is normalized by the total training time of the ResNet50- RS.
PERFORMANCE OF THE TRANSFORMED CNNS
In this section, we apply reparamctrization state-of-the CNNs, then fine-tune the resulting T-CNNs to learn better representations. This method allows to fully disentangle the training of the SA layers from that of the convolutional backbone, which is of practical interest for two reasons. First, it minimizes the time spent training the SA layers, which typically have a slower throughput. Second, it separates the algorithmic choices of the CNN backbone from those of the SA layers, which typically different; for example, CNNs are typically trained with SGD whereas SA layers perform much better with adaptive optimizers such as Adam (Zhang et al., 2019), an incompatibility which may limit the performance of usual hybrid models.
Note that our reparametrization can also be applied halfway through training: this scenario is investigated in Sec. 5. Results suggest that reparametrizing at intermediate times is optimal in terms of sreeddperformance trade-offs. However, for easier reproducibility, we foeus on reparametrizing the fully pre-trained CNNs available in the t iimm package (Wightman, 2019): this avoids having to retrain the various models from scratch.
Training details We applied our method to pre-trained ResNet-RS (Bello et al., 2021) models, using the weights provided by the t timm package (Wightman, 2019). These models are derived from the original ResNet (He et al., 2016), but use improved architectural features and training strategies, enabling them to reach better speed-accuracy trade-off's than EfficientNets.
To minimize computational cost, we restrict our fine-tuning to 50 epochs². Following (Zhang et al., 2019), we use the AdamW optimizer, with a batch size of 1024 The learning rate is warmed up to thenthen annealed using cosine schedule. To encourage the T-CNN to escape the convolutional configuration and learn content-based attention, we use larger learning rate 0. for the gating parameters of Eq. 7 (one could equivalently decrease the temperature of the sigmoid function).
We use the same data augmentation scheme as the DeiT (Touvron et al., 2020), as well as rather large stochastic depth coefficients d, reported in Tab. 1. Hoping that method could be used as an alternative to the commonly used practice of fine-tuning models at higher resolution, we also increase the resolution during fine-tuning (Touvron et al., 2019). In this setting, ResNet50 requires only 6 hours of fine-tuning on 16 V100 GPUs, compared to 33 hours for the original training. For our largest model (ResNet350-RS), the fine-tuning lasts 50 hours.
Performance improvements Results are presented in Tab. 1, where we also report the baseline improvement of fine-tuning in the same setting but without SA. In all cases, our fine-tuning improves top-1 accuracy, with a significant gap over the baseline. To demonstrate the wide applicability of our method, report similar improvements for ResNet-D architectures in SM. E.
Despite the extra fine-tuning epochs and their slower throughput, the resulting T-CNNs also out- performing the original CNNs on the ImageNet validation set at equal training budget, shown
We study how performance depends on the number of fine-tuning epochs in SM. D. CConfrriing the results of (Zhang al., 2019), obtained worse results with SGD.
5

---

Under review as a conference paper at ICLR 2021
Training
Fine-tuning
Backbone
Without SA
With SA
Res.
d,
TTT
Top-1
Res.
dy
TTT
Top-1
TTT
Top-1
ResNe150-RS
160
0.0
1 (ref.)
78.8
224
0.1
1.16
80.4
1.30
81.0
ResNet101-RS
192
0.0
1.39
80.3
224
0.1
1.65
81.9
1.79
82.4
ResNe1152-RS
256
0.0
3.08
81.2
320
0.2
3.75
83.4
4.13
83.7
ResNet200-RS
256
0.1
4.15
82.8
320
0.2
5.04
83.7
5.42
84.0
ResNet270-RS
256
0.1
6.19
83.8
320
0.2
7.49
83.9
7.98
84.3
ResNet350-RS
288
0.1
10.49
84.0
320
0.2
12.17
84.1
12.69
84.5
Table Statistics of the models considered, trained from seratch on ImageNet. Top-1 accuracy is measured on ImageNet- Ik validation set. "TTT" stands for total training time (including fine- tuning), normalized by the total training time of the ResNet50-RS. d, the stochastic depth coefficient used for the various models.
50
noise
Rel. Improvement
Rel. Improvement
blur
40
40
digital
weather
30
20
20
10
10
2
3
5
brightness elastic transform
Jpeg compression glass blur
pixelate
motion blur
2o0m blur defocus blur
shot noise
gaussian noise frost snow impulse noise contrast
fog
Corruption severity
Figure 3: Robustness is most improved for strong and blurry corruption categories. We report the relative improvement between the top-1 accuracy of the T-ResNet50-Re and that of the ResNet50- RS on ImageNet-C,, averaging the different corruption categories (left) and corruption severities (right).
in the leftmost panel Fig. 24, However, the major benefit of the reparamctrization is terms of robustness, as shown in Fig. 2(b) and explained below.
Robustness improvements Recent work has shown that Transformer- architectures offer better robustness and out-of-domain generalization than convolutional architectures (Bhojanapalli et al., 2021; Mao et al., 2021; ; Mahmood et al., 2021; Shao et al., 2021). To investigate whether our fine-tuning procedure is enough to imbue CNNs with these advantages, we evaluate our T-CNNs on various benchmarks:
Common corruptions: we use ImageNet-C (Hendrycks & Dietterich, 2019), dataset containing 15 of randomly generated corruptions, grouped into categories: 'noise". 'blur', 'weather' and 'digital'. Each corruption type has five levels of severity, resulting in 75 distinct corruptions. Note that to avoid distorting the corruptions, which are often pixel-based, we keep resolution of 224 at inference, which disadvantages the large models trained at higher resolutions.
Adversarial robustness: following (Mao et al., 2021), we evaluate the accuracy of our models under two white-bow a (i) single-step FGSM (Goodfellow et al., 2014) and (ii) multi-step Les -PG (Madry et al., 2017) with t 5 steps of size 0.5. Both attackers perturb the input
We estimated the training times of the original ResNet-RS models based on their throughput, for the same hardware as used for the T-ResNet-RS.
We use the toolkit provided by https: thub. cww/bethge ab/ foolbox

---

Under review as a conference paper at ICLR 2021
image with max magnitude 1. We also evaluate models ImageNet-A (Hendrycks et al., 2021), a dataset containing naturally "adversarial" examples from ImageNet. Note however that since this dataset is built from the flaws of a ResNet, it is potential unfair to CNNs.
we use ImageNet-R (Hendrycks et al., 2020), a dataset with various stylized "renditions" ImageNet images ranging from paintings to embroidery, which strongly modify the local image statistics.
Distribution shifts:
The full table of results presented in Tab. of SM. A, and illustrated in Figs. 1 and 2. The T-ResNet-RS substantially outperforms the ResNet-RS on all robustness benchmarks. For example, our T-ResNet1O1-RR, which 50% faster than ResNet200-RS reaches similar or better results all robustness tasks, despite its lower top-1 accuracy on ImageNet-1k: This demonstrates that SA improves robustness more than improves classification accuracy. The most striking improvement is in terms of adversarial robustness, where the smallest T-ResNet-RS is on par with the largest ResNet-RS despite requiring 5 times less compute.
To better understand where the benefits come from, decompose the improvement of the T- ResNet50-RS over the various corruption severeties and categories of ImageNet-C in Fig. 3. We observe that improvement increases almost linearly with corruption severity. Although performance higher in all corruption categories, there a strong variability: the T-CNN shines particularly in tasks where the objects in the image less sharp due lack of contrast, bad weather blurriness. We attribute this to the ability of SA to distinguish shapes in the image, as investigated in Sec 4.
4 DISSECTING THE TRANSFORMED CNNS
In this section, we analyze various observables understand how the representations of a ResNe(270-RS evolve from those of the ResNe1270-RS throughout training.
A
B
C
85
1.0
Layer
Top-1 accuracy
80
Attention span
3
Layer 1
0.8
75
Test Train
Layer 2
0.6
2
Layer 3
0.4
Positions
65
0.2
0.0
20
40
20 Epochs
40
20 Epochs
40
Epochs
Figure 4: The later layers effectively escape the convolutional configuration. A: top-1 accuracy throughout the 50 epochs of fine-tuning of T-ResNet270-RS. B: size of the receptive field of the various heads (thin lines), calculated as 0,," (see Eq. 3). Thick lines represent the average over the heads. C: depicts how much attention the various heads à (thin lines) pay positional information, through the value of (An) (see Eq. 7). Thick lines represent the average over the heads.
Unlearn to better relearn In Fig. 4A, we display the train and test accuracy throughout training The dynamics decompose into two distinct phases: accuracy dips down during the learning rate warmup phase (first epochs of training), then increases back as the learning rate is decayed.
As shown in SM. B, the depth of the dip depends on the learning rate. For too small learning rates, the dip is small, but the test accuracy increases too slowly after the dip; for too large learning rates, the test accuracy increases rapidly after the dip, but the dip is too deep to be compensated for. This suggests that the T-CNN needs to "unlearn" to some extent, a phenomenon reminiscent of the "catapult" mechanism of Lewkowyez al. (2020) which propels models out sharp minima to land in wider minima.
'The train accuracy is lower than the test accuracy due to the heavy data augmentation used during fine-tuning.
7

---

Under review as a conference paper at ICLR 2021
Head
Head
Head
Head
Sager
Layer
Layer
(a) Input image
(b) Attention maps
Figure GPSA layers combine local and global attention a complementary way. We depicted the attention maps of the four GPSA layers of the T-ResNee270-RS, obtained by feeding the image on the left through the convolutional backbone, then selecting a query pixel in the center of the image (red box). For each head h, indicate the value of the gating parameter a (Ap,) in red (see Eq. 7). In each layer, at least one of the heads learns to perform content- attention (a(An) =0)
Escaping the convolutional representation In Fig. 4B, we show the evolution of the "attention span" 1/an (see Eq. 4), which reflects the size of the receptive field of attention head h. On average (thick lines), this quantity increases in the first three layers, showing that the attention span widens, but variability exists among different attention heads (thin lines): some broaden their receptive field, whereas others contract it.
In Fig. 4C, we show the evolution of the gating parameters X of Eq. which reflect how much atten- tion head h pays to position versus content. Interestingly, the first layer stays strongly convolutional on average, as o, (Aa) rapidly becomes close to one (thick blue line). The other layers strongly escape locality, with most attention heads focusing on content information at the end of fine-tuning.
In Fig. 5, we display the attention maps after fine-tuning. A clear divide appears between "convolutional" attention heads, which remain close to their initialization, and the "content-based attention heads, which learn more complex dependencies. Notice that the attention head initially focusing the query pixel (head 5) stays convolutional in all layers. Throughout the layers, the edges of the central object is more and more clearly visible, as observed in (Caron et al., 2021). This supports the hypothesis that robustness gains obtained for blurry corruptions (see Fig. 3) are partly due to the ability of the SA layers to isolate objects from the background.
5 WHEN SHOULD ONE START LEARNING THE SELF-ATTENTIN LAYERS?
We have demonstrated the benefits of initializing T-CNNs from pre-trained CNNs, very compelling procedure given the wide availability pretrained models. But one may ask: how does this compare to training hybrid model from scratch? More generally, given computational budget, how long should the SA layers be trained compared to the convolutional backbone?
Transformed CNN versus hybrid models To answer the first question, we consider a ResNet-50 trained on ImageNet for 400 epochs. We use SGD with momentum 0.9 and a batch size of 1024, warming up the learning rate for epochs before a cosine decay. To achieve strong baseline, we use the same augmentation scheme as in Touvron al. (2020) for the DeiT. Results reported in Tab. 2. In this modern training setting, the vanilla ResNet50 reaches solid performance 79.04% on ImageNet, well above the 77% usually reported in litterature.
The T-CNN obtained by fine-tuning the ResNet for 50 epochs at same resolution obtains a top-1 accuracy of 79.88%, with 15% increase in training time, and 80. 84 resolution 320, with 35% increase in training time. In comparison, the hybrid model trained for 400 epochs in the same setting only reaches 79.95%. in spite of 40% increase in training time. Hence, fine-tuning yields better results than training the hybrid model from scratch.
What is the best time to reparametrize? We now study a scenario between the two extreme cases: what happens if reparametrize halfway through training? To investigate this question in
8

---

Under review as a conference paper at ICLR 2021
Name
t
f2
Train time
Top-1
Vanilla CNN
400
0
2.0k mn
79.04
Vanilla CNN1320
450
0
2.4k mn
79.78
T-CNN
400
50
2.3k mn
79.88
T-CNN1320
400
50
2.7k mn
80.84
Vanilla hybrid
0
400
2.8k mn
79.95
T-CNN"
100
300
2.6k mn
80.44
T-CNN"
200
200
2.4k mn
80.28
T-CNN"
300
100
2.2k mn 79.28
Table 2: The benefit of late reparametrization. We report the top-1 accuracy of ResNet-50 on ImageNet reparameterized at various times 1 during training. 1320 stands for fine-tuning at resolution 320. The models with a keep the same optimizer after reparametrization, in contrast with usual T-CNNs.
systematic way, train the ResNet50 for & epochs, then reparametrize and resume training for another l2 epochs, ensuring that & + f2 = 400 in all cases. Hence, & 400, amounts to the vanilla ResNet50, whereas i corresponds the hybrid model trained from scratch. To study how final performance depends on t in fair setting, we keep the same optimizer and learning rate after the reparametrrizttoon, in contrast with the fine-tuning procedure which uses fresh optimizer.
Results are presented in Tab. 2. Interestingly. the final performance evolves nnn-moootonically. reaching maximum of 80.44 for 4 = 100, then decreasing back down as the SA layers have less and less time to learn. This con-monotonicity is remarkably similar to that observed in d' Ascoli et al. (2019), where reparameterizing a CNN as a FCN in the early stages training enables the FCN to outperform the CNN. Crucially, this result suggests that reparametrizing during training not only saves time, also helps the T-CNN find better solutions.
DISCUSSION
In this work, we showed that complex building blocks such as self-attention layers need not trained from start. Instead, by initializing them from pre-trained convolutional layers. At a time where energy savings and robustness are key stakes, we believe this finding is important.
in compute time while gaining in performance and robustness
On the practical side, our fine-tuning method offers an interesting new direction for practitioners. One limitation of our method is the prohibitive cost of reparametrizing the early stages of CNNs. This cost could however be alleviated by using linear attention methods (Wang et al., 2020), an important direction for future work. Note also that while T-CNNs significantly improve the robustness of CNNs. they do not always reach the performance of end-to-end Transformers such as the DeiT (for example on ImageNet-C, see Fig. 1). Bridging this gap is important next step for hybrid models.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint urXiv:1409.000 2014.
Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. arXiv preprint arXiv:2102.08602, 2021.
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3286-3295, 2019.
Irwan Bello, William Fedus, Xianzhi Du, Ekin D Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. arXiv preprint arXiv: 103.0:2003.0759, 2021.
Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li. Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. arXiv preprint arXiv:2103. 14586, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. arXiv preprint arXiv:2005.12877 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. European Conference on Computer Vision, pp. 104-120. Springer, 2020.
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A2-nets: Double attention networks. arXiv preprint arXiv: 2018
Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer, 2021.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009..4777 2020.
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. arXiv preprint 1xXiv:1911.03584 2019.
Stéphane d' Ascoli, Levent Sagun, Giulio Biroli, and Joan Bruna. Finding the needle the haystack with convolutions: the benefits of architectural bias. In Advances in Neural Information Processing Systems, pp. 9334-9345, 2019.
Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2/03. 10697, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXhv:1810.04805. 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1rXiv:1412.6577 2014.
10

---

Under review as a conference paper at ICLR 2021
Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint drXhv:2104.01136, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 558-567, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer, The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006.1622 2020.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021.
Bycongho Heo. Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Scong Joon Oh. Rethinking spatial dimensions of vision transformers. arXiv preprint arXiv:2103.6633 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: 103.02150,,02531, 2015.
Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3588-3397,, 2018
Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Xiaojie Jin, Anran Wang, and Jiashi Feng. Token labeling: Training 85.4% top-1 accuracy vision transformer with 56m parameters on imagenet, 2021.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mons: Fast autoregressive transformers with linear attention. In International Conference Machine Learning, pp. 5156-5165. PMLR, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Baoppgapagttion applied to handwritten zip code recognition. Neural computation, 1(4):541-551. 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (0(1)):22782333 1998.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.022: 022118. 2020.
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. arXiv preprint arXiv:2004.0822 082499 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
11

---

Under review as a conference paper at ICLR 2021
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv preprint (tt.fx:20066100555 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: htt:::/006.00800 2017.
Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk. On the robustness of vision transformers to adversarial examples. arXiv preprint wxXiv:210400261/ 2021.
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Shaokai Ye, Yuan He, and Hui Xue. Rethinking the design principles robust vision transformer. arXiv preprint arXiv:2105.07926. 2021.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv: / (906. 05909, 2019.
Dominik Scherer, Andreas Müller, and Sven Behnke. Evaluation of Pooling Operations in Con- volutional Architectures for Object Recognition. In Konstantinos Diamantaras, Wlodek Duch, and Lazaros S. Iliadis (eds.), Artificial Neural Networks ICANN 2010, Lecture Notes Com- puter Science, pp. 92-192-122 101, Berlin, Heidelberg, 2010. Springer. ISBN 978-3-642-15825-4 doi: 10.1007/978-3 3-2199978---4---1 15825-4-110000
Jürgen Schmidhuber. Deep learning neural networks: An overview. Neural Networks, 61: 85-117, January 2015. ISSN 0893-6080.
doi: 10.1016fj.mummc22144099000013 URL http: / //www. www.ssscieieeciiiecomco com/ wttp:///aaaeeerr..aaa//////////.
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin- Yu Chen, and Cho-Jui Hsieh. On the adversarial robustness of visual transformers. arXiv preprini arXiv:2103./56700 2021.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:/803.022 2018
Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck Transformers for Visual Recognition. arXiv e-prints, art. 1rXiv:2101.11605, January 2021.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Vidcobert: A joint model for video and language representation learning. In Proceedings of the IEEE International Conference Computer Vision, pp. 7464-7473, 2019.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. arXiv preprint arXiv:1906.06423. 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103.17253: 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and H Linformer Ma. Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
Wei Wen, Feng Yan, Yiran Chen, and Hai Li. Autogrow: Automatic layer growing in deep con- volutional networks. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 833-841, 2020.
Ross Wightman. Pytorch image models.
https: /ttp://///thtbb// wwtp:wwwgttttteeo
pytorch- gytooch--imae-modde mageemodel s, 2019.
12

---

Under review as a conference paper at ICLR 2021
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Masayoshi Tomizuka, Kurt Keutzer, and Peter Vajda. Visual Transformers: Token-based Image Representation and Processing for Computer Vision. arXiv:2006.03671 [cs, eess], July 2020. URL http:://axii..///// abs/2006. 03677. arXiv: 2006.03677.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103. 2021.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers, arXiv preprint urXiv:2103.11816 2021.
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? arXiv preprint 1rXiv::11..033194, 2019.
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10076-10085, 2020.
13

---

Under review as a conference paper at ICLR 2021
Model
Res.
Params
Speed
Flops
IN-IK
IN-C
IN-A
IN-R
FGSM
PGD
Transformers
ViT-B/16
224
M
182
16.9
77.9
52.2
21.9
30.6
14.3
ViT-I/161
224
307 M
55
59.7
76.5
49.3
6.1
17.9
27.8
13.0
ViT-B/16
224
86 M
182
16.9
84.0
65.8
26.7
38.0
31.3
10.3
ViT-L/16
224
307 M
55
59.7
85.1
70.0
28.1
40.6
40.5
16.2
DeiT-S*
224
22 M
544
4.6
79.9
55.4
18.9
31.0
40.7
16.7
DefT-B'
224
87 M
182
17.6
82.0
60.7
27.4
34.6
46.4
21.3
ConViT-s'
224
28 M
296
5.4
81.5
59.5
24.5
34.0
41.0
17.2
V
224
87 M
139
17.7
82.4
61.9
29.0
36.9
51.8
20.8
RVT-S!
224
23.3
4.7
81.9
25.7
47.7
51.8
28.2
RVT-B-
224
91.8 M
17.7
82.6
28.5
48.7
53.0
29.9
CNNs
ResNet50f
224
25 M
736
4.1
76.8
46.1
4.2
21.5
ResNet101
224
45 M
435
7.85
78.0
50.2
6.3
23.0
14.7
2.0
ResNet101x33
224
207 M
62
69.6
80.3
53.4
9.1
24.5
23.6
7.3
ResNet152x4
224
965 M
18
183.1
80.4
54.5
11.6
25.8
33.3
10.5
ResNe150-RS
160
36 M
938
4.6
78.8
36.8
5.7
39.1
28.7
18.4
ResNet101-RS
192
64 M
674
12.1
80.3
44.1
11.8
44.8
32.9
18.8
ResNet152-RS
256
87 M
304
31.2
81,2
49.9
23.4
45.9
41.6
28.5
ResNer200-RS
256
M
225
40.4
82.8
48.1
40.4
24.6
ResNet270-RS
256
130 M
152
54.2
83.8
53.6
26.6
48.7
44.7
30.3
ResNet350-RS
288
164 M
89
87.5
84.0
53.9
34.9
49.7
48.3
34.6
Our Transformed CNNs
T-ResNet50-RS
224
M
447
17.6
81.0
48.0
18.7
42.9
47.2
33.9
T-ResNer101-RS
224
66 M
334
25.1
82.4
52.9
27.7
47.8
50.3
34.2
TRResNe155-RS
320
89 M
128
65.8
83.7
54.5
39.8
50.6
57.3
36.8
T-ResNet200-RS
320
96 M
105
80.2
84.0
$7.0
41.2
$1.1
58.3
36.4
T-ResNet270-RS
320
133 M
75
107.2
84.3
58.6
43.7
36.6
T-ResNe1350-RS
320
167 M
61
130.5
84.5
59.2
44.8
53.8
53.4
36.4
Table Accuracy our models on various benchmarks. Throughput the number of images processed per second V100 GPU batch size 32. For ImageNet-C, we keep resolution of 224 at test time to avoid distorting the corruptions; this disadvantages our large models, which are trained higher resolutions. f: reported from (Mao et al., 2021) (we recalculated ImageNet-C accuracies, as the original paper reports MCE). f: reported from (Bhojanapalli et al., 2021) (in their setup, PGD uses steps with stepsize of 1/8).
A PERFORMANCE TABLE
In Tab. 3, we display the characteristies and the performance of our T-ResNet-RS models and compare them to the original ResNet-RS models as well as several other strong baselines reported Bhojanapalli et al. (2021); Mao et al. (2021).
B CHANGING THE LEARNING RATE
We have shown that learning dynamics decompose into two phases: the learning rate warmup phase, where the test loss drops, then the learning rate decay phase, where the test loss increases again. This could lead one to think that the maximal learning rate too high, and the dip could be avoided by choosing lower learning rate. Yet this not the case, shown in Fig. 6. Reducing the maximal learning rate indeed reduces the dip, but also slows down the increase in the second phase of learning. This confirms that the model needs "unleam" the right amount to find better solutions.
14

---

Under review as a conference paper at ICLR 2021
80.0
Top-1 sccuracy
77.3
75.0
72.3
70.0
1r
0.0005
67.5
10
20
30
40
50
Epochs
Figure 6: The larger the learning rate, the lower the test accuracy dips, but the faster it climbs back up. We show the dynamics of the ResNet50, fine-tuned for 50 epochs at resolution 224, for three different values of the maximal learning rate.
ResNeiRS-50
ResNetRS-101
ResNetRS-152
Top-1 acouracy
82
ResNetRS-200
ResNetRS-270
ResNetRS-350
T-ResNetRS-50
78
T-ResNetRS-101
T-ResNeIRS: -522
T-ResNetRS-200
T-ReaNetRS-22 270
160
192
224
256
298
320
352
384
416
T-ResNetRS-3e 350
Test resolution
Figure Performance different test-time resolutions, for the finetuned models with and without SA. The ResNe150-RS and ResNer101-RS models are finetuned at resolution 224, and all other models are finetuned resolution 320.
C CHANGING THE TEST RESOLUTION
One advantage of the GPSA layers introduced by d' d'Ascoli et al. (2021) is how easily they adapt to different image resolutions. Indeed, the positional embeddings they use are fixed rather than learnt. They simply consist in 3 values for each pair of pixels: their euclidean distance ||ō]], as well as their coordinate distance 8, 82 (see Eq. 4). Our implementation automatically adjusts these embeddings to the input image, allowing us to change the test resolution seamlessly.
In Fig. 7, we show how the top-1 accuracies of our T-ResNet-RS models compares to those of the ResNet-RS models finetuned at same resolution but without SA. At test resolution 416, our T- ResNetRS- reaches an impressive top-1 accuracy of 84.9%, beyond those the best EfficientNets and BotNets Srinivas et al. (2021).
D CHANGING THE NUMBER OF EPOCHS
In Tab. 4, we show how the 1 accuracy of the T-ResNet-RS model changes with the number of fine-tuning epochs. As expected, performance increases significantly fine-tune for longer, yet we chose to set maximum of 50 fine-tuning epuchs to keep the computational cost of fine-tuning well below that of the original training.
15

---

Under review as a conference paper at ICLR 2021
Model
Epochs
Top- acc
ResNe15O-RS
0
79.91
T-ResNet5O-RS
10
80.11
T-ResNet5O-RS
20
80,51
T-ResNet5O-RS
50
81.02
ResNet101-RS
0
81.70
1-ResNee1O1-RS
10
81.54
T-ResNet1O1-RS
20
81.90
T-ResNet1O1-RS
50
82.39
Table 4: Longer fine-tuning increases final performance. We report the top-1 accuracies of models on ImageNet-Ik at resolution 224.
E CHANGING THE ARCHITECTURE
Our framework, which builds on the timm package, makes changing the original CNN architecture very easy. We applied our fine-tuning procedure to the ResNet-D models He et al. (2019) with the exact same hyperparameters, and observed substantial performance gains, similar the ones obtained for ResNet-RS, see Tab. This suggests the wide applicability of our method.
Model
Original res.
Original acc.
Fine-tune res.
Fine-tune acc.
Gain
T-ResNet00-D
224
80.6
320
81.6
+1.0
T-ReeNet101-D
320
82.3
384
83.1
+0.8
T-ReeNet152-D
320
83.1
384
83.8
+0.7
T-ResNer200-D
320
83.2
384
83.9
+0.7
T-ResNet50-RS
160
78.8
224
81.0
+2.8
T-ResNet101-RS
192
81.2
224
82.4
+1.2
T-ResNet1.52RRS
256
83.0
320
83.7
+0.7
T-ResNer200-RS
256
83.4
320
84.0
+0.6
Table 5: Comparing the performance gains of the ResNet-RS and ResNet-D architectures. Top-
I accuracy is measured on ImageNet-Ik validation set. The pre-trained models are taken from the timm library Wightman (2019).
16


Reviews:
[{"review_id": "vEb1zEKoHDE", "reviewer": "Reviewer_zV42", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is simple and straightforward. The motivation makes sense to me and the results look good, however there are several things can be improved.\nThe novelty is somewhat limited, the authors did not really propose new techniques but reused components from the previous paper on a not-so-new setting.\nWhen showing robustness improvements, the authors' motivation is that transformer-based architectures offer better robustness and out-of-domain generalization than convolutional architectures. However, from Fig.1 it seems that the Transformer-based baselines do not clearly outperform the convolutional baselines (ResNet-RS is better sometimes),  also, why there is a significant drop in the FGSM dataset when increasing model size or training time, do the authors have some explanation on these phenomenons?\nWould be great if table 1 can include some transformer baseline for comparison.\n", "summary_of_the_paper": "This paper shows that by leveraging a previous work Gated Positional Self-Attention layer, we can reparameterize the late stages of pre-trained CNNs and make them be hybrid Transformed CNNs. Experiments show that it can boost the performance and robustness, without training transformer block from scratch, reducing the training cost.", "summary_of_the_review": "Overall I think the paper shows an interesting direction and present good results. However, the limited novelty prevents me from giving a higher rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Xy2w8VnhLAN", "reviewer": "Reviewer_ojmG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) The novelty of the proposed method is limited as they do not always reach the performance of end-to-end Transformers. \n2) The results in Table 1 do not show a significant performance improvement. It would be better if the author could discuss situations in which this method is less effective than an end-to-end Transformer. \n3) Since the authors have motioned that the proposed method can save the compute time while gaining in performance and robustness, it needs more evidence that it indeed achieves a lower computational complexity than other hybrid models. \n4) Personally speaking, the organization of this paper needs to be fixed, especially the related work that needs to be reorganized.", "summary_of_the_paper": "This paper shows that complex building blocks such as self-attention layers need not be trained from start. Instead, one can save in compute time while gaining in performance and robustness by initializing them from pre-trained convolutional layers.", "summary_of_the_review": "See the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LUH2XIp9WTJ", "reviewer": "Reviewer_Q4Pp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper is written very well. The authors explain their contributions very clearly, putting them in adequate context, and are upfront about the limitations. The actual methodological contribution, while somewhat incremental, could be potentially quite impactful. In addition, the paper provides some interesting analysis in terms of how the function implemented by the network changes after the self-attention fine-tuning.\n\nLimitations:\n\nThe main limitation of the work is that the experiments are extremely limited in scope. The authors provide results for two ResNet models on ImageNet and three different Imagenet-based robustness tasks. They also provide some results on what happens if finetuning occurs during training, and study the effect of varying the learning rate, but otherwise no other ablation studies are provided. I want to emphasize that the authors justify their choices adequately and thoughtfully, but I believe that additional experiments are warranted to adequately document the potential of the approach. \n", "summary_of_the_paper": "This work proposes an approach to bridge CNNs and vision transformers for image recognition. The idea is to replace the last convolutional stage of a ResNet by a self-attention layer which is initialized from the weights of the convolutional stage. The approach is shown to improve the performance of the CNN, especially in terms of robustness to adversarial examples, corruptions in the input images, and domain shift. ", "summary_of_the_review": "The paper is great, but the experiments are very limited, which I believe it puts it below the bar.\n*****\nAfter the response:\n\nI have decided to upgrade my recommendation. Upon a more careful look through the supplementary material, the paper does include some interesting ablation studies. I think it would good to provide some experiments on a dataset different to ImageNet. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G4xnf9btMYv", "reviewer": "Reviewer_ZjBY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### ***Pros:-***\n- The paper is meticulously written and is structured well. There literature analysis is thorough and covers majority of the related works that inspired the proposed model.\n- The idea explored in the paper of recasting of pre-trained CNN weights to improve training time efficiency is interesting and is well motivated. The main goal of improving train time efficiency is also validated in the experimental section.\n- The experimental analysis is detailed and includes many ablation analysis experiments to analyze different aspects of the proposed model architecture.\n- The experiments also show that proposed model is more robust to common corruptions, distribution shifts or adversarial perturbations than their counterparts with experiments on ImageNet-C, ImageNet-R, ImageNet-A and FGSM/PGD attack types.\n\n### ***Cons:-***\n- One of the concern includes one of the the motivation of the proposed method, which is efficient training time. Since, the training time cost is a one-time cost which incurs only while training the model, it does not seem to be an important aspect of the model. The test-time cost is a much better motivation in comparison as it has many benefits. Having efficient training is indeed an important aspect, but test-time efficiency is much more important than that and current experimental analysis does not include such analysis.\n- The proposed model is not compared with many of the existing model architectures that try to create a hybrid model combining CNN and transformers in interesting manner. Furthermore, most model architecture work provide the benefits of such pre-trained model on any downstream tasks such as detection, segmentation etc, which is also missing from the experimental analysis.\n- The performance comparison is provided with number of parameters vs accuracy graph, which does have its place, but additional comparison with number of FLOPs vs accuracy graph is of more importance as it also provide an important aspect of the model efficiency. I would suggest to include those graph in the main paper compared to having it in the supplementary material as a table.\n- Also it seems that with increase in model capacity the performance of the proposed T-CNN has diminishing returns as CNN vs T-CNN accuracies are very close. A discussion on this should be included to explain when it is beneficial to extend CNNs to T-CNNs.\n\n\n\n\n\n\n\n", "summary_of_the_paper": "The paper explores a hybrid type of model architectures that combines the recently popular transformer based model architectures with already well established convolutional neural networks. Specifically, the proposed hybrid model follows a two-stage training strategy where first a CNN model is trained and the pre-trained weights are used to further improve the training by recasting the weights into a new transformed CNN model termed int he paper as T-CNN. The proposed T-CNN model architecture is able to outperform its CNN counterparts and other models compared in the paper. The representations learnt by the T-CNN are also analyzed in the experimental analysis section.", "summary_of_the_review": "Overall, the paper explores an interesting direction of research. The proposed model architecture seems to be train time efficient and improves both robustness and performance of its counterpart model. There are some concerns that need to be addressed raised in the main review, which should further clarify the technical novelty and motivation for the work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to accelerate training of an architectural hybrid of Transformers and CNNs: first train a CNN and then use the learned parameters to initialize a more general Transformed CNN (T-CNN) model; subsequently continue training the T-CNN.

Reviewers ratings are marginal, with three "marginally above threshold" and one "marginally below threshold".  However, no reviewer makes a compelling argument for acceptance, and all reviewers point to significant weaknesses in the work.  Reviewer ojmG: "novelty of the proposed method is limited" and "do not always reach the performance of end-to-end Transformers".  Reviewer Q4Pp: "experiments are very limited" and also (after rebuttal): "it would good to provide some experiments on a dataset different to ImageNet".  Reviewer ZjBY: "proposed model is not compared with many of the existing model architectures" and (after rebuttal): "would benefit from additional experimental analysis".  Reviewer zV42: "limited novelty prevents me from giving a higher rating".

In summary, while reviewer ratings span either side of above/below the acceptance threshold, the reviewer comments point to limited novelty and limited experimental impact.  Results appear not particularly surprising or significant: while the method provide some savings in training time, it does not seem to ultimately improve top accuracy on tasks and still lags behind the latest vision transformer architectures.  The author response did not substantially change reviewer opinion.  The AC has also taken a detailed look at the paper and does not believe the contribution to be of sufficient significance to warrant acceptance.
Decision: Reject