Year: 2021

Abstract:
Vision Transformers (ViT) have recently emerged as a powerful alternative to convolutional networks (CNNs). Although hybrid models attempt to bridge the gap between these two architectures, the self-attention layers they rely on induce a strong computational bottleneck, especially at large spatial resolutions. In this work, we explore the idea of reducing the time spent training these layers by initializing them from pre-trained convolutional layers. This enables us to transition smoothly from any pre-trained CNN to its functionally identical hybrid model, called Transformed CNN (T-CNN). With only 50 epochs of fine-tuning, the resulting T-CNNs demonstrate significant performance gains over the CNN as well as substantially improved robustness. We analyze the representations learnt by the T-CNN, providing deeper insights into the fruitful interplay between convolutions and self-attention.

text: INTRODUCTION
Since the success of AlexNet in 2012 (Krizhevsky et al., 2017), the field of Computer Vision has been dominated by Convolutional Neural Networks (CNNs) (LeCun et al., 1998; 1989). Their local receptive fields give them a strong inductive bias to exploit the spatial structure of natural images (Scherer et al., 2010; Schmidhuber, 2015; Goodfellow et al., 2016), while allowing them to scale to large resolutions seamlessly. Yet, this inductive bias limits their ability to capture long-range interactions.
In this regard, self-attention (SA) layers, originally introduced in language models (Bahdanau et al., 2014; Vaswani et al., 2017; Devlin et al., 2018), have gained interest as a building block for vision Ramachandran et al. (2019); Zhao et al. (2020). Recently, they gave rise to a plethora of Vision Transformer (ViT) models, able to compete with state-of-the-art CNNs in various tasks Dosovitskiy et al. (2020); Touvron et al. (2020); Wu et al. (2020); Touvron et al. (2021); Liu et al. (2021); Heo et al. (2021) while demonstrating better robustness (Bhojanapalli et al., 2021; Mao et al., 2021). However, capturing long-range dependencies necessarily comes at the cost of quadratic complexity in input size, a computational burden which many recent directions have tried to alleviate (Bello, 2021; Wang et al., 2020; Choromanski et al., 2020; Katharopoulos et al., 2020). Additionally, ViTs are generally harder to train (Zhang et al., 2019; Liu et al., 2020), and require vast amounts of pre-training (Dosovitskiy et al., 2020) or distillation from a convolutional teacher (Hinton et al., 2015; Jiang et al., 2021; Graham et al., 2021) to match the performance of CNNs.
Faced with the dilemma between efficient CNNs and powerful ViTs, several approaches have aimed to bridge the gap between these architectures. On one side, hybrid models append SA layers onto convolutional backbones (Chen et al., 2018; Bello et al., 2019; Graham et al., 2021; Chen et al., 2021; Srinivas et al., 2021), and have already fueled successful results in a variety of tasks (Carion et al., 2020; Hu et al., 2018; Chen et al., 2020; Locatello et al., 2020; Sun et al., 2019). Conversely, a line of research has studied the benefit of introducing convolutional biases in Transformer architectures to ease learning (d’Ascoli et al., 2021; Wu et al., 2021; Yuan et al., 2021). Despite these interesting compromises, modelling long-range dependencies at low computational cost remains a challenge for practitioners.
Contributions At a time when pre-training on vast datasets has become common practice, we ask the following question: does one need to train the SA layers during the whole learning process? Could
one instead learn cheap components such as convolutions first, leaving the SA layers to be learnt at the end? In this paper, we take a step in this direction by presenting a method to fully reparameterize any pre-trained convolutional layer as a Gated Positional Self-Attention (GPSA) layer (d’Ascoli et al., 2021). The latter is initialized to reproduce the mapping of the convolutional layer, but is then encouraged to learn more general mappings which are not accessible to the CNN by adjusting positional gating parameters.
We leverage this method to reparametrize pre-trained CNNs as functionally equivalent hybrid models. After only 50 epochs of fine-tuning, the resulting Transformed CNNs (T-CNNs) boast significant performance and robustness improvements as shown in Fig. 1, demonstrating the practical relevance of our method. We analyze the inner workings of the T-CNNs, and show that they learn more robust representations by combining convolutional heads and attentional heads in a complementary way.
Related work Our work mainly builds on two pillars. First, the idea that SA layers can express any convolution, introduced by Cordonnier et al. (2019). This idea was recently leveraged by d’Ascoli et al. (2021), which initialize the SA layers of end-to-end Transformers as random convolutions to imbue them with a local inductive bias and improve their sample efficiency. Our approach leverages the opposite idea: giving an end-to-end CNN the freedom to escape locality by learning self-attention at late times.
Second, we exploit the following learning paradigm: train a simple and fast model, then reparameterize it as a more complex model for the final stages of learning. This approach was studied from a scientific point of view in d’Ascoli et al. (2019), which shows that reparameterizing a CNN as a fully-connected network (FCN) halfway through training can lead the FCN to outperform the CNN. Yet, the practical relevance of this method is limited by the vast increase in number of parameters required by the FCN to functionally represent the CNN. In contrast, our reparameterization hardly increases the parameter count of the CNN, making it easily applicable to any state-of-the-art CNN. Note that these reparameterization methods can be viewed an informed version of dynamic architecture growing algorithms such as AutoGrow (Wen et al., 2020).
In the context of hybrid models, various works have studied the performance gains obtained by introducing MHSA layers in ResNets with minimal architectural changes (Srinivas et al., 2021; Graham et al., 2021; Chen et al., 2021). However, the MHSA layers used in these works are initialized randomly and need to be trained from scratch. Our approach is different, as it makes use of GPSA layers, which can be initialized to represent the same function as the convolutional layer
it replaces. We emphasize that the novelty in our work is not in the architectures used, but in the unusual way they are blended together.

heading: 1 BACKGROUND
text: Multi-head self-attention The SA mechanism is based on a trainable associative memory with (key, query) vector pairs. To extract the semantic interpendencies between the L elements of a sequence X ∈ RL×Din , a sequence of “query” embeddings Q = WqryX ∈ RL×Dh is matched against another sequence of “key” embeddingsK = WkeyX ∈ RL×Dh using inner products. The result is an attention matrix whose entry (ij) quantifies how semantically relevantQi is toKj :
A = softmax ( QK>√ Dh ) ∈ RL×L. (1)
Multi-head SA layers use several SA heads in parallel to allow the learning of different kinds of dependencies:
MSA(X) := Nh∑ h=1 [SAh(X)]W hout, SAh(X) := A hXW hval, (2)
whereW hval ∈ RDin×Dv andW hout ∈ RDv×Dout are two learnable projections. To incorporate positional information, ViTs usually add absolute position information to the input at embedding time, before propagating it through the SA layers. Another possibility is to replace the vanilla SA with positional SA (PSA), by including a position-dependent term in the softmax (Ramachandran et al., 2019; Shaw et al., 2018). Although there are several ways to parametrize the positional attention, we use encodings rij of the relative position of pixels i and j as in (Cordonnier et al., 2019):
Ahij := softmax ( QhiK h> j + v h> posrij ) . (3)
Each attention head learns an embedding vhpos ∈ RDpos , and the relative positional encodings rij ∈ RDpos only depend on the distance between pixels i and j, denoted denoted as a twodimensional vector δij .
Self-attention as a generalized convolution Cordonnier et al. (2019) shows that a multi-head PSA layer (Eq. 3) with Nh heads and dimension Dpos ≥ 3 can express any convolutional layer of filter size √ Nh, with Din input channels and min(Dv, Dout) output channels, by setting the following: vhpos := −αh ( 1,−2∆h1 ,−2∆h2 , 0, . . . 0 ) rδ := ( ‖δ‖2, δ1, δ2, 0, . . . 0
) Wqry = Wkey := 0
(4)
In the above, the center of attention ∆h ∈ R2 is the position to which head h pays most attention to, relative to the query pixel, whereas the locality strength αh > 0 determines how focused the attention is around its center ∆h. When αh is large, the attention is focused only on the pixel located at ∆h; when αh is small, the attention is spread out into a larger area. Thus, the PSA layer can achieve a convolutional attention map by setting the centers of attention ∆h to each of the possible positional offsets of a √ Nh × √ Nh convolutional kernel, and sending the locality strengths αh to some large value.

heading: 2 APPROACH
text: In this section, we introduce our method for mapping a convolutional layer to a functionally equivalent PSA layer with minimal increase in parameter count. To do this, we leverage the GPSA layers introduced in d’Ascoli et al. (2021).
Loading the filters We want each head h of the PSA layer to functionally mimic the pixel h of a convolutional filterWfilter ∈ RNh×Din×Dout , where we typically have Dout ≥ Din. Rewriting the action of the MHSA operator in a more explicit form, we have
MHSA(X) = Nh∑ h=1 AhX W hvalW h out︸ ︷︷ ︸
W h∈RDin×Dout
(5)
In the convolutional configuration of Eq. 4, AhX selects pixel h of X . Hence, we need to set W h = W hfilter. However, as a product of matrices, the rank ofWh is bottlenecked by Dv. To avoid this being a limitation, we need Dv ≥ Din (since Dout ≥ Din). To achieve this with a minimal number of parameters, we choose Dv = Din, and simply set the following initialization:
W hval = I, W h out = W h filter. (6)
Note that this differs from the usual choice made in SA layers, where Dv = bDin/Nhc. However, to keep the parameter count the same, we share the sameW hval across different heads h, since it plays a symmetric role at initialization.
Note that this reparameterization introduces three additional matrices compared to the convolutional filter: Wqry,Wkey,Wval, each containing Din × Din parameters. However, since the convolutional filter contains Nh ×Din ×Dout parameters, where we typically have Nh = 9 and Dout ∈ {Din, 2Din}, these additional matrices are much smaller than the filters and hardly increase the parameter count. This can be seen from the model sizes in Tab. 3.
Gated Positional self-attention Recent work (d’Ascoli et al., 2021) has highlighted an issue with standard PSA: the fact that the content and positional terms in Eq. 3 are potentially of very different magnitudes, in which case the softmax ignores the smallest of the two. This can typically lead the PSA to adopt a greedy attitude: choosing the form of attention (content or positional) which is easiest at a given time then sticking to it.
To avoid this, the authors suggest to sum the content and positional terms after the softmax, with their relative importances governed by a learnable gating parameter λh (one for each attention head). The resulting Gated Positional Self-Attention (GPSA) layers are parametrized as follows:
Ahij := (1− σ(λh)) softmax ( QhiK h> j ) + σ(λh) softmax ( vh>posrij ) , (7)
where σ : x 7→ 1/(1+e−x) is the sigmoid function. In the positional part, the encodings rij are fixed rather than learnt (see Eq. 4), which makes changing input resolution straightforward (see SM. C) and leaves only 3 learnable parameters per head: ∆1,∆2 and α1.
How convolutional should the initialization be? The convolutional initialization of GPSA layers involves two parameters, determining how strictly convolutional the behavior is: the initial value of the locality strength α, which determines how focused each attention head is on its dedicated pixel, and the initial value of the gating parameters λ, which determines the importance of the positional information versus content. If λh 0 and α 1, the T-CNN will perfectly reproduce the input-output function of the CNN, but may want to greedily stay in the convolutional configuration. Conversely, if λh 0 and α 1, the T-CNN will forget about the input-output function of the CNN. Hence, we choose α = 1 and λ = 1 to lie in between these two extremes, encouraging the T-CNNs to escape locality throughout training.
Architectural details To make our setup as canonical as possible, we focus on ResNet architectures (He et al., 2016), which contain 5 stages, with spatial resolution halfed and number of channels doubled at each stage. Our method involves reparameterizing 3×3 convolutions as GPSA layers with 9 attention heads. However, global SA is too costly in the first layers, where the spatial resolution is large. We therefore only reparameterize the last stage of the architecture, while replacing the first stride-2 convolution by a stride-1 convolution, exactly as in (Srinivas et al., 2021). We also add explicit padding layers to account for the padding of the original convolutions.
1Since α represents the temperature of the softmax, its value must stay positive at all times. To ensure this, we instead learn a rectified parameter α̃ using the softplus function: α = 1
β log(1 + e−βα̃), with β = 5.

heading: 3 PERFORMANCE OF THE TRANSFORMED CNNS
text: In this section, we apply our reparametrization to state-of-the-art CNNs, then fine-tune the resulting T-CNNs to learn better representations. This method allows to fully disentangle the training of the SA layers from that of the convolutional backbone, which is of practical interest for two reasons. First, it minimizes the time spent training the SA layers, which typically have a slower throughput. Second, it separates the algorithmic choices of the CNN backbone from those of the SA layers, which are typically different; for example, CNNs are typically trained with SGD whereas SA layers perform much better with adaptive optimizers such as Adam (Zhang et al., 2019), an incompatibility which may limit the performance of usual hybrid models.
Note that our reparametrization can also be applied halfway through training: this scenario is investigated in Sec. 5. Results suggest that reparametrizing at intermediate times is optimal in terms of speed-performance trade-offs. However, for easier reproducibility, we focus on reparametrizing the fully pre-trained CNNs available in the timm package (Wightman, 2019): this avoids having to retrain the various models from scratch.
Training details We applied our method to pre-trained ResNet-RS (Bello et al., 2021) models, using the weights provided by the timm package (Wightman, 2019). These models are derived from the original ResNet (He et al., 2016), but use improved architectural features and training strategies, enabling them to reach better speed-accuracy trade-offs than EfficientNets.
To minimize computational cost, we restrict our fine-tuning to 50 epochs2. Following (Zhang et al., 2019), we use the AdamW optimizer, with a batch size of 10243. The learning rate is warmed up to 10−4 then annealed using a cosine schedule. To encourage the T-CNN to escape the convolutional configuration and learn content-based attention, we use a larger learning rate of 0.1 for the gating parameters of Eq. 7 (one could equivalently decrease the temperature of the sigmoid function).
We use the same data augmentation scheme as the DeiT (Touvron et al., 2020), as well as rather large stochastic depth coefficients dr reported in Tab. 1. Hoping that our method could be used as an alternative to the commonly used practice of fine-tuning models at higher resolution, we also increase the resolution during fine-tuning (Touvron et al., 2019). In this setting, a ResNet50 requires only 6 hours of fine-tuning on 16 V100 GPUs, compared to 33 hours for the original training. For our largest model (ResNet350-RS), the fine-tuning lasts 50 hours.
Performance improvements Results are presented in Tab. 1, where we also report the baseline improvement of fine-tuning in the same setting but without SA. In all cases, our fine-tuning improves top-1 accuracy, with a significant gap over the baseline. To demonstrate the wide applicability of our method, we report similar improvements for ResNet-D architectures in SM. E.
Despite the extra fine-tuning epochs and their slower throughput, the resulting T-CNNs also outperforming the original CNNs on the ImageNet validation set at equal training budget, as shown
2We study how performance depends on the number of fine-tuning epochs in SM. D. 3Confirming the results of (Zhang et al., 2019), we obtained worse results with SGD.
in the leftmost panel of Fig. 24. However, the major benefit of the reparametrization is in terms of robustness, as shown in Fig. 2(b) and explained below.
Robustness improvements Recent work has shown that Transformer-based architectures offer better robustness and out-of-domain generalization than convolutional architectures (Bhojanapalli et al., 2021; Mao et al., 2021; Mahmood et al., 2021; Shao et al., 2021). To investigate whether our fine-tuning procedure is enough to imbue CNNs with these advantages, we evaluate our T-CNNs on various benchmarks:
• Common corruptions: we use ImageNet-C (Hendrycks & Dietterich, 2019), a dataset containing 15 sets of randomly generated corruptions, grouped into 4 categories: ‘noise’, ‘blur’, ‘weather’, and ‘digital’. Each corruption type has five levels of severity, resulting in 75 distinct corruptions. Note that to avoid distorting the corruptions, which are often pixel-based, we keep a resolution of 224 at inference, which disadvantages the large models trained at higher resolutions.
• Adversarial robustness: following (Mao et al., 2021), we evaluate the accuracy of our models under two white-bow attacks5: (i) single-step FGSM (Goodfellow et al., 2014) and (ii) multi-step L∞-PGD (Madry et al., 2017) with t = 5 steps of size α = 0.5. Both attackers perturb the input
4We estimated the training times of the original ResNet-RS models based on their throughput, for the same hardware as used for the T-ResNet-RS.
5We use the toolkit provided by https://github.com/bethgelab/foolbox.
image with max magnitude = 1. We also evaluate our models on ImageNet-A (Hendrycks et al., 2021), a dataset containing naturally “adversarial” examples from ImageNet. Note however that since this dataset is built from the flaws of a ResNet, it is potential unfair to CNNs.
• Distribution shifts: we use ImageNet-R (Hendrycks et al., 2020), a dataset with various stylized “renditions” of ImageNet images ranging from paintings to embroidery, which strongly modify the local image statistics.
The full table of results is presented in Tab. 3 of SM. A, and illustrated in Figs. 1 and 2. The T-ResNet-RS substantially outperforms the ResNet-RS on all robustness benchmarks. For example, our T-ResNet101-RS, which is 50% faster than ResNet200-RS, reaches similar or better results all robustness tasks, despite its lower top-1 accuracy on ImageNet-1k. This demonstrates that SA improves robustness more than it improves classification accuracy. The most striking improvement is in terms of adversarial robustness, where the smallest T-ResNet-RS is on par with the largest ResNet-RS despite requiring 5 times less compute.
To better understand where the benefits come from, we decompose the improvement of the TResNet50-RS over the various corruption severeties and categories of ImageNet-C in Fig. 3. We observe that improvement increases almost linearly with corruption severity. Although performance is higher in all corruption categories, there is a strong variability: the T-CNN shines particularly in tasks where the objects in the image are less sharp due to lack of contrast, bad weather or blurriness. We attribute this to the ability of SA to distinguish shapes in the image, as investigated in Sec 4.

heading: 4 DISSECTING THE TRANSFORMED CNNS
text: In this section, we analyze various observables to understand how the representations of a TResNet270-RS evolve from those of the ResNet270-RS throughout training.
Unlearn to better relearn In Fig. 4A, we display the train and test accuracy throughout training6. The dynamics decompose into two distinct phases: accuracy dips down during the learning rate warmup phase (first 5 epochs of training), then increases back up as the learning rate is decayed.
As shown in SM. B, the depth of the dip depends on the learning rate. For too small learning rates, the dip is small, but the test accuracy increases too slowly after the dip; for too large learning rates, the test accuracy increases rapidly after the dip, but the dip is too deep to be compensated for. This suggests that the T-CNN needs to “unlearn” to some extent, a phenomenon reminiscent of the “catapult” mechanism of Lewkowycz et al. (2020) which propels models out of sharp minima to land in wider minima.
6The train accuracy is lower than the test accuracy due to the heavy data augmentation used during fine-tuning.
Escaping the convolutional representation In Fig. 4B, we show the evolution of the “attention span” 1/αh (see Eq. 4), which reflects the size of the receptive field of attention head h. On average (thick lines), this quantity increases in the first three layers, showing that the attention span widens, but variability exists among different attention heads (thin lines): some broaden their receptive field, whereas others contract it.
In Fig. 4C, we show the evolution of the gating parameters λh of Eq. 7, which reflect how much attention head h pays to position versus content. Interestingly, the first layer stays strongly convolutional on average, as Ehσ(λh) rapidly becomes close to one (thick blue line). The other layers strongly escape locality, with most attention heads focusing on content information at the end of fine-tuning.
In Fig. 5, we display the attention maps after fine-tuning. A clear divide appears between the “convolutional” attention heads, which remain close to their initialization, and the “content-based” attention heads, which learn more complex dependencies. Notice that the attention head initially focusing on the query pixel (head 5) stays convolutional in all layers. Throughout the layers, the edges of the central object is more and more clearly visible, as observed in (Caron et al., 2021). This supports the hypothesis that robustness gains obtained for blurry corruptions (see Fig. 3) are partly due to the ability of the SA layers to isolate objects from the background.

heading: 5 WHEN SHOULD ONE START LEARNING THE SELF-ATTENTION LAYERS?
text: We have demonstrated the benefits of initializing T-CNNs from pre-trained CNNs, a very compelling procedure given the wide availability of pretrained models. But one may ask: how does this compare to training a hybrid model from scratch? More generally, given a computational budget, how long should the SA layers be trained compared to the convolutional backbone?
Transformed CNN versus hybrid models To answer the first question, we consider a ResNet-50 trained on ImageNet for 400 epochs. We use SGD with momentum 0.9 and a batch size of 1024, warming up the learning rate for 5 epochs before a cosine decay. To achieve a strong baseline, we use the same augmentation scheme as in Touvron et al. (2020) for the DeiT. Results are reported in Tab. 2. In this modern training setting, the vanilla ResNet50 reaches a solid performance of 79.04% on ImageNet, well above the 77% usually reported in litterature.
The T-CNN obtained by fine-tuning the ResNet for 50 epochs at same resolution obtains a top-1 accuracy of 79.88%, with a 15% increase in training time, and 80.84 as resolution 320, with a 35% increase in training time. In comparison, the hybrid model trained for 400 epochs in the same setting only reaches 79.95%, in spite of a 40% increase in training time. Hence, fine-tuning yields better results than training the hybrid model from scratch.
What is the best time to reparametrize? We now study a scenario between the two extreme cases: what happens if we reparametrize halfway through training? To investigate this question in
a systematic way, we train the ResNet50 for t1 epochs, then reparametrize and resume training for another t2 epochs, ensuring that t1 + t2 = 400 in all cases. Hence, t1 = 400, amounts to the vanilla ResNet50, whereas t1 = 0 corresponds to the hybrid model trained from scratch. To study how final performance depends on t1 in a fair setting, we keep the same optimizer and learning rate after the reparametrization, in contrast with the fine-tuning procedure which uses fresh optimizer.
Results are presented in Tab. 2. Interestingly, the final performance evolves non-monotonically, reaching a maximum of 80.44 for t1 = 100, then decreasing back down as the SA layers have less and less time to learn. This non-monotonicity is remarkably similar to that observed in d’Ascoli et al. (2019), where reparameterizing a CNN as a FCN in the early stages of training enables the FCN to outperform the CNN. Crucially, this result suggests that reparametrizing during training not only saves time, but also helps the T-CNN find better solutions.
DISCUSSION
In this work, we showed that complex building blocks such as self-attention layers need not be trained from start. Instead, one can save in compute time while gaining in performance and robustness by initializing them from pre-trained convolutional layers. At a time where energy savings and robustness are key stakes, we believe this finding is important.
On the practical side, our fine-tuning method offers an interesting new direction for practitioners. One limitation of our method is the prohibitive cost of reparametrizing the early stages of CNNs. This cost could however be alleviated by using linear attention methods (Wang et al., 2020), an important direction for future work. Note also that while our T-CNNs significantly improve the robustness of CNNs, they do not always reach the performance of end-to-end Transformers such as the DeiT (for example on ImageNet-C, see Fig. 1). Bridging this gap is an important next step for hybrid models.

heading: A PERFORMANCE TABLE
text: In Tab. 3, we display the characteristics and the performance of our T-ResNet-RS models and compare them to the original ResNet-RS models as well as several other strong baselines reported in Bhojanapalli et al. (2021); Mao et al. (2021).

heading: B CHANGING THE LEARNING RATE
text: We have shown that the learning dynamics decompose into two phases: the learning rate warmup phase, where the test loss drops, then the learning rate decay phase, where the test loss increases again. This could lead one to think that the maximal learning rate is too high, and the dip could be avoided by choosing a lower learning rate. Yet this is not the case, as shown in Fig. 6. Reducing the maximal learning rate indeed reduces the dip, but it also slows down the increase in the second phase of learning. This confirms that the model needs to “unlearn” the right amount to find better solutions.

heading: C CHANGING THE TEST RESOLUTION
text: One advantage of the GPSA layers introduced by d’Ascoli et al. (2021) is how easily they adapt to different image resolutions. Indeed, the positional embeddings they use are fixed rather than learnt. They simply consist in 3 values for each pair of pixels: their euclidean distance ‖δ‖, as well as their coordinate distance δ1, δ2 (see Eq. 4). Our implementation automatically adjusts these embeddings to the input image, allowing us to change the test resolution seamlessly.
In Fig. 7, we show how the top-1 accuracies of our T-ResNet-RS models compares to those of the ResNet-RS models finetuned at same resolution but without SA. At test resolution 416, our TResNetRS-350 reaches an impressive top-1 accuracy of 84.9%, beyond those of the best EfficientNets and BotNets Srinivas et al. (2021).

heading: D CHANGING THE NUMBER OF EPOCHS
text: In Tab. 4, we show how the top-1 accuracy of the T-ResNet-RS model changes with the number of fine-tuning epochs. As expected, performance increases significantly as we fine-tune for longer, yet we chose to set a maximum of 50 fine-tuning epochs to keep the computational cost of fine-tuning well below that of the original training.

heading: E CHANGING THE ARCHITECTURE
text: Our framework, which builds on the timm package, makes changing the original CNN architecture very easy. We applied our fine-tuning procedure to the ResNet-D models He et al. (2019) with the exact same hyperparameters, and observed substantial performance gains, similar to the ones obtained for ResNet-RS, see Tab. 5. This suggests the wide applicability of our method.

Review summaries:
This work proposes an approach to bridge CNNs and vision transformers for image recognition. The idea is to replace the last convolutional stage of a ResNet by a self-attention layer which is initialized from the weights of the convolutional stage. The approach is shown to improve the performance of the CNN, especially in terms of robustness to adversarial examples, corruptions in the input images, and domain shift. 

The paper explores a hybrid type of model architectures that combines the recently popular transformer based model architectures with already well established convolutional neural networks. Specifically, the proposed hybrid model follows a two-stage training strategy where first a CNN model is trained and the pre-trained weights are used to further improve the training by recasting the weights into a new transformed CNN model termed int he paper as T-CNN. The proposed T-CNN model architecture is able to outperform its CNN counterparts and other models compared in the paper. The representations learnt by the T-CNN are also analyzed in the experimental analysis section.