CoAtNet: Marrying Convolution and Attention for All Data Sizes
Authors: Zihang Dai
Hanxiao Liu
Quoc V Le
Mingxing Tan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Hybrid
Transformer
Image Recognition

Abstract: Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks. In this work, we show that while Transformers tend to have larger model capacity, their generalization can be worse than convolutional networks due to the lack of the right inductive bias. To effectively combine the strengths from both architectures, we present CoAtNets(pronounced "coat" nets), a family of hybrid models built from two key insights: (1) depthwise Convolution and self-Attention can be naturally unified via simple relative attention; (2) vertically stacking convolution layers and attention layers in a principled way is surprisingly effective in improving generalization, capacity and efficiency. Experiments show that our CoAtNets achieve state-of-the-art performance under different resource constraints across various datasets: Without extra data, CoAtNet achieves 86.0% ImageNet top-1 accuracy; When pre-trained with 13M images from ImageNet-21K, our CoAtNet achieves 88.56% top-1 accuracy, matching ViT-huge pre-trained with 300M images from JFT-300M while using 23x less data; Notably, when we further scale up CoAtNet with JFT-3B, it achieves 90.88% top-1 accuracy on ImageNet, establishing a new state-of-the-art result.

CoAtNet: Marrying Convolution and Attention for All Data Sizes
Zihang Dai, Hanxiao Liu, Quoc V. Le, Mingxing Tan
Google Research, Brain Team
(zihangd, hanxiaol, qvl, tanmingzingg@@ooggee..o
Abstract
Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks. this work, we show that while Transformers tend have larger model capacity, their generalization can be worse than convolutional networks due to the lack of the right inductive bias. To effectively combine the strengths from both architectures, (pronounced "coat" nets), a family of hybrid models built from two key insights: (1) depthwise Convolution and self-Attention can be naturally unified via simple relative attention; (2) vertically stacking convolution layers and attention layers in principled way is surprisingly effective improving generalization, capacity and efficiency. Experiments show that our CoAtNets achieve satte-of-the-ale perfor- mance under different resource constraints across various datasets: Without extra data, CoAtNet achieves 86.0% ImageNet top-1 accuracy: When pre-trained with 13M images from ImageNet-2IK,, our CoAtNet achieves 88.56% top-I accuracy, matching ViT-huge pre-trained with 300M images from JFT-300M while using 23x less data; Notably, when we further scale up CoAtNet with JFT-3B, achieves 90.88% top-1 accuracy on ImageNet, establishing new state-of-the-ar result.
present CoAtNets
Introduction
Since the breakthrough of AlexNet [1], Convolutional Neural Networks (ConvNets) have been the dominating model architecture for computer vision [2. 3, 4, 5]. Meanwhile, with the success of self-attention models like Transformers [6] in natural language processing [7, 8], many previous works have attempted to bring in the power of attention into computer vision [9, 10, 11, 12]. More recently, Vision Transformer (ViT) [13] has shown that with almost' only vanilla Transformer layers, one could obtain reasonable performance on ImageNet- 1K [14] alone. More importantly, when pre-trained large-scale weakly labeled JFT-300M dataset [15], VIT achieves comparable results to state-of-the-as (SOTA) Conv indicating that Transformer models potentially have higher capacity at scale than ConvNets.
While ViT has shown impressive results with enormous JFT 300M training images, its performance still falls behind ConvNets in the low data regime. For example, without extra JFT-300M pre-train pre-training the ImageNet accuracy of VIT is still significantly lower than ConvNets with comparable model size [5] (see Table 13). Subsequent works use special regularization and stronger data augmentation to improve the vanilla VIT [16, 17, 18), yet none of these ViT variants could outperform the SOTA convolution-only models ImageNet classification given the same amount of data and computa- tion [19, 20]. This suggests that vanilla Transformer layers may lack certain desirable inductive biases possessed by ConvNets, and thus require significant amount of data and computational resource to compensate. Not surprisingly, many recent works have been trying to incorporate the induc- tive biases of ConvNets into Transformer models, by imposing local receptive fields for attention
The initial projection stage can seen as an aggressive sawn-iampling convolutional stem.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

layers [21, 22] or augmenting the attention and FFN layers with implicit or explicit convolutional operations [23, 24, 25]. However, these approaches are either ad-hoc or focused on injecting a particular property, lacking systematic understanding of the respective roles of convolution and attention when combined.
In this work, we systematically study the problem of hybridizing convolution and attention from two fundamental aspects in machine learning - generalizationn and model capacity. Our study shows that convolutional layers tend to have better generalization with faster converging speed thanks to their strong prior of inductive bias, while attention layers have higher model capacity that can benefit from larger datasets. Combining convolutional and attention layers can achieve better generalization and capacity; however, a key challenge here how to effectively combine them to achieve better trade-offs between accuracy and efficiency. In this paper, we investigate two key insights: First, we observe that the commonly used depthwise convolution can be effectively merged into attention layers with simple relative attention; Second, simply stacking convolutional and attention layers, in proper way, could be surprisingly effective to achieve better generalization and capacity. Based on these insights, we propose simple yet effective network architecture named CoAtNet, which enjoys the strengths from both ConvNets and Transformers.
Our CoANNet achieves SOTA performances under comparable resource constraints across different data sizes. Specifically, under the low-data regime, CoAtNet inherits the great generalization property of ConvNets thanks to the favorable inductive biases. Moreover, given abundant data, CoAtNet not only enjoys the superior scalability of Transformer models, but also achieves faster convergence and thus improved efficiency. When only ImageNet-I K is used for training, CoAtNet achieves 86.0% top-1 accuracy, matching the prior art NENet [20] under similar computation resource and training conditions. Further, when pre-trained on ImageNet-21k with about 10M images, CoAtNet reaches 88.56% top-1 accuracy when finetuned on ImageNet- IK, matching the ViT-Huge pre-trained on JFT-300M, 23x larger dataset. Finally, when JFT-3B is used for pre-training, CoAtNet exhibits better efficiency compared to ViT, and pushes the ImageNet-Ie IK top-1 accuracy to 90.88% while using 1.5x less computation of the prior art set by ViT-G/14 [26].
2 Model
In the section, we focus on the question of how to "ooptimall"" combine the convolution and trans- former. Roughly speaking, we decompose the question into two parts:
1. How to combine the convolution and self-attention within one basic computational block?
2. How to vertically stack different types of computational blocks together to form a complete network?
The rationale of the decomposition will become clearer as we gradually reveal our design choices.
2.1 Merging Convolution and Self-A Attention
For convolution, we mainly focus on the MBConv block [27] which employs depthwise convolu- tion [28] to capture the spatial interaction. A key reason of this choice is that both the FFN module in Transformer and MBCony employ the design of "inverted bottleneck", which first expands the channel size of the input by 4x and later project the the 4x-wide hidden state back to the original channel size to enable residual connection.
Besides the similarity of inverted bottleneck, we also notice that both depthwise convolution and self-attention can be expressed as per-dimension weighted sum of values a pre-defined receptive field. Specifically, convolution relies fixed kernel to gather information from local receptive field
= Σ JEC()
Wi-j ② xy (depthwise convolution),
where I4, Vi E RD are the input and output at position respectively, and L(i) denotes a local neighborhood of e.g., 3x3 grid centered at i in image processing.

---

In comparison, self aelf-attention allows the receptive field to be the entire spatial locations and computes the weights based on the re-normalized pairwise similarity between the pair (zy, x,):²
exp (x.[xg)))))))
V/
Kkeg keg exp (ar xT zj (self-attention))
JEG
Aid
where G indicates the global spatial space. Before getting into the question how to best combine them, it is worthwhile to compare their relative strengths and weaknesses, which helps to figure out the good properties we hope to retain.
First of all, the depthwise convolution kernel wy-j is input-independent parameter of static value, while the attention weight As,, dynamically depends on the representation of the input. Hence, much easier for the self-attention to capture complicated relational interactions between different spatial positions, property that we desire most when processing high-level concepts. However, the flexibility comes with risk of easier overfitting, especially when data is limited.
Secondly, notice that given any position pair j). the corresponding convolution weight wi-; only cares about the relative shift between them, i.e. - j. rather than the specific values of i or This property often referred to translation equivalence, which has been found to improve generalization under datasets of limited size [29]. Due to the usage of absolution positional embeddings, standard Transformer (ViT) lacks this property. This partially explains why ConvNets are usually better than Transformers when the dataset is not enormously large.
Finally, the size of the receptive field one of the most crucial differences between self-attention and convolution. Generally speaking, a larger receptive field provides more contextual information, which could lead to higher model capacity. Hence, the global receptive field has been a key motivation to employ self-attention in vision. However, large receptive field requires significantly more computation. In the case of global attention, the complexity is quadratic w.r.t spatial size, which has been fundamental trade-off in applying self-attention models.
Table 1: Desirable properties found in convolution or self-attention.
Properties
Convolution
Self-Attention
Translation Equivariance
Input-adaptive Weighting
Global Receptive Field
Given the comparison above, an ideal model should be able to combine the 3 desirable properties in Table With the similar form depthwise convolution in Eqn. (1) and self-attention in Eqn. (2), a straightforward idea that could achieve this is simply to global static convolution kernel with the adaptive attention matrix, either after or before the Softmax normalization, i.c.,
post E (Σ(
exp (x]Tz) JEG Exeg exp (x, mx)
exp (I. 8T + +w:++))) wi-j) Eteg exp (xr as + wi-k)
wi-s
Zz
V.
Σ
Fz-- (3)
Interestingly, while the idea seems overly simplified, the pre-normalization version Tee corresponds to a particular variant relative self-attention [30, 31]. In this case, the attention weight A.,, is decided jointly by the Wi-y of translation equivariance and the input-adaptive x! Ky, which can enjoy both effects depending their relative magnitudes. Importantly, note that in order to enable the global convolution kernel without blowing up the number of parameters, we have reloaded the notation of w;-j as scalar (i.e., w E RO(1)) rather than vector in Eqn. (1). Another advantage of the scalar formulation of w is that retrieving w;-j for all (i,j) is clearly subsumed by computing the pairwise dot-product attention, hence resulting minimum additional cost (see Appendix A. I). Given the benefits, we will use the Transformer block with the pre-normalization relative attention variant in Eqn. (3) as the key component of the proposed CoAtNet model.
TTo simplify the presentation, we deliberately omit the multi-head query, key and value projections for now. In the actual implementation, always use the multi-head projections.
3

---

2.2 Vertical Layout Design
After figuring out a neat way to combine convolution and attention, we next consider how to utilize it to stack an entire network.
As we have discuss above, the global context has quadratic complexity w.r.t. the spatial size. Hence, if we directly apply the relative attention in Eqn. (3) to the raw image input, the computation will be excessively slow due to the large number of pixels in any image of common sizes. Hence, to construct a network that is feasible in practice, we have mainly three options:
(A) Perform some down- down-sampling to reduce the spatial size and employ the global relative attention after the feature map reaches manageable level.
(B) Enforce local attention, which restricts the global receptive field G in attention to local field L just like in convolution [22, 21].
(C) Replace the quadratic Softmax attention with certain linear attention variant which only has linear complexity w.r.t. the spatial size [12, 32, 33].
We briefly experimented with option (C) without getting reasonably good result. For option (B), we found that implementing local attention involves many non-trivial shape formatting operations that requires intensive memory access. On our accelerator of choice (TPU), such operation turns out be extremely slow [34], which not only defeats the original purpose of speeding up global attention, but also hurts the model capacity. Hence, as some recent work has studied this variant [22, 21], will focus on option (A) and compare our results with theirs in our empirical study (Section 4).
For option (A), the down-sampling can be achieved by either (I) convolution stem with aggressive stride (c.g., stride 16x16) as in VIT or (2) a multi-stage network with gradual pooling as in ConvNets. With these choices, we derive search space of variants and compare them in controlled experiments.
When the ViT Stem used, we directly stack Transformer blocks with relative attention, which we denote as VIT
When the multi-stage layout is used, we mimic ConvNets to construct a network of 5 stages (SO, S1, S2, S3 & 84), with spatial resolution gradually decreased from SO to S4. At the beginning of each stage, we always reduce the spatial size by 2x and increase the number of channels (see Appendix A.I for the detailed down-sampling implementation).
The first stage SO is a simple 2-layer convolutional Stem and S1 always employs MBCоnw blocks with squeeze-excitation (SE), as the spatial size is too large for global attention. Starting from S2 through $4, consider either the MBConv the Transformer block, with a constraint that convolution stages must appear before Transformer stages. The constraint is based on the prior that convolution is better at processing local patterns that are more common in early stages. This leads to 4 variants with increasingly more Transformer stages, C-C-C-C, C-C-C-T, C-C-T-T and C-T-T-T, where C and T denote Convolution and Transformer respectively.
To systematically study the design choices, we consider two fundamental aspects generalization capability and model capacity: For generalization, are interested in the gap between the training loss and the evaluation accuracy. If two models have the same training loss, then the model with higher evaluation accuracy has better generalization capability, since it can generalize better to unseen evaluation dataset. Generalization capability is particularly important to data efficiency when training data size is limited. For model capacity, we measure the ability to fit large training datasets. When training data is abundant and overfitting is not an issue, the model with higher capacity will achieve better final performance after reasonable training steps. Note that, since simply increasing the model size can lead to higher model capacity, to perform a meaningful comparison, we make sure the model sizes of the variants are comparable.
To compare the generalization and model capacity, we train different variants of hybrid models on ImageNet-IK (1.3M) and JFT (>300M) dataset for 300 and 3 epochs respectively, both without any regularization or augmentation. The training loss and evaluation accuracy on both datasets are summarized in Figure 1.
From the ImageNet- IK results, a key observation is that, in terms of generalization capability (i.e.. gap between train and evaluation metrics), we have
C-C-C-C C-C-C-T C-C-T-T C-T-T-T > VIT

---

0.52
5.03
0.22
Eodlaate Pretienh
70%
32%
30%
609
205
62c
1006
1200
1986
(a) ImageNet-1K IK
(b) JFT
Figure Comparison for model generalization and capacity under different data size. For fair comparison, all models have similar parameter size and computational cost.
Particularly, VIT is significantly worse than variants by large margin, which we conjecture is related to the lack of proper low-level information processing in its aggressive down-sampling Stem. Among the multi-stage variants, the overall trend is that the more convolution stages the model has, the smaller the generalization gap is.
As for model capacity, from the JFT comparison, both the train and evaluation metrics at the end of the training suggest the following ranking:
C-C-T-T = C-T-T-T > VIT C-C-C-T > C-C--CC
Importantly, this suggests that simply having more Transformer blocks does NOT necessarily mean higher capacity for visual processing. On one hand, while initially worse, VIT Rm. ultimately catch up with the two variants with more MBConv stages, indicating the capacity advantage of Transformer blocks. On the other hand, both C-C-T-T and C-T-T-T clearly outperforming VIT ReL. suggest that the VIT stem with an aggressive stride may have lost too much information and hence limit the model capacity. More interestingly. the fact that C-C-T-T C-T-T-T indicates the for processing low-level information, static local operations like convolution could be as capable as adaptive global attention mechanism, while saving computation and memory usage substantially.
Finally, to decide between C-C-T-T and C-T-T-T, we conduct another transferability test - we finetune the two JFT pre-trained models above ImageNet-1K K for 30 epochs and compare their transfer performances. From Table 2, it turns out that C-C-T-T achieves clearly better transfer accuracy than C-T-T-T, despite the same pre-training performance.
Table Transferability test results.
Metric
C-C-T-T C-T-T-T
Pre-training Precision @1 (JFT)
34.40
34.36
Transfer Accuracy 224x224
82.39
81.78
Transfer Accuracy 384x384
84.23
84.02
Taking generalization, model capacity, transferability and efficiency into consideration, we adapt the C-C-T-T multi-stage layout for CoAtNet. More model details included in Appendix A.1.
Related Work
Convolutional network building blocks. Convolutional Networks (ConvNets) have been the dom-
inating neural architectures for many computer vision tasks. Traditionally. regular convolutions, such
ResNet blocks [3], are popular in large-scale ConvNets; in contrast, depthwise convolutions [28] are popular in mobile platforms due to its lower computational cost and smaller parameter size [27]. Recent works show that improved inverted residual bottlenecks (MBConv [27, 35]), which is built upon depthwise convolutions, can achieve both high accuracy and better efficiency [5, 19]. As
discussed in Section 2, due to the strong connection between MBConv and Transformer blocks. this paper mostly employs MBCony as convolution building blocks.
Rigorously speaking, this test examines not only the transferability but also the generalization.
5

---

Self-attention and Transformers. With the key ingredients of self-attention, Transformers have been widely adopted for neural language processing and speech understanding. As an early work, stand-alone self-attention network [34] shows self-attention alone can work well for different vision tasks, though with some practical difficulties. Recently, ViT [13] applies vanilla Transformer to ImageNet classification, and achieves impressive results after pre-training
large-scale JFT dataset. However, VIT still largely lags behind state-of-the-art ConvNets when training data is limited. Following that, many recent works have been focused on improving vision Transformers for data efficiency and model efficiency. For a more comprehensive review of vision Transformers, we refer readers to the dedicated surveys [36, 37].
Relative attention. Under the general name of relative attention, there have been various variants in literature [30, 38, 39, 34, 40, 31J. Generally speaking, we can separate them into two categories: (a) the input-dependent version where the extra relative attention score a function of the input states (1,,, ; - j), and (b) the input-independent version f(i - j). The variant in CoAtNet belongs to the input-independent version, and is similar to the one used in T5 [31], but unlike T5, we neither share the relative attention parameters across layers nor use the bucketing mechanism. As a benefit of the input independence, obtaining fii j) for all (i.j) pairs is computationally much cheaper than the input-dependent version on TPU. In addition, at inference time, this only needs to be computed once and cached for future use. A recent work [22] also utilizes such an spput-nndependett parameterization, but restricts the receptive field to local window.
Combining convolution and self-attention. The idea combining convolution and self-attention for vision recognition is not new. A common approach is to augment the ConvNet backbone with explicit self-attention or non-local modules [9, 10, 11, 12]. or to replace certain convolution layers with standard self-attention [1] []]] or more flexible mix of linear attention and convolution [41]. While self-attention usually improves the accuracy, they often come with extra computational cost and hence often regarded as add-on to the Conv/Nets, similar
squeeze-and-excitatii [42] module. In comparison, after the success of ViT and ResNet- ViT [13], another popular line of research starts with Transformer backbone and tries incorporate explicit convolution or some desirable properties of convolution into the Transformer backbone [25, 24, 23, 22, 21, 43, 44].
While our work also belongs to this category, we show that our relative attention instantiation is natural mixture of depthwise convolution and content- based attention with minimum additional cost. More importantly, starting from the perspectives of generalization and model capacity, we take systematic approach to the vertical layout design and show how and why different network stages prefer different types of layers. Therefore, compared to models that simply use off-the-shelf ConvNet as the stem layer, such as ResNet- ViT [13], CoAtNet also scales the Convolution stage (S2) when the overall size increases. On the other hand, compared to models employing local attention [22, 211, CoAtNet consistently uses full attention for S3 & S4 to ensure the model capacity, as S3 occupies the majority of the computation and parameters.
4 Experiments
In this section, we compare CoAtNet with previous results under comparable settings. For complete- ness, all the hyper- parameersss not mentioned here are included in Appendix A.2.
4.1 Experiment Setting
CoAtNet model family. To compare with existing models of different sizes, we also design a family of CoAtNet models as summarized in Table 3. Overall, we always double the number of channels from S1 to S4, while ensuring the width of the Stem to be smaller or equal to that of S1. Also, for simplicity, when increasing the depth of the network, we only scale the number of blocks in S2 and S3,
Evaluation Protocol. Our experiments focus on image classification. To evaluate the performance of the model across different data sizes, we utilize three datasets of increasingly larger sizes, namely ImageNet- K (1.20 (1.28M images), ImageNet-21K (12.7M images) and JFT (300M images). Following previous works, we first pre-train our models on each of the three datasets at resolution 224 for 300, 90 and 14 epochs respectively. Then, we finetune the pre-trained models on ImageNet- -K at the desired

---

Table 3: L denotes the number of blocks and denotes the hidden dimension (#channels). For all
Conv and MBConv blocks, we always use the kerne size 3. For all Transformer blocks, we set the size of each attention head to 32, following [22]. The expansion rate for the inverted bottleneck always 4 and the expansion (shrink) rate for the SE is always 0.25.
Stages
Size
CoAtNet-0
1
CoAtNet-2
CoAtNet-3
CoAtNet-4
SO-Conv
1/2
L=2 D=64
L=2 D=64
L=2
D=128
L=2
D=192
L=2
D=192
S1-MbConv
1/4
L=2 D-96
L=2 D=96
L=2
D=128
L=2
D=192
L=2
D=192
S2 MB-CBB
1/8
L-3 D-192
L-6
D-192
L-6
D-256
L-6
D-384
L-12
D-384
63-TFMRs
1/16
L-5 D-384
L-14
D-384
L-14
D-512
L-14
D-768
L-28
D-768
S4-TFMRe
1/32
L-2
D-768
L-2
D-768
L-2
D-1024
L-2
D-1536
L-2
D-1536
resolutions for 30 epochs and obtain the corresponding evaluation accuracy. One exception the ImageNet-11 K performance at resolution 224, which can be directly obtained at the end of pre-training. Note that similar to other models utilizing Transformer blocks, directly evaluating models pre-trained on ImageNet- K at larger resolution without finetuning usually leads to performance drop. Hence, finetuning is always employed whenever input resolution changes.
Data Augmentation & Regularization. In this work, we only consider two widely used data augmentations, namely Rand [45] and MixUp [46], and three common techniques, including stochastic depth [47], label smoothing [48] and weight decay [49]. to regularize the model. Intuitively, the specific hyper-pane parameters of the augmentation and regularization methods depend on model size and data scale, where strong regularization is usually applied for larger models and smaller dataset.
Under the general principle, a complication under the current paradigm is how to adjust the regular- ization for pre-training and finetuning as data size can change. Specifically, we have an interesting observation that if a certain type of augmentation is entirely disabled during pre-training, simply turning it during fine-tuning would most likely harm the performance rather than improving. We conjecture this could be related to data distribution shift. As a result, for certain runs of the proposed model, we deliberately apply RandAngment and stochastic depth of small degree when pre-training on the two larger datasets, ImageNet21 IKK and JFT. Although such regularization can harm the pre-training metrics, this allows more versatile regularization and augmentation during finetuning, leading to improved down-strean performances.
4.2
Main Results
CoMMien
89.55 (MT R04 Protrast
50 Connlet
2
Calt
J SWiTTRV
EttNetv2
TET
ConVIT
SwinTFM
PrT
Delf
Imugehel
50
10
20
25
30
35
50
100
150
200
220
300
Figure 2: Accuracy-to-FLOPs scaling curve un- der ImageNet- K only setting at 224x224.
Figure Accuracy-to-Parrant scaling curve un- der ImageNet-21K + ImageNet-1K 1K settiing.
ImageNet-1K The experiment results with only the ImageNet- IK dataset are shown in Table 4. Under similar conditions, the proposed CoAtNet models not only outperform VIT variants, but also match the best convolution-only architectures, i.e., EfficientNet-V2 V2 and NFNets. Additionally, we also visualize the all results resolution 224x224 in Fig. 2. As we can see, CoAtNet scales much better than previous model with attention modules.
7

---

Table 4: Model performance on ImageNet. 1K only denotes training on ImageNet-1K only; 21K+1K
denotes pre-training ImageNet-21K and finetuning on ImageNet- IK; PT-RA denotes applying Ragment during 21K pre-training, and E150 means 150 epochs of 21K pre-training, which is longer than the standard 90 epochs. More results are in Appendix A.3.
Models
Eval Size
#Params
#FLOPs
ImageNet Top-1 Accuracy
only
21K+1K
EfficienNNetBB?
600
66M
84.7
Conv Only
EfficientNetV2-L
4802
121M
53B
85.7
86.8
NFNet-F3
4162
255M
114.8B
85.7
NFNet-F5
544²
377M
289.8B
86.0
DeïT-B
3842
86M
55.4B
83.1
ViT-L/16
384
304M
190.7B
85.3
ViT-Stem TFM
CaiT-S-36
384
68M
48.0B
85.0
DeepViT-L
224
55M
12.5B
83.1
Swin-B
384
88M
47.0B
84.2
86.0
stage TFM
Swin-L
384
197M
103.9B
86.4
BotNet-T7
384
75.1M
45.8B
84.7
Lant
320
84.8
Con++TFM
T2T-V1T-24
224
64.1M
15.0B
82.6
CvT-21
3842
32M
24.9B
83.3
CvT-W24
384
277M
193.2B
CoAtNet-O
224
25M
4.2B
81.6
CoAtNet-1
224
42M
8.4B
83.3
CoAINet-2
224
75M
15.7B
84.1
87.1
CoAtNet-3
224
168M
34.7B
84.5
87.6
CoAtNet-0
3842
25M
13.4B
83.9
CoAINet-1
384
42M
27.4B
85.1
Con++TFM
CoAtNot-2
3842
75M
49.8B
85.7
87.1
(ours)
CoAINes-3
3842
168M
107.4B
85.8
87.6
CoAINet-4
3842
275M
189.5B
87.9
PT-RA
3842
275M
189.5B
88.3
+ PT-RA-E150
3842
275M
189,5B
88.4
2
512
75M
96.7B
85.9
87.3
3
5122
168M
203.1B
86.0
87.9
CoAINet-4
5122
275M
360.9B
88.1
+ PT-RA
512²
275M
360.9B
88.4
+ PT-RA-EI50
512
275M
360.9B
88.56
from Table 4 and Fig. 3, when ImageNet-21K is used for pre- training, the advantage of CoAtNet becomes more obvious, substantially outperforming all previous models. Notably, the best CoAtNet variant achieves a top-1 accuracy of 88.56%, matching the ViT- H/14 performance of 88.55%, which requires pre- pre-training the 2.3x larger ViT model on a 23x larger proprietary weakly labeled dataset (JFT) for 2.2x more steps. This marks dramatic improvement in both data efficiency and computation efficiency.
ImageNet-21K As
JFT Finally, in Table 5, we further evaluate CoAtNet under the large-scale data regime with JFT- 300M and JFT-3B. Encouragingly, our CoAtNet-4 can almost match the best previous performance with JFT-300M set by WFNet-F4+, while being 2x more efficient in terms of both TPU training time and parameter count. When scale up the model to consume similar training resource as NFNet-F4+,, CoAtNet-5 reaches 89.77% on top-1 accuracy, outperforming previous results under comparable settings.
Moreover, as we further push the training resource towards the level used by ViT-G/14 and utilize the same JFT-3B dataset of even larger size [26], with over 4x less computation, CoAtNet-6 is able to

---

Table 5: Performance Comparison large-scale JFT dataset. TPUv3-core-dee denotes the pre- training time, Top-I Accuracy denotes the finetuned accuracy on ImageNet. Note that the last 3 rows larger dataset JFT-3B [26] for pre-training, while others use JFT-300M [15]. See Appendix A.2
for the size details Co T: Down-sampling the MBCony block is achieved by stride-2
Depthwise Convolution. :
ViT-G/14 computation consumption is read from Fig. I of the paper [26].
Models
Eval Size
#Params
#FLOPs
TPUv3-core-days
Top-1 Accuracy
+ ViT-L/16
3842
330M
87.12
ViT-L/16
5122
307M
364B
0.68K
87.76
ViT-H/11
5182
632M
1021B
2.5K
88.55
NFNet-F4+
512²
527M
367B
1.86K
89.2
CoAINet-3/
3842
168M
114B
0.58K
88.52
CoAtNet-3t
512²
168M
214B
0.58K
88.81
4
5122
275M
361B
0,95K
89.11
CoAtNer-5
5122
688M
812B
1.82K
89.77
ViT-G/14
5182
1,84B
5160B
>30K
90.45
CoAtNer-6
5122
1.47B
1521B
6.6K
90.45
CoAtNet-7
512²
2.44B
2586B
20.1K
90.88
match the performance of ViT-G/14 of 90.45%, and with 1.5x less computation, CoAtNet-7 achieves 89.77% top-1 accuracy 90. 90.88% achieving the new state-of-the-art performance.
4.3 Ablation Studies
In this section, we will ablate our design choices for CoAtNet
Firstly, we study the importance of the relative attention from combining convolution and attention into a single computation unit. Specifically, we compare two models, one with the relative attention and the other without, under both the ImageNet-1K alone and ImageNet-21 K transfer setting. As we can see from Table 6, when only the ImageNet- 1K is used, relative attention clearly outperforms the standard attention, indicating better generalization. In addition, under the ImageNet-21 K transfer setting, the relative attention variant achieves a substantially better transfer accuracy, despite their very close pre-training performances. This suggests the main advantage of relative attention in visual processing is not in higher capacity but in better generalization.
Table Ablation on relative attention.
Setting
Metric
With Rel-Attn
Without Rel-Attn
Accuracy (224²)
84.1
83.8
ImageNet- 1K
Accuracy (3842)
85.7
85.3
ImageNet-21K
Pre-train Precision @ (224?)
53.0
52.8
ImageNet-IK
Finetune Accuracy (3842)
87.9
87.4
Table 7: Ablation on architecture layout.
Setting
Models
Layout
Top-1 Accuracy
VO: CoAtNet-2
[2, 2, 6, 14,
84.1
ImageNet-IK
V1: s2 + S3
[2, 2, 2, 18,
2]
83.4
V2: S2 + S3
[2, 2, 8, 12, 2]
84.0
ImageNet-21K ImageNet- -K
VO: CoAtNet-3
[2, 2, 6, 14,
2]
53.0 - 87.6
V1: S2 + S3
[2, 2, 2, 18,
53.0
87.4
Secondly, as S2 with MBConw blocks and S3 with relative Transformer blocks occupy most of the computation the CoAtNet, question to ask is how to split the computation between (MBConv) and S3 (Transformer) to achieve good performance. In practice, it boils down to deciding the number of blocks to have in each stage, which we will refer as "layout" design. For this purpose, compare few different layouts that we experimented with in Table 7,

---

Table 8: Ablation on head size and normalization type.
Sefting
Models
Image Size
Top-1 Accuracy
CoAtNet-2
224
84.1
ImageNet-1K
Head size: 32 - 64
224
83.9
Norm type: BN - LN
2242
84.1
ImageNet-21K
CoAtNet-3
3842
87.9
ImageNet-1K
Norm type: BN + LN
3842
87.8
If we keep the total number of blocks in S2 and S3 fixed and vary the number in each stage, observe that VO is sweet spot between V1 and V2. Basically, having more Transformer blocks in S3 generally leads better performance until the number of MBConv blocks in S2 is small to generalize well.
To further evaluate whether the sweet spot also holds in the transfer setting, where higher capacity is often regarded more important, we further compare VO and V1 under the ImageNet- 21K transferring ImageNet- K setup. Interestingly, despite that V1 and VO have the same performance during ImageNet-21K: pre-training. the transfer accuracy
clearly falls behind VO. Again, this suggests the importance of convolution in achieving good transferability and generalization.
Lastly, we study two choices of model details, namely the dimension of each attention (default to 32) head well as the type of normalization (default to BatchNorm) used in MBCony blocks. From Table 8, we can see increasing head size from 32 to 64 can slightly hurt performance, though it actually improves the TPU speed by significant amount. In practice, this will be a quality-speed trade-off one can make. On the other hand, BatchNorm and LayerNorm have almost the same performance, while BatchNorm 10 20% faster on TPU depending on the per-core batch size.
Conclusion
In this paper, we systematically study the properties of convolutions and Transformers, which leads to a principled way to combine them into new family of models named CoAtNet. Extensive experiments show that CoAtNet enjoys both good generalization like ConvNets and superior model capacity like Transformers, achieving state-of-the-art performances under different data sizes and computation budgets.
Note that this paper currently focuses on ImageNet classification for model development. However, we believe our approach is applicable to broader applications like object detection and semantic segmentation. will leave them for future work.
10

---

References
[1] Alex Krizhevsky, Πla Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097-1105, 2012.
[2] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[4] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 1-9, 2015.
[5] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, 2019.
[6] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv:1706.03762 2017.
[7] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 2018.
04805,
[8] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005. 14000000.411.5. 14165, 2020.
[9] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 7794-7803, 2018
[10] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3286-3295, 2019.
[11] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. arXiv preprint 110. 2021.
[12] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3531-3539, 2021.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929,, 2020.
[14] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lece, 2009.
[15] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852. 2017.
[16] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficien image transformers & distillation through attention. arXiv preprint arXiv:2012. /2877, 2020.
[17] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103. 2021.
[18] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint 1188, 2021.

---

[19]
Mingxing Tan and Quoe V Le. Efficientmetv2: Smaller models and faster training, ICML, 2021.
[20] Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan. High-performance large- scale image recognition without normalization. arXiv preprint (tt::2022.0617 2021.
[21] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. arXiv preprint arXiv:2103 12731, 2021.
[22] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
[23] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang, Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103. 15808, 2021.
[24] Ben Graham, Alazeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. arXiv preprint 2rX1vv2104.01136, 2021.
[25] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch imagenet. arXiv preprint arXiv:2 1986, 2021.
[26] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. arXiv preprint arXiv:2106. 04560, 2021.
[27] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Llang-Chiel Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[28]
Laurent Sifre. Rigid-motion scattering for image classification. Ph.D. thesis section 6.2, 2014.
[29]
Mirgahney Mohamed, Gabriele Cesa, Taco S Cohen, and Max Welling. A data and compute efficient design for limited-resources deep learning. arXiv preprint arXiv:2004.0960 2020.
[30] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv preprint arXiv: 02155, 2018.
[31] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019.
[32] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnss: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5166 PMLR, 2020.
[33] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprini arXiv:2009. 14794, 2020.
[34] Prajit Ramachandran, Niki Parmar, Ashish Vaswani. Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv:1906.05909, 2019.
[35] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-awarc neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820. 2019.
[36] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on visual transformer. arXiv preprint arXiv:2012. /2556, 2020.
[37] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. arXiv preprint arXiv:2101.00 169, 2021.
[38] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew Hoffman, Monica Dinculescu, and Douglas Eck, Music 2018.
transformer. arXiv preprint
12

---

[39] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint 1901:02900000// 2019.
[40] Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: A unified understanding of transformer's attention via the lens of kernel. arXiv preprint arXiv:1908. 11755, 2019.
[41] Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. arXiv preprint miXiv:2102.08602 2021.
[42] Jie Hu, Li Shen, and Gang Sun. Squeere-and-excitaiie networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[43] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXiv preprint arXiv:2103.1 1186, 2021.
[44] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.1 12 122, 2021.
[45] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[46] Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1.X.::1710004112 2017.
[47] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinherger. Deep networks with stochastic depth. In European conference computer vision, pages 646-661. Springer, 2016.
[48] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[49] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X 2017.
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European conference on computer vision, pages 630-645. Springer, 2016.
[51] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 2016.
[52] Zihang Dai, Guokun Lai, Yiming Yang, and Quoe V Le. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. arXiv preprint arXiv:2006.032316 2020.
13


Reviews:
[{"review_id": "ZQPUOqGjmZi", "reviewer": "Reviewer_BJnY", "summary": "The paper presents a new architecture combining convolution and transformers block presented as state of the art in terms of FLOPs and accuracy on ImageNet with 3 types of pre-training: ImageNet-1k only, ImageNet-22k and JFT-300M.\nIn order to explain the logic behind the construction of its architecture the paper tries to answer the following questions:\n- How to combine the convolution and self-attention within one basic computational block?\n- How to vertically stack different types of computational blocks together to form a complete network?\nThe paper proposes in the attention modules of its model to associate them with a convolution this is presented as \"the key component of the proposed CoAtNet model.\"\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The idea is interesting\n- The results on ImageNet are good.Moreover, the experience seems to be well executed given that the training hyperparameters have been adjusted for each model.\n\nWeakness:\n- Some statements in the paper seem to be incorrect:\n\n    1) \"CoAtNet achieves 86.0% ImageNet top-1 accuracy without extra data,  and 89.77% with extra JFT data, outperforming prior arts of both convolutional networks and Transformers\" L13 . But the paper Meta Pseudo Labels [1] reach 90.2%  ImageNet top-1 accuracy with JFT data, CaiT[2] and NFNets[3] reach 86.5%  ImageNet top-1 accuracy without extra data.  So this assertion of the paper seem incorrect.\n    2) \"Subsequent works use special regularization and stronger data augmentation to improve the vanilla ViT [16, 17, 18], yet none of these ViT variants could outperform the SOTA convolution-only models on ImageNet classification given the same amount of data and computation [19, 20]\" L31 . However the reference 17 (\"Going deeper with image transformers\" ) gives similar results than reference 20  with similar amount of data and less computation.  So this assertion of the paper seem incorrect.\n   3) \"When only ImageNet-1K is used for training, CoAtNet achieves 86.0% top-1 accuracy, matching the best public record set by a ConvNet variant NFNet\" L57 . NFNet-F5+SAM reach 86.3%  top-1 accuracy and NFNet-F6+SAM reach 86.5%  top-1 accuracy. So this assertion of the paper seem incorrect.\n\n- Significance of the results: There is only ImageNet results there is no evaluation on ImageNet-v2 or ImageNet-real and there is no standard deviation reported. This does not allow to evaluate easily the significance of the results and to know if the method is overfitted.\n\n- Performance comparison: The efficiency of the models is mainly measured in FLOPs and number of parameters but these metrics are far from perfect and it would be interesting to complete them with inference speed or memory consumption at inference time.\nPre-training time Table 5 is measured for few models and only for a pre-training on JFT it would be more informative to have this with more models and on a more widely used dataset.\n\n- Ablation: \n1) The paper state \"we will use the Transformer block with the pre-normalization relative attention variant in Eqn. (3) as the key component of the proposed CoAtNet model.\"  L146.  Nevertheless it seems that there is no ablation between this approach and the classical attention. It would be interesting to see how this impacts the performance of the model.\n\n2) Most of the recent paper with vision transformers use the DeiT[5] training procedure. It seems that the parameters used for CoaTNet are different. It would be interesting to know how this impacts performance.\n\n- Related work:  The Convit  paper [4] is quite similar to the CoatNet paper.  Convit have quite similar approach to the relative attention presented in the CoatNet paper. Convit proposes an architecture with blocks closer to convolution at the beginning and transformers block at the end. Moreover, it shows that their approach is more sample efficient than ViT/DeiT. Indeed, they compares it to ViT according to the number of training data which is quite similar to what is done in the CoatNet paper. So the ideas and message of these two papers seem quite similar. \nIt would be interesting to discuss this paper and compare the CoatNet method with ConViT in Figure 1.\n\n\n[1] Pham et al., Meta Pseudo Labels\n\n[2] Touvron et al., Going deeper with Image Transformers\n\n[3] Brock et al., High-Performance Normalizer-Free ResNets\n\n[4] Ascoli et al., ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases\n\n[5] Touvron et al., Training data-efficient image transformers & distillation through attention", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes the authors adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "KYkD-r3nTc", "reviewer": "Reviewer_5AC9", "summary": "In this paper, the authors provide extensive empirical study results on how to combine transformer blocks and convs for best performance of image recognition. The authors mainly search for 4 different configurations with different numbers of conv-stages and numbers of transformer stages. The conv stages always come before transformer stages. The authors finally present a series of models based on the search results. The final models outperforms existing ViT based or conv-based models on multiple dataset-size and resolution settings. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**[originality: low]**\n - This paper does not propose a new method or architecture building block. \n - The training techniques, search method, and evaluation methods are all standard. \n - Combining conv and attention is also not new (e.g. non-local neural network (Wang et al., CVPR 2018) does this). \n - The method presented in 3.1 is effectively a relative positional encoding similar to what's proposed in [37].\n\nThus overall, I don't see this paper to be particularly original or novel. \n\n**[quality: medium - high]**\n\nOn the other hand, the empirical study presented in this paper can be quite useful. The choice of ablation study and analysis gives useful insight on ViT+conv-based vision model design. The results convincingly demonstrate the strength of the proposed method. I also find the ablation study or analysis (e.g. Table 1) useful. \n\n**[clarity: medium]**\n - While in Table 3, a summary of the proposed models is provided, more details about the architecture specifications are required. At the current level of details, reproducing any results would be difficult. \n - The writing is overall clear and easy to read. \n - While 3.1 is interesting to read, the similarity to relative positional encoding is vague. I think a more straightforward and concise description and a side by side comparison to commonly-used positional encoding would make the method easier to understand.\n\n**[significance: high]**\n\nThe area of ViT-related model design is promising for computer vision research and introduces many interesting open questions. This paper provides useful empirical study that can serve as useful reference for future research. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the main limitation is the vague description of the exact model specification. Regarding societal impact, the research seems computationally quite expensive to conduct (thus potentially consuming much energy). Other than that, I don't see this paper to be particular concerning compared to other deep network design papers.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours"}, "extra_text": {}}, {"review_id": "IN9Cl6bJoQ5", "reviewer": "Reviewer_qHHe", "summary": "This paper proposes to marry convolution with self-attention in order to combine the strengths from both architectures (good inductive bias, larger model capacity). By vertically stacking depth-wise convolution layers and self-attention layers, the resulting CoAtNet gets better generalization ability, capacity, and efficiency. The author carefully designs and tests the layout choices and finally chooses C-C-T-T as the architecture of CoAtNet. From the experimental results, the proposed CoAtNet achieves 86.0% ImageNet top-1 accuracy with 167M parameters and 512 as input size and outperforms previous Convolution and Vision-Transformer baselines. Further extensive experiments and ablation studies show that CoAtNet enjoys good generalization ability like Convolution Networks and superior model capacity as Transformers, and possesses better efficiency over Transformers.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The paper is generally well-written and easy to follow.\n2) This paper works on an interesting problem—combing Convolution and Self-attention to get the strengths from both operations. The resulting CoAtNet effectively solves the drawbacks of Vision Transformers (need more data and larger parameters to train) and enjoys good generalization ability, model capacity, and efficiency.\n3) Extensive experiments are conducted to confirm the good CoAtNet layout and architecture design, the effectiveness and efficiency of CoAtNet over previous Convolution-based and Transformer-based approaches.\n4) The proposed CoAtNet reached a high performance of 86.0% ImageNet top-1 accuracy without extra data and compares to NFNet-F5, CoAtNet requires less than a half number of parameters and is faster. If trained with ImageNet 21K data and JFT data, CoAtNet could reach 88.56% and 89.77% top-1 accuracy respectively, and demonstrate huge data efficiency compared to ViT.\n\nWeaknesses and Questions:\n1) CoAtNet only explores ImageNet classfication. Compared with the papers (such as PVT/Swin/BoT) which perform extensive experiments on detection, segmentation, self-supervised learning, CoAtnet is very weak in this perspective.\n2) The main idea of unifying convolution with transformers has been explored in the following NLP papers. The author needs to discuss the relationship between them. In my opinion, the core idea of CoAtNet is the same as proposed in NLP research. I am prone to weakly reject this paper as lack of novelty and lack of discussion with paper in NLP:\n[1] Zhao, Guangxiang, Xu Sun, Jingjing Xu, Zhiyuan Zhang, and Liangchen Luo. \"Muse: Parallel multi-scale attention for sequence to sequence learning.\" arXiv preprint arXiv:1911.09483 (2019).\n[2] Wu, Felix, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. \"Pay less attention with lightweight and dynamic convolutions.\" arXiv preprint arXiv:1901.10430 (2019).\n[3] Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models. Tyler A. Chang, Yifan Xu, Weijian Xu and Zhuowen Tu. ACL-IJCNLP 2021.\n3) A minor concern: The author said that \"CoAtNets achieve state-of-the-art performance under different resource constraints across various datasets.\" However, the author forgot to compare with the following ViL paper. When we set 224 as the input size, then compare ViL-S (Top-1 Acc: 82.0, #Params: 24.6M, #FLOPs: 4.9B) and CoAtNet-0 (Top-1 Acc: 81.6,  #Params: 25M, #FLOPs: 4.2B). CoAtNet-0 seems not to outperform ViL-S.\n[4] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. arXiv preprint arXiv:2103.15358, 2021\n4) The code is not provided with the supplementary materials, but the checklist said yes.\n5) Why pre-normalization is better than post-normalization is not very clear to me, could you elaborate more on this?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "2sjGXZkApVG", "reviewer": "Reviewer_PojV", "summary": "The paper proposes a fusing strategy of convolution (C) and self-attention (SA) layers to get the benefits of both words namely 1. Translation Equivariance (C), 2. Input-adaptive Weighting (SA) and 3. Global Receptive Field (SA). In particular, the authors claimed to improve the generalization and model-capacity of the proposed model through this Conv-Attention module where they have added a weight scalar along with the computed QK^T and then normalized them. Interestingly the authors pointed out this is quite similar to the idea of positional embedding in transformers and referred few NLP papers on that front. Then the authors applied various existing techniques of optimal speed up of attention-based models for better training efficiency and compared among them. The authors did an excellent literature survey and provided thorough results in terms of accuracy performance of the proposed CoAtNet (though I have few concerns and recommendations, see later) on large-scale datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "> Minor issue, please proofread again to resolve typo issues, for example: \"Particularly, VITREL is significantly worse other variants by a large margin\"..(L 198). a 'than' is missing here.\n\n> A study of incremental learning to further justify the better model capacity of CoAtNet (or in general attention supported networks) over ConvNets would be interesting.  \n\n> The authors said: we also notice that both depthwise convolution and self-attention can be expressed as a weighted sum of values in a pre-defined receptive field (L 107-108).. isn't this valid for any convolution instead? So, from this part of the explanation, it is not clear why the authors shortlist DWC?\n\n> Is the following observation true for self-attentiontion-based vision models with any type of positional embedding? \" Due to the usage of absolution positional embeddings, standard Transformer (ViT) lacks this property.\" (L 125-126), I mean even for attention with specialized pos. embeddings like axial attention [1]?\n\n> The axial attention [1], stand-alone self-attention [2] papers (and there are few others) also talked about and pointed to the issue of the translational equivariance of attention modules and proposed similar pos. embedding layers infused with the Q,K or Q, K, V embedding layers. Thus the finding and the proposed solution does not fall under the category of \"novel\" approach. The authors deployed the same in Vision transformers which are much larger in param count and thus deploying the idea and validating at such large scale deserves praise, however, I am skeptical about terming the finding as novel and thus concerned about the contribution.\n\n> Please cite [3] in line 155-156.\n\n> The finding of line 178-181 looks quite similar to that of [2]. \n\n> I did not understand what might be the reason for Conv+TFM to surpass the performance of COnv only on ImageNet 1K? And if it is so, are the authors hinting at the fact that its better to use conv (or at least no gain in using Conv+TFM) for medium-scale datasets (like ImageNet 1K)?\n\n> Isn't the results of Table 2 is counterintuitive as per the paper's description? If Transformer layers generalize better, then why don't models with higher no of T layers would transfer better (or faster) as well? It would be good it the authors provide explanations to that.\n\n> In another line of research authors of [4] had tried to make the properties of conv learn to SA via a form of distillation. It would be good to see how this solution works as compared to CoAtNet.\n\n[1] Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation, ECCV 2020.\n[2] Stand-Alone Self Attention for Vision, NeurIPS 2019.\n[3] Attention Augmented Convolutional Networks, ICCV 2019.\n[4] AttentionLite: Towards Efficient Self-Attention Models for Vision, ICASSP 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There might be environmental impact, but no social impact to my understanding.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "15"}, "extra_text": {"ethical_concerns": "Not much among what is listed. However, I would be happy to see a separate flag for research asking huge compute and thus energy-budget. This is for the conference to keep a close eye on the count of such kind of research. I am not against this line of research, however, I am in favor of keeping a watch on it, so that we can know its impact if it starts affecting on a large scale. I also commit to not judge the paper's contribution with this in mind. \n\nI thus would like to request the NeurIPS PC members to add another ethical concern with the name \"environmental impact\" (if there is no such track already)."}}]

Author rebuttal:

Meta-review: This paper has received slightly negative ratings - the main concern being the lack of novelty, as well as several ambiguous claims. Following the rebuttal and discussion phase, the reviewers acknowledged the quality of the ImageNet results, but some questions remained unanswered. For this reason, I recommend rejection, and encourage the authors to take the reviewer's comments into account to improve the manuscript and resubmit to another venue. 
Decision: Accept (Poster)