CoAtNet: Marrying Convolution and Attention for All Data Sizes
Authors: Zihang Dai
Hanxiao Liu
Quoc V Le
Mingxing Tan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Hybrid
Transformer
Image Recognition

Abstract: Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks. In this work, we show that while Transformers tend to have larger model capacity, their generalization can be worse than convolutional networks due to the lack of the right inductive bias. To effectively combine the strengths from both architectures, we present CoAtNets(pronounced "coat" nets), a family of hybrid models built from two key insights: (1) depthwise Convolution and self-Attention can be naturally unified via simple relative attention; (2) vertically stacking convolution layers and attention layers in a principled way is surprisingly effective in improving generalization, capacity and efficiency. Experiments show that our CoAtNets achieve state-of-the-art performance under different resource constraints across various datasets: Without extra data, CoAtNet achieves 86.0% ImageNet top-1 accuracy; When pre-trained with 13M images from ImageNet-21K, our CoAtNet achieves 88.56% top-1 accuracy, matching ViT-huge pre-trained with 300M images from JFT-300M while using 23x less data; Notably, when we further scale up CoAtNet with JFT-3B, it achieves 90.88% top-1 accuracy on ImageNet, establishing a new state-of-the-art result.

CoAtNet: Marrying Convolution and Attention for All Data Sizes
Zihang Dai, Hanxiao Liu, Quoc V. Le, Mingxing Tan
Google Research, Brain Team
(zihangd, hanxiaol, qvl, tanmingzingg@@ooggee..o
Abstract
Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks. this work, we show that while Transformers tend have larger model capacity, their generalization can be worse than convolutional networks due to the lack of the right inductive bias. To effectively combine the strengths from both architectures, (pronounced "coat" nets), a family of hybrid models built from two key insights: (1) depthwise Convolution and self-Attention can be naturally unified via simple relative attention; (2) vertically stacking convolution layers and attention layers in principled way is surprisingly effective improving generalization, capacity and efficiency. Experiments show that our CoAtNets achieve tatte-of-the-ale perfor- mance under different resource constraints across various datasets: Without extra data, CoAtNet achieves 86.0% ImageNet top-1 accuracy: When pre-trained with 13M images from ImageNet-2IK,, our CoAtNet achieves 88.56% top-I accuracy, matching ViT-huge pre-trained with 300M images from JFT-300M while using 23x less data; Notably, when we further scale up CoAtNet with JFT-3B, achieves 90.88% top-1 accuracy on ImageNet, establishing new state-of-the-ar result.
present CoAtNets
Introduction
Since the breakthrough of AlexNet [1], Convolutional Neural Networks (ConvNets) have been the dominating model architecture for computer vision [2. 3, 4, 5]. Meanwhile, with the success of self-attention models like Transformers [6] in natural language processing [7, 8], many previous works have attempted to bring in the power of attention into computer vision [9, 10, 11, 12]. More recently, Vision Transformer (ViT) [13] has shown that with almost' only vanilla Transformer layers, one could obtain reasonable performance on ImageNet- 1K [14] alone. More importantly, when pre-trained large-scale weakly labeled JFT-300M dataset [15], VIT achieves comparable results to state-of-the-as (SOTA) Conv indicating that Transformer models potentially have higher capacity at scale than ConvNets.
While ViT has shown impressive results with enormous JFT 300M training images, its performance still falls behind ConvNets in the low data regime. For example, without extra JFT-300M pre-train pre-training the ImageNet accuracy of VIT is still significantly lower than ConvNets with comparable model size [5] (see Table 13). Subsequent works use special regularization and stronger data augmentation to improve the vanilla VIT [16, 17, 18), yet none of these ViT variants could outperform the SOTA convolution-only models ImageNet classification given the same amount of data and computa- tion [19, 20]. This suggests that vanilla Transformer layers may lack certain desirable inductive biases possessed by ConvNets, and thus require significant amount of data and computational resource to compensate. Not surprisingly, many recent works have been trying to incorporate the induc- tive biases of ConvNets into Transformer models, by imposing local receptive fields for attention
The initial projection stage can seen as an aggressive sawn-iampling convolutional stem.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

layers [21, 22] or augmenting the attention and FFN layers with implicit or explicit convolutional operations [23, 24, 25]. However, these approaches are either ad-hoc or focused on injecting a particular property, lacking systematic understanding of the respective roles of convolution and attention when combined.
In this work, we systematically study the problem of hybridizing convolution and attention from two fundamental aspects in machine learning - generalization and model capacity. Our study shows that convolutional layers tend to have better generalization with faster converging speed thanks to their strong prior of inductive bias, while attention layers have higher model capacity that can benefit from larger datasets. Combining convolutional and attention layers can achieve better generalization and capacity; however, a key challenge here how to effectively combine them to achieve better trade-offs between accuracy and efficiency. In this paper, we investigate two key insights: First, we observe that the commonly used depthwise convolution can be effectively merged into attention layers with simple relative attention; Second, simply stacking convolutional and attention layers, in proper way, could be surprisingly effective to achieve better generalization and capacity. Based on these insights, we propose simple yet effective network architecture named CoAtNet, which enjoys the strengths from both ConvNets and Transformers.
Our CoANNet achieves SOTA performances under comparable resource constraints across different data sizes. Specifically, under the low-data regime, CoAtNet inherits the great generalization property of ConvNets thanks to the favorable inductive biases. Moreover, given abundant data, CoAtNet not only enjoys the superior scalability of Transformer models, but also achieves faster convergence and thus improved efficiency. When only ImageNet-I K is used for training, CoAtNet achieves 86.0% top-1 accuracy, matching the prior art NENet [20] under similar computation resource and training conditions. Further, when pre-trained on ImageNet-21k with about 10M images, CoAtNet reaches 88.56% top-1 accuracy when finetuned on ImageNet- IK, matching the ViT-Huge pre-trained on JFT-300M, 23x larger dataset. Finally, when JFT-3B is used for pre-training, CoAtNet exhibits better efficiency compared to ViT, and pushes the ImageNet-Ie IK top-1 accuracy to 90.88% while using 1.5x less computation of the prior art set by ViT-G/14 [26].
2 Model
In the section, we focus on the question of how to "ooptimall"" combine the convolution and trans- former. Roughly speaking, we decompose the question into two parts:
1. How to combine the convolution and self-attention within one basic computational block?
2. How to vertically stack different types of computational blocks together to form a complete network?
The rationale of the decomposition will become clearer as we gradually reveal our design choices.
2.1 Merging Convolution and Self-A Attention
For convolution, we mainly focus on the MBConv block [27] which employs depthwise convolu- tion [28] to capture the spatial interaction. A key reason of this choice is that both the FFN module in Transformer and MBCony employ the design of "inverted bottleneck", which first expands the channel size of the input by 4x and later project the the 4x-wide hidden state back to the original channel size to enable residual connection.
Besides the similarity of inverted bottleneck, we also notice that both depthwise convolution and self-attention can be expressed as per-dimension weighted sum of values a pre-defined receptive field. Specifically, convolution relies fixed kernel to gather information from local receptive field
= Σ JEC()
Wi-j ② xy (depthwise convolution),
where I4, Vi E RD are the input and output at position respectively, and L(i) denotes a local neighborhood of e.g., 3x3 grid centered at i in image processing.

---

In comparison, self aelf-attention allows the receptive field to be the entire spatial locations and computes the weights based on the re-normalized pairwise similarity between the pair (zy, x,):²
exp (x.[xg)))))))
V/
Kkeg keg exp (ar xT zj (self-attention))
JEG
Aid
where G indicates the global spatial space. Before getting into the question how to best combine them, it is worthwhile to compare their relative strengths and weaknesses, which helps to figure out the good properties we hope to retain.
First of all, the depthwise convolution kernel wy-j is input-independent parameter of static value, while the attention weight As,, dynamically depends on the representation of the input. Hence, much easier for the self-attention to capture complicated relational interactions between different spatial positions, property that we desire most when processing high-level concepts. However, the flexibility comes with risk of easier overfitting, especially when data is limited.
Secondly, notice that given any position pair j). the corresponding convolution weight wi-; only cares about the relative shift between them, i.e. - j. rather than the specific values of i or This property often referred to translation equivalence, which has been found to improve generalization under datasets of limited size [29]. Due to the usage of absolution positional embeddings, standard Transformer (ViT) lacks this property. This partially explains why ConvNets are usually better than Transformers when the dataset is not enormously large.
Finally, the size of the receptive field one of the most crucial differences between self-attention and convolution. Generally speaking, a larger receptive field provides more contextual information, which could lead to higher model capacity. Hence, the global receptive field has been a key motivation to employ self-attention in vision. However, large receptive field requires significantly more computation. In the case of global attention, the complexity is quadratic w.r.. spatial size, which has been fundamental trade-off in applying self-attention models.
Table 1: Desirable properties found in convolution or self-attention.
Properties
Convolution
Self-Attention
Translation Equivariance
Input-adaptive Weighting
Global Receptive Field
Given the comparison above, an ideal model should be able to combine the 3 desirable properties in Table With the similar form depthwise convolution in Eqn. (1) and self-attention in Eqn. (2), a straightforward idea that could achieve this is simply to global static convolution kernel with the adaptive attention matrix, either after or before the Softmax normalization, i.c.,
exp (x]Tz) JEG Exeg exp (x, mx)
exp (I. 8j + +w:++))) wi-j) Eteg exp (xr as + wi-k)
post E
wi-s
Zz
V.
Σ
Fz-- (3)
Interestingly, while the idea seems overly simplified, the pre-normalization version Tee corresponds to a particular variant relative self-attention [30, 31]. In this case, the attention weight A.,, is decided jointly by the Wi-y of translation equivariance and the input-adaptive x! Ky, which can enjoy both effects depending the global convolution kernel without blowing up the number of parameters, we have reloaded the notation of w;-j as scalar (i.e., w E RO(1)) rather than vector in Eqn. (1). Another advantage of the scalar formulation of w is that retrieving w;-j for all (i,j) is clearly subsumed by computing the pairwise dot-product attention, hence resulting minimum additional cost (see Appendix A. I). Given the benefits, we will use the Transformer block with the pre-normalization relative attention variant in Eqn. (3) as the key component of the proposed CoAtNet model.
their relative magnitudes. Importantly, note that in order to enable
TTo simplify the presentation, we deliberately omit the multi-head query, key and value projections for now. In the actual implementation, always use the multi-head projections.
3

---

2.2 Vertical Layout Design
After figuring out a neat way to combine convolution and attention, we next consider how to utilize it to stack an entire network.
As we have discuss above, the global context has quadratic complexity w.r.t. the spatial size. Hence, if we directly apply the relative attention in Eqn. (3) to the raw image input, the computation will be excessively slow due to the large number of pixels in any image of common sizes. Hence, to construct a network that is feasible in practice, we have mainly three options:
(A) Perform some down- down-sampling to reduce the spatial size and employ the global relative attention after the feature map reaches manageable level.
(B) Enforce local attention, which restricts the global receptive field G in attention to local field L just like in convolution [22, 21].
(C) Replace the quadratic Softmax attention with certain linear attention variant which only has linear complexity w.r.t. the spatial size [12, 32, 33].
We briefly experimented with option (C) without getting reasonably good result. For option (B), we found that implementing local attention involves many non-trivial shape formatting operations that requires intensive memory access. On our accelerator of choice (TPU), such operation turns out be extremely slow [34], which not only defeats the original purpose of speeding up global attention, but also hurts the model capacity. Hence, as some recent work has studied this variant [22, 21], will focus on option (A) and compare our results with theirs in our empirical study (Section 4).
For option (A), the down-sampling can be achieved by either (I) convolution stem with aggressive stride (c.g., stride 16x16) as in VIT or (2) a multi-stage network with gradual pooling as in ConvNets. With these choices, we derive search space of variants and compare them in controlled experiments.
When the ViT Stem used, we directly stack Transformer blocks with relative attention, which we denote as VIT
When the multi-stage layout is used, we mimic ConvNets to construct a network of 5 stages (SO, S1, S2, S3 & 84), with spatial resolution gradually decreased from SO to S4. At the beginning of each stage, we always reduce the spatial size by 2x and increase the number of channels (see Appendix A.I for the detailed down-sampling implementation).
The first stage SO is a simple 2-layer convolutional Stem and S1 always employs MBCоnw blocks with squeeze-excitation (SE), as the spatial size is too large for global attention. Starting from S2 through $4, consider either the MBConv the Transformer block, with a constraint that convolution stages must appear before Transformer stages. The constraint is based on the prior that convolution is better at processing local patterns that are more common in early stages. This leads to 4 variants with increasingly more Transformer stages, C-C-C-C, C-C-C-T, C-C-T-T and C-T-T-T, where C and T denote Convolution and Transformer respectively.
To systematically study the design choices, we consider two fundamental aspects generalization capability and model capacity: For generalization, are interested in the gap between the training loss and the evaluation accuracy. If two models have the same training loss, then the model with higher evaluation accuracy has better generalization capability, since it can generalize better to unseen evaluation dataset. Generalization capability is particularly important to data efficiency when training data size is limited. For model capacity, we measure the ability to fit large training datasets. When training data is abundant and overfitting is not an issue, the model with higher capacity will achieve better final performance after reasonable training steps. Note that, since simply increasing the model size can lead to higher model capacity, to perform a meaningful comparison, we make sure the model sizes of the variants are comparable.
To compare the generalization and model capacity, we train different variants of hybrid models on ImageNet-IK (1.3M) and JFT (>300M) dataset for 300 and 3 epochs respectively, both without any regularization or augmentation. The training loss and evaluation accuracy on both datasets are summarized in Figure 1.
From the ImageNet- IK results, a key observation is that, in terms of generalization capability (i.e.. gap between train and evaluation metrics), we have
C-C-C-C C-C-C-T C-C-T-T C-T-T-T > VIT

---

0.52
5.03
0.22
Eodlaate Pretienh
70%
32%
30%
609
205
62c
1006
1200
1986
(a) ImageNet-1K IK
(b) JFT
Figure Comparison for model generalization and capacity under different data size. For fair comparison, all models have similar parameter size and computational cost.
Particularly, VIT is significantly worse than variants by large margin, which we conjecture is related to the lack of proper low-level information processing in its aggressive down-sampling Stem. Among the multi-stage variants, the overall trend is that the more convolution stages the model has, the smaller the generalization gap is.
As for model capacity, from the JFT comparison, both the train and evaluation metrics at the end of the training suggest the following ranking:
C-C-T-T = C-T-T-T > VIT C-C-C-T > C-C--CC
Importantly, this suggests that simply having more Transformer blocks does NOT necessarily mean higher capacity for visual processing. On one hand, while initially worse, VIT Rm. ultimately catch up with the two variants with more MBConv stages, indicating the capacity advantage of Transformer blocks. On the other hand, both C-C-T-T and C-T-T-T clearly outperforming VIT ReL. suggest that the VIT stem with an aggressive stride may have lost too much information and hence limit the model capacity. More interestingly. the fact that C-C-T-T C-T-T-T indicates the for processing low-level information, static local operations like convolution could be as capable as adaptive global attention mechanism, while saving computation and memory usage substantially.
Finally, to decide between C-C-T-T and C-T-T-T, we conduct another transferability test - we finetune the two JFT pre-trained models above ImageNet-1K K for 30 epochs and compare their transfer performances. From Table 2, it turns out that C-C-T-T achieves clearly better transfer accuracy than C-T-T-T, despite the same pre-training performance.
Table Transferability test results.
Metric
C-C-T-T C-T-T-T
Pre-training Precision @1 (JFT)
34.40
34.36
Transfer Accuracy 224x224
82.39
81.78
Transfer Accuracy 384x384
84.23
84.02
Taking generalization, model capacity, transferability and efficiency into consideration, we adapt the C-C-T-T multi-stage layout for CoAtNet. More model details included in Appendix A.1.
Related Work
Convolutional network building blocks. Convolutional Networks (ConvNets) have been the dom-
inating neural architectures for many computer vision tasks. Traditionally. regular convolutions, such
ResNet blocks [3], are popular in large-scale ConvNets; in contrast, depthwise convolutions [28] are popular in mobile platforms due to its lower computational cost and smaller parameter size [27]. Recent works show that improved inverted residual bottlenecks (MBConv [27, 35]), which is built upon depthwise convolutions, can achieve both high accuracy and better efficiency [5, 19]. As
discussed in Section 2, due to the strong connection between MBConv and Transformer blocks. this paper mostly employs MBCony as convolution building blocks.
Rigorously speaking, this test examines not only the transferability but also the generalization.
5

---

Self-attention and Transformers. With the key ingredients of self-attention, Transformers have been widely adopted for neural language processing and speech understanding. As an early work, stand-alone self-attention network [34] shows self-attention alone can work well for different vision tasks, though with some practical difficulties. Recently, ViT [13] applies vanilla Transformer to ImageNet classification, and achieves impressive results after pre-training
large-scale JFT dataset. However, VIT still largely lags behind state-of-the-art ConvNets when training data is limited. Following that, many recent works have been focused on improving vision Transformers for data efficiency and model efficiency. For a more comprehensive review of vision Transformers, we refer readers to the dedicated surveys [36, 37].
Relative attention. Under the general name of relative attention, there have been various variants in literature [30, 38, 39, 34, 40, 31J. Generally speaking, we can separate them into two categories: (a) the input-dependent version where the extra relative attention score a function of the input states (1,,, ; - j), and (b) the input-independent version f(i - j). The variant in CoAtNet belongs to the input-independent version, and is similar to the one used in T5 [31], but unlike T5, we neither share the relative attention parameters across layers nor use the bucketing mechanism. As a benefit of the input independence, obtaining fii j) for all (i.j) pairs is computationally much cheaper than the input-dependent version on TPU. In addition, at inference time, this only needs to be computed once and cached for future use. A recent work [22] also utilizes such an spput-nndependett parameterization, but restricts the receptive field to local window.
Combining convolution and self-attention. The idea combining convolution and self-attention for vision recognition is not new. A common approach is to augment the ConvNet backbone with explicit self-attention or non-local modules [9, 10, 11, 12]. or to replace certain convolution layers with standard self-attention [1] []]] or more flexible mix of linear attention and convolution [41]. While self-attention usually improves the accuracy, they often come with extra computational cost and hence often regarded as add-on to the Conv/Nets, similar
squeeze-and-excitatii [42] module. In comparison, after the success of ViT and ResNet- ViT [13], another popular line of research starts with Transformer backbone and tries incorporate explicit convolution or some desirable properties of convolution into the Transformer backbone [25, 24, 23, 22, 21, 43, 44].
While our work also belongs to this category, we show that our relative attention instantiation is natural mixture of depthwise convolution and content- based attention with minimum additional cost. More importantly, starting from the perspectives of generalization and model capacity, we take systematic approach to the vertical layout design and show how and why different network stages prefer different types of layers. Therefore, compared to models that simply use off-the-shelf ConvNet as the stem layer, such as ResNet- ViT [13], CoAtNet also scales the Convolution stage (S2) when the overall size increases. On the other hand, compared to models employing local attention [22, 211, CoAtNet consistently uses full attention for S3 & S4 to ensure the model capacity, as S3 occupies the majority of the computation and parameters.
4 Experiments
In this section, we compare CoAtNet with previous results under comparable settings. For complete- ness, all the hyper- parameersss not mentioned here are included in Appendix A.2.
4.1 Experiment Setting
CoAtNet model family. To compare with existing models of different sizes, we also design a family of CoAtNet models as summarized in Table 3. Overall, we always double the number of channels from S1 to S4, while ensuring the width of the Stem to be smaller or equal to that of S1. Also, for simplicity, when increasing the depth of the network, we only scale the number of blocks in S2 and S3,
Evaluation Protocol. Our experiments focus on image classification. To evaluate the performance of the model across different data sizes, we utilize three datasets of increasingly larger sizes, namely ImageNet- K (1.20 (1.28M images), ImageNet-21K (12.7M images) and JFT (300M images). Following previous works, we first pre-train our models on each of the three datasets at resolution 224 for 300, 90 and 14 epochs respectively. Then, we finetune the pre-trained models on ImageNet- -K at the desired

---

Table 3: L denotes the number of blocks and denotes the hidden dimension (#channels). For all
Conv and MBConv blocks, we always use the kerne size 3. For all Transformer blocks, we set the size of each attention head to 32, following [22]. The expansion rate for the inverted bottleneck always 4 and the expansion (shrink) rate for the SE is always 0.25.
Stages
Size
CoAtNet-0
1
CoAtNet-2
CoAtNet-3
CoAtNet-4
SO-Conv
1/2
L=2 D=64
L=2 D=64
L=2
D=128
L=2
D=192
L=2
D=192
S1-MbConv
1/4
L=2 D-96
L=2 D=96
L=2
D=128
L=2
D=192
L=2
D=192
S2 MB-CBB
1/8
L-3 D-192
L-6
D-192
L-6
D-256
L-6
D-384
L-12
D-384
63-TFMRs
1/16
L-5 D-384
L-14
D-384
L-14
D-512
L-14
D-768
L-28
D-768
S4-TFMRe
1/32
L-2
D-768
L-2
D-768
L-2
D-1024
L-2
D-1536
L-2
D-1536
resolutions for 30 epochs and obtain the corresponding evaluation accuracy. One exception the ImageNet-11 K performance at resolution 224, which can be directly obtained at the end of pre-training. Note that similar to other models utilizing Transformer blocks, directly evaluating models pre-trained on ImageNet- K at larger resolution without finetuning usually leads to performance drop. Hence, finetuning is always employed whenever input resolution changes.
Data Augmentation & Regularization. In this work, we only consider two widely used data augmentations, namely Rand [45] and MixUp [46], and three common techniques, including stochastic depth [47], label smoothing [48] and weight decay [49]. to regularize the model. Intuitively, the specific hyper-pane parameters of the augmentation and regularization methods depend on model size and data scale, where strong regularization is usually applied for larger models and smaller dataset.
Under the general principle, a complication under the current paradigm is how to adjust the regular- ization for pre-training and finetuning as data size can change. Specifically, we have an interesting observation that if a certain type of augmentation is entirely disabled during pre-training, simply turning it during fine-tuning would most likely harm the performance rather than improving. We conjecture this could be related to data distribution shift. As a result, for certain runs of the proposed model, we deliberately apply RandAngment and stochastic depth of small degree when pre-training on the two larger datasets, ImageNet21 IKK and JFT. Although such regularization can harm the pre-training metrics, this allows more versatile regularization and augmentation during finetuning, leading to improved down-strean performances.
4.2
Main Results
CoMMien
89.55 (MT R04 Protrast
50 Connlet
2
Calt
J SWiTTRV
EttNetv2
TET
ConVIT
SwinTFM
PrT
Delf
Imugehel
50
10
20
25
30
35
50
100
150
200
220
300
Figure 2: Accuracy-to-FLOPs scaling curve un- der ImageNet-1K K only setting at 224x224.
Figure Accuracy-to-Parrant scaling curve un- der ImageNet-21K + ImageNet-1K 1K settiing.
ImageNet-1K The experiment results with only the ImageNet- IK dataset are shown in Table 4. Under similar conditions, the proposed CoAtNet models not only outperform VIT variants, but also match the best convolution-only architectures, i.e., EfficientNet-V2 V2 and NFNets. Additionally, we also visualize the all results resolution 224x224 in Fig. 2. As we can see, CoAtNet scales much better than previous model with attention modules.
7

---

Table 4: Model performance on ImageNet. 1K only denotes training on ImageNet-1K only; 21K+1K
denotes pre-training ImageNet-21K and finetuning on ImageNet- 1K; PT-RA denotes applying Ragment during 21K pre-training, and E150 means 150 epochs of 21K pre-training, which is longer than the standard 90 epochs. More results are in Appendix A.3.
Models
Eval Size
#Params
#FLOPs
ImageNet Top-1 Accuracy
only
21K+1K
EfficienNNetBB?
600
66M
84.7
Conv Only
EfficientNetV2-L
4802
121M
53B
85.7
86.8
NFNet-F3
4162
255M
114.8B
85.7
NFNet-F5
544²
377M
289.8B
86.0
DeïT-B
3842
86M
55.4B
83.1
ViT-L/16
384
304M
190.7B
85.3
ViT-Stem TFM
CaiT-S-36
384
68M
48.0B
85.0
DeepViT-L
224
55M
12.5B
83.1
Swin-B
384
88M
47.0B
84.2
86.0
stage TFM
Swin-L
384
197M
103.9B
86.4
BotNet-T7
384
75.1M
45.8B
84.7
Lant
320
84.8
Con++TFM
T2T-V1T-24
224
64.1M
15.0B
82.6
CvT-21
3842
32M
24.9B
83.3
CvT-W24
384
277M
193.2B
CoAtNet-O
224
25M
4.2B
81.6
CoAtNet-1
224
42M
8.4B
83.3
CoAINet-2
224
75M
15.7B
84.1
87.1
CoAtNet-3
224
168M
34.7B
84.5
87.6
CoAtNet-0
3842
25M
13.4B
83.9
CoAINet-1
384
42M
27.4B
85.1
Con++TFM
CoAtNot-2
3842
75M
49.8B
85.7
87.1
(ours)
CoAINes-3
3842
168M
107.4B
85.8
87.6
CoAINet-4
3842
275M
189.5B
87.9
PT-RA
3842
275M
189.5B
88.3
+ PT-RA-E150
3842
275M
189,5B
88.4
2
512
75M
96.7B
85.9
87.3
3
5122
168M
203.1B
86.0
87.9
CoAINet-4
5122
275M
360.9B
88.1
+ PT-RA
512²
275M
360.9B
88.4
+ PT-RA-EI50
512
275M
360.9B
88.56
from Table 4 and Fig. 3, when ImageNet-21K is used for pre- training, the advantage of CoAtNet becomes more obvious, substantially outperforming all previous models. Notably, the best CoAtNet variant achieves a top-1 accuracy of 88.56%, matching the ViT- H/14 performance of 88.55%, which requires pre- pre-training the 2.3x larger ViT model on a 23x larger proprietary weakly labeled dataset (JFT) for 2.2x more steps. This marks dramatic improvement in both data efficiency and computation efficiency.
ImageNet-21K As
JFT Finally, in Table 5, we further evaluate CoAtNet under the large-scale data regime with JFT- 300M and JFT-3B. Encouragingly, our CoAtNet-4 can almost match the best previous performance with JFT-300M set by WFNet-F4+, while being 2x more efficient in terms of both TPU training time and parameter count. When scale up the model to consume similar training resource as NFNet-F4+,, CoAtNet-5 reaches 89.77% on top-1 accuracy, outperforming previous results under comparable settings.
Moreover, as we further push the training resource towards the level used by ViT-G/14 and utilize the same JFT-3B dataset of even larger size [26], with over 4x less computation, CoAtNet-6 is able to

---

Table 5: Performance Comparison large-scale JFT dataset. TPUv3-core-dee denotes the pre- training time, Top-I Accuracy denotes the finetuned accuracy on ImageNet. Note that the last 3 rows larger dataset JFT-3B [26] for pre-training, while others use JFT-300M [15]. See Appendix A.2
for the size details Co T: Down-sampling the MBCony block is achieved by stride-2
Depthwise Convolution. :
ViT-G/14 computation consumption is read from Fig. I of the paper [26].
Models
Eval Size
#Params
#FLOPs
TPUv3-core-days
Top-1 Accuracy
+ ViT-L/16
3842
330M
87.12
ViT-L/16
5122
307M
364B
0.68K
87.76
ViT-H/11
5182
632M
1021B
2.5K
88.55
NFNet-F4+
512²
527M
367B
1.86K
89.2
CoAINet-3/
3842
168M
114B
0.58K
88.52
CoAtNet-3t
512²
168M
214B
0.58K
88.81
4
5122
275M
361B
0,95K
89.11
CoAtNer-5
5122
688M
812B
1.82K
89.77
ViT-G/14
5182
1,84B
5160B
>30K
90.45
CoAtNer-6
5122
1.47B
1521B
6.6K
90.45
CoAtNet-7
512²
2.44B
2586B
20.1K
90.88
match the performance of ViT-G/14 of 90.45%, and with 1.5x less computation, CoAtNet-7 achieves 89.77% top-1 accuracy 90. 90.88% achieving the new state-of-the-art performance.
4.3 Ablation Studies
In this section, we will ablate our design choices for CoAtNet
Firstly, we study the importance of the relative attention from combining convolution and attention into a single computation unit. Specifically, we compare two models, one with the relative attention and the other without, under both the ImageNet-1K alone and ImageNet-21 K transfer setting. As we can see from Table 6, when only the ImageNet- 1K is used, relative attention clearly outperforms the standard attention, indicating better generalization. In addition, under the ImageNet-21 K transfer setting, the relative attention variant achieves a substantially better transfer accuracy, despite their very close pre-training performances. This suggests the main advantage of relative attention in visual processing is not in higher capacity but in better generalization.
Table Ablation on relative attention.
Setting
Metric
With Rel-Attn
Without Rel-Attn
Accuracy (224²)
84.1
83.8
ImageNet- 1K
Accuracy (3842)
85.7
85.3
ImageNet-21K
Pre-train Precision @ (224?)
53.0
52.8
ImageNet-IK
Finetune Accuracy (3842)
87.9
87.4
Table 7: Ablation on architecture layout.
Setting
Models
Layout
Top-1 Accuracy
VO: CoAtNet-2
[2, 2, 6, 14,
84.1
ImageNet-IK
V1: s2 + S3
[2, 2, 2, 18,
2]
83.4
V2: S2 + S3
[2, 2, 8, 12, 2]
84.0
ImageNet-21K ImageNet- -K
VO: CoAtNet-3
[2, 2, 6, 14,
2]
53.0 - 87.6
V1: S2 + S3
[2, 2, 2, 18,
53.0
87.4
Secondly, as S2 with MBConw blocks and S3 with relative Transformer blocks occupy most of the computation the CoAtNet, question to ask is how to split the computation between (MBConv) and S3 (Transformer) to achieve good performance. In practice, it boils down to deciding the number of blocks to have in each stage, which we will refer as "layout" design. For this purpose, compare few different layouts that we experimented with in Table 7,

---

Table 8: Ablation on head size and normalization type.
Sefting
Models
Image Size
Top-1 Accuracy
CoAtNet-2
224
84.1
ImageNet-1K
Head size: 32 - 64
224
83.9
Norm type: BN - LN
2242
84.1
ImageNet-21K
CoAtNet-3
3842
87.9
ImageNet-1K
Norm type: BN + LN
3842
87.8
If we keep the total number of blocks in S2 and S3 fixed and vary the number in each stage, observe that VO is sweet spot between V1 and V2. Basically, having more Transformer blocks in S3 generally leads better performance until the number of MBConv blocks in S2 is small to generalize well.
To further evaluate whether the sweet spot also holds in the transfer setting, where higher capacity is often regarded more important, we further compare VO and V1 under the ImageNet- 21K transferring ImageNet- K setup. Interestingly, despite that V1 and VO have the same performance during ImageNet-21K: pre-training. the transfer accuracy
clearly falls behind VO. Again, this suggests the importance of convolution in achieving good transferability and generalization.
Lastly, we study two choices of model details, namely the dimension of each attention (default to 32) head well as the type of normalization (default to BatchNorm) used in MBCony blocks. From Table 8, we can see increasing head size from 32 to 64 can slightly hurt performance, though it actually improves the TPU speed by significant amount. In practice, this will be a quality-speed trade-off one can make. On the other hand, BatchNorm and LayerNorm have almost the same performance, while BatchNorm 10 20% faster on TPU depending on the per-core batch size.
Conclusion
In this paper, we systematically study the properties of convolutions and Transformers, which leads to a principled way to combine them into new family of models named CoAtNet. Extensive experiments show that CoAtNet enjoys both good generalization like ConvNets and superior model capacity like Transformers, achieving state-of-the-art performances under different data sizes and computation budgets.
Note that this paper currently focuses on ImageNet classification for model development. However, we believe our approach is applicable to broader applications like object detection and semantic segmentation. will leave them for future work.
10

---

References
[1] Alex Krizhevsky, Πla Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097-1105, 2012.
[2] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[4] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 1-9, 2015.
[5] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, 2019.
[6] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv:1706.03762 2017.
[7] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 2018.
04805,
[8] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005. 14000000.411.5. 14165, 2020.
[9] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 7794-7803, 2018
[10] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3286-3295, 2019.
[11] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. arXiv preprint 110. 2021.
[12] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3531-3539, 2021.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929,, 2020.
[14] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lece, 2009.
[15] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852. 2017.
[16] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficien image transformers & distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
[17] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103. 2021.
[18] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint 1188, 2021.

---

[19]
Mingxing Tan and Quoe V Le. Efficientmetv2: Smaller models and faster training, ICML, 2021.
[20] Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan. High-performance large- scale image recognition without normalization. arXiv preprint (tt::2022.0617 2021.
[21] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. arXiv preprint arXiv:2103 12731, 2021.
[22] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
[23] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang, Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103. 15808, 2021.
[24] Ben Graham, Alazeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. arXiv preprint 2rX1vv2104.01136, 2021.
[25] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch imagenet. arXiv preprint arXiv:2 1986, 2021.
[26] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. arXiv preprint arXiv:2106. 04560, 2021.
[27] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Llang-Chiel Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[28]
Laurent Sifre. Rigid-motion scattering for image classification. Ph.D. thesis section 6.2, 2014.
[29]
Mirgahney Mohamed, Gabriele Cesa, Taco S Cohen, and Max Welling. A data and compute efficient design for limited-resources deep learning. arXiv preprint arXiv:2004.0960 2020.
[30] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv preprint arXiv: 02155, 2018.
[31] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019.
[32] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5166 PMLR, 2020.
[33] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprini arXiv:2009. 14794, 2020.
[34] Prajit Ramachandran, Niki Parmar, Ashish Vaswani. Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv:1906.05909, 2019.
[35] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-awarc neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820. 2019.
[36] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on visual transformer. arXiv preprint arXiv:2012. /2556, 2020.
[37] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. arXiv preprint arXiv:2101.00 169, 2021.
[38] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew Hoffman, Monica Dinculescu, and Douglas Eck, Music 2018.
transformer. arXiv preprint
12

---

[39] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint 1901:02900000// 2019.
[40] Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: A unified understanding of transformer's attention via the lens of kernel. arXiv preprint arXiv:1908. 11755, 2019.
[41] Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. arXiv preprint miXiv:2102.08602 2021.
[42] Jie Hu, Li Shen, and Gang Sun. Squeere-and-excitaiie networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[43] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXiv preprint arXiv:2103.1 1186, 2021.
[44] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.1 12 122, 2021.
[45] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[46] Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1.X.::1710004112 2017.
[47] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinherger. Deep networks with stochastic depth. In European conference computer vision, pages 646-661. Springer, 2016.
[48] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[49] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X 2017.
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European conference on computer vision, pages 630-645. Springer, 2016.
[51] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 2016.
[52] Zihang Dai, Guokun Lai, Yiming Yang, and Quoe V Le. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. arXiv preprint arXiv:2006.032316 2020.
13


Reviews:
[{"review_id": "pbGTmPFalDA", "reviewer": "Reviewer_err4", "summary": "This paper proposes CoAtNet, a hybrid model that can take the advantage of convolution and self-attention. The efficacy of CoAtNet can be attributed to two components, relative self-attention and vertical layout design. The proposed relative self-attention is essentially a combination of self-attention and relative positional encoding. Based on that, the CoAtNet further applies some other tweaks including relacing the self-attention into convolutions in the early stages, adding SE modules etc.\nBased on the above observations, CoAtNet achieves state-of-the-art performance on ImageNet.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n\n- The novelty of this paper is somewhat limited. The relative attention proposed in this work is very similar to the positional encoding designed in LambdaNet [1]. Or it could be regarded as a 2D-version relative positional encoding proposed in HaloNet [2] and SASA [3]. It would be preferable if the author could concretely compare with the above modules in their paper.\n\n- The vertical layout design can also be found in some existing works like LambdaNet [1] and AANet [4]. To me, it is not surprising that convolution can achieve a better cost-accuracy tradeoff in the early stages. \n\n---\n\nQuality:\n\n- The design of CoAtNet is backed by extensive and comprehensive experiments. I appreciate the efforts that the authors made to analyze the generalization and model capacity of convolution and self-attention (Figure 1). But I am curious about whether the relative attention itself can well exploit the information given by the inductive biases. It would be more clear to add another experiment replacing all $3\\times 3$ convolutions in ResNet with the relative attention designed in this paper.\n\n\n---\n\n\nSignificance: \n\n- In general, this paper should be impactful in its corresponding field due to the comprehensive experiments and strong performance. However, as it does not extend to other downstream tasks like object detection, the impact of this paper may be somewhat limited in other fields of computer vision.\n\n---\n\nClarity: \n\n- This paper is well written and easy to understand. However, it may be hard to reproduce as there are lots of tweaks (e.g. SE, Stem, MBConv, Pre-activation, Relative Attention...) introduced in just one network. It would be preferable if the authors promise to release the code of the proposed method. Meanwhile, the indexing of the relative self-attention in Eq (3) makes me a bit confused. It would be better to have a figure here demonstrating the entire process.\n\n---\n\nReferences\n\n[1] Bello I. Lambdanetworks: Modeling long-range interactions without attention. ICLR 2021\n\n[2] Vaswani A, Ramachandran P, Srinivas A, et al. Scaling local self-attention for parameter efficient visual backbones. CVPR 2021\n\n[3] Ramachandran P, Parmar N, Vaswani A, et al. Stand-alone self-attention in vision models. NeurIPS 2019\n\n[4] Bello I, Zoph B, Vaswani A, et al. Attention augmented convolutional networks. ICCV 2019", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations:\n\n- Some of my concerns are given in the main review. See above.\n\n- It would be preferable to provide a fair comparison in terms of throughput with other networks listed in Table 4.\n\n- It would be more clear to provide the pseudocode of the relative self-attention in the appendix.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "mOgHB_Jtsrd", "reviewer": "Reviewer_3fnX", "summary": "This paper explores how to combine convolution and self-attention for vision models. It points out that depthwise convolution and self-attention can be unified as relative attention. In addition, it proposes a family of hybrid models, named CoAtNets, which vertically stack convolution layers and self-attention layers. Experiments show that CoAtNets strike a balance between generalization and model capacity and these models achieve state-of-the-art performance in different data regimes. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper systematically analyzes how to hybrid convolution and attention. It studies this from aspects of model generalization ability and model capacity.  \n\nIt shows depthwise convolution and self-attention can be simply merged as relative self-attention module. This unified view is interesting but not so innovative.\n\nIn addition, CoAtNets are proposed by searching in 5 variants. Rules are designed to evaluate the generalization and capacity of models. These strategies are straightforward but inspiring. They may be useful for future network design.\n\nExtensive experiments show CoAtNets achieve both generalization and capacity. Their SOTA performances are impressive.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. This paper regards relative self-attention as the hybrid of convolution and self-attention. This understanding is interesting. However, as relative position bias have been discussed in many previous works, it is hard to say this is the contribution of this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "DFb6E4HnTji", "reviewer": "Reviewer_VNGC", "summary": "This paper proposes an architecture CoAtNet that combines convolution and relative attention. The paper claims that CoAtNet can have the advantages of both operations and can achieve better or similar performance in all data sizes. Experiments show state-of-the-art classification performance on ImageNet-1K, ImageNet-21K, and the larger JFT dataset with the less computational resource.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n+ The paper is clear and easy to understand.\n+ The paper presents many good practices, which can be very useful to the community and shed light on the design of convolution and attention mixtures.\n+ The experiments are comprehensive and as many details as possible are included in the paper. This makes other researchers easily reproduce the results.\n+ The proposed CoAtNet achieves very good performance, outperforming previous SOTA methods such as Swin.\n\nCons:\n- The paper has limited novelty. Using the mixture of convolution and attention has been attempted by many previous works such as BoTNet [11]. The contribution of the work is just to find a better combination, thus being incremental.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mentioned the limitation is that they only focus on the classification. I think the authors should include more, for example, the models' actual latency on TPUs/GPUs.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "1t3GAGy-NdP", "reviewer": "Reviewer_euiw", "summary": "This paper presents a system and experimental studies on combining convolutions with transfomers.  Multiple placements of a convnet->stacked transformer transition are evaluated, finding about three conv stages and two transformer stages are best in most cases.  A set of shift-parameterized weights are added to attention similarities in the transformers, which conceptually combines convolution and attention operations in these stages as well.  The system achieves high accuracy for FLOPs on ImageNet, in cases both with and without extra data and pretraining, and the effect of different parts of the system are evaluated with ablations in resource allotment.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a good set of studies and explanations of the effect of conv->transformer stacked models.  The eventual model arrived at is similar to the hybrid ResNet50-ViT stride 16 model described in Visual Transformers [13], though with differences around the structure of the convolution stages, using depthwise conv blocks.  Section 3.2 provides a more comprehensive study around conv->transformer placement, evaluating different numbers of conv stages from stride 8 up to 32.\n\nCombining convolutional (shift-dependent) weights in the attention weights is also an interesting approach, which the experiments show has positive effect.  I would have liked to see a comparison between the two variants mentioned in the description (\"pre\" and \"post\", only \"pre\" is used).\n\nAnother recent work that combines convnets and transformers is PVT https://arxiv.org/pdf/2102.12122.pdf, which I think would be interesting to compare with.  The hybrid ResNet50-ViT in [13] is also not mentioned, and I think would also be good to add to Table 5 and Figures 2 and 3.\n\nThis is a performant system that is explained well, including an interesting relative attention module that combines convlution-like weights directly into attention values.  Much of the general architecture ends up similar to recent other hybrid approaches, with additional studies validating these choices.\n\n\n\nAdditional comments and questions:\n\n- would be good to also compare with PVT https://arxiv.org/pdf/2102.12122.pdf\n\n- for multi-headed attention, are the weights w_{i-j} the same or different for each head?\n\n- sec 3.1 footnote 2:  Says this omits q k v projections for now, but I don't see them re-introduced later.  If these projections are used in the actual system, I think it would be good to either include, or mention that they are used possibly with multihead attention reference.\n\n\n- sec 3.2:  how is 2x downsampling performed for stages S2..5, for either C or T variants?  This is in appendix A.1, but I think key sentences or equations of A.1 including downsampling placements would be better included the main text, as they help explain questions that came up immediately for me when reading 3.2.\n\n\n- Table 7:  How does moving layers between S2/S3 affect FLOPs?  Also text says V1 has better imnet21k pretraining accuracy but both v0 and v1 in the table show 53.0.\n\n\n- 3.2 option (A) --- PVT and CvT also use a form of downsampling, within each attention layer on kv maps (in addition to striding between stages), which is different from the approach here that is only studying downsampling between stages.  I think this could be good to contrast.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The rebuttal addressed all of the reviewers concerns, and all reviewers recommend acceptance. The AC agrees with this recommendation.
Decision: Accept (Poster)