Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper explores how to combine convolution and self-attention for vision models. It points out that depthwise convolution and self-attention can be unified as relative attention. In addition, it proposes a family of hybrid models, named CoAtNets, which vertically stack convolution layers and self-attention layers. Experiments show that CoAtNets strike a balance between generalization and model capacity and these models achieve state-of-the-art performance in different data regimes. 
main_review: This paper systematically analyzes how to hybrid convolution and attention. It studies this from aspects of model generalization ability and model capacity.  

It shows depthwise convolution and self-attention can be simply merged as relative self-attention module. This unified view is interesting but not so innovative.

In addition, CoAtNets are proposed by searching in 5 variants. Rules are designed to evaluate the generalization and capacity of models. These strategies are straightforward but inspiring. They may be useful for future network design.

Extensive experiments show CoAtNets achieve both generalization and capacity. Their SOTA performances are impressive.

limitations_and_societal_impact: 1. This paper regards relative self-attention as the hybrid of convolution and self-attention. This understanding is interesting. However, as relative position bias have been discussed in many previous works, it is hard to say this is the contribution of this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a system and experimental studies on combining convolutions with transfomers.  Multiple placements of a convnet->stacked transformer transition are evaluated, finding about three conv stages and two transformer stages are best in most cases.  A set of shift-parameterized weights are added to attention similarities in the transformers, which conceptually combines convolution and attention operations in these stages as well.  The system achieves high accuracy for FLOPs on ImageNet, in cases both with and without extra data and pretraining, and the effect of different parts of the system are evaluated with ablations in resource allotment.

main_review: This is a good set of studies and explanations of the effect of conv->transformer stacked models.  The eventual model arrived at is similar to the hybrid ResNet50-ViT stride 16 model described in Visual Transformers [13], though with differences around the structure of the convolution stages, using depthwise conv blocks.  Section 3.2 provides a more comprehensive study around conv->transformer placement, evaluating different numbers of conv stages from stride 8 up to 32.

Combining convolutional (shift-dependent) weights in the attention weights is also an interesting approach, which the experiments show has positive effect.  I would have liked to see a comparison between the two variants mentioned in the description ("pre" and "post", only "pre" is used).

Another recent work that combines convnets and transformers is PVT https://arxiv.org/pdf/2102.12122.pdf, which I think would be interesting to compare with.  The hybrid ResNet50-ViT in [13] is also not mentioned, and I think would also be good to add to Table 5 and Figures 2 and 3.

This is a performant system that is explained well, including an interesting relative attention module that combines convlution-like weights directly into attention values.  Much of the general architecture ends up similar to recent other hybrid approaches, with additional studies validating these choices.



Additional comments and questions:

- would be good to also compare with PVT https://arxiv.org/pdf/2102.12122.pdf

- for multi-headed attention, are the weights w_{i-j} the same or different for each head?

- sec 3.1 footnote 2:  Says this omits q k v projections for now, but I don't see them re-introduced later.  If these projections are used in the actual system, I think it would be good to either include, or mention that they are used possibly with multihead attention reference.


- sec 3.2:  how is 2x downsampling performed for stages S2..5, for either C or T variants?  This is in appendix A.1, but I think key sentences or equations of A.1 including downsampling placements would be better included the main text, as they help explain questions that came up immediately for me when reading 3.2.


- Table 7:  How does moving layers between S2/S3 affect FLOPs?  Also text says V1 has better imnet21k pretraining accuracy but both v0 and v1 in the table show 53.0.


- 3.2 option (A) --- PVT and CvT also use a form of downsampling, within each attention layer on kv maps (in addition to striding between stages), which is different from the approach here that is only studying downsampling between stages.  I think this could be good to contrast.


limitations_and_societal_impact: -
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes CoAtNet, a hybrid model that can take the advantage of convolution and self-attention. The efficacy of CoAtNet can be attributed to two components, relative self-attention and vertical layout design. The proposed relative self-attention is essentially a combination of self-attention and relative positional encoding. Based on that, the CoAtNet further applies some other tweaks including relacing the self-attention into convolutions in the early stages, adding SE modules etc.
Based on the above observations, CoAtNet achieves state-of-the-art performance on ImageNet.
main_review: Originality: 

- The novelty of this paper is somewhat limited. The relative attention proposed in this work is very similar to the positional encoding designed in LambdaNet [1]. Or it could be regarded as a 2D-version relative positional encoding proposed in HaloNet [2] and SASA [3]. It would be preferable if the author could concretely compare with the above modules in their paper.

- The vertical layout design can also be found in some existing works like LambdaNet [1] and AANet [4]. To me, it is not surprising that convolution can achieve a better cost-accuracy tradeoff in the early stages. 

---

Quality:

- The design of CoAtNet is backed by extensive and comprehensive experiments. I appreciate the efforts that the authors made to analyze the generalization and model capacity of convolution and self-attention (Figure 1). But I am curious about whether the relative attention itself can well exploit the information given by the inductive biases. It would be more clear to add another experiment replacing all $3\times 3$ convolutions in ResNet with the relative attention designed in this paper.


---


Significance: 

- In general, this paper should be impactful in its corresponding field due to the comprehensive experiments and strong performance. However, as it does not extend to other downstream tasks like object detection, the impact of this paper may be somewhat limited in other fields of computer vision.

---

Clarity: 

- This paper is well written and easy to understand. However, it may be hard to reproduce as there are lots of tweaks (e.g. SE, Stem, MBConv, Pre-activation, Relative Attention...) introduced in just one network. It would be preferable if the authors promise to release the code of the proposed method. Meanwhile, the indexing of the relative self-attention in Eq (3) makes me a bit confused. It would be better to have a figure here demonstrating the entire process.

---

References

[1] Bello I. Lambdanetworks: Modeling long-range interactions without attention. ICLR 2021

[2] Vaswani A, Ramachandran P, Srinivas A, et al. Scaling local self-attention for parameter efficient visual backbones. CVPR 2021

[3] Ramachandran P, Parmar N, Vaswani A, et al. Stand-alone self-attention in vision models. NeurIPS 2019

[4] Bello I, Zoph B, Vaswani A, et al. Attention augmented convolutional networks. ICCV 2019
limitations_and_societal_impact: Limitations:

- Some of my concerns are given in the main review. See above.

- It would be preferable to provide a fair comparison in terms of throughput with other networks listed in Table 4.

- It would be more clear to provide the pseudocode of the relative self-attention in the appendix.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an architecture CoAtNet that combines convolution and relative attention. The paper claims that CoAtNet can have the advantages of both operations and can achieve better or similar performance in all data sizes. Experiments show state-of-the-art classification performance on ImageNet-1K, ImageNet-21K, and the larger JFT dataset with the less computational resource.

main_review: Pros:

+ The paper is clear and easy to understand.
+ The paper presents many good practices, which can be very useful to the community and shed light on the design of convolution and attention mixtures.
+ The experiments are comprehensive and as many details as possible are included in the paper. This makes other researchers easily reproduce the results.
+ The proposed CoAtNet achieves very good performance, outperforming previous SOTA methods such as Swin.

Cons:
- The paper has limited novelty. Using the mixture of convolution and attention has been attempted by many previous works such as BoTNet [11]. The contribution of the work is just to find a better combination, thus being incremental.

limitations_and_societal_impact: The authors mentioned the limitation is that they only focus on the classification. I think the authors should include more, for example, the models' actual latency on TPUs/GPUs.
review_title: null

Meta-review: The rebuttal addressed all of the reviewers concerns, and all reviewers recommend acceptance. The AC agrees with this recommendation.
Decision: Accept (Poster)