Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a “transposed” self-attention module, it has linear complexity in the token side, which is more efficient on high-resolution images. The performance of XCiT on ImageNet classification  is promising, it also shows strong results on detection/segmentation and self-supervised pre-training.
main_review: This paper has clear paper writing, nice reference/related work, and very solid experiments. The novelty is also good for me.

Here are some suggestions: 

This paper seems like a new version of SENet[1], the proposed Cross-Covariance Attention (XCA) actually does a self-attention at channel level, which is more complex than SENet's way (and the results is better).  It achieves spatial interaction with 3x3 dw-convs. It makes sense.

The idea of XCA is good. It can reduce the complexity of self-attention. But I have several questions. If the authors can solve my concerns in rebuttal, I am happy to increase my score. 

(1) How about the results on detection/segmentation of combining XCA with general-purpose Vision Transformers such as PVT[2] and Swin Transformer[3]? 
Because XCiT can only return features with the same scales, and manually resize them to multiple scales, which is not elegant and may hurt performance.
This paper has a strong potential to handle high-resolution images, which is important for object detection and semantic segmentation tasks. Choosing PVT or Swin as a baseline is a better choice.

(2) How about using regular self-attention for patch interaction instead of using 3x3 convs? Can it further improve the performance? Or use efficient attention in PVT(sparse manner) or Swin(local manner).

(3) Miss the ablation study about comparison XCA and Squeeze-and-Excitation. 


Reference

[1] Squeeze-and-Excitation Networks

[2] Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

[3] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes the idea of bypassing the calculation of NxN attention scores and directly deriving the weighted average of values in a self-attention by first calculating DxD cross-covariance matrix of K^TQ and then averaging over the D features of V instead of its tokens. Since this method of attention does not involve cross-token communication, the paper further proposes to use a Local Patch Interaction module in which a depth-wise 3x3 convolution enables cross-token communication.
main_review: Manuscript: The paper is very well-written and easy to follow. While the paper provides results on multiple tasks and datasets, the comparison to state-of-the-art is not complete and very little ablation is provided.

Novelty: The core idea of the paper is interesting. However, intuitive justification for some ideas is missing in the paper. It is important to cover the following questions in the paper:
1. The intuition behind a conventional attention layer is clear. What is the equivalent intuition behind XCA attention layer?
2. The paper claims that adding LPI helps with accuracy and convergence. This might be due to the Batch Normalization in this layer. Have the authors tried only a Batch Normalization (which also implicitly brings cross-sample stats in) after the XCA modules?
3. The paper argues against MLP-Mixer for deriving attention scores by learnable weights. Related to this argument, an interesting ablation to consider is applying a channel-wise MLP instead of LPI after the attention module to observe the effect of such projection. This, while still quadratic in computation, is necessary to support such argument. It essentially shows what happens if we replace the MLP-2 module in MLP-Mixer with an XCA.
limitations_and_societal_impact: The societal impact has not been discussed in the paper. However, the limitations are mentioned and discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a modification of vision transformers by introducing Cross-Covariance Attention (XCA) and Local Patch Interaction (LPI) which substantially reduce the computational overhead and enhances the stability of training. The authors also adopt the DeiT training strategy with distillation using a convolutional teacher to improve the performance. Experiments on Imagenet classification, self-supervised training with DINO and many downstream tasks (COCO object detection and instance segmentation, ADE20K semantic segmentation) show the superior performance. 
main_review: The core idea of Cross-Covariance Attention (XCA) to make transformer linearity in this paper was originally proposed in [56]. This submission use it to make the whole transformer model work with L2-Normalization and temperature scaling.
On the other hand, the Local Patch Interaction (LPI) is good but do not bring any theoratical contribution or give exciting intuition despite the 1.2% improvement on ImageNet (Table 4).
Another contribution is that the final model performs better than original ViT on various tasks including self-supervised training with DINO and many downstream tasks.
It also achieves state-of-the-art performance by using the DeiT training strategy.
However, this paper does not bring any new idea but incorporates several different things together. I vote for rejection.

[56] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.
limitations_and_societal_impact: See above
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an efficient method to compute self-attention in Transformers without sacrificing accuracy. The idea is to compute the covariance self-attention over the feature dimension followed by a 3x3 conv to model the patch interaction. With additional techniques such as feature grouping and L2 norm, the method shows competitive accuracy on ImageNet compared to the recent Vision Transformer models. This method has linear complexity to the input patches and thus can be used for high-resolution images on the detection and segmentation tasks.
main_review: **Originality**: High the method is new in vision transformers.

**Quality**: Good. The method is well motivated and explained. The main claim appears to be supported by strong empirical results.

**Clarity**: Great.

**Significance**: High. This fundamental method enables linear computation of self-attention over high-resolution images which may impact many works in object detection and segmentation.

Strengths:
+ Novel and efficient method that is well-motivated by covariance computation and 1x1 conv operator.
+ The problem is non-trivial for which the authors develop additional techniques in order to obtain promising results.
+ Extensive experiments are conducted over multiple tasks to support its efficiency and accuracy.

Weaknesses:
- The performance on ImageNet is not state-of-the-art without distillation. It is okay but is it fair to include the distillation numbers in Table 2 as many of the baselines do not? What is the performance for the teacher conv model for the models in Table 1?

- The comparison to other efficient attention computation methods (in Table A of Appendix) should be included in the main paper. Would the results still be consistent on other popular Vision Transformers other than DeiT? 

- As the advantage of the method is on high-resolution images, why not compare with other baselines in Table A on object detection or segmentation?

My preliminary rating is 7 but I hope the authors can address the above questions.


=====
Post-rebuttal

Thank you for answering the questions. My questions are mainly about clarity and presentation, and the author made them clearer in the rebuttal. I keep my rating and hope the authors would make the changes stated in the rebuttal.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: Initially, the paper received three acceptance and one rejection. One reviewer had concern on the novelty of the paper given its similarity with EA. Upon rebuttal and discussion, the reviewer converged with the other reviewers on the novelty of this paper. The AC agrees with the reviewers and recommends to accept the paper. The authors are encouraged to improve final version of the paper following the suggestions from the reviewers.
Decision: Accept (Poster)