Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel vision transformer architecture called CrossFormer which focuses on the cross-scale ability in the attention module. Specifically, the proposed CrossFormer introduce Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA) to model cross-scale interactions. CEL has the capability to merge multi-scale features with various kernel sizes, while LSDA enables the self-attention to capture feature interactions of both short distance and long distance. Experiments demonstrate that the proposed CrossFormer achieves performance improvement on image classification, object detection, instance segmentation and semantic segmentation tasks.
main_review: Pros:
1. The Cross-scale Embedding Layer (CEL) extends the single-scale feature maps used in prior works of ViTs (e.g. ViT, DeiT, PVT, Swin)  to multi-scale feature maps. The mixture of multi-scale features enable a richer context, especially for dense prediction tasks like object detection, instance segmentation and semantic segmentation. In addition, the CEL employs a series of kernel with the same stride and decreasing number of output dimensions, which is able to control the total computation budgets.

2. In the self-attention mechanism, the proposed CrossFormer introduces a long short distance attention (LSDA), which includes a long distance attention (LDA) and a short distance attention (SDA). The SDA is very similar to the shifted window attention in Swin, which splits feature maps into windows (denoted by groups in this paper) and performs self-attention in each window (group). Different from Swin, the LDA module has a dilated window where each embedding in the window has a large interval to its adjacent embeddings. The proposed LDA can model long range interactions while still keeping a linear complexity in computation.

3. In the experiment section, the proposed approach achieves state-of-the-art performance on all tasks. Specifically, on dense prediction task like detection and segmentation, the proposed CrossFormer obtains decent improvement over prior works with similar computation budget.

Cons / Questions:
1. The proposed CrossFormer focuses on better modeling of cross-scale information for ViTs: CEL enables multi-scale features and LSDA enables long distance interaction (which implies interaction among large-scale features). However, there are already some works on the cross-scale information such as [1]. Although the prior works may not use the identical way, it is still needed to have some discussion in the manuscript to compare the alternatives to model cross-scale information and the benefits from the proposed CrossFormer.

    [1] Co-Scale Conv-Attentional Image Transformers, Xu et al, ICCV 2021

2. The proposed LSDA may have some limitations in its design: Table 1 shows that each stage in CrossFormer has the same choice of (G, I) and Appendix A.1 shows that G*I is the size of the feature map. Thus, when the size is large (e.g. stage 1, size=56), since G is alway 7, interval I will be large (e.g. stage 1, I=7), which causes the elements in the window (group) are very scattered, and the behavior of the LDA could be very different from SDA in the same stage. I wonder if such large gap between LDA and SDA could lead to some potential issues. In addition, perhaps ablation study on LDA with lower interval is needed, since it may work better together with SDA.

3. This paper also proposes a dynamic position bias (DPB), which aims to replace the relative position bias (RPB) used in Swin and early works in NLP (Shaw et al). However, Table 6(b) does not show that the proposed DPB is able to outperform RPB, which limits the use of DPB. In addition, it is claimed that DPB has the capability to support different window (group) size. It would be better to have a case that demonstrates this benefit.

4. In the empirical experiments, on the image classification task, the proposed CrossFormer obtains similar performance as the best of prior works. Thus, I wonder if it is possible to apply the proposed CEL and LSDA to prior works (e.g. RegionViT or NesT) to further improve the performance?

summary_of_the_review: This paper proposed CrossFormer that aims to model cross-scale information through Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). The design of both modules is reasonable, and it demonstrates performance improvement on dense prediction tasks. However, due the potential limits of LSDA and DPB and missing comparison with some prior works, I would like to give a "marginally above the acceptance threshold" rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two new techniques: Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). Cross-scale Embedding Layer samples patches with convolution kernels of different sizes and concatenates them as one embedding, so that patch embeddings contain information of multiple scales. Long Short Distance Attention includes a short attention module and a long attention module. The short attention is computed in a local region. The long one covers the whole image but is computed sparsely. These two new designs enable the network to learn cross-scale features. Also, a dynamic position bias is designed to make the network suitable for inputs of different resolutions.
main_review: Pros:
1. The idea of CEL is sound and its design is neat. As this layer can be plugged in other structures easily, it can also boost other models.
2. Thorough experiments verify the effectiveness of the proposed techniques. And CrossFormer achieve good performance on several vision taks.

Cons:
1. Some comparisons and discussions with related works are missed. For example, CrossViT[1] also combine features from patches of different sizes and Shuffle Transformer[2] proposes a spatial shuffle mechanism for transformer layer, which is similar with the long attention module in this paper.

[1]  Chen, C. F., Fan, Q., & Panda, R. (2021). Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103.14899.

[2]  Huang, Z., Ben, Y., Luo, G., Cheng, P., Yu, G., & Fu, B. (2021). Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer. arXiv preprint arXiv:2106.03650

summary_of_the_review: Overall, the ideas of CEL and LSDA are sound and solid experiments support most claims in the paper. However, some designs are not new and some related works are missed. I would rate this as a borderline paper (marginally above the acceptance threshold).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a novel multi-stage vision transformer architecture, named CrossFormer, which aimed at building interaction among features from different scales. The proposed architecture follows the layout of Swin transformer but replaces the shifted windows transformer block with the proposed short-distance attention and long-distance attention. A cross-scale module is proposed to replace the patch embedding or patch merging layer in the Swin transformer. Comprehensive experiments are conducted on image classification, objection detection, and segmentation tasks. Results shows the effectiveness of the proposed architecture. 
main_review: Pros:
- How to build the interaction among different scale features is an interesting problem in computer vision. The proposed method provides an easy way to build the multi-scale feature interaction for transformer-based architecture. 
- Comprehensive experiments are conducted, and the results showed the effectiveness of the proposed architecture.
- The idea of the paper is clear and easy to follow. 

Cons:
- The short-distance attention is essentially the local window attention in the Swin-transformer, which is not new.
- The results in Table 6(b) showed that the proposed dynamic positional embedding has a similar performance with relative positional embedding. It's hard to say that the proposed dynamic positional embedding (DPE) is a contribution. The original relative positional embedding is also learned and dynamic and is also the function of relative coordinates. So, the problem becomes how to define the function.  However, the authors did not show how and why the function in the proposed DPE is superior to original relative positional embedding. 
- The way to generate the multi-scale features is not novel. In the previous works [3, 4], the mixed convolution, which is especially the same as the proposed Cross-scale Embedding Layer. 
- The long-distance attention is like the method of interleaved group or shuffle, which are proposed in the paper of [1,2] over the spatial features. I think the authors should carefully discuss the relation between them. 

[1] ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

[2] Interleaved Group Convolutions

[3] Going Deeper with Convolutions

[4] MixConv: Mixed Depthwise Convolutional Kernels

summary_of_the_review: The idea of the paper is clear. Comprehensive experiments are performed, and superior results over several benchmarks demonstrated the effectiveness of the proposed architecture. My main concern is that the idea of the proposed components is not new in the CNN-based works. The overall novelty of the paper is limited.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes a pyramidal vision transformer that introduces cross-scale patch embeddings, dynamic relative position biases and "Long-Short Attention". The model shows superior performance on imagenet classification and especially on dense prediction tasks. Ablations on classification are provided hinting that the multi-scale embeddings are the most important factor contributing to the overall superior results to similar, previous architectures.
main_review: ## Strengths
* Overall simplicity and elegance of the introduced components. I think the paper strikes a good balance between introduced complexity and achieved performance.
* Results are very strong on a number of vision tasks, and in particular on dense prediction tasks
* Well presented and easy to understand.

## Weaknesses
* Context to prior work:
  * In principle a fully connected layer over a large patch can learn to represent patches of multiple smaller scales as well. The Cross-scale embedding layer in this paper is essentially just a more constraint version of it. Taking 32x32 patches (4x4 stride) and projecting those to D is strictly more expressive than the procedure described in the paper. This is not to say that it is not useful as it clearly provides inductive bias, but it is less expressive which is the opposite of what the paper says. Hence I don't agree with the blank statement: "They fail to build the interactions among features of different scales, whereas such an ability is very vital for a lot of vision tasks." A more subtle discussion on inductive biases would be more appropriate and I would ask the authors to change that. Personal side-note: A more interesting take on scale (IMO) would be to introduce a mechanism that enforces scale invariance (at least in some range).
  * Long-short distance attention is basically a version of sparse attention (strided attention) that was introduced for instance in [1]. There are also other papers, e.g., [2, 3], that introduced very similar sparse attention mechanisms, however, mostly with a focus on text but they easily generalize to images. So the novelty of the presented mechanism here is limited (at best) and the absence of proper references to similar work is a bit concerning. A more thorough study and comparison with prior work should be added.
* Novelty (Framing): The individual components are not very novel (see points before), and it should be made clear in the paper. It's, however, not a big issue for me because finding the right combination of these components in order to strike the right balance between complexity and performance is more important and as such also novel.

## Questions
* Ablations on a dense prediction task to see where the improvements really come from. The authors hypothesize they are due to cross scale patches. These ablations are important to also verify that it isn't the overall training setup or other subtle differences in the architecture that give the advantage.
* A strange choice is that smaller patches project to larger dimensions, leading to larger compression. Was this a deliberate choice and ablated?
* Setting G and I to different values and not making G divisible by I is a bit odd, I wonder why such values were chosen.

* [1] https://arxiv.org/abs/1904.10509
* [2] https://arxiv.org/abs/2007.14062
* [3] https://arxiv.org/abs/2004.05150
* [4] https://arxiv.org/abs/2103.15358
summary_of_the_review: The paper is overall well written and easy to understand. I think there are some strong contributions that are a bit oversold (in my view). However, I am leaning towards accepting the paper given that some of the concerns and questions are properly addressed. I am willing to raise my score in case those concerns are addressed.

UPDATE after rebuttal. I am willing to raise my score after reading the rebuttal and the updates to the paper. I think the paper is a good and clean contribution as it is conceptually simpler (in my view) than most of the previous works while showing stronger performance.
review_title: null

Meta-review: The paper proposes several modifications to vision transformers: multiscale features, a variant of factorized attention, and "dynamic position bias". The proposed architecture with these modifications achieves strong results on classification, detection, and segmentation.

After considering the authors' responses, all reviewers are positive about the paper (reviewer K7wS mentioned upgrading to weak accept, but apparently forgot to do so). Main pros include clean architecture and strong empirical results. The main con is the somewhat limited novelty.

Overall, I recommend acceptance. While each of the proposed modifications might not be that unique, they are reasonably new in the context of transformers and their combination makes for a clean architecture that performs very well in practice.
Decision: Accept (Poster)