Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presented a new Transformer-based model for visual recognition. The key idea is to use a checkerboard-like pattern (dilated splitting) to divide the 2D feature plan into multiple smaller partitions, where the standard self-attention can be efficiently computed on each of these partitions (glance branch). A local depthwise convolution was also added to link the partitions (gaze branch). To evaluate the proposed model, the authors presented extensive results across several visual recognition tasks, including image classification, semantic segmentation, and object detection. The results are promising.

main_review: **Originality**

The key idea of the glance branch (dilated splitting) is well suited for 2D images and quite interesting. The combination of the glance / gaze branch is convincing.
 
My main concern is the missing link to previous works on efficient Transformer models in the ML / NLP community. Specifically, there has been quite some recent effort on developing linear time self-attention / Transformer blocks, e.g.,  Linformer [1], Longformer [2], Performer [3], Reformer [4], Nystromformer [5], Informer [6] and Big Bird [7]. As one of the main arguments of this paper is an efficient Transformer model, these previous works seem quite relevant and should be discussed and/or compared. My understanding is that the proposed method is tailored for 2D visual data, while previous approaches are more general and can also be applied to 2D data. Further, some of the ideas proposed in the paper can be found in these previous works.  For example, using sparse attention from different local pattern was discussed in [7] while using depthwise convolution in parallel to self-attention was presented in [5]. It thus remains unclear to me how the proposed method compares to these previous approaches.

[1] Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768.

[2] Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.

[3] Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., ... & Weller, A. (2020). Rethinking attention with performers. In ICLR.

[4] Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020). Reformer: The efficient transformer. In ICLR.

[5] Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., & Singh, V. (2021). Nystr\" omformer: A Nystr\" om-Based Algorithm for Approximating Self-Attention. In AAAI.

[6] Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021, May). Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI.

[7] Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., ... & Ahmed, A. (2020, July). Big Bird: Transformers for Longer Sequences. In NeurIPS.


**Quality**

The paper is technically sound. Experiments are thorough. And the results are solid.

A main motivation of the model is the visual attention in the human visual system. Yet I don’t see a clear link between self-attention (computation of self similarity between features) and human visual attention (selection of relevant sensory information). I do think that the idea of dilated splitting stems from dilated kernels in 2D convolution. And that seems to be a better perspective. 

**Clarity**

The paper is reasonably well written. The key idea is nicely illustrated. 

**Significance**

The paper addressed the design of vision Transformers, a trendy topic in the field. The proposed model has the potential to serve as a common backbone for many vision tasks. The downside is that the proposed model seems to only have minor performance gains and similar computational costs when compared to Swin Transformer in [24], though I recognize that [24] is an arXiv paper and not yet published. 

**Post-rebuttal Update**

Thanks for the authors' response. I've read other reviews and responses. A main concern shared among reviewers is the comparison to Swin-Transformer [24]. The clarification of details and the additional results provided in the rebuttal have addressed my concern. Further, given that [24] appeared on arXiv around two months before the NeurIPS deadline, I don't think this paper should get penalized for not beating the results in [24]. That being said, I encourage the authors to clarify the experiment settings in the paper, and to improve the presentation quality. Overall, I think this is a paper with an interesting idea and good results and I am happy to increase my rating. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This paper proposes a Glance and Gaze vision transformer to overcome the complexity problem common in vision transformers. The proposed transformer has two parallel branches - a glance branch that accounts for long-term dependencies and a gaze branch to account for local context. The authors compare their work on ImageNet-1k, COCO and ADE20K data sets for image classification, object recognition and semantic segmentation. Reference [24] (Swin Transformer) was the prior art on which the experimental set-up was taken from. Comparisons with a number of other vision transformer settings were made.
main_review: The idea of Glance and Gaze has an intuitive appeal and this is appreciated by the reviewer. However, there are a number of issues with this paper:

1) The paper has numerous grammatical errors and typos which made it hard to follow. 
2) The main issue with this paper is that it is unable to beat the accuracy of [24] which they have taken as baseline. For example, consider the last four rows of Table 1 which compares Swin with the proposed GG. The complexity is identical and accuracy similar. Also, results with ImageNet-22k are not shown.
3) The Swin-T and Swin-S numbers listed in Table 2 for ADE20K task are lower than the ones reported in [24]. Why is this? The original paper [24] reports mIoU of 46.1% (Swin-T) and 49.3% (Swin-S) which are higher than those obtained by GG which further buttresses the comment in 2). FPS numbers are not reported. Not sure what is going on. 
4) Table 3 for object detection has the same issue as in Table 2 (see Comment 3). The numbers reported by [24] are higher that those cited and compared with. Also, since [24] is the main prior work for establishing a comparison framework, there needs to be comparison with other object detection frameworks such as ATSS, Rev Points V2...

It is important that the authors clarify the issue of numbers from prior art they are quoting in the paper (see comments 3 and 4) and expand the set of comparisons they are making to match up with reference [24]. The scope of the paper needs to cover at least the same data sets and metrics as in [24].

limitations_and_societal_impact: To some extent.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a novel attention mechanism for the transformer architecture. Inspired by human gaze, authors propose to split the self-attention operation which is typically done across all patches in two parallel processes: one which reasons locally and another one that reasons globally. Authors evaluate their method in standard datasets for segmentation, classification and detection and ablate the main design decisions.
main_review: Summary: The paper introduces one single idea, describes it very well and show its effectiveness through evaluation in multiple tasks on standard benchmarks. For those reasons, I believe the paper should be accepted.

Strengths:
- I really like how the block splits the standard self-attention operation into local and global attention in a simple but effective way. I believe that as transformers are becoming a more relevant architecture over time, the community will need to move towards more efficient architectures to process images in transformers.

- Authors evaluate their model in three tasks: detection, segmentation and classification. I believe this makes the paper very strong, as it shows that the proposed block is effective not only for the standard classification task (which is no affected by local changes in the image) but also works well for detection and segmentation.

-   The results of the evaluation in the three task show that the performance of the proposed method improves previous proposal. Furthermore, authors report an extensive list of baselines which helps the reader put the result in perspective.

- Authors ablate the most relevant design decisions of the paper. Specially, I like that the block also improves performance when used in other backbones Table 4.c.

- The paper is very well written and explains very well the approach proposed. 

Weaknesses:

- I am missing some qualitative evaluation on what each block is doing on the operation. I think this would be very interesting for the reader to gain a bit of understanding of the model. This could be archived by providing some qualitative results of the attention, or even examples where the proposed model improves the baselines. 

- In L58, the sentence in italic is a bit confusing and it appears to be the name of a paper (which is not). I would recommend the authors edit that part of the paper to make this clearer.

- Have the authors considered processing the gaze part without a convolution and with a transformer-like approach? I would be curious to see if there a solution without involving convolution and how does that perform to the proposed method. 

- I think more discussion about the computational reduction would be also interesting and make the paper stronger. For instance, how would this approach scales to large-resolution images compared to traditional transformer approaches (ViT). 


limitations_and_societal_impact: Authors discuss limitations and societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a glance and gaze module to replace standard self-attention with higher efficiency in terms of memory. The paper validated their methods in image classification, object detection, and semantic segmentation.
main_review: 
1. This paper achieved a better trade-off between FLOPs and accuracy than vanilla ViT.

2. Runtime: Even though the proposed method has the same FLOPs as Swin; however, it seems that it needs more folding/unfolding operations which could slow down the network in runtime. 


Questions

1. See above point 2. What is the runtime speed compared to Swin? 

2. Why do the authors use stronger data augmentation in object detection? in PVT and Swin, for 1x schedule, they do not use multi-scale training. What is the performance if disabling multi-scale training?

3. Why the ablation study does not use the same settings, do those observations still hold if using the same settings?

4. Why the proposed model (Gaze + Glance) is 80.28% in Table 4b while 80.38% in Table 4a?

5. In Table 4c, what is the performance of DeiT without repeated augmentation?

6. Does the proposed method scale better? E.g. from Swin-S to Swin-B, the performance does not improve too much. How about the proposed method, GG-B?

7. It seems that the proposed method achieved better improvement for the smaller model only (GG-T) while the gain over Swin-S is limited. I wonder what is the explanation.


limitations_and_societal_impact: Yes
review_title: null

Meta-review: The paper introduces an efficient transformer architecture with a “glance” branch to model long-range dependencies and a “gaze” branch to account for local context. Three reviewers recommend acceptance, highlighting that the idea is novel, the paper is well-written, and the results are solid. One reviewer, despite appreciating the intuitive appeal of the glance and gaze idea, recommends rejection primarily because of the discrepancy between the Swin baseline numbers being compared with and the actual numbers in the Swin paper referenced in the manuscript. However, the AC agrees with the other three reviewers and the authors based on their response that the comparison with Swin transformers is fair, as the same setting was used to compare both models. The paper is technically sound, has an interesting idea, and the results are promising. The authors should carefully proofread the paper and add the discussion in the rebuttal to the final version.
Decision: Accept (Poster)