Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new ViT architecture by adopting the concepts of pyramid structure and employ regional-to-local attention instead of global self-attention as in standard vision transformers. The regional and local concept is linked to patch size i.e. region consists of larger patches in comparison to local. A regional token is associated a set of non-overlapping local tokens while computing reginal-to-local attention. This attention is computed using a regional self-attention comprising all reginal tokens to extract global information, followed by a local self- attention mechanism that pass information between the reginal token and the linked local tokens via self attention. The approach is evaluated image recognition, object and keypoints detection, semantic segmentation and action recognition. The performance is comparable to the state-of-the-art.
main_review: +ives
The paper is well-written and easy to follow. 
The experimental evaluation on various dataset is very impressive. However, the accuracy improvement is not convincing.
The accompanied source code.

-ives
There is a significant advancement using part-hierarchies (Zheng et al TIP 2019), Attention Pyramid (Ding et al. TIP 2021), attention-driven hierarchical multi-scale (Warton et al., ArXiv 2021, I am aware that this paper is added recently)  and patches are all you need (this is also under review, https://github.com/tmp-iclr/convmixer). It is unclear how the regions and patches are novel. To me, it is how adapted in transformers.

It is unclear how memory saving is 73% when you are computing self-attention among regions followed by local self-attention between regions and local patches.

Is dimension of C for features representing local patches and regional patches same?

The ablation studies is very comprehensive. However, missing ablation study without self-attention between regions (first step of R2L attention)

It is also unclear on the 2nd step of the which one is used as key for multi-head attention is it  local patches? If so what could be the outcome if we use the region as keys.

The overall accuracy of the model is good but not significantly better than the SotA approaches. The accuracy is better for + models in which the window size is 14 and the model has higher GFLOPS and parameters.
summary_of_the_review: The idea of large patches to capture high-level shape, as well as concentrate on detailed texture and parts information using smaller patches is very good. However, the justification could have been improved. The paper is well-written and the experimental evaluation is comprehensive. However, the improvement in accuracy is not significant in comparison to the way the novelty of R2L attention is described. There is an advancement to the transformer model however the impact is limited. The related work is very nicely done.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces local inductive bias into vanilla vision transformer by adopting the pyramid structure and employing regional-to-local attention. The regional-to-local (R2L) attention is the main contribution of the paper. Compared to the vanilla self-attention, R2L has two steps: 1) self-attention on regional tokens; 2) self-attention on local tokens and their corresponding regional token. 
main_review: ### Advantages
1) This paper is well written and easy to follow.
2) The experiments are sufficient to evaluate the proposed method.

### Disadvantage
Since the R2L attention has two steps, I am concerned about the throughput of RegionViT. FLOPs can not directly reflect the throughput. It is better to show the throughput comparison for Table 3.  
summary_of_the_review: The idea sounds direct and reasonable, although introducing local inductive bias into ViT is not and there have been lots of paper to achieve it. The paper is well-written and shows efficient experiments to evaluate the proposed RegionViT.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a new spin on the ViT architecture. Similar to other recent developments, it proposes a pyramidal style architecture where subsequent layers process the input image at decreasing resolutions. This allows the approach to be significantly more memory and compute efficient compared to the original ViT architecture, while still obtaining solid results on several tasks and benchmarks. At the core of the method is a regional-to-local attention approach, where very coarse regional tokens are processed by a normal multi-head attention yielding global attention and each coarse regional patch is additionally represented by a set of smaller local patches, which are also processed together with the corresponding regional token to get finer-grained local attention. Each layer in the network follows this principle and between different sections of the network both the regional and local tokens are downsampled with strided conv layers. 
main_review: **Strengths:**
- Overall the paper is fairly easy to follow and it's easy to understand how the approach works, including a fair amount of detail to potentially reproduce results. The provided code, albeit I did not run it, seems like it should work.
- In many of the presented experiments, RegionViT does indeed obtain on par or state-of-the-art results when compared to similarly sized models
- Quite some relevant and interesting ablations are presented.

**Weaknesses:**
- For me the main weakness of the paper is that it seems fairly incremental in a longer line of ideas. First there was ViT, followed by more efficient training from DeiT including many augmentations, several works then proposed a pyramid-like architectures, and finally this work is a small increment on how to best realize the pyramid architecture including some smaller tweaks. The scores indeed do improve, but sometimes there are quite some different variants (vanilla, +, + w/PEG) that are not consistently listed for the different tasks, partially raising the question how large the margin to the previous models really is. While I don't doubt that the approach has some merit, I would find the paper significantly more valuable if the focus of the experiments would be to dive deeper into how to best design the pyramid structure of the network and less on listing many different tasks with some table showing state of the art results. Now this paper presents yet another new proposal on how to change change the ViT architecture, but I gain little insight into why it really performs better. Shifting the focus to this part of the paper would be more interesting in my opinion. For example it would be interesting to discuss some of the ablation results in more details, or to do a more thorough investigation into the size of the local and global patches. One could even ask the question if two levels (local+regional) are optimal, or an additional super-regional or global attention could also be useful?
- I was a bit surprised by the fairly low results from the ImageNet 21k transfer experiments. Of course it is clear that the FLOP count and parameter count is significantly lower than the better models in Table 4a, however, it does raise the question if RegionViT can indeed be scaled to such an extend that it matches the state of the art. Given that one big part of method is the reduced complexity, it is also not surprising that the model is computationally less heavy, but it would be a drawback if there is some performance ceiling it hits due to the specific architecture. Here it would be good to create some even bigger model to see if the larger models can be matched or outperformed. Also the other transfer experiments in Table 4 are not super convincing.
- The keypoint detection and action recognition sections take up quite some space and especially Table 6 does not list any convincing baselines. At the same time these experiments don't add huge value to the overall story in my opinion. I would recommend to use this space to perform more relevant and interesting ablations/experiments/discussions instead and move these to the supplementary material.
- Lastly, the writing is not the best in some parts of the paper. For a potential final version, I would recommend that the paper is proofread by a native speaker.


**Open questions:**
- Table 1 makes some statements about overlapping vs non-overlapping patches. The way the local tokenization works (using 1 or 3 convs)), I would say it is somewhat of a stretch so say that the patches are strictly non-overlapping.
- I was fairly surprised about weights being shared between RSA and LSA, as well as in the downsampling convs. I don't really see a larger number of parameters as a huge issue for a model and the FLOP count/inference speed should be the same if these weights would not be shared. Actually ablating this would be very interesting. I would kind of expect that the local and regional tokens contain slightly different things and having separate weights could potentially improve the results?
- The paper states that this model saves 73% of memory. I'm wondering if this is an empirical measure, or it was derived based on the complexity analysis? I guess this factor might also be different for the Ti/S/B/L models?
- I find it quite surprising that the regional tokens don't have a huge effect on the overall performance (Ablation 1). Is the remaining strong performance based on the pyramidal structure of RegionViT? Given that the pure local attention never is performed on overlapping areas that is the only reason I would see why the model still remotely works.
summary_of_the_review: Overall the paper proposes a fairly interesting pyramidal modification to ViT. It seems like a somewhat incremental approach though. While the performance is empirically shown, there is little that can really be learned about the underlying approach from the experiments. Nothing is really wrong with the paper, I do think it could be significantly better though. As such I recommend the paper to be accepted, but I do think it has a lot more potential that is not uncovered at the moment and I would actually be very happy to read such an improved version of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new architecture that adopts the pyramid structure and employs a novel regional-to-local attention for vision applications. The regional-to-local attention reduces the memory complexity as compared to standard self-attention. 

main_review: + The performance of this work is promising. 
-  The novelty of this work is somewhat trivial, the issue of the interaction between global and local information has been proposed by previous work, not only in image classification but also in other downstream tasks. And the proposed methods are not novel enough when compare with the previous ones.
- Several grammatical mistakes are made. e.g. In abstract, the regional self-attention extract global information among... (extracts); 

summary_of_the_review: This work aims to solve the interaction between global and local information in a ViT architecture and to save the computation at the same time, the performance is good. However, I am concerned with the novelty of this work and the contribution may be limited.
review_title: null

Meta-review: The paper proposed a new architecture called Regional-to-Local attention for the vision transformers. The idea is easy to understand, the model adopts the pyramid structure and adds a regional to local attention instead of using the global attention. The architecture is well-motivated and the paper is generally well written.

The main concerns from the reviewers are mostly clarification questions. The authors did a good job addressing them. Apart from those, most reviewers raise the novelty issue of such architecture, which I would think is a drawback of this paper.

I am leaning towards the acceptance of this paper mainly because of its experimental results. It is the best in my batch and I think there is a significant improvement over the previous approaches.
Decision: Accept (Poster)