Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the efficient Vision Transformers. The authors proposed the HiLo attention which combines the window-based self-attention and spatial reduction self-attention. The experimental results on serval datasets demonstrate the effectiveness of the proposed method.

strengths_and_weaknesses: + The proposed HiLo attention is somehow reasonable.
+ Compared to the baselines, the proposed methods could bring constant improvements 
+ The abundant experiments are introduced to prove the effectiveness of the proposed method.

The main concerns are listed below. 
- The core contribution is HiLo attention, however, as shown in Table 5, Compared to adding ConvFFN, the improvements HiLo attention brings are minor. 
- In Figure 4, the HiLo attention is equivalent to spatial reduction attention (SRA) when the alpha=1.0. SRA could achieve ~81.9 Top-1 Accuracy on ImageNet with 3.7G FLOPs. Table 4 shows that HiLo (alpha=0.9) attention achieves 82.0 with 3.7G FLOPs. The improvement is just 0.1%, which is too weak.
- some references are missing.
 For window-based self-attention, there are some methods [a][b][c] are not included.
 To study high/low frequencies in images, Octave Convolution[d] is heavily related to the proposed method.

[a] MixFormer: Mixing Features across Windows and Dimensions, CVPR 2022.

[b] Transformer in Transformer, NeurIPS 2021.

[c] Rethinking Spatial Shuffle for Vision Transformer, arXiv 2021.

[d] Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution, ICCV 2019.
questions: Please refer to the Questions in the Weaknesses.

How do you design the number of multi-head in attention when splitting the channels? Because the split features may not be divisible by the head_dim.

limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses efficient Vision Transformers (ViTs) design. The paper argues that while previous works on designing efficient ViTs have considered the theoretical asymptotic computational complexity and computational complexity measured in floating point operations (FLOPS) and memory, those metrics do not capture the actual running time and throughput. Specifically, the paper argues that previous methods might require low number of FLOPs (or lower asymptotic complexity), but in practise their implementation is not hardware friendly thus slow when running on GPU. The paper proposes to benchmark FLOPS, memory consumption and actual running time (on GPU) and further proposes a ViT design that performs favourably in those metrics while providing high accuracy when used as a backbone in various vision tasks, namely: image classification, object detection, instance segmentation and semantic segmentation. 

The proposed ViT architecture is based on separating the MultiHead Self Attention (MSA) heads into 2 groups - one group performs local window self attention to capture local fine grained details characterised by high frequencies and the second group performs global self attention on a downscaled (in practice - average pooling in each high res window) version of the feature map to capture global structures characterised by low frequencies. The total number of MSA heads are divided between the groups such that 1-alpha of the heads belong to the first group (local windowed self attention on the full resolution feature map) and alpha of the heads belong to the second group (global attention on the downscaled feature map). Their method is thus dubbed HiLo to denote the different attention branches working on High and Low frequencies. Regarding the value of alpha - the authors provide an experiment to measuring the effect of different choices of alpha and when measuring on the various benchmarks alpha is set to 0.9, so in practice 10% and 90% of the MSA heads belong to the high and low frequencies branch, respectively. Note that in the low frequency brach, keys and values are computed on the downscaled feature map, but the queries still come from the high frequency branch. Also, to further speed up the method, the authors replace explicit positional encoding by adding a layer of 3x3 depth-wise convolution in each Feed Forward block. 

Finally, to demonstrate the effectiveness of their approach, the authors compare their methods to other ViT architecture in classification on ImageNet 1K as well as when using their architecture as a backbone (weights are initialised from the ImageNet trained model) in object detection and instance segmentation (measured on COCO) and semantic segmentation (measured on ADE20K). The experiments demonstrate relatively high speed, low number of FLOPS and high accuracy of the proposed method compared to other ViT architectures and efficient attention mechanisms. 

strengths_and_weaknesses: Strength: 
1. The paper highlights an important limitation of previous methods - that using previously proposed metrics such as memory footprint and number of FLOPs (and theoretic asymptotic complexity) are only proxies to the metric of running time (or throughput) and in practice methods that perform favourably in those metrics (FLOPs, memory) might actually have slow running time due to not being "hardware friendly". To the best of my knowledge, very few papers report throughput (an exception is the Swin paper) and addressing this limitation by providing an architecture design with"hardware friendliness" in mind is an important contribution of the paper. 
2. The proposed method is simple and tackles efficient ViT design from a viewpoint that has not been considered before. The architecture separates handling of local and global information is simple, yet novel and sound.
3. The paper is well written and easy to follow. 
4. The paper provide extensive experiments and visualisations. 

Weaknesses:
1. In terms of technical contribution - while the proposed architecture is new, all the ideas presented were previously explored. For example, computing local windowed self attention have been previously used in Swin and Focal transformer (in different ways). Computing global self attention on pooled versions of the feature map have been previously explored in PVT and Focal transformer (this is also true for the removal of positional encoding and replacing with depth-wise convolutions). The paper does combine those ideas in a way that was not previously suggested, but in my opinion this doesn't suffice for a Neurips paper (in terms of technical novelty). 
2. Regarding the experiments - the methods compared against the proposed architecture are relatively old and are missing several strong recent papers, for example: XCiT, CaiT and CoAtNet to name a few. While HiLo performs favourably agains older methods, more recent works (as mentioned above) report higher accuracy than HiLo. Of course, their speed might be lower, but this needs to be tested and compared against HiLo to support the authors claims. 

**Edit**: The rebuttal addresses my main concerns and the weaknesses I described above. I accept the authors claims that the proposed design principals have not been considered before and specially IMO designing an efficient architecture with high throughput is a significant contribution. Also, the experiments do demonstrate that the proposed method is on par with previous contributions (in my comment I was addressing performance when fine-tuning on 324x324) while being more efficient. I have changed the final rating accordingly, I would like to thank the authors again for the detailed rebuttal.
questions: 1. Suggestion - I would suggest to compare the proposed method against more recent ViT architectures. 
2. Question - In the low frequency branch, why do the queries (Q matrix) still computed from the high resolution feature map? did you try to completely separate the two branches such that the queries will also be computed from the pooled feature map, in addition to K and V? 
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a novel vision Transformer, LITv2, which directly targets the speed on GPUs, instead of theoretical FLOPs. To achieve this, the authors improve the efficiency of ViT by targeting the architecture design, attention mechanism and positional encoding. Specifically, they first adopt the same design principle of LIT [29] to get rid of the early MSAs, then propose a novel efficient self-attention mechanism, HiLo, which disentangles the high/low frequency patterns in an attention layer with two groups of heads, where one group captures high frequencies via local window self-attention and another group of heads model the global relationship between the average-pooled low-frequency keys from each window and each query position in the input feature map. Comprehensive benchmarking has shown the advantage of the proposed HiLo over other attention schemes. Furthermore, the authors also propose to replace the time-consuming relative positional encoding (RPE) with zero-padding positional information from convolutions for further speedup on dense prediction tasks. Extensive experiments on both ImageNet and downstream tasks have shown that LITv2 achieves a better speed-accuracy trade-off than previous SoTA ViTs. 
strengths_and_weaknesses: ### Strength

1. The problem that the paper tackles for the current ViT design is significant. As most recent works focus only on theoretical FLOPs, the direct speed metric (e.g. throughput) on hardware is more important and informative for the community, especially for the efficient design of self-attention.

2. The proposed HiLo attention is well-motivated. The idea of disentangling different frequencies is technically sound and has been well validated by visualisations. The complexity analysis of HiLo is clear and the comparison with other efficient attention mechanisms is also comprehensive.

3. Simple architecture but works. The proposed model is backed by impressive results on both ImageNet and downstream tasks. The experiments are quite strong and the comparisons with other methods are comprehensive. Surprisingly, the proposed model achieves faster speed and uses less memory footprint than representative CNNs, with competitive performance on ImageNet. This is a significant step for general ViT backbones.

4. The structure of this paper is well-written and easy to follow. Nice figures, which clearly present the idea and claims.

### Weakness

1. The authors have indicated that by training a slightly larger window size the model can achieve better efficiency. However, it could be difficult to determine the optimal window size on dense prediction tasks. How would the speed-accuracy trade-off change if directly evaluating a trained model (e.g. trained with window size 2) with a different window size? 

2. Swin Transformer also adopts RPE, but it is not slow on GPUs. So why removing RPE can significantly improve the speed of LITv1 on COCO?

3. It would be better if the COCO results based on LITv2-B can be moved into the main manuscript.
questions: See the weakness.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work introduces HiLo Attention to vision transformers on top of LIT[1] for 2D images. The proposed HiLo is composed of High-frequency attention (Hi-Fi) and Low-frequency attention (Lo-Fi). By splitting two branches in every transformer block and combining the output of the two branches, these two attention modules process the high-frequency information and low-frequency information in the image. This work achieves competitive results in Image Classification on ImageNet-1K, Object Detection and Instance Segmentation on COCO, and Semantic Segmentation on ADE20K. The work also shows the visualization of low-frequency features and high-frequency features.




[1] Pan, Zizheng, et al. "Less is more: Pay less attention in vision transformers." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 2. 2022.
strengths_and_weaknesses: Strengths:

1. This work addresses the problem of modeling local and global features in vision transformers, which is good motivation from the perspective of Fourier transform.

2. This work achieves state-of-the-art performance on different tasks e.g. Image Classification on ImageNet-1K, Object Detection and Instance Segmentation on COCO, and Semantic Segmentation on ADE20K.

3. Many implementation details are mentioned in the paper e.g. positional encoding, and low-FLOP models not always being fast models

Weaknesses:

1. There is little discussion or proof to show the connection between low/high frequency and local/global features. The visualization of low/high frequency feature (fig 5) is confusing for reading to understand.

2. Looking at the Figure 4, the best model with alpha=0.8 is only roughly higher than the model with alpha=1.0. In the case of alpha=1.0, there is no global attention (or low-frequency feature) is involved. This ablation study looks very confusing. Does it mean global attention (or low-frequency attention) is useless?

3. [Minor concern] Although comparing FLOPs is not a fair speed comparison, the throughput of a certain kind of GPU (RTX 3090) doesn't lead to a strong conclusion. 
questions: 1. What is the strict definition of low-frequency/high-frequency attention? How do you define the relationship between high/low attentions with local/global attentions

2. How do you generate the figure 5 ? (visualization of high/low feature)

3. [Minor] How do you compare your method and results with C-SWin[1]?




[1] Dong, Xiaoyi, et al. "Cswin transformer: A general vision transformer backbone with cross-shaped windows." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

limitations: None
review_title: null

Meta-review: metareview: Initially, this paper received diverging reviews. The authors did a good job addressing the reviewers' concerns, by adding additional comparisons to more SOTA ViT backbones and benchmarking throughput on a variety of GPU platforms. The AC agrees with the reviewer that the concerns have been sufficiently addressed and recommends acceptance.
confidence: Certain
Decision: Accept