Fast Vision Transformers with HiLo Attention
Authors: Zizheng Pan
Jianfei Cai
Bohan Zhuang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vision Transformers
Image recognition

Abstract: Vision Transformers (ViTs) have triggered the most recent and significant breakthroughs in computer vision. Their efficient designs are mostly guided by the indirect metric of computational complexity, i.e., FLOPs, which however has a clear gap with the direct metric such as throughput. Thus, we propose to use the direct speed evaluation on the target platform as the design principle for efficient ViTs. Particularly, we introduce LITv2, a simple and effective ViT which performs favourably against the existing state-of-the-art methods across a spectrum of different model sizes with faster speed. At the core of LITv2 is a novel self-attention mechanism, which we dub HiLo. HiLo is inspired by the insight that high frequencies in an image capture local fine details and low frequencies focus on global structures, whereas a multi-head self-attention layer neglects the characteristic of different frequencies. Therefore, we propose to disentangle the high/low frequency patterns in an attention layer by separating the heads into two groups, where one group encodes high frequencies via self-attention within each local window, and another group encodes low frequencies by performing global attention between the average-pooled low-frequency keys and values from each window and each query position in the input feature map. Benefiting from the efficient design for both groups, we show that HiLo is superior to the existing attention mechanisms by comprehensively benchmarking FLOPs, speed and memory consumption on GPUs and CPUs. For example, HiLo is 1.4× faster than spatial reduction attention and 1.6× faster than local window attention on CPUs. Powered by HiLo, LITv2 serves as a strong backbone for mainstream vision tasks including image classification, dense detection and segmentation. Code is available at https://github.com/ziplab/LITv2.

Fast Vision Transformers with HiLo Attention
Zizheng Pan
Jianfei Cai
Bohan Zhuang
Department of Data Science & AI, Monash University, Australia
Abstract
Vision Transformers (ViTs) have triggered the most recent and significant break- throughs in computer vision. Their efficient designs are mostly guided by the indirect metric of computational complexity, i.e., FLOPs, which however has clear gap with the direct metric such as throughput. Thus, we propose to use the direct speed evaluation the target platform the design principle for effi- cient ViTs, Particularly, we introduce LITv2, simple and effectivv ViT which performs favourably against the existing state-of-the-ar methods across spec- trum of different model sizes with faster speed. At the core of LITv2 is a novel self-attention mechanism, which we dub HiLo. HiLo is inspired by the insight that high frequencies in an image capture local fine details and low frequencies focus on global structures, whereas multi-head self-attention layer neglects the characteristic of different frequencies. Therefore, we propose to disentangle the high/low frequency patterns in an attention layer by separating the heads into two groups, where group encodes high frequencies via self-attention within each local window, and another group encodes low frequencies by performing global attention between the everage-pooles low-frequency keys and values from each window and each query position the input feature map. Benefiting from the efficient design for both groups, show that HiLo is superior to the existing attention mechanisms by comprehensively benchmarking FLOPs, speed and mem- ory consumption GPUs and CPUs. For example, HiLo is 1.4x faster than spatial reduction attention and 1.6x faster than local window attention on CPUs. Powered HiL.o, LITv2 serves strong backbone for mainstream vision tasks including image classification, dense detection and segmentation. Code is available https::/// / thub. com/ htmmziplabb/IIVV
Introduction
Real-world applications usually require a model to have an optimal speed and accuracy trade-off under limited computational budget, such UAV and autonomous driving. This motivates substantial works toward efficient vision Transformer (VIT) design, such as PVT [51], Swin [32] and Focal Transformer [60], among others. To measure the computational complexity, widely adopted metric in recent ViT design is the number of float-point operations, i.e., FLOPs. However, FLOPs is an indirect metric, which can not directly reflect the real speed on the target platform. For example, Focal-Tiny is much slower than Swin-Ti Ti on GPUs although their FLOPs are comparable.
In general, the discrepancy between the indirect metric (FLOPs) and the direct metric (speed) in recent ViTs can be attributed to two main reasons. First, although self-attention is efficient on low-resolution feature maps, the quadratic complexity in both memory and time makes it much slower on high-resolution images due to intensive memory access cost [34], where fetching data from off-chip DRAM can be ppeed-consuming. Second, some efficient attention mechanisms in ViTs have low theoretical complexity guarantee but are actually slow on GPUs due to particular operations that
CCoresponding author. E-mail: bohan. bowan.zhuanan www..aaaggngmmnss..dde.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(1 a).aA
High Frequency Attention (HI-F))
Scaled
Coreal
Projection
OKV
Produut
Avg Pool
Concat
Projection
Input
Sealed
Concat
Dot/Procuct
Allendon
Low Frequency Attention (Lo-Fi)
Figure 1: Framework of HiLo attention. N₂, refers to the total number of self-attention heads at this layer. a denotes the split ratio for high/low frequency heads. Best viewed in color.
are not hardware-frienee friendly or cannot be parallelized, such as the multi-scale window partition [60], recursion [44] and dilated window [20].
With these observations, in this paper we propose to evaluate ViT by the direct metric, ie., throughput, not only FLOPs. Based on this principle, we introduce LITv2, novel efficient and accurate vision Transformer that outperforms most statc-of-thh-arr (SoTA) ViTs on standard benchmarks while being practically faster on GPUs LITv2 is bulit upon LITv1 [36], a simple ViT bascline which removes all multi-head self-attention layers (MSAs) in the early stages while applying standard MSAs ss in the later stages. Benefit from this design, LITvI is faster than many existing works on ImageNet classification due to no computational cost from the early MSAs while the later MSAs only need to process downsampled low-resoluting feature maps. However, the standard MSA still suffers from huge computational cost on high-resolution images, especially for dense prediction tasks.
To address this problem, we propose novel efficient attention mechanism, termed HiLo. HiLo is motivated by the fact that natural images contain rich frequencies where high/low frequencies play different roles in encoding image patterns, i.e., local fine details and global structures, respectively, A typical MSA layer enforces the same global attention across all image patches without considering the characteristics of different underlying frequencies. This motivates to propose separate an MSA layer into two paths where one path encodes high-frequency interactions via local self-attention with relatively high-resolution feature maps while the other path encodes wwwfrrquency interactions via global attention with down-sampled feature maps, which leads to great efficiency improvement.
Specifically, HiLo employs two efficient attentions to disentangle High/Low frequencies in feature maps. As shown in Figure 1, in the upper path, we allocate a few heads to the high frequency attention (Hi-Fi) to capture fine-gru ane-grained high frequencies by local window self-attention (e.g.. 2 windows), which is much more efficient than standard MSAs. The lower path, implementing the ow-frequency attention (Lo-Fi), first applies average pooling each window to obtain freeuereeeccy signals. Then, we allocate the remaining heads for Lo-Fi to model the relationship between each query position in the input feature map and the perage-ppoless low- cow-frqueecyy keys and values from each window. Benefit from the reduced length of keys and values, Lo-Fi also achieves significant complexity reduction. Finally, we concatenate the refined high/low-frequene subsequent layers. Since both Hi-Fi and Lo-Fi are not equipped with time-consuming operations such as dilated windows and recursion, the overall framework of HiLo is fast on both CPUs and GPUs. We show by comprehensive benchmarks that HiL.o achieves advantage over the existing attention mechanisms in terms of performance, FLOPs, throughput and memory consumption.
features and forward the resulting output into
Besides, we find the fixed relative positional encoding in LITvI dramatically slows down its speed on dense prediction tasks due the interpolation for different image resolutions. For better efficiency, we propose to adopt one 3 x 3 depthwise convolutional layer with zero- prrodpadding in each FFN to incorporate the implicitly learned position information from zero-pase gero-padding [27]. Moreover, the 3 x 3 convolutional filters simultaneously help to enlarge the receptive field of the early multi-layer perceptron (MLP) blocks in LITvl. Finally, we conduct extensive experiments on ImageNet, COCO and ADE20K to evaluate the performance of L.TV2. Comprehensive comparisons with SoTA models show that our architecture achieves competitive performance with faster throughput, making ViTs more feasible to run low-latency applications for real-world scenarios.

---

2
Related Work
Vision Transformers. Vision Transformers are neural networks that adopt self-attention mechanisms into computer vision tasks. In [18], Dosovitskiy et al. propose VIT for image classification, which inherits the similar architecture from a standard Transformer [48] in natural language processing (NLP) tasks. Since then, subsequent works have been proposed to improve VIT by incorporating more convolutional layers [54, 61], introducing pyramid feature mäps [51, 32], enhancing the locality [62], well automatically searching well- well-performed architecture [5, 3] with neural architecture search (NAS). Some others also seek for token pruning to accelerate the inference speed of ViTs [37] or applying VIT into low-level vision tasks [47]. Compared to existing works, this paper focuses general ViT-based backhone for computer vision (CV) tasks and aims to achieve better efficiency on GPUs while maintaining competitive performance.
Efficient attention mechanisms. Efficient attention mechanisms aim to reduce the quadratic com- plexity of standard MSAs. Existing efforts in NLP can be roughly categories into low-rank decom- position [50], kemelization [28, 39], memory [40] and sparsity mechanism [10]. However, simply adopting these method usually performs suboptimally in CV tasks [32, 63]. In CV, representative efficient self-attention mechanisms includes spatial reduction attention (SRA) [51], local window attention [32, 26] and Twins attention [12]. However, they only focus either local global attention at the same layer. To address this problem, TNT [21] introduced additional global tokens and MixFor [6] mixed local window attention with depthwise convolutional layers. Some other attention mechanisms consider both simultancously, such as Focal [60] and QuadTree [44]. However, due to the inefficient operations which are not frrledl and cannot be reflected in FLOPs (e.g., multi-scale window partition, recursion), they are slow on GPUs even compared to standard MSA. To this end, the proposed HiLo attention simultaneously captures rich local-globe information at the same MSA layer and is faster and more memory- effocieen compared to the existing works.
F Frequency domain analysis in vision. The frequeney domain analysis in CV has been well studied in the literature. According to [13, 16], the low frequencies in an image usually capture global structures and color information while the high frequencies contain fine details of objects (e.g., sharp edges). Based on this insight, a plethora of solutions have been proposed for image super-resolution [66, 19), generalization [25], image re-scaling [56] and neural network compression [59, 7J. Furthermore. Octave convolution [9] targeted convolutional layers and proposed to locally applies convolution on high/low-resolltion feature maps, separately. Different from it, the proposed HiLo is novel attention mechanism that captures both local and global relationships with self-attention.
3 Background
Multi-head self-attention. Transformers are built upon multi-head self-attention, which enables to capture long-range relationships for tokens at different positions. Specifically, let X E R be the input sequence into a standard MSA layer, where N is the length of the input sequence and D) refers to the number of hidden dimensions. Each self-attention head calculates the query Q key K and value V matrices with linear transformation from X,
Q XW K XW,, V= XW..
(I)
where Wg. WR. W. e DxD, are learnable parameters and Dj is the number of hidden dimensions for head. Next, the output of a self-attention head is weighted sum over N value vectors,
QK
SA (X) = Softmax(
J
(2)
VD
For an MSA layer with N₁, heads, the final output is computed by a linear projection of the concate- nated outputs from each self- self-attention head, which can be formulated by
MSA(X)
concat (X) NE[Ns]
(3)
where W, E R(NxxDx) is learnable parameter. In practice, D is usually equal to N₂, x Dp- Overall, a standard MSA layer have the computational cost of 42 + 2N2 where 2/2 comes from Eq. (2), 3.N 2 and N 2 comes from Eq. (L) and Eq. (3), respectively.
3

---

Transformer blocks. A standard vision Transformer as described in [18] consists of patch embedding layer, several blocks and a prediction head. Let / be the index of block. Then each block contains an MSA layer and a desition-wise feed-forward network (FFN), which can expressed as
X = X + M))(L)(XX-1))), = + FFN(LL(((-1)),
(4)
(5)
where LN denotes the LayerNorm [2] and an FFN consists of two FC layers with GELU [24] non- linearity in between. Recent works on VIT have proposed to divide the blocks into several stages (typically 4 stages) to generate pyramid feature maps for dense prediction tasks. Furthermore, to reduce the computational cost on high-resolution feature maps in the early stages, the MSA in Eq. (4) has been replaced with efficient alternatives, such as SRA [51] and W-MSA [32].
Bottlenecks of LITv1. Recent studies have shown that the MSA layers in the early stages in a model still focus on local patterns [14]. With the same observation, LITv1 [36] removes all early MSAS (i.e., exclude Eq. (4) in each block) while applying standard MSAs at the later stages. This design principle has achieved better efficiency with competitive performance on ImageNet compared to PVT [51] and Swin [32]. However, LITv1 still has two main bottlenecks in speed: 1) Given a high-resolution image, the standard MSAS in the later stages still result in huge computational cost. 2) The fixed relative positional encoding [32] dramatically slows down the speed when dealing with different image resolutions. This is due interpolating the fixed-size positional encoding for each different image resolution. In the next section, we describe a novel attention mechanism with zero padding positional encoding to comprehensively accelerate LITv1.
4
Method
4.1
HiLo Attention
We propose to separately process high/low frequencies in feature map at an attention layer. We name the new attention mechanism as HiLo, which is depicted in Figure 1. Essentially, the u attention branch (Lo-Fi) is to capture the global dependencies of the input (image/features), which does not need a high-resolution feature map but requires global attention. On the other hand, the high-frequency attention branch (Hi-Fi) is to capture the fine detailed local dependency, which requires high-resolution feature map but can be done via local attention. In the next, we describe the two attentions in detail.
Intuitively, as high frequencies encode local details of objects, it can be redundant and computationally expensive to apply global attention on a feature map. Therefore, we propose to design Hi-Fi to capture fine -na-grained high frequencies with local window self-attention (e.g.. 2 windows), which saves significant computational complexity. Furthermore, we employ the simple von-overiap window partition in Hi-Fi, which is more tordware--riendi compared to the time-consuming operations such as window shifting [32] or multi-scale window partition [60].
attention.
Low-frequency attention. Recent studies have shown that the global attention in MSA helps to capture low frequencies [38]. However, directly applying MSA to high-resolution feature maps requires huge computational cost. As averaging is low-pass filter [49], Lo-Fi firstly applies average pooling to each window get low-frequency signals in the input X. Next, the average-pooled feature maps are projected into keys E TRNN DD. and values E N/ where is the window size. The queries Q in Lo-Fi still comes from the original feature map X. We then apply the standard attention to capture the rich low-frequency information in feature maps. Note that due to the spatial reduction of K and V. Lo-Fi simultancously reduces the complexity for both Eq. (1) and Eq. (2).
Head splitting. A naive solution for head assignment number of heads as the standard MSA layer. However, doubling heads results in more computational cost. In order to achieve better efficiency, HiLo separates the same number of heads in an MSA into two groups with a split ratio o, where (1 a NN, heads will be employed for Hi-Fi and the other NN,, heads are used for Lo-Fi. By doing so, as each attention has a lower complexity than a standard MSA, the entire framework of Hilo guarantees low complexity and ensures high throughput on GPUs. Moreover, another benefit of head splitting is that the learnable parameter W. can be decomposed into two smaller matrices, which helps to reduce model parameters. Finally, the output of HiLo is
allocate both Hi-Fi and Lo-Fi the same

---

3 2
1e9
s 4
1.25
Hi-FI
Hi-Fi
MSA
1.5
2.0
Lo-Fi
1.00
Lo-Fi
HiLo w/ 5-2
HiLo w/ s=4
1.5
0.75
1L0P%
FLOP
1.0
1.0
0.50
0.5
0.5
0.25
0.0
0.00
0.0
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
Resclution (a)
Resolution (b)
Resolution (c)
Figure 2: FLOPs comparison for Hi-Fi and Lo-Fi under different image resolutions and equal number of heads (Figures and b). larger window size helps HiLo achieve better efficiency on high- resolution images (Figure c).
concatenation of the outputs from each attention
HiLo(X) Lo-Fi(());
(6)
where [:] denotes the concatenation operation.
Complexity Analysis. Without loss of generality, we assume Hi-Fi and Lo-Fi have an equal number of heads (i.e., 0.5) and the feature map has equal width and height. Then, Hi-Fi and Lo-Fi have a computational cost of NN D2 + ND and ( + + N2 + + NND, respectively. Derivation for this result can be found in the supplementary material. As shown in Figure 2-(a) and (b), under small input image resolution and a small value of (e.g.. 8 = 2). both Hi-Fi and Lo-Fi are comparably efficient. However, with much higher resolution, Lo-Fi will result in huge computational cost still has quadratic complexity in terms of N in Eq. (2), f.e., 1/2 N2D. In this case, slightly increasing (e-g., $ = 4) helps Lo-Fi achieve better efficiency while preserving the accuracy. Combining the two attentions together, a larger window size also helps the overall framework of HiLo to reduce more FLOPS high-resolution images, shown in Figure 2-(c). Thus, suggest practical guideline for adopting HiLo into existing frameworks: increasing the window size in order to ger better efficiency on high-resolution images. We further show in Section 5.2 that this principle helps LITv2 achieve a better speed and accuracy trade-off on downstream tasks, e.g., dense object detection.
4.2 Positional Encoding
Positional encoding is essential to self-attention due to its permutation-invariint property. LITvl, the later MSAs adopt the same relative positional encoding (RPE) scheme Swin [32]. This approach has significantly improves Swin by 0.7% in Top-1 accuracy on ImageNet compared to using absolute positional encoding [32]. However, on dense prediction tasks, the fixed RPE has to be interpolated for different image resolutions, which dramatically slows down the training/inference speed of LITv As recent study [27] has shown that position information can be implicitly learned from zero- zero-padding prod-pdding in CNNs, we propose to adopt one layer x 3 depthwise convolutional layer with zero-pase pro-ppadiig in each FFN replace the time-consuming RPE. Notably, due to the elimination of early MSAs, the early blocks in LITvl only have FENs left, which results in a tiny receptive field of 1. To this end, we show in Section 5.4 that the 3 x convolutional filters adopted in each FFN also improve LITv2 by simultancously enlarging the receptive field in the early stages.
4.3 Model Architecture
LITv2 has three variants: LITv2-S, LITv2-M and LITv2-B, corresponding to the small, medium and base settings in LITv1, respectively. For a fair comparison, we keep the network width and depth as the same as LITvI. The overall modifications are simply two steps: 1) Adding one layer of depthwise convolution with zero-padding in each FFN and removing all relative positional encodings all MSAs. 2) Replacing all attention layers with the proposed Hio. attention. Detailed architecture configurations can be found in the supplementary material.
5

---

Experiment
In this section we conduct experiments validate the effectiveness of the proposed vITv2. Following common practice [51, 32, 12, 60), we experiment LITv2 on three tasks, including image classification on ImageNet-Il I [43], object detection and instance segmentation on COCO [31] and semantic segmentation on ADE20K [65].
Table 1: Image classification results on ImageNet-IK, By default, the FLOPs, throughput and memory consumption measured based the resolution 224 224. We report the throughput and training/test time memory consumption with a batch size of 64. Throughput is tested on one NVIDIA RTX 3090 GPU and averaged over 30 runs. ResNet results
from "ResNet Stikes Back" [53].
"T 384" means a model is finetuned at the resolution 384 x 384. "OOM" means "out-of-memory"
Param
FLOPs
Throughput
Train Mem
Test Mem
Top-1
Model
(M)
(G)
(imgs/s)
(GB)
(GB)
(%)
ResNet-50 [53]
4.1
1,279
7.9
2.8
80.4
ConvNext-Ti [33]
4.5
1,079
8.3
1.7
82.1
PVT-S [51]
25
3.8
1.007
6.8
1.3
79.8
Swin-Ti [32]
28
4.5
961
6.1
1.5
81.3
CvT-13 [54]
20
4.5
947
6.1
1.5
81.6
Focal-Tiny [60]
29
4.9
384
12.2
3.3
82.2
Twins-PCPVT-S [12]
24
3,8
998
6.8
1.2
81.2
LITv1-S [36]
27
4.1
1,298
5.8
1.2
81.5
LITv2-S
3.7
1,471
5.1
1.2
82.0
ResNet- 101 [53]
45
7.9
722
10.5
3.0
81.5
ConvNext-S [33]
8.7
639
12.3
1.8
83.1
PVT-M [51]
44
6.7
680
9.3
1.5
81.2
Twins-SVT-B [12]
56
8.3
621
9.8
1.9
83.2
Swin-S [32]
50
8.7
582
9.7
1.7
83.0
LITv1-M [36]
48
8.6
638
12.0
1.4
83.0
LITv2-M
49
7.5
812
8,8
1.4
83.3
ResNet- 152 [53]
60
11.6
512
13.4
2.9
82.0
-onvNext-B [33]
89
15.4
469
16.9
2.9
83.8
Liins-SVT-LLBT [12]
99
14.8
440
13.7
3.1
83.7
Swin-B [32]
15.4
386
13.4
2.4
83.3
LITv1-B [36]
15.0
444
16.4
2.1
83.4
LITv2-B
87
13.2
602
12.2
2.1
83.6
DeiT-Bt 384 [45]
55,4
159
39.9
2.5
83.1
Swin-Bt 384 [32]
88
47.1
142
OOM
6.1
84.5
LITv2-Bt 384
87
39.7
198
35.8
4.6
84.7
5.1
Image Classification on ImageNet-1KK
We conduct image classification experiments on ImageNet- 1 [43], a large-scale image dataset which contains 11.M training images and 50K validation images from IK categories. We measure the model performance by Top-1 accuracy. Furthermore, we report the FLOPs, throughput, as well as training/test memory consumption and several representative SoTA ViTs [51, 32, 54, 60, 12]. Note that this paper does not consider mobile-level architectures [8, 35]. Instead, we focus models with the similar model size. Besides, we are also not directly comparable with NAS-based methods [3, 5] as LITv2 is manually designed.
GPUs. We compare with two CNN-based models [53, 33]
Implementation details. All models are trained for 300 epochs from scratch on 8 V100 GPUs. At training time, we the total batch size as 1,024. The input images are resized and randomly cropped into 224 x 224. The initial learning rate is set to 10 and the weight decay is set to 5 x 10 We use AdamW optimizer with cosine decay learning rate scheduler. All training strategies including the data augmentation are same as in LITvI. For Hilo, the window size is set to 2. The split ratio set 0.9, which chosen from a simple grid search on ImageNet- 1K The depthwise convolutional layers in FFNs set with kernel size of 3 x 3, stride of and zero padding size of 1.

---

Table Object detection and instance segmentation performance on the COCO va12017 split using the RetinaNet [30] and Mask R-CNN [22] framework. AP and APM denote the bounding box AP and mask AP, respectively. "*" indicates the model adopts a local window size of 4 in HiLo.
RetinaNet
Mask C-N
Backbone
Params
FLOPs (G)
FPS
AP
Params
FLOPs (G)
FPS
AP
APm
ResNet-50 [23]
38M
239
18.5
36.3
44M
260
27.1
38.0
34.4
PVT-S [51]
34M
273
13.0
40.4
44M
292
16.2
40.4
37.8
Swin-T [32]
38M
251
17.0
41.5
48M
270
21.1
42.2
39.1
Twins-SVT-S [12]
34M
225
15.5
43.0
44M
244
20.4
43.4
40.3
LITv1-S [36]
39M
305
3.3
41.6
48M
324
3.2
42,9
39.6
LITv2-S
38M
242
18.7
44.0
47M
261
18.7
44.9
40.8
LITv2-S#
38M
230
20.4
43.7
47M
249
21.9
44.7
40.7
ResNet-101 [23]
57M
315
15.2
38.5
63M
336
20,9
40.4
36.4
PVT-M [51]
54M
348
10.5
41.9
64M
367
10.8
42.0
39.0
Swin-S [32]
60M
343
13.3
44.5
69M
362
15.8
44.8
40.9
Twins-SVT-B [12]
358
10.8
45.3
76M
377
12.7
45.2
41.5
LITv2-M
59M
348
12.2
46.0
68M
367
12.6
46.8
42.3
LJTv2-M
59M
312
14.8
45.8
68M
315
16.0
46.5
42,0
ResNeXt101-64x44 [58]
96M
473
10.3
41.0
102M
493
12.4
42.8
38.4
P5T
71M
439
9.5
42.6
81M
457
8.3
42.9
39.5
Swin-B [32]
98M
488
11.0
44.7
107M
507
11.3
45.5
41.3
Twins-SVT-LL12) [12]
111M
504
45.7
120M
524
10.1
45.9
41.6
LITv2-B
97M
481
9.5
46.7
106M
500
9.3
47.3
42.6
LITv2-B
97M
430
11.8
46.3
106M
449
11.5
46.8
42.3
Results. In Table 1, we report the experiment results ImageNet- IK. First, compared LITv1 base- lines, LITv2 achieves consistent improvement on Top-1 I accuracy while using less FLOPs. Moreover, benefit from HiLo, LITv2 achieves faster throughput and significant training time memory reduction (e.g., 13%, 27%, 36% inference speedup for the small, medium and base settings, respectively) compared to LITv1. Second, compared to CNNs, LITv2 models outperform all counterparts of ResNet and ConvNext in terms of FI.OPs, throughput and memory consumption while achieving comparable performance. Last, compared to SoTA ViTs, LITv2 surpasses many models in terms of throughput and memory consumption with competitive performance. For example, under the similar amount of FLOPs, LITv2-S achieves faster inference speed than PVT-S and WVPnS-PCPVTT-S with better performance. Although Focal-Tiny achieves better Top-I accuracy than LITv2-S, it runs much slower (i.e., 384 vs. 1,471 images/s) and requires a large amount memory to train. Besides, when finetuning on higher resolution, LITv2-B outperforms both DeiT-B and Swin-B with a faster throughput and lower complexity.
5.2 Object Detection and Instance Segmentation on COCO
In this section, we conduct experiments COCO 2017, common benchmark for object detection and instance segmentation which contains 11K images for the training set and ~5K images for the validation set. Following common practice [12, 51], we experiment with two detection frameworks: RetinaNet [30] and Mask R-CNN [22]. We measure model performance by Average Precision (AP).
Implementation details. All backbones are initialized with pretrained weights ImageNet- IK. We train each model on 8 GPUs with xx schedule (12 epochs) and total batch size of 16. For fair comparison, Note that we pretrain LITv2 with a local window size of 2 and = 0.9 on ImageNet-1K IK. Under the same a, a larger window size helps to achieve lower complexity and thus improves the speed at high resolution, as explained Section I. In this case, we also train models with a slightly larger window size of s 4 for better efficiency, which we denote with "*" By default, FLOPs evaluated based on the input resolution of 1280 x 800. FPS is measured on one RTX 3090 GPU based on the mmdetection [4] framework.
adopt the same training strategy and hyperparameter settings as in LITvl [36].
Results. In Table we report the experimental results on COCO. general, LITv2 outperforms LITv1 by large margin in almost all metrics. Besides, our LITv2 significantly surpasses ResNet terms of AP, though runs slightly slower in some cases. More importantly, our LITv2 beats all the compared SoTA VITs, achieving the best AP with compelling fast inference speed. Furthermore, by adopting a larger window size (i.e., & = 4), LITv2 achieves better efficiency with a slightly performance drop.

---

Table 4: Performance comparisons with other efficient attention mechanisms ViTs based on LITv2-S. We report the Top-1 accuracy on ImageNet- K and mloU on ADE20K.
ImageNet-IK
ADE20K
Method
Params
FLOPs
Throughput (images/s)
Train Mem
Test Mem
Top-1
Params
FLOPs
mioU
(M)
(G)
(GB)
(GB)
(%)
(M)
(G)
(%)
MSA
28
4.1
1,293
6.5
1.2
82.3
32
46.5
43.7
SRA [51]
32
4.0
1,425
5.1
1.3
81.7
35
42.4
42.8
W-MSA [32]
4.0
1,394
1.2
81.9
32
42.7
41.9
T-MSA [12]
30
4.0
1,462
5.0
1.3
81.8
33
42.5
44.0
HiLo
28
3.7
1,471
5.1
1.2
82.0
31
42.6
44.3
Throughpu (imgals)
1500
Train Memiory (GB)
Test Memory (GB)
FLOPs (6)
1000
500
250
350
Resolution
Resolution
Resolution
MSA
SRA
T-MSA
QuadTree HiLo (Ours)
Performer
Figure 3: Comparison with other attention mechanisms based LITv2-S. We report the FLOPs, throughput, and training/test time memory consumption. Evaluations are based on batch size of 64 on one RTX 3090 GPU. The black cross symbol means "out-of-ne -flloff-mmory"..
5.3 Semantic Segmentation on ADE20K
In this section, we evaluate LITv2 on the seman- tic segmentation task. We conduct experiments on ADE20K [65], widely adopted dataset for semantic segmentation which has ~20 train- ing images, ~2K validation images and ~33 test images. Following prior works, we adopt the framework of Semantic FPN [29] and mea- sure the model performance by mloU, We train each model on GPUs with total batch size of 16 with 80K iterations. All backbones are initialized with pretrained weights on ImageNet- IK. The stochastic depth for the small, medium and base models of LIT'2 are 0.2, 0.2 and 0.3, respectively. All other training strategies are the same as in LITv] [36].
Table 3: Semantic segmentation performance of different backbones on the ADE20K validation set. FLOPs is evaluated based on the image resolution of 512 x 512.
Params
FLOPPs
mloU
Backbone
FPS
(M)
(G)
(%)
ResNet-50 [23]
45.4
36.7
PVT-S [51]
28
40
38.7
39.8
Swin-Ti [32]
32
46
39.6
41.5
Twins-SVT-S [12]
34.5
43.2
LITv1-S [36]
46
18.1
41.7
LITv2-S
31
41
42.6
44.3
ResNet- 101 [23]
36.7
38.8
PVT-M [51]
55
29.7
41.6
Swin-S [32]
53
70
24.4
45.2
Results. In Table we compare LITv2 with ResNet and representative ViTs on ADE20K. In general, LITv2 achieves fast speed while outper- forming many SoTA models. For example, our LITv2-S. LITv2-M and LITv2-B surpass Swin- Ti, Swin-S and Swin-B by 2.8%, 0.5% and 1.2% mloU with higher FPS, respectively.
Twins-SVT-B [12]
67
28.0
45.3
LITv2-M
28.5
45.7
PVT-L [51]
65
71
20.5
42.1
Swin-B [32]
107
107
25.5
46.0
Twins-SVT-L. (12)
104
102
25.9
46.7
LITY2-B
90
93
27.5
47.2
5.4 Ablation Study
In this section, we provide ablation studies for LITv2, including the comparison with other efficient attention variants, the effect of a in HiLo, well as the effect of architecture modifications. By default, the throughput and memory consumption are measured on one RTX 3090 GPU with a batch size of 64 under the resolution of 224 224.
Comparing HiLo with other attention mechanisms. Based on LITv2-S, we compare the per- formance of HiLo with other efficient attention mechanisms ImageNet-1KK, including spatial reduction attention (SRA) in PVT [51], shifted-window based attention (W-MSA) in Swin [32] and alternated local and global attention (T-MSA) in Twins [12]. In our implementation, directly

---

Table 5: Effect of architecture modifications based on LITvI-S. "ConvFNN" means we add one layer of
82.0
3.95
Top-1 Accuracy (%)
81.8-
3.90
3 3 depthwise convolutional layer into each FFN.
"RPE" refers to relative positional encoding [32].
3.85
81.6
ImageNet-IK
COCO (RetinaNet)
Name
3.80
FLOPs Mem Top-1
FLOPs
81.4
(G)
(GB)
(%)
(G)
FPS
AP
3.75
LITvI-S [36]
4.1
5.8
81.5
305
3.3
41.6
81.2
3.70
ConvFFN
4.1
6.5
82.5
306
3.1
45.1
0.00
0.25
0.50
0.75
1.00
+ Remove RPE
4.1
6,5
82.3
306
13.3
44.7
alpha
Figure Effect of a based on LITv2-S.
+ HiLo
3.7
5.1
82.0
224
18.7
44.0
Hi-Fi
Lo-Fi
Figure 5: Frequency magnitude (14 X 14) from 8 output channels of Hi-Fi and Lo-Fi in LITv2-B. The magnitude is averaged over 100 samples. The lighter the color, the larger the magnitude. A pixel that is closer to the centre means lower frequency. Visualization code can be found in the supplementary material.
replace HiLo with each compared method. The results are reported in Table In general, HiLo reduces more FLOPs while achieving better performance and faster speed than the compared methods. Furthermore, in Figure 3, we provide comprehensive benchmarks for more attention mechanisms based on different image resolutions, including Focal [60]. Quad'Tree [44] and Performer [11]. Suf- fering from weak parallelizability, they are even slower than that of using standard MSAs on GPUs. Compared to them, HiLo achieves competitive results in terms of the FLOPs, throughput and memory consumption. Morcover, we conduct experiments based on ADE20K and Semantic FPN and show that HiLo achieves more performance gain than other attention mechanisms on the downstream dense prediction task.
Effect of a. As shown in Figure 4, since the complexity of Lo-Fi is lower than Hi-Fi under the resolution of 224 x 224 and the window size 2, a larger a helps to reduce more FLOPs as we allocate more heads to Lo-Fi. Moreover, we found HiLo performs badly with a 0, in which case only the Hi-Fi left and Hil.o only focuses on high frequencies. We speculate that low frequencies play important role in self-attention. For other values of a, find the performance difference is around 0.2%, where a = 0.9 achieves the best performance. However, it is worth noting that although the pure Lo-Fi branch (a 1.0) can achieve competitive results ImageNet- IK, figh-ffrequeny signals play an important role in capturing fine object details, which is particularly important for dense prediction tasks such semantic segmentation. For example, with 0.9. LITv2-S based Semantic FPN achieves more performance gain (+0.6%) than that of using a = 1.0 (43.7%).
Effect of architecture modifications. Based on LITv2-S, we explore the effect of architecture modifications. As shown Table 5, benefit from the enlarged receptive field in the early stages, adoption of depthwise convolutions improves the performance on both ImageNet and COCO. Next, by removing the relative positional encoding, we significantly improve FPS on dense prediction tasks with slightly performance drop both datasets. Also note that since depthwisc convolutions have encoded positional information by zero paddings [27]. the climination RPE does not result in significant performance drop compared to prior works [32]. Finally, benefit from HiLo, we achieve more gains in model efficiency on both ImageNet and COCO.
Spectrum analysis of HiL.o In Figure 5, we visualize the magnitude of frequency component [42] by applying Fast Fourier Transform (FFT) to the output feature maps from Hi-Fi and Lo-Fi attentions, respectively. The visualisation indicates that Hi-Fi captures more high frequencies and Lo-Fi mainly focuses on low frequencies. This strongly aligns with our aim of disentangling high and low frequencies in feature maps at single attention layer.

---

Table Speed and performance comparisons between LITv2-S and other recent ViTs on different
GPUs. All throughput results are averaged over 30 runs with a total batch size of 64 and image resolution 224 224 on one GPU card. We also report the Top-1 accuracy on ImageNet- IK.
Model
Params (M)
FLOPs (G)
A100
V100
RTX 6000
RIX 3090
Top-1 (%)
ResNet-50 [53)
26
4.1
1,424
1.123
877
1,279
80.4
PVT-S [51]
25
3.8
1,460
798
548
1,007
79.8
Twins-PCPVT:S) [12]
24
3.8
1,455
792
529
998
81.2
Swin-Ti [32]
28
4.5
1,564
1,039
710
961
81.3
TNT-S [21]
24
5.2
802
431
298
534
81.3
CvT-13 [54]
20
4.5
1,595
716
379
947
81.6
CoAINet-0 [15]
25
4.2
1.538
962
643
1,151
81.6
CaiT-XS24 [46]
27
5.4
991
484
299
623
81.8
PVTv2-B2 [52]
25
4.0
1,175
670
451
854
82.0
XCIT-S12 [1]
26
4.8
1,727
761
504
1,068
82.0
ConvNext-TI [33]
28
4.5
1,654
762
571
1,079
Focal-Tiny [60]
29
4.9
471
372
261
384
82.2
LITv2-S
28
3.7
1,874
1,304
928
1,471
82.0
Intelit Core 18-10900X CPU @ 3.70GH2
NVIDIA GeForce RTX 3090
5000
Throughpus (imgs/s)
Throughhut (imgsis).
6010
800
4551
600
018
4000
3500
200
225
132
3000
3180)
2314
2012
HiLo
Local Window
SRA
XCA
Unformer
shifted Window
MSA
Crass Window
Per formerr
DAT
Facel
Hortet
QuadTred
VAN
2500
HiLD
SRA
XCA
Linfarmer
Local Window
Shifted Window
MSA
Cross Window
VAN
Hortet
QuadTree
Perfomer
DAY
Focal
Figure 6: Throughput comparisons with more attention mechanisms on CPUs and GPUs based single attention layer and 14x 14 feature maps.
Speed and performance comparisons with more ViTs different GPUs. We compare the inference speed with more models and on more types of GPUs. Table reports the results. It shows that LITv2-S still achieves consistent faster throughput (images/s) than many ViTs NVIDIA A100, Tesla V100, RTX 6000, and RTX 3090. It is also worth noting that under similar performance (82.0%), LITv2-S 2.1 faster than PVTv2- 2-22 [52]. 1.7x faster than XCTT-S12 [1] and ConvNext-T Ti [33], and 3.5x faster than Focal-Timg Tiny [60] on V100, which is another common GPU version for speed test previous works [32. 45, 33, 64].
Throughput comparisons with more attention mechanisms on CPUs and GPUs. In Figure 6, we show that HiLo is consistently faster than many attention mechanisms [51, 32, 10, 11, 17, 55, 60, 41, 44, 20, 18] on both CPUs and GPUs. In particular, under CPU testing, Hil.o 1.4x faster than SRA [51], 1.6x faster than local window attention [32] and 17.4x faster than VAN [20]. Detailed benchmark configurations can be found in the supplementary material.
6 Conclusion and Future Work
In this paper, we have introduced LITv2, novel efficient vision Transformer backbone with fast speed on GPUs and outperforms most SoTA models on ImageNet and downstream tasks. have also presented HiLo attention, the core of LITv2 which helps to achieve better efficiency especially on high-resolution images. With competitive performance, HiLo achieves great advantage over the existing attention mechanisms across FLOPs, throughput and memory consumption. Future work may include incorporating convolutional stem [57] and overlapping patch embedding [52] for better performance, extending HiLo more tasks such speech recognition and video processing.
Limitations and societal impact. HiLo adopts a head splitting ratio assign different numbers of heads into Hi-Fi and Lo-Fi. In our experiments, this ratio is determined by grid search on ImageNet (i.e., a = 0.9). However, different tasks may have different importance on high and low frequencies. Thus, the optimal value of is task-specific and needs to be set manually. Besides, our work potentially brings some negative societal impacts, such as the huge energy consumption and carbon emissions from large-scale training GPU clusters.
10

---

References
A. Ali, H. Touvron, M. Caron, P. Bojanowski. M. Douze, A. Joulin, Laptev, N. Neverova, G. Synnaeve. Verbeek, and Jégou. Xeit: Cross-covariance image transformers. In NIPS, pages 20014-20027, 2021.
[2] J. Ba, J. Kiros, and G. E. Hinton. Layer normalization. ArXiv, abs/1607.06450, 2016.
[3] B. Chen, P. Li, C. Li, B. Li, L. Bai, C. Lin, M. Sun, J. Yan, and W. Ouyang. Glit: Neural architecture search for global and local image transformer. ICCV, 2021.
[4] K. Chen, J. Wang. J. Pang. Y. Cao, Y. Xiong, X. Li S. Sun, W. Feng. Z. Liu, J. Xu, Z. Zhang, D. Cheng.
C. Zhu, Cheng. Q. Zhao, X. Lu, R. Zhu, Wu, Dai, Wang. Shi, W. Ouyang. C. Loy, and
D. Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint X.X::1906.0715..
2019.
[5] M. Chen, H. Peng, J. Fu, and H. Ling. Autoformer: Searching transformers for visual recognition. In ICCV, pages 12250-12260, 2021.
[6] Q. Chen, Q. Wu, J. Wang, Q. Hu, T. Hu, F. Ding, J. Cheng, and J. Wang. Mixformer: Mixing features across windows and dimensions. In CVPR, pages 5239-5249, 2022.
[7] W. Chen, J. T. Wilson, S. Tyree. K. Q. Weinberger. and Y. Chen. Compressing convolutional neural networks in the frequency domain. In KDD, pages 1475 1484, ACM, 2016.
[8] Y. Chen, X. Dai, D. Chen, M. Liu, X. Dong, L. Yuan. and Z. Liu. Mobile-former: Bridging mobilenet and transfommer. In CVPR, pages 5260-5269, 2022.
[9] Y. Chen, H. Fan, B. Xu, Z. Yan, Y. Kalantidis, M. Rohrbach, S. Yan, and J. Feng. Drop octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In ICCV, pages 3434-3443. IEFF, 2019.
[10] R. Child, S. Gray, A. Radford, and Sutskever. Generating long sequences with sparse transformers. arXiv preprint arxiv: 1904. 10509, 2019.
[11] M. Choromanski, V. Likhosherstov, Dohan, Song, A. Gane, Sarlós, Hawkins. J. Q. Davis. A. Mohiuddin, L. Kaiser, D. B. Belanger, L. Colwell, and A. Weller. Rethinking attention with performers. In ICLR, 2021.
[12] X. Chu, Z. Tian, Y. Wang, B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurlPS, pages 9355-9999 13666, 2021.
[13] J. W. Cooley, P. A. W. Lewis, and P. D. Welch. The fast fourier transform and its applications. JEEE Transactions on Education, 12(1):27-34, 1969.
[14] J. Cordonnier, A. Loukas, and M. Jaggi. On the relationship between self oeff-attention and convolutional layers. ICLR, 2020.
[15] Z. Dai, H. Liu. Q. V. Le, and M. Tan. Coatnet: Marrying convolution and attention for all data sizes. In NeurIPS, pages 3965-3977, 2021.
[16] G. Deng and I.. Cahill, An adaptive gaussian filter for noise reduction and edge detection. In 1993 IEEE Conference Record Nuclear Science Symposium and Medical Imaging Conference, pages 1615-1619 vol.3, 1993.
[17] X. Dong. J. Bao, D. Chen, W. Zhang, N. Yuan, D. Chen, Guo. Cswin transformer: A general vision transformer backbone with cross cross-shaped windows. In CVPR, pages 12114-12124 2022.
[18] A. Dosovitskiy, L. Beyer, A. Kolesnikov. D. Weissenbor,, X. Zhai, T. Unterthiner. M. Dehghani, Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[19] M. Fritsche, S. Gu, and R. Timofte. Frequency separation for real-world super-resolution. ICCVW, Oct 2019.
arXiv:2202.0977 2022. [20] M.-H. Guo, C.-Z. Lu, Z.-N. Liu, M.-M. Cheng, and S.-M. Hu. Visual attention network. arXiv preprint
[21] K. Han. A. Xiao, E. Wu. J. Guo, C. Xu, and Y. Wang. Transformer in transformer. In NewrlPS, pages 15908 15919, 2021.

---

[22] K. He. G. Gkioxari, P. Dollár, and R. B. Girshick. Mask R-CNN. In ICCV, pages 2980-2988, 2017.
[23] K. He, X. Zhang. S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[24] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08115. 2016.
[25] J. Huang, D. Guan, A. Xiao, and S. Lu. Rda: Robust domain adaptation via fourier adversarial attacking. In ICCV, pages 8988-8999, 2021.
[26] Z. Huang, Y. Ben, G. Luo, P. Cheng, G. Yu, and B. Fu. Shuffle transformer: Rethinking spatial shuffle for vision transtormer. arXiv preprint arXiv:2106,0 S0k:::2006006500, 2021.
[27] M. A. Islam, S. Jia, and N. D. B. Bruce. How much position information do convolutional neural networks encode? In ICLR, 2020.
[28] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are mns: Fast autoregressive transformers with lincar attention. In ICML, pages 5156-5165, PMLR, 2020.
[29] Kirillov, B. Ginshick, He, and P. Dollár. Panoptic feature pyramid networks. CVPR, pages 6399-6408, 2019.
[30] T. Goyal, Girshick, He, and P. Dollár. Focal loss for dense object detection. In ICCV, pages 19999-007) 2017.
[31] Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: common objects in context. In Fleet, Pajdla, B. Schiele, and T. Tuytelaars, editors, ECCV, pages 740-755, 2014.
[32] Z. Liu, Lin, Y. Cao, H. Hu, Wei, Zhang, S. Lin, and Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 9992-10002, 2021.
[33] Z. Liu, H. Mao. C.-Y. Wu, C. Feichtenhofer. T. Darrell, and S. Xie. A convnet for the 2020s. In CVPR, pages 11966-131 11966, 2022.
[34] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, pages 116-131, 2018.
[35] S. Mchta and M. Rastegari. Mobilevit: Light-weight, general- purpose, and mobile-friendly vision trans- former. In ICLR, 2022.
[36] 7. Pan, B. Zhuang. H. He, J. Liu, and J. Cai. Less is more: Pay less attention in vision transformers. In AAAI, pages 2035-20020 2033-2043, 2022.
[37] Z. Pan, B. Zhuang, J. Liu, H. He, and Cai. Scalable visual transformers with hierarchical pooling. In ICCV, pages 377-386, 2021.
[38] N. Park
Kim. How do vision transformers work? In ICLR, 2022.
[39] H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. A. Smith, and L., Kong. Random feature attention. In ICLR, 2021.
[40] J. W. Rae, A. Potapenko, S. M. Jayakumar, and T. P. Lillicrap. Compressive transformers for long-range sequence modelling. In ICLR, 2020.
[41] Y. Rao, W. Zhao, Y. Tang, J. Zhou, S.-L. Lim, and J. Lu. Horet: Efficient high-order spatial interactions with recursive gated convolutions. In NewrIPS, 2022.
[42] Y. Rao, W. Zhao, Z. Zhu, J. Lu, and J. Zhou. Global filter networks for image classification. In NeurlPS, pages 980-993, 2021.
[43] O. Russakovsky, Deng, Su, Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bersstein, et al. Imagenet large scale visual recognition challenge. IJCV, pages 211-252, 2015.
[44] S. Tang. J. Zhang, Zhu, and Tan. Quadtree attention for vision transformers. ICLR, 2022.
[45] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In ICML, pages 10347-10357, 2021.
[46] H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou. Going deeper with image transformers. In ICCV, pages 32 42. IEEE, 2021.
12

---

[47] Z. Tu, H. Talebi. H. Zhang. F. Yang. P. Milanfar, A. Bovik, and Y. Li. Maxim: Multi-axis mlp for image processing. CVPR. pages 5759-5770, 2022.
[48] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I., Jones, A. N. Gomez, L., Kaiser, and I. Polosukhin. Attention is all you need. In NeurlPS, pages 5998- 6008- 2017.
[49] E. Voigtman and J. D. Winefordner. Low-pass filters for signal averaging. Review of Scientific Instruments, 57(5):957-966, 1986.
[50] S. Wang, B. Li, M. Khabsa, H. Fang, and H. Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
[51] W. Wang, E. Xie, X. Li, D.-. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyramid vision transformer: versatile backhone for dense prediction without convolutions. In ICCV, pages 548-558, 2021.
[52] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pvtv2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):1-10, 2022.
[53] R. Wightman, H. Touvron, and H. Jégou. Resnet strikes back: An improved training procedure in timm. CoRR, abs/2110.00476, 2021.
[54] H. Wu, Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang. Cvt: Introducing convolutions vision transformers. In ICCV, pages 22-31, 2021.
[55] Z. Xia, X. Pan, S. Song, L. E. Li, and G. Huang. Vision transformer with deformable attention. In CVPR, pages 1794-4803. 2022.
[56] M. Xiao, S. Zheng, C. Liu, Y. Wang, D. He, G. Ke, J. Bian, Z. Lin, and Liu. Invertible image rescaling. In ECCV, pages 126-144, 2020.
[57] T. Xiao, M. Singh, Mintun, T. Darrell Dollár, and Girshick. Early convolutions help transformers see better. NeurlPS, pages 30392-30400, 2021.
[58] S. Xic, R. B. Girshick, P. Dollár, 7. Tu, and K. He. Aggregated residual transformations for deep neural networks. CVPR, pages 5987-5995, 2017.
[59] K. Xu, M. Qin, F. Sun, Y. Wang, Y. Chen, and Ren. Learning the frequency domain. In CVPR, pages 1737-1746. Computer Vision Foundation IEEE, 2020.
[60] J. Yang, C. Li, Zhang, X. Dai, B. Xiao, L. Yuan, and Gao. Focal self-attention for local-global interactions in vision transformers. NeurlPS, pages 30008-30022, 2021.
[61] Yuan, Guo, Z. Liu, Zhou, F. Yu, and W. Wu. Incorporating convolution designs into visual transformers. ICCV, pages 359-568, 2021.
[62] L. Yuan, Y. Chen, Wang, Yu, Y. Shi, F. E. Tay, J. Feng, and S. Yan. tokene-to-token vit: Training vision transformers from scratch on imagenet. ICCV, pages 538-53 047,557742 2021.
[63] P. Zhang, X. Dai, Yang. B. Xiao, L. Yuan, L. Zhang, and Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In ICCV, pages 2978-2988, 2021.
[64] Q Zhang, Y. Xu, J. Zhang, and Tao. Vsa: Leaming varied-sizz size window attention vision transformers. ECCV, 2022.
[65] B. Zhou. Zhao, Puig, Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ADE20K dataset. IJCV, pages 302-321, 2019.
[66] Y. Zhou, W. Deng, T. Tong, and Q. Gao. Guided frequency separation network for real-world super- resolution, In CVPRW, pages 173111331,2 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] Code and pretrained models are included as a URL in the abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No] We use the same random seed as in recent works for fair comparison.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 5.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 5.
(b) Did you mention the license of the assets? [No] The license of the public datasets can be found in their websites.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] Code and pretrained models included URL in the abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [No] We use public datasets (e.g., ImageNet [43] and ADE20K [65]). (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] No such concerns as we are using widely adopted public datasets.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "wZipyx_eCfk", "reviewer": "Reviewer_xTew", "summary": "The paper addresses efficient Vision Transformers (ViTs) design. The paper argues that while previous works on designing efficient ViTs have considered the theoretical asymptotic computational complexity and computational complexity measured in floating point operations (FLOPS) and memory, those metrics do not capture the actual running time and throughput. Specifically, the paper argues that previous methods might require low number of FLOPs (or lower asymptotic complexity), but in practise their implementation is not hardware friendly thus slow when running on GPU. The paper proposes to benchmark FLOPS, memory consumption and actual running time (on GPU) and further proposes a ViT design that performs favourably in those metrics while providing high accuracy when used as a backbone in various vision tasks, namely: image classification, object detection, instance segmentation and semantic segmentation. \n\nThe proposed ViT architecture is based on separating the MultiHead Self Attention (MSA) heads into 2 groups - one group performs local window self attention to capture local fine grained details characterised by high frequencies and the second group performs global self attention on a downscaled (in practice - average pooling in each high res window) version of the feature map to capture global structures characterised by low frequencies. The total number of MSA heads are divided between the groups such that 1-alpha of the heads belong to the first group (local windowed self attention on the full resolution feature map) and alpha of the heads belong to the second group (global attention on the downscaled feature map). Their method is thus dubbed HiLo to denote the different attention branches working on High and Low frequencies. Regarding the value of alpha - the authors provide an experiment to measuring the effect of different choices of alpha and when measuring on the various benchmarks alpha is set to 0.9, so in practice 10% and 90% of the MSA heads belong to the high and low frequencies branch, respectively. Note that in the low frequency brach, keys and values are computed on the downscaled feature map, but the queries still come from the high frequency branch. Also, to further speed up the method, the authors replace explicit positional encoding by adding a layer of 3x3 depth-wise convolution in each Feed Forward block. \n\nFinally, to demonstrate the effectiveness of their approach, the authors compare their methods to other ViT architecture in classification on ImageNet 1K as well as when using their architecture as a backbone (weights are initialised from the ImageNet trained model) in object detection and instance segmentation (measured on COCO) and semantic segmentation (measured on ADE20K). The experiments demonstrate relatively high speed, low number of FLOPS and high accuracy of the proposed method compared to other ViT architectures and efficient attention mechanisms. \n", "questions": "1. Suggestion - I would suggest to compare the proposed method against more recent ViT architectures. \n2. Question - In the low frequency branch, why do the queries (Q matrix) still computed from the high resolution feature map? did you try to completely separate the two branches such that the queries will also be computed from the pooled feature map, in addition to K and V? ", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. The paper highlights an important limitation of previous methods - that using previously proposed metrics such as memory footprint and number of FLOPs (and theoretic asymptotic complexity) are only proxies to the metric of running time (or throughput) and in practice methods that perform favourably in those metrics (FLOPs, memory) might actually have slow running time due to not being \"hardware friendly\". To the best of my knowledge, very few papers report throughput (an exception is the Swin paper) and addressing this limitation by providing an architecture design with\"hardware friendliness\" in mind is an important contribution of the paper. \n2. The proposed method is simple and tackles efficient ViT design from a viewpoint that has not been considered before. The architecture separates handling of local and global information is simple, yet novel and sound.\n3. The paper is well written and easy to follow. \n4. The paper provide extensive experiments and visualisations. \n\nWeaknesses:\n1. In terms of technical contribution - while the proposed architecture is new, all the ideas presented were previously explored. For example, computing local windowed self attention have been previously used in Swin and Focal transformer (in different ways). Computing global self attention on pooled versions of the feature map have been previously explored in PVT and Focal transformer (this is also true for the removal of positional encoding and replacing with depth-wise convolutions). The paper does combine those ideas in a way that was not previously suggested, but in my opinion this doesn't suffice for a Neurips paper (in terms of technical novelty). \n2. Regarding the experiments - the methods compared against the proposed architecture are relatively old and are missing several strong recent papers, for example: XCiT, CaiT and CoAtNet to name a few. While HiLo performs favourably agains older methods, more recent works (as mentioned above) report higher accuracy than HiLo. Of course, their speed might be lower, but this needs to be tested and compared against HiLo to support the authors claims. \n\n**Edit**: The rebuttal addresses my main concerns and the weaknesses I described above. I accept the authors claims that the proposed design principals have not been considered before and specially IMO designing an efficient architecture with high throughput is a significant contribution. Also, the experiments do demonstrate that the proposed method is on par with previous contributions (in my comment I was addressing performance when fine-tuning on 324x324) while being more efficient. I have changed the final rating accordingly, I would like to thank the authors again for the detailed rebuttal.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ceDxNLbuWa3", "reviewer": "Reviewer_Rjxy", "summary": "This work introduces HiLo Attention to vision transformers on top of LIT[1] for 2D images. The proposed HiLo is composed of High-frequency attention (Hi-Fi) and Low-frequency attention (Lo-Fi). By splitting two branches in every transformer block and combining the output of the two branches, these two attention modules process the high-frequency information and low-frequency information in the image. This work achieves competitive results in Image Classification on ImageNet-1K, Object Detection and Instance Segmentation on COCO, and Semantic Segmentation on ADE20K. The work also shows the visualization of low-frequency features and high-frequency features.\n\n\n\n\n[1] Pan, Zizheng, et al. \"Less is more: Pay less attention in vision transformers.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 2. 2022.", "questions": "1. What is the strict definition of low-frequency/high-frequency attention? How do you define the relationship between high/low attentions with local/global attentions\n\n2. How do you generate the figure 5 ? (visualization of high/low feature)\n\n3. [Minor] How do you compare your method and results with C-SWin[1]?\n\n\n\n\n[1] Dong, Xiaoyi, et al. \"Cswin transformer: A general vision transformer backbone with cross-shaped windows.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n", "limitations": "None", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This work addresses the problem of modeling local and global features in vision transformers, which is good motivation from the perspective of Fourier transform.\n\n2. This work achieves state-of-the-art performance on different tasks e.g. Image Classification on ImageNet-1K, Object Detection and Instance Segmentation on COCO, and Semantic Segmentation on ADE20K.\n\n3. Many implementation details are mentioned in the paper e.g. positional encoding, and low-FLOP models not always being fast models\n\nWeaknesses:\n\n1. There is little discussion or proof to show the connection between low/high frequency and local/global features. The visualization of low/high frequency feature (fig 5) is confusing for reading to understand.\n\n2. Looking at the Figure 4, the best model with alpha=0.8 is only roughly higher than the model with alpha=1.0. In the case of alpha=1.0, there is no global attention (or low-frequency feature) is involved. This ablation study looks very confusing. Does it mean global attention (or low-frequency attention) is useless?\n\n3. [Minor concern] Although comparing FLOPs is not a fair speed comparison, the throughput of a certain kind of GPU (RTX 3090) doesn't lead to a strong conclusion. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "b-NSXlEZCBi", "reviewer": "Reviewer_zE7q", "summary": "This paper studies the efficient Vision Transformers. The authors proposed the HiLo attention which combines the window-based self-attention and spatial reduction self-attention. The experimental results on serval datasets demonstrate the effectiveness of the proposed method.\n", "questions": "Please refer to the Questions in the Weaknesses.\n\nHow do you design the number of multi-head in attention when splitting the channels? Because the split features may not be divisible by the head_dim.\n", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The proposed HiLo attention is somehow reasonable.\n+ Compared to the baselines, the proposed methods could bring constant improvements \n+ The abundant experiments are introduced to prove the effectiveness of the proposed method.\n\nThe main concerns are listed below. \n- The core contribution is HiLo attention, however, as shown in Table 5, Compared to adding ConvFFN, the improvements HiLo attention brings are minor. \n- In Figure 4, the HiLo attention is equivalent to spatial reduction attention (SRA) when the alpha=1.0. SRA could achieve ~81.9 Top-1 Accuracy on ImageNet with 3.7G FLOPs. Table 4 shows that HiLo (alpha=0.9) attention achieves 82.0 with 3.7G FLOPs. The improvement is just 0.1%, which is too weak.\n- some references are missing.\n For window-based self-attention, there are some methods [a][b][c] are not included.\n To study high/low frequencies in images, Octave Convolution[d] is heavily related to the proposed method.\n\n[a] MixFormer: Mixing Features across Windows and Dimensions, CVPR 2022.\n\n[b] Transformer in Transformer, NeurIPS 2021.\n\n[c] Rethinking Spatial Shuffle for Vision Transformer, arXiv 2021.\n\n[d] Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution, ICCV 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HJG86cW29Nj", "reviewer": "Reviewer_VgYF", "summary": "The paper proposes a novel vision Transformer, LITv2, which directly targets the speed on GPUs, instead of theoretical FLOPs. To achieve this, the authors improve the efficiency of ViT by targeting the architecture design, attention mechanism and positional encoding. Specifically, they first adopt the same design principle of LIT [29] to get rid of the early MSAs, then propose a novel efficient self-attention mechanism, HiLo, which disentangles the high/low frequency patterns in an attention layer with two groups of heads, where one group captures high frequencies via local window self-attention and another group of heads model the global relationship between the average-pooled low-frequency keys from each window and each query position in the input feature map. Comprehensive benchmarking has shown the advantage of the proposed HiLo over other attention schemes. Furthermore, the authors also propose to replace the time-consuming relative positional encoding (RPE) with zero-padding positional information from convolutions for further speedup on dense prediction tasks. Extensive experiments on both ImageNet and downstream tasks have shown that LITv2 achieves a better speed-accuracy trade-off than previous SoTA ViTs. ", "questions": "See the weakness.", "limitations": "Yes.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n1. The problem that the paper tackles for the current ViT design is significant. As most recent works focus only on theoretical FLOPs, the direct speed metric (e.g. throughput) on hardware is more important and informative for the community, especially for the efficient design of self-attention.\n\n2. The proposed HiLo attention is well-motivated. The idea of disentangling different frequencies is technically sound and has been well validated by visualisations. The complexity analysis of HiLo is clear and the comparison with other efficient attention mechanisms is also comprehensive.\n\n3. Simple architecture but works. The proposed model is backed by impressive results on both ImageNet and downstream tasks. The experiments are quite strong and the comparisons with other methods are comprehensive. Surprisingly, the proposed model achieves faster speed and uses less memory footprint than representative CNNs, with competitive performance on ImageNet. This is a significant step for general ViT backbones.\n\n4. The structure of this paper is well-written and easy to follow. Nice figures, which clearly present the idea and claims.\n\n### Weakness\n\n1. The authors have indicated that by training a slightly larger window size the model can achieve better efficiency. However, it could be difficult to determine the optimal window size on dense prediction tasks. How would the speed-accuracy trade-off change if directly evaluating a trained model (e.g. trained with window size 2) with a different window size? \n\n2. Swin Transformer also adopts RPE, but it is not slow on GPUs. So why removing RPE can significantly improve the speed of LITv1 on COCO?\n\n3. It would be better if the COCO results based on LITv2-B can be moved into the main manuscript.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept