FasterViT: Fast Vision Transformers with Hierarchical Attention
Authors: Ali Hatamizadeh
Greg Heinrich
Hongxu Yin
Andrew Tao
Jose M. Alvarez
Jan Kautz
Pavlo Molchanov
Venue: ICLR 2024 poster (2024)
Keywords: Vision Transformers
Classification
Detection
Instance Segmentation
Semantic Segmentation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: A general computer vision backbone to effectively learn both short and long-range spatial information.

Abstract: We design a new family of hybrid CNN-ViT neural networks, named FasterViT, with a focus on high image throughput for computer vision (CV) applications. FasterViT combines the benefits of fast local representation learning in CNNs and global modeling properties in ViT. Our newly introduced Hierarchical Attention (HAT) approach decomposes global self-attention with quadratic complexity into a multi-level attention with reduced computational costs. We benefit from efficient window-based self-attention. Each window has access to dedicated carrier tokens that participate in local and global representation learning. At a high level, global self-attentions enable the efficient cross-window communication at lower costs. FasterViT achieves a SOTA Pareto-front in terms of accuracy and image throughput. We have extensively validated its effectiveness on various CV tasks including classification, object detection and segmentation. We also show that HAT can be used as a plug-and-play module for existing networks and enhance them. We further demonstrate significantly faster and more accurate performance than competitive counterparts for images with high resolution. Code is available at https://github.com/NVlabs/FasterViT.

Published as a conference paper at ICLR 2024
FASTERVIT: FAST VISION TRANSFORMERS WITH HIERARCHICAL ATTENTION
Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, Jose M. Alvarez,
Jan Kautz, Pavlo Molchanov NVIDIA zadeh, pmol ghanov) Beaeenoovvnniiiinnnn @nvidia.ciom
ABSTRACT
We design a new family of hybrid CNN-ViT neural networks, named FasterViT, with focus on high image throughput for computer vision (CV) applications. VisterViT combines the benefits of fast local representation learning in CNNs and global modeling properties in ViT. Our newly introduced Hierarchical Attention (HAT) approach decomposes global self-attention with quadratic complexity into a multi-level attention with reduced computational costs. We benefit from efficient window-based self-attention. Each window has access to dedicated carrier tokens that participate in local and global representation learning. At a high level, global self-attentions enable the efficient cross- wossdw communication lower costs. Faster ViT achieves a SOTA Pareto-front in terms of accuracy and image throughput. We have extensively validated its effectiveness classification, object detection and segmentation. We also show that HAT can be used plug-and-play module for existing networks and enhance them. We further demonstrate significantly faster and more accurate performance than competitive counterparts for images with high resolution.
various CV tasks including
Code is available at https:://// / thub. con////aabs//attee///
INTRODUCTION
Vision Transformers (ViTs) (Dosovitskiy al., 2020) have recently become popular in computer vision and achieved superior performance in various applications such as image classification (Liu al., 2021; Dong et al., 2022; Lin et al., 2017), object detection (Zhang al., 2021b; Fang et al., 2021) and semantic segmentation (Xie et al., 2021; Cheng et al., 2021). In addition to learning more uniform lo- cal and global representations across their archi- tecture when compared to Convolutional Neu- ral Networks (CNNs), ViTs scale properly to large-scale data and model sizes (Raghu et al., 2021; Paul & Chen, 2022). Recently, several efforts (He et al., 2022; Xie et al., 2022) have also shown the exceptional capability of ViTs in self-supervised learning of surrogate tasks such as masked image modeling which may signifi- cantly enhance the performance of downstream applications. Despite these advantages, lack of inductive bias in pure ViT models may require more training data and impede performance (Xu et al., 2021b). 1b). Hybrid architectures, which con- sist of both CNN and ViT-based components, could address this problem and achieve compet- itive performance without needing large-scale training datasets (Dosovitskiy al., 2020) or other techniques such as knowledge distillation (Touvron
FisterViT Vis
Fister
84
VisterViT:
Accuracy 19)
V
Maie
Thaneler
Texl
1006
21:3
80
1758
1000
2000
3000
4000
5000
Throughput (imgisec)
Famanitiiouri
Dideenunn pcali
Teire GG23
Caret
Conmen 1212)
2820
Kreser (ron)
See 1822)
NTs mahree p470
Figure 1: Comparison of image throughput and ImageNet- 1K Top-1 accuracy. Throughput mea- sured on A100 GPU with batch size of 128.

---

Published as a conference paper at ICLR 2024
et al,, 2021a). An integral component of ViTs is the self-attention mechanism (Vaswani et al., 2017; Dosovitskiy ct al., 2020) which enables modeling of both short and long-range spatial dependencies. However, the quadratic computational complexity of self-attention significantly impacts the efficiency and hinders its use for applications with high-resolution images. In addition, contrary to the isotropic architecture (i.., same feature resolution with no downsampling) of the original ViT model, learning feature representations in a multi-scale manner lypically yields better performance (Fan et al., 2021; Wang et al., 2022), specifically for downstream applications (e.g. detection, segmentation).
To address these issues, Swin Transformer (Liu et al., 2021) proposed a multi-scale architecture in which self-attention is computed in local windows, and window- shifting allows for interaction of different regions. However, due to the lim- ited receptive field of these local regions and small area of coverage in window shifting (Liu et al., 2021; Lin et al., 2017), cap- turing cross wondww interactions and model- ing the long-range spatial dependencies become challenging for large-resolution input features. Furthermore, using self- attention blocks in early stages with larger resolution may impact the im- age throughput due to the increased number of local windows. Recently, the Swin Transformer V2 model (Liu al., 2022a) was proposed to address training instabilities on high-resolution images by improving the self-attention mecha- nism. However, in addition to having a lower image throughput compared to the Swin Transformer (Liu et al., 2021), Swin Transformer V2 still relies on the original window-shifting mechanism for cross-interaction of different windows, which becomes less effective with large image sizes.
Figure 2: Visualization of the proposed Hierarchical Attention in the feature space. By performing local win- dow attention and hierarchical attention can achieve global information propagation reduced costs.
Olbal
XD
In this work, we attempt to address these issues and propose novel hybrid architecture, denoted as ViTt which is tailored for high-resolution input images, while maintaining a fast image throughput. VisterViT consists of four different stages in which the input image resolution is reduced by using a strided convolutional layer, while doubling the number of feature maps. We propose to leverage residual convolutional blocks in the high-resolution stages of the architecture (i.e., stage in later stages (i.e., stage 3, 4). This strategy allows for fast generation of high-level tokens which can be further processed with the transformer-basee blocks. For each transformer block, we use an interleaved pattern of local and, newly proposed, Hierarchical Attention blocks to capture both short and long-range spatial dependencies and efficiently model the cross- window interactions. Specifically, our proposed Hierarchical Attention (see Fig. 2) learns carrier tokens as a summary of each local window and efficiently models the gross-interaction between these regions. The computational complexity of the Hierarchical Attention grows almost linearly with input image resolution, as the number of regions increases, due to the local windowed attention being the compute bottleneck. Hence, it is an efficient, yet effective way of capturing long-range information with large input features.
1, 2). while employing transformer-blowe
We have extensively validated the effectiveness of the proposed Vas model on various image tasks and datasets such as ImageNet-Ik for image classification, MS COCO for object detection and instance segmentation and ADE20K dataset for semantic segmentation. FasterViT achieves state-of-the-ar performance considering the trade-off between performance (e.g.. ImageNet- IK top-1 accuracy) and image throughput (see Fig. 1). To demonstrate the scalability of FasterViT for larger datasets, we have also pre-trained FisterViT on ImageNet-21K K dataset and achieved state-of-the-art performance when fine-tuning and evaluating on larger-scale resolutions.
The summary of our contributions is as follows:
We introduce VasterViT, which is a novel hybrid vision transformer architecture designed for an optimal trade-off between performance and image throughput. Faster scales effectively higher resolution input images for different dataset and model sizes.
We propose the Hierarchical Attention module which efficiently captures the cross-window interactions of local regions and models the long-range spatial dependencies.
FasterViT achieves a new SOTA Pareto front in terms of image throughput and accuracy trade-off and is significantly faster than comparable ViT-based architectures yielding signifi-
2

---

Published as a conference paper at ICLR 2024
cant speed-up compared to recent SOTA models. It also achieves competitive performance on downstream tasks of detection and instance segmentation on MS COCO dataset and semantic segmentation on ADE20K dataset.
2 RELATED WORK
Vision Transformers. Oriented from the language processing domain, the first application of transformer architecture to vision task immediately offers an inspiring demonstration of the high efficacy of attention across image patches across varying scenarios (Dosovitskiy et al., 2020). The appealing strength of vision transformer and its architecture and logic simplicity has therefore triggered a quickly evolving literature in the past two years, where ViT performance quickly boosted by an erupting new set of innovations: network-wise leveraging knowledge distillation for data-efficient training as in DeiT (Touvron et al., 2021a), hybriding convolution and self-attention for enhanced inductive biases as in LeViT (Graham et al., 2021), imposing CNN-inspired pyramid rules on ViTs (Wang et al., 2021; 2022), along with component-win wise improvements such as improved token utilization as in T2T-ViT (Yuan et al., 2021), enhanced positional embedding (Chu et al., 2023), local window attention as shown in the inspiring work of the Swin family (Liu et al., 2021; and CSwin (Dong et al., 2022), global attention in GCVIT (Hatamizadeh et al., 2023), among many other architectural insights (Chu et al., 202 la; Zhang et al., 2021a; Yuan et al., 2022). Along with the increasing capacity comes the increasing computation burden. As similarly facing challenges in scaling up the models in language tasks (e.g., from BERT-Large 0.3B (Devlin et al., 2019), to Megatron- L 8.3B (Shoeybi al., 2019), and Switch- Transforeerl.66 6.6T (Fedus et al., 2022)), scaling up vision transformers is also highly challenging but highly important task (Dai et al., 2021; Liu et al., 2022a) due to the antention-extensinn nature of transformers, urging efficiency for pervasive usage.
Towards Enhanced Efficiency. Boosting up ViT efficiency has therefore been a very vibrant area. One stream of approach roots in the efficient deep learning literature that cuts down on network complexity leveraging popular methods such as efficient attention (Bolya et al., 2022; Lu et al., 2021; Cai et al., 2022), network compression (Chen et al., 2021b;c; Liang et al., 2022; Yang et al., 2021a), dynamic inference (Yin et al., 2022; Rao et al., 2021), operator adaptation (Molchanov al., 2022), token merging and manipulations (Marin al., 2021; Xu al., 2022), etc. These methods can yield off-the-shelf speedups on target ViT backbones, but are also limited to the original backbone's 's accuracy and capacity. Another stream of work, on the other hand, focuses on designing new ViT architectures with enhanced efficiency as an original design objective. For example, EfficientFormer (Li et al., 2022) entails mobile applications through cimenssioncoonsiseett re-design of transformer block and removing redundant architectural components. VisFormer (Chen et al., 2021d) transits computation extensive transformer to a convolutional counterpart for enhanced vision efficiency. CrossViT (Chen et al., 2021a) learns multi-scale features and utilizes amall/large-pate backed tokens that are channeled by efficient attention, offering linear time and memory complexity. Even with such rapid progress in literature, enabling efficient ViTs remains significant challenge, where we next further push the Pareto front of faster ViT on top of prior art by a large margin. Note that we focus the second stream of architectural redesign for efficiency boost, and consider a joint exploration with the first acceleration stream of method like compression as orthogonal and fruitful future work.
Global Leff-Attention A number of efforts have introduced global self-attention to capture more contextual information. In NLP (i.e., ID), BigBird (Zaheer al., 2020) and LongFormer (Beltagy et al., 2020) proposed to select special tokens (i.e. non-learnable) as global tokens to attend to other tokens via sliding-windor window dense self-attention. computer vision. Edge ViT (Pan al., 2022), Twins (Chu et al., 2021a) and Focal Transformer (Yang et al., 2021b) proposed hierarchical-like attention mechanisms which rely on heuristic token aggregation in the forms of pooling ( (Yang et al., 2021b) or linear projection (Pan et al., 2022; Chu et al., 2021a). There are three key differences between these efforts and our proposed hierarchical attention: (1) as opposed to using a pre-defined mechanism to select the global tokens (e.g., random), we propose to learn these tokens (i.e., carrier token) via summarizing the role of each region in the input feature space (2) we propose learnable token aggregation and propagation mechanisms by computing self-attention among carrier tokens (3) as opposed to using dense/dilated self-attention, our proposed HAT uses local window- self-attention and has smaller computational complexity.
3

---

Published as a conference paper at ICLR 2024
Stage xNNg
Stage x/N
Slage 3 K/Ng
Stage X.N
#xxxx
W 2
JI
J
X80
input
Corv (52)
Conv (52)
Conv Block
Coty Block
Heraachicc Attenton
Herarchical Attention
Hoad
CT
Figure 3: Overview of the VisterVIT architecture. We use multi-scale architecture with CNN and transformer-basee blocks in stages 1, 2 and 3, 4, respectively. Best viewed in color.
FASTERVIT
3.1 DESIGN PRINCIPALS
We next detail our VisterViT architecture, offering Pareto accuracy-latency trade-off. We focus highest throughput for computer vision tasks on mainstream off-the-shelf hardware such as GPUs that excel in parallel computing. Computation in this case involves of streaming multiprocessors (SMs) with CUDA and Tensor cores as computation units. requires frequent data transfer for calculation and can be impacted by data movement bandwidth. As such, operations bounded by computation are math-limited, while those bounded by memory transfer are memory-limited, requires careful balance between the two maximize throughput.
In hierarchical vision models, spatial dimension of intermediate representation shrinks as inference proceeds. Initial network layers mostly have larger spatial dimensions and fewer channel (e.g.. 112x making them memorr-bbund. This makes a better fit for compute-intensive operations, such dense convolution instead of cepth-wise/sparse counterparts that impose extra transfer cost. Also operations not representable in matrix manipulation forms, e.g., non-linearity, pooling, batch normalization, are also nemory-bound and shall be minimized for usage. On the contrary, later layers tend to be math-limited with computationally expensive operations. For example, hierarchical CNNs have feature maps of size 14x 14 with high dimensional kernels. This leaves room for more expressive operations such as Layer Normalization, small effect on throughput. Guided by these insights we design a novel architecture that will benefit all stages from accelerated computing hardware.
souecze-and-excil
or attention, with fairly
3.2 ARCHITECTURE
Our overall design shown in Fig. 3. It exploits convolutional layers the earlier stages that operate on higher resolution. The second half of the model relies novel hierarchical attention layers to reason spatially across the entire feature maps. In this design, we optimize the architecture for compute and throughput, As result, the first half of the network and downsampling blocks make use of dense convolutional kernels. We also avoid squeeze-and and-aadddeeeee excitation operators and minimize Layer Normalization for higher resolution stages (i.e., 1, 2), as these layers tend be mimteelimiedd. Later stages (i.e., 3, 4) in the architecture tend to be math-limited on compute compared the memory transfer cost. As result, applying multi-head attention will not be a bottleneck.
GPU hardware spends more time
3.3 FASTERVIT COMPONENTS
Stem An input image x E IR HX x3; is converted into overlapping patches by two consecutive 3 convolutional layers, each with stride of 2, which project them into D-dimensional embedding. The embedded tokens are further batch-normalized (loffe & Szegedy, 2015) and use the ReLU activation function after each convolution.
Downsampler Blocks VisterViT follows the hierarchical structure: the spatial resolution is reduced by 2 between stages by downsampling block. We apply 2D layer normalization on spatial features, followed by convolutional layer with kernel of 3 x 3 and stride of two.
4

---

Published as a conference paper at ICLR 2024
Loc
Hanarchical Attertion Block
repeated
Figure 4: Proposed Hierarchical Attention block. Carrier tokens (CT) learn summary of each local window and facilitate global information exchange between local windows. Local window tokens only have access dedicated subset of CT for efficient attention. CT undergo full self-attention to enable cross-window attention. "Attention" stands for MHSA (Vaswani et al., 2017), MLP for multi-layer perceptron. Best viewed in color.
Conv Blocks Stage and consist of residual convolutional blocks, which are defined
x GELLU(BN(Covvsx<()))
X (N(Conv + x.
where BN denotes batch normalization (Ioffe & Szegedy, 2015). Following the design principles, these convolutions are dense.
Hierarchical Attention In this work, we propose novel formulation of windowed attention, summarized in Fig and detailed presentation in Fig 4. We start with local windows introduced in Swin Transformer (Liu al., 2021). Then, we introduce notion of carrier tokens (CTs) that play the summarizing role of the entire local window. The first attention block is applied on CTss to summarize and pass global information. Then, local window tokens and CTs are concatenated, such that every local window has access only to its own set of CTs. By performing self attention on concatenated tokens we facilitate local and global information exchange reduced cost. By alternating sub- global (CTs) and local (windowed) self attention we formulate a concept of hierarchical attention. Conceptually, CTs can be further grouped into windows and have higher order of carrier tokens.
Assume we are given input feature map E Hx in which H, W and d denote the height, width and number of feature maps, let us set
W for simplicity. We first partition the input n²² feature map into n X n local windows with n = gr,
where l is the window size, as:
x xp))
(2)
The key idea of our approach is the formulation of carrier tokens (CTs) that help to have an attention footprint much larger than a local window at low cost. At first, we initialize CT's by pooling to T, = 2 tokens per window:
X = 3x))
(3)
Set AvgPool (Kc);
where Convax3 represents efficient positional encoding inspired by (Chu et al., 2021b) and used in Twins (Chu al., 2021a). Ket and AvgPool denote the carrier tokens and feature pooling operation, respectively; is set to 1, but can be changed to control latency. The current approach with conv+pooling gives flexibility with the image size. These pooled tokens represent a summary their respective local windows, we set I. << k. The procedure of CT initialization is performed only once for every resolution stage. Note that every local window & has unique set of carrier tokens, &ct.1- such that &et {80
In every HAT block, CTs undergo the attention procedure:
Ect det + 71 MHSA(LN((&.)),,
&ct = Set + 72 MLP:-ao-
where LN represents layer normalization (Ba et al., 2016), MHSA represents multi-head self atten- tion (Vaswani et al., 2017), y is learnable per-channel scale multiplier (Touvron et al., 2021b), ML.P Ml.Pd-dddd a 2-leyer ML.P with GeLU (Hendrycks & Gimpel, 2016) activation function.
5

---

Published as a conference paper at ICLR 2024
(a)
(b) Windowed
(c) Hierarchical
(d) Twins
BigBird Bellngy ct (2020) Zaheer al. (2020)
(e) LongForee
attention
attention
Attention (ours)
Chu ct (2021a)
Figure 5: Attention map comparison for a feature map of size x x d. D- no attention, normal token attention, carrier token attention, random token attention. Full attention (a) has complexity of O(Hd)), windowed attention significantly reduces to O but lacks global context.
Next, in order to model short-long-range spatial information, we compute the interaction between the local and carrier tokens, &1 and &tt.1, respectively. At first, local features and CT's are concatenated. Each local window only has access to its corresponding CTs:
Xw = Concat &1, &ct.1).
(5)
These tokens undergo another set of attention procedure:
Sww &w 71 MHSA(LN($&)) )),
(6)
&w = &w + 72" MLP MLPP--uu-aLN((sk,
Finally, tokens are further split back and used in the subsequent hicrarchical attention layers:
81, & Split(Sw),
Procedures described in Equations 4-7 are iteratively applied for number of layers in the stage. To further facilitate long-shot-range interaction, we perform global information propagation, similar to the one in (Pan et al., 2022) in the end of the stage. Finally, the output of the stage is computed as:
X Upsample,21.-nrr(Saee
+
(8)
MHSAs performed in Eq. and 6 are token position invariant, however, the location of features in the spatial dimension are clearly informative. To address this, we first add absolute positional bias directly to CTs and local window tokens. We are inspired by Swin (Liu et al., 2022a) and employ 2-layer MLP to embed absolute 2D token location into feature dimension. Then, to facilitate image-like locality inductive bias we enhance the attention with log space relative positional bias from Sw (Liu et al., 2022a) (2-layer MLP). It ensures that the relative position of tokens contribute to shared attention patterns. This approach yields flexibility regarding image size, as the positional encoding is interpolated by the MLP, and hence trained model can be applied to any input resolution.
An attention map comparison between efficient global-local self attention is shown Fig. 5. The proposed hierarchical attention splits full attention into local and sub- sub-global, both compressible to dense attentions. Carrier tokens participate in both attentions and facilitate information exchange.
Complexity Analysis of HAT The key features of the efficiency of our approach are (i separation of attentions and (ii) local windows only have access to their CTs. The complexity of the most conventional and popular full attention O(H dd). Partitioning the feature size into windows of size k. and running the attention, simplifies the attention to O(k2 d) as proposed in (Liu et al., 2021). It well known that such windowed attention is more efficient but lacks global feature interaction. Our approach takes this one step further and is based on carrier tokens that summarize and interact over the entire feature map, to remedy for missing global communication. Given L total carrier tokens per window, local window complexity is O((k2 + dd). Local (windowed) attention is followed by attention on carrier tokens with complexity O(( H² d).. The total cost of both attentions 2 LH3d + L(d))
An orthogonal approach for multilevel attention is to provide access to subsampled global information inside local attention. For example, Twins (Chu al., 2021a) subsamples global feature map and

---

Published as a conference paper at ICLR 2024
Table 1:
Comparison of classification benchmarks on ImageNet-Ike 1K dataset (Deng et al., 2009).
Image throughput is measured on A100 GPUs with batch size of 128.
Model
Image Size #Param FL.PPs Throughput Top-1
(Px)
(M)
(G)
(Img/Sec) (%)
Conv-Based
ConvNeXt-T Liu et al. (2022b)
224
28.6
3196
82.0
ConvNeXI-S Liu et al. (2022b)
224
50.2
8.7
2008
83.1
ConvNeXI-B Liu et al. (2022h)
224
88.6
15.4
1485
83.8
RegNerY-040 Radosavovic
(2020)
288
20.6
3227
83.0
ResNetV2-101 Wightman et al. (2021)
224
44.5
7.8
4019
82.0
EfficientNerV2-S Tan & Le (2021)
384
21.5
8.0
1735
83.9
Transformer- Based
Swin-T Liu et al. (2021)
224
28.3
4.4
2758
81.3
Swin-S Liu ct al. (2021)
224
49.6
8.5
1720
83.2
SwinV2-T Liu el. (2022a)
256
28.3
1674
SwinV2-S Liu ct (2022a)
256
49.7
8.5
1043
83.8
SwinV2-B Liu et al. (2022a)
256
87.9
15.1
535
84.6
Twins-B Chu et. (2021a)
224
56.1
1926
DeiT3-L
224
304.4
59.7
535
84.8
PoolFormer-M58 Yu ct al. (2022)
224
73.5
11.6
884
82.4
Hybrid
CoaT-Lite-S S Xu al. (2021a)
224
19.8
4.1
2269
82.3
CrossViT-B Chen al. (2021a)
240
105.0
20.1
1321
82.2
Visformer-S Chen et al. (2021d)
224
40.2
3676
82.1
VigeV Pan et al. (2022)
224
13.1
1.9
4254
81.0
EfticientFormer-L77 Li al. (2022)
224
82.2
10.2
1359
83.4
ViT-B et al. (2022)
224
120.0
507
84.9
Max Tu et al. (2022)
224
212.0
43.9
376
85.1
Faster ViT
Faster VTT-0
224
31.4
3.3
5802
82.1
FasterViT-
224
53.4
5.3
4188
83.2
FasterVIT-2
224
75.9
8.7
3161
84.2
VasterViT-3
224
159.5
18.2
1780
84.9
Vis-
224
424.6
36.6
849
85.4
FasterViT-5
224
957.5
113.0
449
85.6
FasterViT-6
224
1360.0
142.0
352
85.8
uses it as key and value for local window attention. It has a complexity of O(22H2d H¹ d) (from the paper). Under the
size of the local window (k)), and H, E- for Twins. HAT gets more efficient with higher resolution, example, for H 32, k 8, with L we get 088) for HAT, whereas O(16) for Twins.
get the difference of
OLL + H2r,2 for HAT and O #²
4 RESULTS
4.1 IMAGE CLASSIFICATION
In Table I, we demonstrate quantitative comparison between the performance of FasterViT models and a variety of different hybrid, conv and Transformer-based networks on ImageNet- dataset. Comparing to Conv-based architectures, we achieve higher accuracy under the same throughput, for example, we outperform ConvNeXt-T by 2.2%. Considering the accuracy and throughput off ViT models are significantly faster than Trasse models such as the family of Swin

---

Published as a conference paper at ICLR 2024
Table 2: Object detection and instance segmentation benchmarks using Cascade Mask R-CNN (He et al., 2017) on MS COCO dataset (Lin et al., 2014). All models employ 3x schedule. All model statistics are reported using a input test resolution of 1280 800.
Backbone
Throu. im/sec
Aph
Apeack
Box
50
75
Mask
50
75
Swin-T T' et al. (2021)
50.4
69.2
54.7
43.7
66.6
47.3
ConvNeXt-T Liu et al. (2022b)
166
50.4
69.1
54.8
43.7
66.5
47.3
DeiT-Smal/116 Touvron et al. (2021a)
269
48.0
67.2
51.7
41.4
64.2
44.3
Faster VTT-2
287
52.1
71.0
56.6
45.2
68.4
49.0
Swin-S Liu et al. (2021)
119
51.9
70.7
56.3
45.0
68.2
48.8
X101-32 Xie et (2017)
124
48.1
66.5
52.4
41.6
63.9
45.2
ConvNeXt-S Liu et al. (2022b)
128
51.9
56.5
45.0
68.4
49.1
ViT-3
159
52.4
56.7
45.4
68.7
49.3
X101-64 Xie et al. (2017)
86
48.3
66.4
52.3
41.7
64.0
45.1
Swin-B Liu et al. (2021)
90
51.9
70.5
56.4
45,0
68.1
48.9
ConwNeXt-B Liu et al. (20226)
101
52.7
71.3
57.2
45.6
68.9
49.5
VisterViT-4
117
52.9
71.6
57.7
45.8
69.1
49.8
Transformers (Liu et al., 2021; 2022a). Furthermore, compared to hybrid models, such as the recent Efficient- Former (Li et al., 2022) and MaxViT (Tu et al., 2022) models, FasterViT on aver- age has a higher through- put while achieving a better ImageNet top-1 performance. To validate the scalability of the proposed model, we pre-trained FasterViT-4 on ImageNet-21K dataset and fine-tuned it on various im- age resolutions ImageNet-1 1K dataset. As shown in Table 3, Faster ViT-4 has better accuracy- throughput trade-off compared to other counterparts.
Model
Image Size (Px)
#Param
FL.OPs
Throughput Top-1 (Img/Sec)
(M)
(G)
(%)
VIT-L/16
(2021)
307.0
190.7
149
85.2
Swin-L Liu at. (2021)
224
197.0
34,5
787
86.3
Siin-L.FLiucee (2021)
384
197.0
103.9
206
87.3
ConvNeXt-L Liu al. (2022b)
224
198.0
34.4
508
86.6
ConvNeXt-T. Liu et. (2022b)
384
198.0
101.0
172
87.5
FasterViT-44
224
424.6
36.6
849
86.6
ViT-er
424.6
87.5
Table 3: ImageNet-21K pretrained classification benchmarks on ImageNet-1K dataset (Deng et al., 2009). Image throughput is mea- sured on A100 GPUs with batch size of 128. denotes models that pre-trained ImageNet-21K dataset.
4.2 DENSE PREDICTION TASKS
In Table 2, we present object detection and instance segmentation benchmarks on MS COCO dataset (Lin et al., 2014) with Cascade Mask R-CNN (He et al., 2017) network. We observe that VisterViT models have better accuracy-throughie trade-off when compared to other coun- terparts. Specifically, VasterViT-4 outper- forms ConvNeXt-B and Swin-B by +0.2 and +1.0 in terms of box AP and +0.3 and +1.0 in terms of mask AP, while being 15% and 30% faster in terms of throughput, re- spectively. We also conduct additional ob- ject detection experiments with ViT- ImageNet-21K KK pre-trained backbone and the state-of-the-art DINO (Zhang et al., 2022) model and achieve high detection accuracy of 58.7 7 box AP. Table S.6. we present the semantic seg- mentation benchmarks with UPerNet (Xiao et al., 2018) network for experiments conducted on ADE20K dataset (Zhou et al., 2017). Similar to previous tasks, FasterViT models benefit from better performance-throughhe trade-off
Model
Throughput
FLOPs (G) loU(ss/ms)
Swin-T Liu et (2021)
350
945
44.5/45.8
ConvNeXi-T Liu at. (20226)
363
939
/46.7
VisterViT-2
377
974
47.2/48.4
Twins SVT-B Cluu et al. (2021a)
204
47.7/48.9
Swin-S Liu et al. (2021)
219
1038
47.6/49,5
ConsNeXi-S Liu (20226)
234
1027
Vis-erViT-3
254
1076
48.7/49.7
Twins SVT-L Chu et al. (2021a)
164
48.8/50.2
Swin-B Liu et (2021)
172
1188
48. 1/49.7
ConsNeXi-B Liu et (20226)
189
1170
149.9
FasterViT-4
202
1290
49.1/50.3
Table 4: Semantic segmentation on ADE20K (Zhou et al., 2017) with UPerNet (Xiao et al., 2018).
8

---

Published as a conference paper at ICLR 2024
5 ABLATION
EdgeViT and Twins As shown in Table 5, we performed comprehensive ablation study to validate the effectiveness of HAT by replacing all attention layers with attention mech- anisms EdgeViT (Pan et al., 2022) and Twins (Chu et al., 2021a) in the 3rd and 4th stages. For all model variants, Faster- ViT models with HAT achieve better ac- curacy, sometimes by a significant margin. Twins achieves a higher throughput due its small kernel size (i.e. k = 2), however, this significantly limits its accuracy. The better performance of HAT is attributed to its learnable information aggregation/ppooagattion via CTs, and direct access dedicated CTs in windowed attention.
Model
Attention
FLOPs
Thrlmy//Sc)
Top-1
ViT-erViT-0 Twins (Chu et al, 2021a)
3.0
6896
80.8
PasterViT-O
EigeViT (Pan et. 2022)
3.2
$928
81.0
Faster
ViT-o VIT-O HAT
3.3
5802
82.1
FasterViT-1 Twins (Chu al, 2021a)
4.7
4949
82.1
FusterViT-1 EdgoVIT (Pan et al, 2022)
4.8
4188
82.5
ViT- HAT
5.3
4344
83.2
FasterViT-2 Twins (Chu al,, 2021a)
8.0
3668
82.9
ViT-2 Edg (Pan al, 2022)
8.5
3127
83.4
3161
Table 5: Ablation study on the effectiveness of HAT compared to EdgeViT (Pan et al., 2022) and Twins (Chu et al., 2021a) self-attention mechanisms. All attention blocks are replaced with the indicated attention type.
Carrier Token Size We investi- gated the effect of carrier token size and window size the accuracy and image throughput of the model. We observed that increasing the car- rier token size can improve the perfor- mance the cost decreased through- put, sometimes by significant margin. In addition, increasing the window size slightly improves the Top-1 accuracy while also decreasing the throughput. In fact, increasing the window size does not scale properly to higher resolution images due to its significant impact efficiency. As result, HAT is more effective and efficient mechanism that can be employed to model long-range spatial dependencies without sacrificing the throughput. Please refer supplementary materials for more details.
Pretrain
Finetune
W8, 1256
W12,
1384
W16, 1512
W24, 1768
Model
aec
inus
in/s
in/s
im/s
SwinV2-T Liu et al. (2022a)
81.8
1674
83.2
573
83.8
168
84,2 72
SwinV2-S Liu (2022a)
83.7
84.8
338
85.4
153
VisterViT-2
84.3
2500
85.3
984
85.5
489
85.6
155
SwinV2-B Lii al. al. (2022a)
84.2
499
85.1
251
85.6
115
ViTterViT-425 256
85.3
86.1
86.0
44
Table 6: Quantitative comparison between higher resolution fine-tuning of FisterViT and S2in FisterViT is more accurate on average by 0.9%, and faster by 2x.
Plug-and-Play HAT We employed HAT as plug-and-play module with Swin-T model Table This change results in +0.9 and +0.4% improvement terms of mloU and Top-1 accuracy on ImageNet classifica- tion and ADE20K segmentation tasks. In addition, improvements on MS COCO by +0.5 box AP and +0.6 mask AP on object detection and instance segmentation tasks, respectively. addition, we also provide throughput comparisons and show that HAT can be efficiently used with existing architectures with minimal overhead. Hence, validates the effectiveness of HAT as a standalone self-attention.
ImageNet top-1
COCO
ADE20K
Thr
Aphss
Ap
Thr
mloU
Thr
Swin-T
81.3
2758
50.4
43.7
161
44.5
350
Swin-T HAT
81.7
2721
50.9
44.3
150
45.4
338
Table Ablation study on the effectiveness of HAT as plug-and wogg-and-play module with Swin-'T model for various CV tasks. Thr stands for throughput and is measure in image/sec.
6 CONCLUSION
In this work, we have presented a novel hybrid model, denoted FasterViT, which achieves SOTA Pareto-from in terms of ImageNet Top-1 accuracy and throughput. We have extensively validated the effectiveness of Vis in downstream dense prediction tasks such as object detection, instance segmentation and semantic segmentation. Our benchmarks demonstrate better accuracy-thres throuuytth trade-off in comparison to counterpart models such XonvNeXt and Swin Transformer.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENT
We thank Amanda Moran, Christopher Lamb, Sivakumar Thottakara, Sudeep Sabnis, Ranjitha Prasanna and other members of NVIDIA NGC team for providing highly-optimized GPU cloud infrastructures which were used for training and evaluation of FasterViT models.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 17Xi::6607.06450. 2016. 5, 18, 19
Iz Beltagy. Matthew E Peters, and Arman Cohan. Longformer: The long-documen transformer. arXiv preprint Xxx::20044001500, 2020. 3.6
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman, Hydra attention: Efficient attention with many heads. European Conference on Computer Vision, pp. 35-49. Springer, 2022.
Han Cai, Chuang Gan, and Song Han. Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition. arXiv preprint 2ww.:2205.11756, 2022. 3
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vi-
sion transformer for image classification. In Proceedings the IEEE/CVF international conference
computer vision, pp. 357-366, 2021a. 3, 7
Kai Chen, Jiangmiao Pang, Jiaqi Wang. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
4974 4983, 2019. 20
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12270-12280, 2021b.
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances Neural Information Processing Systems, 34:19974-19988, 2021c. 3
Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 589-598, 2021d. 3,7,
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34: 34:17864-17875.
2021. 1
Xiangxiang Chu, Zhi Tian, Yuqing Wang. Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34, 2021a. 5, 6, 7, 18
Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia. Do we really need explicit position encodings for vision transformers? CoRR, abs/2 10882, 2021b. 5
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. In ICLR 2023, 2023. URL http::/ss////ppeeeeeee/eee n forum? -
3
Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-3977, 2021. 3
10

---

Published as a conference paper at ICLR 2024
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A hurge-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009. 7, 8, 15, 16
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped win- dows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12124-1211 12134, 2022. 1,3
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2020. 2,
Jiawei Du, Zhou Daquan, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. In Advances in Neural Information Processing Systems. 19
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6824-6835 6835, 2021.
Yuxin Fang, Bencheng Liao, Xinggang Wang. Jiemin Fang. Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. Advances in Neural Information Processing Systems, 34:26183-26111 2021.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 2022. 3
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:20/0.01412,, 2020. 19
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer convnet's clothing for faster inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12259-12260 2021. 3
Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global context vision transformers. International Conference on Machine Learning. pp. 22633-12646 PMLR,
2023. 3
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017. 8, 15
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv.1606.08415. 2016. 5
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness. A critical analysis of -ittrf-disttibution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021a. 15, 16
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271. 2021b. 15, 16

---

Published as a conference paper at ICLR 2024
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. pp. 448-444455 456. PMLR, 2015. 5,5
Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang. and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 35: 19934--922 2022. 3, 7,8
Youwei Liang, Chongjian GE, Zhan Tong. Yibing Song, Jue Wang, and Pengtao Xie. EVit: Expe- diting vision transformers via token reorganizations. In International Conference on Learning Reprexentations, 2022. URL https: htwp//ppevvvieeeeeeeeeeeeeeeeeee net/ forum did jywdd=-jjyyxxnn__..
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 8, 15, 20
Tsung Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017. 1,2
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 10012-10022, 2021. 1, 3, 6, 8. 15, 16, 18, 20
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang. Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12009-12019, 2022a. 2, 3, 6, 8, 9, 16, 18, 19
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1997611986, 2022b. 7, 8, 15, 16
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017. 15, 18
Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. Soft: softmax-free transformer with linear complexity. Advances in Neural Information Processing Systems, 34:21297-21309, 2021. 3
Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, Anish Prabhu, Mohammad Rastegari, and Oncel Tuzel. Token pooling in vision transformers. arXiv preprinl arXiv:2110003860, 2021.
Pavlo Molchanov, Jimmy Hall, Hongxu Yin, Jan Kautz, Nicolo Fusi, and Arash Vahdat. Lana: latency aware network acceleration. In European Conference on Computer Vision, pp. 137-156. Springer, 2022. 3
Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martinez. Edgevits: Competing light-weight cnns on mobile devices with vision transformers. In ECCV, 2022. 3, 6, 7, 18
Sayak Paul and Pin- Yu Chen. Vision transformers are robust learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2071 2071-2081, 2022.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 10428-10436, 2020.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34, 2021.
12

---

Published as a conference paper at ICLR 2024
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cui Hsieh. Dynamic ViT: Efficient vision transformers with dynamic token sparsification. In NewrIPS, 2021. 3
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. -5890 PMLR, 2019. 15, 16
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Ima Im: Training multi-billion parameter language models using model parallelism. arXiv preprint 1rXiv: 190.0005,,, 2019. 3
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In International Conference on Machine Learning, pp. 10096-10106. PMLR, 2021. 7
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021a. 1,3,8
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference
computer vision, pp. 32 32-42,2222 2021b. 5
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV. pp. 516-533. Springer, 2022. 19
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pp. 459-4799 Springer, 2022. 7,8
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008, 2017. 5,
Wenhai Wang, Enze Xie, Xiang Li, Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 568-578, 2021. 3
Wenhai Wang. Enze Xie, Xiang Li, Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415-424, 2022. 2,
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arX1v:2110.00476, 2021. 7
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference Computer Vision (ECCV), pp. 418-434, 2018. 8, 15
Enze Xie. Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systemx, 34: 10777-12000. 2021. 1
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 1492-1500, 2017.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
13

---

Published as a conference paper at ICLR 2024
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9981-9990, 2021a. 7
Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2964-22222222 29722 2022.
3
Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances Neural Information Prucessing Systems, 34: 22522-28535, 2021b. 1
Huanrui Yang, Hongxu Yin, Pavlo Molchanov, Hai Li, and Jan Kautz. Nvit: Vision transformer compression and parameter redistribution. arXiv preprint arXiv:2110.04869 2021a. 3
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao.
Focal attention for long-range interactions in vision transformers. Advances in Neural Information Processing Systems, 34, 2021b. 3
Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-ViT Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2022. 3
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint 1rXiv:190..00966 2019. 15, 18
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10819-10829, 2022. 7, 20
Li Yuan, Yunpeng Chen, Tao Wang. Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token ViT: Training vision transformers from scratch imagenet. In ICCV, 3
Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. Volo: Vision outlooker for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2022. 3
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 1177772231177,,,18 2020. 3,
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2233.03605, 2022. 20
Pengehuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding, In Proceedings of the IEEF/CVF international conference on computer vision, pp. 2998-3008, 2021a a.
Zixiao Zhang, Xiaoqiang Lu, Guojin Cao, Yuting Yang, Licheng Jiao, and Fang Liu. Vit-yolo: Transformer-he based yolo for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2799-2808, 2021b.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633 633,611,226 2017. 8, 15, 20
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
B TRAINING SETTINGS
Image Classification We employ the ImageNet-1K dataset (Deng et al., 2009) for classification that includes 1.2M and 50K training and validation images. The dataset has 1000 categories and we report the performance in terms of top-1 accuracy. In addition, we use ImageNet-21K dataset which has 14M images with 21841 classes for pretraining.
We train all VasterViT models by using LAMB optimizer (You et al., 2019) optimizer for 300 epochs with a learning rate of 5e-3 and a total batch size of 4096 using 32 A100 GPUs. For data augmentation, we follow same strategies as previous efforts (Liu al., 2022b; 2021). We also use Exponential Moving Average (EMA) which often improves the performance. Further details on training settings can be found in the appendix. For pre-training on ImageNet-21K, we train the models for 90 epochs with learning rate of 4e-3. In addition, we fine-tune the models for 60 epochs with learning rate of 7e-5.
Detection and Segmentation We used the MS COCO dataset (Lin et al., 2014) to finetune Cascade Mask-RCNN network (He et al., 2017) with pretrained Vis backbones, For this purpose, we trained all models with AdamW (Loshchilov & Hutter, 2017) optimizer with initial learning rate of le-4, a 3 x schedule, weight decay of 5e-2 and a total batch size of 16 on A100 GPUs.
Semantic Segmentation For semantic segmentation, we employed ADE20K dataset (Zhou et al., 2017) finetune UperNet network (Xiao et al., 2018) with pre-trained FasterViT backbones. Specifically, we trained all models with Adam-W (Loshchilov & Hutter, 2017) optimizer and by using a learning rate of 6e-5, weight decay of le-2 and total batch size of 16 on 8 A100 GPUs.
C ROBUSTNESS ANALYSIS
In this section, we analyze the robustness of FasterViT models different datasets. We test FasterViT model variants on ImageNet-A Hendrycks et al. (2021b), ImageNet-R Hendrycks et al. (2021a) and ImageNetV2 Recht et al. (2019) datasets. In addition, we did not perform any fine-tuning and simply employed the pre-trained ImageNet- K Deng et al. (2009) weights for each model. As shown in Table S.2, ViT demonstrates promising robustness performance on various datasets for each model variant. Specifically, VisterViT-3 outperforms comparable models such as ConvNeXi-B and Swin-B Liu et al. (2022b) by +7.5% and +8.4% on ImageNet-A Hendrycks al. (2021b), +0.6% and +5.3% on ImageNet-R Hendrycks et al. (2021a) and +1.3% and +2.7% on ImageNetV2 Recht et al. (2019), respectively. For larger models, FasterViT-4 outperforms Le- Liu al. (2022b) by +7.9%, +2.6% and +1.5% on ImageNet-A Hendrycks et al. (2021b), ImageNet-R Hendrycks et al. (202 la) and ImageNetV2 Recht et al. (2019), respectively, hence validating the effectiveness of the proposed model in various benchmarks. Similar trends can be observed for smaller models.
D ABLATION
D.1 COMPONENT-WISE STUDY
Table S.1 shows per component ablation. Two settings are considered: (i) when the model trained without the component, (ii) when the component is disabled after the model is trained. The first shows if the model can operate well without the component, while the second cases shows the components is used in the final model.
We observe that changing the window resolution to 14 x 14 in the 3rd stage (effectively removing HAT by have full global window) improves the model accuracy by +0.1% while scarifying 10% of throughput. Even though this setup shows better accuracy, it does not scale to high resolution, and HAT is required. Removing the HAT block from the architecture results in -0.24% accuracy drop for re-trained model and - 1.49% for post training study at the benefit of 8% throughtput improvement. CT attention is another block of high importance, resulting in 3385% post training
15

---

Published as a conference paper at ICLR 2024
Table S.2: Robustness analysis of ImageNet-1K Deng al. (2009) pretrained FasterViT models on ImageNet-A Hendrycks et al. (2021b), ImageNet-R Hendrycks et al. (2021a) and ImageNetV2 Recht al. (2019) datasets.
Model
Size #Param FLOPs
Throughput Clean (Img/Sec)
A
R
V2
(Px)
(M)
(G)
(%)
(%)
(%)
(%)
FasterViT-0
224
31.4
3.3
5802
82.1
23,9 45.9 70.9
FasterViT-1
224
53.4
5.3
4188
83.2
31.2
47.5 72.6
Swin-T Liu et al. (2021)
224
28.3
4.4
2758
81.3
21.6 41.3 69.7
ConvNeXt-T Liu al. (2022b)
224
28.6
3196
82.0
24.2 47.2 71.0
ConvNeXt-S Liu et al. (2022b)
224
50.2
8.7
2008
83.1
31.3
49.5 72.4
VisterViT-2
224
75.9
8.7
3161
84.2
38.2
49.6 73.7
Swin-S Liu et al. (2021)
224
49.6
8.5
1720
83.2
32.5 44.7 72.1
Swin-B Liu et al. (2021)
224
87.8
15.4
1232
83.4
35.8 46.6 72.3
ConvNeXt-B Liu et al. (2022b)
224
88.6
15.4
1485
83.8
36.7 51.3 73.7
FasterViT-3
224
159.5
18.2
1780
84.9
44.2 51.9 75.0
ConvNeXt-L Liu al. (2022b)
224
198.0
34.4
508
84.3
41.1 53.4 74.2
VasterViT-4
224
424.6
36.6
849
85.4
49.0 56.0 75.7
FasterViT-5
224
975.5
113.0
449
85.6
52.7 56.9 76.0
PasterViT-6
224
1360.0
142.0
352
85.8 53.7 57.1 76.1
removal. Attention bias is important component of our system, resulting in -0.31% drop in the re-training scenario. Removing CT propagation, results in the requirement to pool and propagate features at every layer (similar to ViT), that costs 7% of total inference and in lower accuracy -0.16%. CT initialization is important to the network, as accuracy drops by -0.48% in post-training removal. Removing all components and having only CNN plus windowed vanilla transformer results in -0.46%.
D.2 SWINV2 COMPARISON
In the Table 6 compare the performance of V2 Liu et al. (2022a) and Faster- VIT models large image resolution. The initial model is pretrained with an image resolution 256 px for 300 epochs on ImageNet- K. Then models are fine-tuned on larger resolution (I) for an 30 epochs with various window sizes (W). Faster- VIT consistently demonstrates higher image throughput, sometimes by significant mar- gin compared to Swin Transformer V2 model. Hence validating the effectiveness of the proposed hierarchical attention for high input resolution.
Ablation
Trained from scratch
Post training
Throughpul ratio
removal
HAT black
-0.24%
-1.49%
1.08
CT attention
-0.13%
-3855%
1.00
Attention Bias
-0.31
-8.90%
1.00
CT propagation
-0.16%
0,93
ID pos bias
-0.07%
-24.85%
1.00
CT initialization Window 14x 14
-0.05%
-0.48%
1.00
+0.10%
0.90
Table S.1: Ablation study on the effectiveness of differ- ent components of HAT.
E ATTENTION MAPS
In Fig. S.I, we have illustrated the full attention maps of stage 3 layers for different FasterViT model variants. For this purpose, we use input images of size 224 x 224 x 3 and ImageNet-1K Deng et al. (2009) trained VisterViT models. For each model, from the top the bottom rows, we show the attention maps from the first to the final layer with an interval of quarter of the total number of layers at stage (e.g. layers 1, 4, 9 and 12 for FisterViT-4).
16

---

Published as a conference paper at ICLR 2024
(a) VasterViT-O
(b) VasterViT-1
(c) FasterViT-2
(d) FasterViT-3
(e) FasterViT-4
Figure S.1: 1: (a) FasterViT-O. (b) FasterViT-1. (c) FasterViT-2. (d) FasterViT-3 (e) FasterViT-4, Full attention map visualizations of stage 3 for VisterViT model variants. From top to bottom, we visualize attention maps of first last layers with an interval of quarter length of the number of layers in stage 3 for each model. We visualize the attention maps of the same input image for all cases to facilitate comparability.
In particular, Stage for this illustration serves an important purpose, since we use local atten- tion windows of 7 7 with input features that have a resolution of 14 14. Hence, attention is computed in 4 local regions after window par- titioning and carrier tokens are designated to each corresponding window. Each illustrated attention map has a size of size 53 53 consist- ing of concatenation of 4 4 carrier tokens and 49 x 49 local handowwbbsser attention. The car- rier tokens are shown in in the top left position of each map. We observe that for all models, all tokens will attend to the carrier tokens with different patterns.
Figure S.2: Comparison of image throughput and ImageNet- Top-1 accuracy with TensorRT post- training model optimization. For all models, throughput measured A100 GPU with batch size of 1.
85
sccunss 190
83
82
For BasterViT-O and FasterViT-1 models, from the first to the last layers, all tokens transition to attend the the carrier tokens (i.e. vertical bar on the left side). In the last layers, in addition to all tokens attending to the carrier tokens, we see more global attention pattern, hence showing cross interaction between different regions.
500
1002
Semn
17

---

Published as a conference paper at ICLR 2024
For FasterViT-2,, FasterViT-3 and FasterViT-4 models, starting from the first layers, all tokens attend to both carrier and local tokens. In the last layers however, the attention pattern shifts from local to global. As discussed in this work and also shown in these illustrations, carrier tokens serve an integral role in modeling cross-region interactions and capturing long-range spatial de- pendencies.
F TENSORRT LATENCY
All throughput numbers and insights presented in the main paper were computed using Py PyTorc v1.13. In order to demonstrate the scalability with post-training optimization techniques,
we compared throughput using the TensoRT (TRT) framework for batch size 1, as illustrated in Fig S.2. VasterViT is still considerably faster than other models, making a good choice to meet various efficient inference design targets.
Figure S.3: Learned positional biases for attentions in the 3rd stage of FasterViT-4 model finetuned for 512 512 px. Each kernel corresponds for single head in the multi-headed attention. Visualizations demonstrate that model learns positional dependent features, while also sharing the pattern between pixels.
bias
G ATTENTION BIAS
We follow the concept of relative positional bias in the attention from Swin Liu et al. (2021). Particularly, we use the implementation with MLP from Swin Liu al. (2022a), where relative coordinate shift in is transformed to the positional bias in the attention via 2-layer network. This allows the model learn relative position aware kernels, and to introduce image inductive bias. We visualize learned positional
biases of the MLP in VisterViT-4 finetuned for 512 with window size of 16x 16 pixels in Fig S.3. The visualization shows diverse set of kernels learned by FisterViT model.
H FASTERVIT PROFILING
In Fig. S.4, we provide detailed wise profiling of FasterViT-2 using NVIDIA DLSIM. As expected, stage 3 (HAT) has the highest latency, FLOPs and memory footprint since it is composed of considerably more layers compared to other stages.
I DESIGN INSIGHTS
Layer normalization Ba al. (2016). We found to be critical for transformer blocks (stage and 4). Replacing it with batch normalization leads to accuracy drop of 0.7%. The LN performs cross token normalization and affects cross-channel interaction.
No feature map reshaping. In our architecture, we have removed windowing and de-windowing functions from transformer layers. They are usually used perform convolutions between layers (like in Twins Chu et al. (2021a), EdT Pan et al. (2022), Visformer Chen et al. (2021d)), or window shifting (Swin Liu al. (2021), Swin Liu et al. (2022a)). We perform windowing only once once in stages 3 and 4, and keep data as tokenized with channel last. This leads to throughput improvement of 5% for PyTorch and 10% for TensorRT.
LAMB optimizer You et al. (2019). We observed incredible stability of LAMB You et al. (2019) optimizer for training our biggest models (FasterVIT-3 and VasterViT-4), more widely used AdamW Loshchilov & Hutter (2017) was leading to NaNs for some trainings. We attribute this to joined usage of batch normalization and layer normalization et al. (2016) in the same model.
18

---

Published as a conference paper at ICLR 2024
Positional bias. We employ 1D positional bias for local and carrier tokens, as well as 2D relative attention bias by MLP introduced in
Embedding
Stage
Stage 2
Stage
Stage
Latency
SwinV2 Liu et al. (2022a). For 1D) bias we re- move log scale. This approach yields flexibility to the image size, as positional encoding is in- terpolated by MLP if resolution change. Those positional biases are quick to compute, however, will block all cores in GPUs until positional bi- ases computed, and will significantly impact the throughput. To address this, we propose to pre-compute positional biases for given feature resolution and skip the MLP bottleneck, leading to 6% throughput gain.
FLOPs
Memory
Layers
Figure S.4: FasterViT-2 profiling benchmarks.
Stage 3 (HAT) dominates over all metries.
Drop-out. We found that conventional drop-out on MLP layers and attention has a negative effect on the final accuracy even for big models that overfit. Stochastic depth is helpful; in contrary recent trends, we found that a small probability (up to 30%) works better than 65% like in DEIT3 Touvron al. (2022). Better regularization can be achieved by increased weight decay. For example, model with drop-path rate of 50% and weight decay of 0.05 achieves 84.91 %, while model 4 with drop-path rate of 30% and weight decay of 0.12 achieves 85..
MESA Du et al.. It is shown to be useful to prevent overfitting of larger models at little overhead. MESA simplified version of SAM Foret et al. (2020) that forces optimization have sharper minima at the convergence, naive implementation slows down training by 2x. In MESA, authors propose simply apply knowledge distillation loss with respect to the EMA weight computed during training, the training overhead is almost not noticeable. We enable it after 25% of the training, coefficient is set proportionally to the model size in range 0.25
Intermediate LN. V2 Liu et al. (2022a) argues that intermediate LN Ba et al. (2016) help to stabilize training of large models, we saw accuracy degradation of this approach.
Outpul (Downs. Rate)
Cone BN-ReLUU
Cone- Bene-BN-ReLU C.64, 8:2
Cane-BN-Be C.64, S:2
Con--BN-ReLLU
Stem
112=112 (2x)
C:32, 9:2
x
x
C:64. S:2
Conv-BN-ReLL RcLL C:128
C996
C:196
56
IN-21 Cam; C. 120, N22
IN.11%,Com C 1922 82
IN-11, Cam,,
IN- Cann, 6339,,
(4x)
C:160
x1.
ResBlock
C:192
x
KesBlock
C:256
x3.
ResBlack
C:392
3,
LN-20, Comy;
C300, 8:2
LN-2D C381,
LN-2D Comy,
13512,32 3:2
LN-2D.
Coun, C3768,
28
(8x)
ResBlock
C:320
x
ResBlock
C:984
x
ResBlock
C:512
x
ReaBlack
C:768
LN.DD, C3400, 5:2
LN-2D, Com:
C706,
C1004, 5:2
LN:20, Comy, C:1568
Stage
(16x)
C:600, heud
HAT
x 8.
C:768, heabs
HAT
8.
C:1024, head:
HAT
12,
C:1568, head:16
HAT
12,
LN-2D, C:7280,
IN C-1536,
IN-2D, Camy, C:7048.
LN-2D, Cans, C33
Stage
(32x)
C:1280, 16 16
HAT
x..
C:1536, head: 16
HAT
5.
C:2048, head:
HAT
5.
C:3136, head:32
HAT
x
Table S.3: FasterViT architecture configurations. BN and LN-2D denote Batch Normalization and 2D Layer Normalization, respectively. HAT denotes Hierarchical Attention block.
J ARCHITECTURE DETAILS
In Table S.3, we show the different architecture configurations of the FasterViT model variants.
K CARRIER TOKEN SIZE
In Table S.4, we investigate the effect of carrier token size and window size on accuracy and latency of the model. We observe that increasing the carrier token window size can improve the performance at the cost of increased latency, sometimes by a significant mar- gin. The 2x2 carrier token window size offers a great trade-off between accuracy and
19

---

Published as a conference paper at ICLR 2024
Table S.5: MS COCO dataset (Lin et al., 2014) object detection results with DINO (Zhang et al.,
2022) model. $ denotes models that are pre-trained on ImageNet-22KK dataset.
Backbone
Model
Epochs FLOPs (G) Throughput Aphox
Swin-I.7 (Liu et al., 2021)
HTC++ (Chen et al., 2019)
72
1470
57.1
Swin-I.3 (Liu al., 2021) FasterViT--*
DINO (Zhang al., 2022)
36
1285
71
58.5
DINO (Zhang et al., 2022)
36
1364
84
58.7
latency. In addition, increasing the win- dow size from 7 to 14 increases the Top-1 accuracy by +0.2%. However, expected, increases the latency by 10%. Hence, this shows the advantage of leveraging car- rier token as an efficient mechanism to capture long-range contextual information. We also note that although increasing the window size results in better performance, it does not scale properly to higher resolu- tion images. As result, HAT a more effective and efficient mechanism that can be employed without sacrificing image throughput.
Window Size Carrier Token Size
Latency Ratio Top-1 (%)
7
84,2
7
1.05
83.9
0.47
84.9
0.9
84.4
Table S.4: Effect of window and carrier token size on latency and Top-1 accuracy.
L DOWNSTREAM EXPERIMENTS
We provide additional experiments for both object detection and semantic segmentation with more models, across different sizes, to demonstrate the effectiveness and efficiency of our work. Firstly, in Table S.5, we present additional object detection experiments with DINO on MS-COCO dataset. The DINO model with FasterViT-4 is 18.30% faster than its counterpart with Swin-L backbone in terms of image throughput and outperforms by +0.1 in terms box AP.
We also added semantic segmentation study on the ADE20K dataset with the FPN network, as shown below. Specifically, compare against PoolFormer and PVT back- bones. In this experiment, the model with ViT-- backbone outperforms coun- terpart PoolFormer-St of mloU while also being 8.38% faster in terms of image throughput. Similarly, the model with Faster ViT-2 backbone signifi- cantly outperforms PoolFomer-Mis counterpart by +1.1 terms of mIOU while being 10.05% faster. We have added these experiments the manuscript. We believe that the above experiments validate the effectiveness of Faster ViT as an efficient backbone for downstream tasks such as segmentation and detection across different model sizes.
Backbone
Model Throughput mloU
PoolFomer-S36 (Yu et al., 2022)
FPN
453
42.0
VasterViT-1
FPN
491
42.7
PoolFomer-M33 (Yu et al., 2022) ViT-2
FPN
368
42.4
FPN
405
43.5
by +0.7 in terms
Table S.6: Semantic segmentation on ADE20K (Zhou et al., 2017) with FPN network.
M IMPACT OF CONV-BLOCKS ON THROUGHPUT
We conducted an additional ablation study demonstrate the effect of conv-based block on both accuracy and throughput as shown below. According our experiments, replacing Conv-based blocks with Transformer-based counterparts significantly reduces the throughput while also reducing the accuracy. As expected, the Conv-based blocks are more efficient than the transformer counterparts for processing larger input sizes. The model with conv-based blocks also has higher accuracy compared counterparts due to incorporating inductive biases such locality.
to their fully-trana transformer- The combination of Conv-based (stage and 2) and transformer baassdereer-beeed (stage and 4) architecture as presented in Faster strikes the right balance between accuracy and efficiency.
20

---

Published as a conference paper at ICLR 2024
N THROUGHPUT ON DIFFERENT PLATFORMS
In order to validate the effectiveness of Vis on different platforms, we present additional throughput comparisons on different hardware such as NVIDIA V100, NVIDIA TITAN RTX and NVIDIA A6000 GPUs, Jetson Nano and Intel(R) Xe E5-2699 v4 CPU. For all compar- isons, we use batch size of 128, unless otherwise stated. Our benchmarks show that ViT achieves Pareto-front for ImageNet Top-1 and throughput trade-off, hence validating the effectiveness and scal- ability of our model to different hardware platforms.
Model
Top-1
Throughput
FasterViT-O
82.1
5802
FasterVil-O wo Conv-block
VasterViT-1
83.2
VisterViT-1 wo Conv-block
82.8
VisterViT-2
VasterViT-2 wo Conv-block
FusterViT-3
1780
ViT- wo Conv-block
1397
VisterViT-4
85,4
FasterViT-4 wo Conv-block
712
Table S.7: Effect of Conv-based stages on throughput and accuracy of different FasterViT models.
84
Accuracy (%)
83
82
81
80
79
500
1000
1500
3000
Throughput (img/sec)
FesterViT(ours) Convieet: (2022)
Efficeenteeriz 12021)
12021) DelT
CrossViT (2021)
Visformer (2021)
RegNetr 12020)
(2022) FastVIT (2023)
E#ddenFormer (2022)
MaxiiT (2022)
FAN (2022)
NextViT (2022)
Figure S.5: Comparison of image throughput and ImageNet- 1K Top-1 accuracy on NVIDIA V100 GPU for batch size of 128.
In Fig. S.5, we demonstrate the throughput and accuracy trade-off on V100 GPU and observe that FisterViT achieves a Pareto front. Additionally, in Fig. S.6, we illustrate the same comparison for NVIDIA TITAN RTX GPU, which is considered as enthusiast-class graphics card. Surprisingly, we see that Vis attains a Pareto front on this platforms as well.
In addition, shown in Fig. S.7, report the throughput for all models using NVIDIA A6000 GPU to confirm the scalability our proposed architecture to various types of hardware. On A6000 GPU, FisterVTT still demonstrates strong performance and achieves SOTA Pareto front except for an EfficientNetV2 variant which achieves comparable performance to FasterViT-2.
In addition to GPU hardware, have also measured throughput CPU device well NVIDIA Jetson Nano which considered as an embedded system. Fig. S.8, we demonstrate measurement for Top-1 and image throughput Intel(R) Xeon(R) E5-2698 v4 CPU. On this device, still
21

---

Published as a conference paper at ICLR 2024
86
85
84
83
Accuracy
80
79
500
1000
1500
2000
2500
3000
Throughput (img/sec)
FasterViT(ours)
EffkcentNewv22
Twins
(2021)
Convttex: (2022)
Swin (20211
beif (2021)
Visformrr (2023)
RegNett 12020)
SwinV2 (2022)
PooFormer (2022)
EehdeenForrer (2022)
MackiT (2022)
FAN (2022)
FextViT (2023)
Nestvll (2022)
Figure S.6: Comparison of image throughput and ImageNet-IK Top-1 accuracy on NVIDIA TITAN RTX GPU for batch size of 128.
observe a dominant performance from different Faster variants. However, two variants from EfficientNetV2 and RegNet' models achieve comparable performance counterpart FasterViT models. In Fig. S.9, we present the throughput and accuracy tradeoff for NVIDIA Jetson Nano. Surprisingly, all Vis variants demonstrate strong performance.
22

---

Published as a conference paper at ICLR 2024
86
84
Accuracy (%6)
83
82
79
500
1000
1500
2000
2500
3000
3500
4000
Throughput (img/sec)
FasterViT(curs) Convtiex: (2022)
EffiientRiiv2 1202
Twins 1202 1)
Crowvit (2:021)
Dell
Visfarmer (2021)
RegNety 12020)
(2022) FastVIT (2023)
(2022)
MaxiiT (2022)
FAN (2022)
NextViT (2022)
Figure S.7: Comparison of image throughput and ImageNet- IK Top-1 accuracy on NVIDIA A6000 GPU for batch size of 64.
86
84
Accuracy (%))
83
82
81
80
20
40
100
120
Throughput (img/sec)
FesterViT(ours) Convieet: (2022)
Efficintreaa 12021)
Twins 12021)
Crossvit (20211
DelT
Vistormer (2021)
RegNetr 12020)
EmneenFormer (2022)
MaxiiT (2022)
FAN (2022)
FastViT (2023)
NextViT (2022)
Figure S.8: Comparison of image throughput and ImageNet- Top-1 accuracy on Intel(R) Xeon(R) 25-2698 v4 CPU for batch size of 128.
23

---

Published as a conference paper at ICLR 2024
86
85
84
Accuracy (%)
82
80
Throughput (img/sec)
(asterViT(ourr)
Swin (2021)
DeiT (2021)
Visfomer (2022)
Regbatt (2:02.)
(2:02)
FastViT (2023)
NextVVT (2022)
EffiientNett2 (2021)
Twins (2023)
CrossvIT (2021)
Comparison of image throughput and ImageNet- Top-1 accuracy NVIDIA Jetson Figure S.9:
Nano for batch size of 1.
24


Reviews:
[{"review_id": "QHfTJqGVug", "reviewer": "Reviewer_B19F", "summary": "This paper proposes a hybrid conv+transformer network, optimizing throughputs for high-resolution image processing.  It proposes a Hierarchical Attention to combine local window tokens and carrier tokens (one per window).  Results are mostly on ImageNet and ImageNet21k, and the results in Figure 1 are quite impressive.", "questions": "1. Could you provide more intuition and insights why simply adding one CT token per window will significantly improve model quality?\n2. Could you show the comparisons of FastViT with and without hierarchical attention for ImageNet, e.g., one is vanilla SwinTransformer-style attention and one is HAT?\n3. In Table 7, could you also add the throughput comparison?  I am curious about the overhead of HAT.\n4. Could you add latency for more hardwares (different GPUs and CPUs)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Intuitive idea:  combing conv + attention is not new, and is often considered more efficient than pure conv or pure transformer for image processing. Window attention is also not new, but the hierarchical window attention is interesting.\n2. Impressive results:  results in Figure 1 are pretty impressive. FastViT outperforms other models by pretty good margin on ImageNet.\n3. Well written paper and easy to follow.", "weaknesses": "1. It is unclear how significant the proposed hierarchical attention (HAT) is.  Table 5 shows this HAT is better than Twins and EdgeViT; however, Table 7 show only marginal gains when comparing to the vanilla SwinTransformer if treating HAT as a plug-and-play module.\n2. The idea of HAT is not well motivated.  Though it shows good empirical results, it is unclear why simply adding a per-window CT token can significantly improve quality.  Would be nice to add a few more ablation studies or insights.\n3. All latencies are measured on A100 GPU. It would be nice to measure the latency on more diverse hardware platforms, such as different kinds of GPUs, and CPUs.\n4. Some image texts (e.g. in Figure 4 and 5) are too small to read. I recommend enlarging the text in these images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sREUNz6nHj", "reviewer": "Reviewer_tsUS", "summary": "This paper proposes a fast ViT architecture using hierarchical attention module (HAT). The HAT module is a modified version of window attention by introducing a new carrier token that summarizes the information of a local window. In this way, the model can preserve a certain level of global information while being more efficient than conventional global attention. With such architectural changes, the proposed FasterViT is able to achieve fast inference on modern GPUs and produces good accuracy on classification and downstreaming tasks.", "questions": "- While the proposed HAT is claimed to be more efficient, only benchmarks on A100 GPUs using a batch 128 are provided. Due to various software and hardware differences, conclusion drawn from the current benchmark may not generalize. For example, it is well known that depthwise conv is more efficient on mobile CPU than on GPU and some ops such as reshape/transpose on CPU is slower due to L2 cache limitation. Even on GPU only, the performance may also vary on different variants. Given that most recent works on efficient ViT have presented benchmark results on multiple platforms - GPU, CPU and DSP, the evaluation from this work is worse and incomplete. I would like to see more benchmarks otherwise the current results are not so convincing.\n- The comparison with existing models is not complete. Although in Figure 1, the proposed model is compared to many recent works in terms of image classification task, the comparison on other downstreaming tasks such as detection and segmentation is quite limited. For example, it is only compared with 3 other models (which are not even designed for efficiency) on the semantic segmentation task.\n- The ablation study only shows that the HAT module can be plugged into other model architecture and improves the performance marginally, but misses analysis on the module itself: how do you determine the architecture for the model variants from 0 to 6? Are there any guidelines or empirical results? How does the latency change with respect to changing the parameters in HAT, such as window size, conv kernel size, pooling size, etc? The current setting seems quite ad-hoc.\n- The parameter count and flops of the proposed model seems on the higher end. Under similar latency, it's much larger than most models. This may prevent it from being adopted in realistic scenarios, where there may be restrictions on model size and memory footprint. How do you aim to resolve this?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is overall well written and organized. The motivation of proposing the HTA module is technically sound - it is a good practice to do more memory intensive operations in early stages while putting the computational intensive operations to the later stage. This is also verified in the experiments, where the proposed model is more GPU friendly compared to existing models.\n- The HAT module is not unnecessarily complex and intuitively easy to implement. It is also shown that it can be a plug-in replacement for conventional attention blocks, which makes it more flexible.", "weaknesses": "The major weakness of this work is experiment. The paper claims that with HAT and all the optimization regarding model architecture, the model has much less complexity compared to conventional attention and it compares the proposed model to a few efficient ViT. However, the comparison is on A100 GPU only and there is no comparison on any other platforms. Some recent works, such as EfficientFormer, FastViT and NextViT compared in Figure 1, all benchmarked their models on different mobile platforms. Therefore, it is unclear whether the proposed change applies similarly to other platforms, which could be very different from A100 GPu due to various software and hardware optimizations. In addition, the ablation study is not sufficient. To really understand the HAT module, a more detailed analysis on it should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QA63Wk2e6j", "reviewer": "Reviewer_Wvdn", "summary": "The paper introduces a novel family of hybrid CNN-ViT neural networks called FasterViT, designed for high image throughput in computer vision applications. FasterViT leverages the strengths of both CNNs and ViT by introducing a Hierarchical Attention (HAT) approach that decomposes global self-attention into a multi-level attention with reduced computational costs. This approach efficiently combines local and global representation learning. FasterViT consists of four stages, reducing input image resolution while doubling feature maps in the early stages and employing transformer blocks in later stages. The proposed HAT mechanism efficiently captures long-range spatial dependencies and cross-window interactions. Extensive validation on various computer vision tasks, including image classification, object detection, and segmentation, demonstrates that FasterViT achieves state-of-the-art performance in terms of accuracy and image throughput, especially for high-resolution images, outperforming competitive models like Swin Transformer V2.", "questions": "- The authors should better explain the difference between the proposed attention and window local approaches line Swin.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- FasterViT is tailored for high-resolution input images and demonstrates faster image throughput compared to competitive models, particularly in handling images with higher resolutions.\n- The proposed HAT approach efficiently decomposes global self-attention into a multi-level attention mechanism, reducing computational complexity and enabling effective local and global representation learning. Overall, the idea of carrier tokens is novel and interesting.\n- The paper extensively validates FasterViT on various computer vision tasks, including image classification, object detection, and semantic segmentation, showcasing its state-of-the-art performance across a wide range of applications.", "weaknesses": "- The comparisons in Table 5 and 7 demonstrate minor improvements in terms of accuracy, while the throughput in Table 5 is reduced when using HAT. \n- Performance is compared on A100 GPUs. More platforms should be used to see if throughput results are consistent. At present results are not conclusive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jb66dyETnt", "reviewer": "Reviewer_oF4Y", "summary": "The paper introduces a new vision transformer architecture, FasterViT, aiming to improve the GPU throughput for vision tasks. The key innovations of the paper include i) using convolutional blocks for the early stages of the model, which are typically memory bound; ii) a new hierarchical attention module (HAT) that interleaves global attention over carrier tokens and self-attention within local windows. Experimental results show that, compared to prior works, FasterViT achieves a better trade-off between accuracy and GPU throughput.", "questions": "N.A.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(S1) [Method] the idea of incorporating convolutional blocks for memory-bound stages is interesting, which is actually similar to the design of MobileViT.\n\n(S2) [Performance] Compared to the latest approaches, FasterViT presents improved throughputs on A100.\n\n(S3) [Ablation] Ablations show that the proposed HAT is a plug-and-play module that can also improve the performance of other models\n\n(S4) [Writing] The paper is easy to follow.", "weaknesses": "(W1) The proposed HAT looks similar to the hierarchical attention proposed in EdgeViT (Pan et al. 2022). While comparisons with EdgeViT is included, it would be beneficial if the author could further clarify the key difference between HAT and the attention block in EdgeViT (maybe just a small paragraph)\n\n(W2) If the reviewer understood correctly, FasterViT incorporated two new designs to improve the throughout i) conv-blocks in memory-bound stages ii) HAT. Clarifying how each element contributes to the throughput would be insightful. Here, the impact of ii) is shown in Table 5, an ablation study for i) would be a valuable addition.\n\n(W3) The authors are encouraged to expand the evaluation to include accelerators beyond A100, such as different GPUs, or even NPUs, so as to provide a broader performance perspective.\n\n\nThe reviewer appreciates the improved performance of FasterViT, but has some concerns about clarity and evaluation. At this moment, the reviewer would like to rate the paper as weak accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)