EfficientFormer: Vision Transformers at MobileNet Speed
Authors: Yanyu Li
Geng Yuan
Yang Wen
Eric Hu
Georgios Evangelidis
Sergey Tulyakov
Yanzhi Wang
Jian Ren
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vision Transformer

Abstract: Vision Transformers (ViT) have shown rapid progress in computer vision tasks, achieving promising results on various benchmarks. 
However, due to the massive number of parameters and model design, e.g., attention mechanism, ViT-based models are generally times slower than lightweight convolutional networks. Therefore, the deployment of ViT for real-time applications is particularly challenging, especially on resource-constrained hardware such as mobile devices. Recent efforts try to reduce the computation complexity of ViT through network architecture search or hybrid design with MobileNet block, yet the inference speed is still unsatisfactory. This leads to an important question: can transformers run as fast as MobileNet while obtaining high performance? To answer this, we first revisit the network architecture and operators used in ViT-based models and identify inefficient designs. Then we introduce a dimension-consistent pure transformer (without MobileNet blocks) as a design paradigm. Finally, we perform latency-driven slimming to get a series of final models dubbed EfficientFormer. Extensive experiments show the superiority of EfficientFormer in performance and speed on mobile devices. Our fastest model, EfficientFormer-L1, achieves $79.2\%$ top-1 accuracy on ImageNet-1K with only $1.6$ ms inference latency on iPhone 12 (compiled with CoreML), which runs as fast as MobileNetV2$\times 1.4$ ($1.6$ ms, $74.7\%$ top-1), and our largest model, EfficientFormer-L7, obtains $83.3\%$ accuracy with only $7.0$ ms latency. Our work proves that properly designed transformers can reach extremely low latency on mobile devices while maintaining high performance. 

EfficientFormer: Vision Transformers at MobileNet Speed
Yanyu Li
Geng Yuan 1,2,1
Yang Wen'
Ju Hu'
Georgios Evangelidis'
Sergey Tulyakov
Yanzhi Wang
Jian Ren
1 Snap Inc.
"Nottheastern University
Abstract
Vision Transformers (ViT) have shown rapid progress in computer vision tasks, achieving promising results on various benchmarks. However, due to the massive number of parameters and model design, e.g, attention mechanism, ViT-based models are generally times slower than lightweight convolutional networks. There- fore, the deployment of VIT for real-time applications is particularly challenging, especially on resource-conse constrained hardware such as mobile devices. Recent efforts try to reduce the computation complexity of VIT through network architecture search or hybrid design with MobileNet block, yet the inference speed still unsatisfactory. This leads an important question: as MobileNet while obtaining high performance? To answer this, we first revisit the network architecture and operators used in ViT-based models and identify inefficient designs. Then we introduce a dimension-consistent pure transformer (without MobileNet blocks) as a design paradigm. Finally, perform latency- driven slimming to get series final models dubbed EffficietFFrmmer Extensive experiments show the superiority of Efficien(Former in performance and speed on mobile devices. Our fastest model, EfficiendFormer-LI. achieves 79.2% top-1 accu- racy on ImageNet-JK JK with only 1.6 inference latency on iPhone 12 (compiled with CoreML), which runs as fast as MobileNetV2x11 (1.6 ms, 74.7% top-1), 1), and our largest model, EfficientFormer-L7, obtains 83.3% accuracy with only 7.0 ms latency. Our work proves that properly designed transformers can reach extremely low latency on mobile devices while maintaining high performancd
transformers
as fast
Introduction
The transformer architecture (1, initially designed for Natural Language Processing (NLP) tasks, introduces the Multi-Head Self Attention (MHSA) mechanism that allows the network to model long-term dependencies and is easy to parallelize. In this context, Dosovitskiy al. 2 adapt the attention mechanism to 2D images and propose Vision Transformer (ViT): the input image divided into non-overlapping patches, and the inter-patch representations MHSA without inductive bias. ViTs demonstrate promising results compared to convolutional neural networks (CNNs) on computer vision tasks. Following this success, several efforts explore the potential of ViT by improving training strategies [3] [] introducing architecture changes [o 7 redesigning attention mechanisms 9. and elevating the performance various vision tasks such
learned through
as classification [0 I 12), segmentation [3 14] and detection [5 [6]
On the downside, transformer models usually times slower than competitive CNNs 17 There are many factors that limit the inference speed of ViT, including the massive number of
'These authors contributed equally.
Code and models are available at http:///hhhlooooo//aa/..//////aa
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

84
ElfideetFormer-177
83
EfficienFoomme-L33
82
EfficientNet-83
LeVIT-256
PoolFormer-S38
81
Conviixer-768
PoolFamer-S24
80
ImageNet- Top-1
79
EfficennFormenL1
ResMLP-S24
Convmixer
Dell-Small
78
Resnet50
77
EfficientNee-BO0
PoolFomer-S12
76
75
MobilaNetV2xx,44
MobileVIT-XS
74
73
72
MobileNetV2
Dell-Tiny
71
2
3
4
10
11
12
13
Latency (ms)
Figure 1: Inference Speed Accuracy. All models are trained ImageNet- and measured by iPhone 12 with CoreMLTools to get latency. Compared to CNNs, EfficientFormer-LI runs 40% faster than EfficientNet-BO. while achieves 2.1% higher accuracy. For the latest Mobile EfficienForrer--.7 runs 0.2 ms faster with 8.5% higher accuracy.
parameters, quadratic- guadratie-inereasing computation complexity with respect token length, non-fot non-foldable normalization layers, and lack of compiler level optimizations (e.g., Winograd for CNN 19].. The high latency makes transformers impractical for real-world applications hardware, such as augmented or virtual reality applications on mobile devices and wearables. As
resource-constraing
result, lightweight CNNs 20
21 22
remain the default choice for real-time inference.
To alleviate the latency bottleneck of transformers, many approaches have been proposed. For instance, some efforts consider designing new architectures operations by changing the linear layers with convolutional layers (CONV) 23 combining self-attention with MobileNet blocks (24] or introducing sparse attention 125
to reduce the computational cost, while other efforts or pruning 29 to improve efficiency. Although the computation-perfomet performance trade-off has been improved by existing works, the fundamental question that relates to the applicability of transformer models remains unanswered: Can powerful vision transformers run at MobileNet speed and become default option for edge applications? This work provides a study towards the answer through the following contributions:
26
leverage network searching algorithm
2
First, we revisit the design principles of VIT and its variants through latency analysis (Sec. B Fol- lowing existing work [ we utilize iPhone 12 as the testbed and publicly available CoreML 30 as the compiler, since the mobile device is widely used and the results can be easily reproduced.
analysis, identify inefficient designs and operators in ViT and propose new dimension-consistee design paradigm for vision transformers (Sec. 4.
Second, based
Third, starting from a supernet with the new design paradigm, we propose simple yet effective latency-driven slimming method to obtain new family of models, namely, EfficientFormers
(Sec. 4.2
We directly optimize for inference speed instead of MACs or number of parameters 31
2
K3
Our fastest model, EfficientFormer-1... achieves 79.2% top-1 1 accuracy on ImageNet- 1K 34 clas- sification task with only 1.6 ms inference time (averaged over 1,000 runs). which runs as fast as VobileNetV22x1.. and wields 4.5% higher top-1 accuracy (more results in Fig. [ and Tab. [ The promising results demonstrate that latency no longer an obstacle for the widespread adoption of vision transformers. Our largest model, FfficientFormerr-L.. achieves 83.3% accuracy with only 7.0 ms latency, outperforms ViTxMobileNet hybrid designs (MobileViT-XS. 74.8%. 7.2ms) by large margin. Additionally, observe superior performance by employing EfficientFormer as

---

the backbone in imge detection and segmentation benchmarks (Tab. We provide a preliminary answer to the aforementioned question, VITs can achieve ultra fast inference speed and wield powerful performance the same time. We hope our EfficientFormer can serve as a strong baseline and inspire followup works on the edge deployment of vision transformers.
2 Related Work
Transformers are initially proposed to handle the learning of long sequences in NLP tasks M. Dosovitskiy et al. (2) and Carion al. [15] adapt the transformer architecture to classification and detection, respectively, and achieve competitive performance against CNN counterparts with stronger training techniques and larger-scale datasets. DeiT 33 further improves the training pipeline with the aid of distillation, eliminating the need for large-scale pretraining [35 Inspired by the competitive performance and global receptive field of transformer models, follow-up works are proposed to refine the architecture 36 377. explore the relationship between CONV nets and ViT Other research efforts explore the essence of attention mechanism and propose insightful variants of token mixer, e.g., local attention [ spatial MLP
38
39
40
and
adapt VIT to different computer vision tasks
45
40
47 48), and pooling-mixer (6)
Despite the success in most vision tasks, ViT-based models cannot compete with the well- lightweight CNNs
21 to when the inference speed the major concern [50] [1]
[2]] especially on
resource-constrrr constraiee edge devices [ To accelerate ViT, many approaches have been introduced
with different methodologies, such as proposing new architectures or modules 5
54
55
56
57
58
re-thinking self-attention and sparse-attention mechanisms [59 60 61 62,
63
64
65
and utilizing
search algorithms that are widely explored in CNNs to find smaller and faster ViTs GO
28
67
design to accelerate vision transformer. However, order to perform MHSA, the 4D features need to be frequently reshaped into flat patches, which is still expensive to compute on edge resources (Fig. 2 Likewise, MobileViT [[8] introduces hybrid architecture that combines lightweight MobileNet blocks (with point-wise and depth-wise CÓNV) and MHSA blocks; the former is placed at early stages in the network pipeline extract low-level features, while the latter is placed in late stages to enjoy the global receptive field. Similar approach has been explored by several works 24 23 as straightforward strategy to reduce computation.
Recently, LeViT [ proposes a
boundary of pure vision transformers instead of relying on hybrid designs, and directly optimize for mobile latency. Through our detailed analysis (Sec. B we propose a new design paradigm (Sec. 4.J). which can be further elevated through architecture search (Sec. 4.2
Different from existing works, we aim at pushing the
3 On-Device Latency Analysis of Vision Transformers
Most existing approaches optimize the inference speed of transformers through computation com- plexity (MACs) or throughput (images/sec) obtained from server GPU 23 28] While such metrics do not reflect the real on-device latency. To have clear understanding of which operations and design choices slow down the inference of ViTs on edge devices, we perform a comprehensive latency analysis over a number of models and operations, as shown in Fig. whereby the following observations are drawn.
Observation 1: Patch embedding with large kernel and stride is a speed bouteneck on mobile devices.
Patch embedding is often implemented with a non-overlapping convolution layer that has large kernel size and stride 3 (55) A common belief is that the computation cost of the patch embedding layer in transformer network is unremarkable or negligible k. However, our comparison in Fig. 2 between models with large kernel and stride for patch embedding, i.e., DeiT-S [ and PoolFormer-S24 [6]. and the models without it, i.e., ViT--22 (3 and EfficientFormer, shows that patch embedding is instead a speed bottleneck mobile devices.
Large-kernel convolutions are not well supported by most compilers and cannot be accelerated through existing algorithms like Winograd [19]. Alternatively, the non-overlapping patch embedding can be replaced by convolution stem with fast downsampling 68 23 that consists of several
hardware-efficient 3x x 3 convolutions (Fig. g
3

---

Patch Embedding Attention a Linear (Proj. MLP)
Reshape Norm Activation Others Total
MobileNe(Y2x1.4
(74.%)
EfficientNet-B0
(77.1%)
ResNet50
(78.5%)
ResMLP-S24
(79.45))
MobileVIT-XS
(74.8%)
DefT-Small
(81.2%)
PoolFormer-S24
(80.3%)
Fffmerr-t
(82.4%)
FfficrentFormer...
(79.2%)
5
7
Latency (ms)
Figure 2: Latency profiling, Results are obtained on iPhone 12 with CoreML. The on-device speed for CNN (MobileNerV2x11. ResNet50, and EfficientNet-BO), ViT-based models (DeiT-Small, LeViT-256, PoolFormer-S24 and FfficientFormer), and various operators are reported. The latency of models and operations denoted with different color. (-) the top-1 accuracy on ImageNet- K. LiT uses HardSwish which is not well supported by CoreML, replace it with GeLU for fair comparison.
Observation 2: Consistent feature dimension is important for the choice of token mixer. MHSA is not necessarily speed bottleneck.
Recent work extends ViT-based models the MetaFormer architecture [6 consisting of MLP blocks and unspecified token mixers. Selecting a token mixer is an essential design choice when building ViT-based models. The options many- the conventional MHSA mixer with global receptive field, more sophisticated shifted window attention [ or a non- non-parametric operator like pooling [
We narrow the comparison to the two token mixers, pooling and MHSA, where we choose the former for its simplicity and efficiency, while the latter for better performance. More complicated token mixers like shifted window S are currently not supported by most public mobile compilers and we leave them outside our scope. Furthermore, we not use depth- wise convolution replace pooling 70 as we focus on building architecture without the aid of lightweight convolutions.
To understand the latency of the two token mixers, we perform the following two comparisons:
we observe that the Reshape operation is bottleneck for LeViT-256. The majority of ViT-255 is implemented with CONV on 4D tensor, requiring frequent reshaping operations when forwarding features into MHSA since the attention has to be performed attention heads). The extensive usage of Reshape limits the speed of LeViT on mobile devices (Fig. 2
First, by comparing PoolFormer-s24 6 and LeViT-256 23
patchified 3D tensor (discarding the extra dimension of
On the other hand, pooling naturally suits the 4D tensor when the network primarily consists of CONV-based implementations. egg., CONV x as MLP implementation and CONV stem for downsampling. As result, PoolFormer exhibits faster inference speed.
Second, by comparing DeiT-Small 3 and Le 23 we find that MHSA does not bring significant overhead on mobiles if the feature dimensions are consistent and Reshape is not required. Though much more computation intensive, DeiT-Small with consistent 3D feature can achieve comparable speed the new ViT variant, i.e., LeViT-256.
In this work, we propose a dimension dimension-connsiseen network (Sec. 4.I with both 4D feature implemen- tation and 3D MHSA, but the inefficient frequent Reshape operations eliminated.
Observation CONV-BN is more latency-favorable than (GN)-Linear and the accuracy draw- back is generally acceptable.

---

HW
NW 32
XX3xHXW
Cx
Bx Cx 8
Bx C3 X
16 x
16a
B
x
Input
CONY stom
Eubedding
Rubedding
Output
MBAD
MB
MB 4D
MB 3D
MB 3D
Stagel Nix
Stage2 N
Stage3 No
N33
Stage4: No
MB 4D
MBAD
bias
(CONY x
Posling
[coN] ::11
Lineor
Lánear
Linear
CeL
Lincar
BN
Figure 3: Overview of EfficientFormer. The network starts with convolution stem as patch embed- ding. followed by MetaBlock (MB). The MB1D and MB3D contain different token mixer configurations, i.e., local pooling or global multi-head self-attention, arranged dimension-consisten manner.
Choosing the MLP implementation is another essential design choice. Usually, one of the two options is selected: layer normalization (LN) with 3D linear projection (proj.) and CONV 1 x with batch normalization (BN). CONV-BN more latency favorable because BN can be folded into the preceding convolution for inference speedup, while dynamic normalizations, such as LN and GN, still collects running statistics the inference phase, thus contributing to latency. From the analysis of DeiT-Small and PoolFormer-S24 in Fig. and previous work [17], the latency introduced by LN constitutes around 10% 20% latency of the whole network.
Based on our ablation study in Appendix Tab, CONV-BN only slightly downgrades performance compared to GN and achieves comparable results to channel-wise LN. this work, we apply CONV- BN as much as possible (in all latent 4D features) for the latency gain with negligible performance drop, while using LN for the features, which aligns with the original MHSA design in ViT and yields better accuracy.
Observation 4: The latency of nonlinearity is hardware and compiler dependent.
Lastly, we study nonlinearity, including GeLU, ReLU, and HardSwish. Previous work 177 suggests GeLU is not efficient hardware and slows down inference. However, we observe GeLU is well supported by iPhone 12 and hardly slower than its counterpart, ReLU, On the contrary, HardSwish surprisingly slow in our experiments and may not be well supported by the compiler (Le ViT-256 latency with HardSwish 44.5 ms while with GeLU 11.9 ms). We conclude that nonlinearity should be determined case-by case-by-case basis given specific hardware and compiler at hand. We believe that most of the activations will be supported in the future. In this work, we employ GeLU activations.
Design of EfficientFormer
Based the latency analysis, we propose the design of EfficientFormer, demonstrated in Fig. 5) The network consists of patch embedding (PatchEmbed) and stack of meta transformer blocks, denoted MB:
y [[ MB, (PatchEeeebb B.H.WWW
where Xo is the input image with batch size as and spatial size as [,, W]. is the desired output, and is the total number of blocks (depth). MB consists of unspecified token mixer (TokenMizer) followed by a MLP block and can be expressed as follows:
X MB, (X,) MLP (IP((ooeeMixer((X)
where XqiE00 is the intermediate feature that forwarded into the 5th MB. We further define Stage (or S) as the stack of several MetaBlocks that processes the features with the same spatial size, such as Ni x in Fig, denoting has N2 MetaBlocks. The network includes Stages. Among each Stage, there
5

---

is an embedding operation to project embedding dimension and downsample token length, denoted as Embedding in Fig. B With the above architecture, EfficientFormer is a fully wransformer-basee model without integrating MobileNet structures. Next, we dive into the details of the network design, specifically, the architecture details and the search algorithm.
4.1 Dimension-Consisten Design
With the observations in Sec. we propose a dimension consistent design which splits the network into a 4D partition where operators are implemented in CONV-net style (MB42), and a 3D partition where linear projections and attentions are performed over 3D tensor to enjoy the global modeling power of MHSA without sacrificing efficiency (MB3D), as shown Fig. 3 Specifically, the network starts with 4D partition, while 3D partition is applied in the last stages. Note that Fig. i just an instance, the actual length of 4D and 3D partition is specified later through architecture search.
First, input images are processed by CONV stem with two 3 x convolutions with stride as patch embedding,
X1 C. PatchEmbed (2,) B.B.H.W...
(3)
where Cj the channel number (width) of the j th stage. Then the network starts with MB4D with simple Pool mixer to extract low level features,
I PooloolA X
W
X; x,
Bf. w
(4)
B,C,, 1 X+1 21- 2FFT = Conv B (Conv (I:)) +I,
W
where Conv B.G refers to whether the convolution is followed by BN and GeLU, respectively. Note here we do not employ Group Layer Normalization (LN) before the Pool mixer in (6), since the 4D partition is CONV-BN based design, thus there exists a BN in front of each Pool mixer.
After processing all the MBAD blocks, we perform a one-time reshaping to transform the features size and enter 3D partition. MB3 follows conventional ViT structure, as Fig. Formally,
B.
HW G
C
HW
I = Linear(MHSA LMMaaaMMMSA(LLLnnear(((((AA
())) + X
(5)
(Linear B. HW C2
II, where Linearg denotes the Linear followed by GeLU, and
X
Q KT VC,
MHSA K, V)= Sof tmax
+ b) V,
(6)
where Q, K, V represents query, key, and values learned by the linear projection, and is parameter- ized attention bias as position encodings.
4.2 Latency Driven Slimming
Design of Supermet. Based on the dimension-cconsitenn design, we build supernet for searching efficient models of the network architecture shown in Fig. 3 (Fig. shows an example of searched final network). In order represent such supernet, we define the MetaPath (MP), which is the collection of possible blocks:
MP M. (MB Tt):
MP MP..j-3,4 (MB .BB 7z).
where / represents identity path, j denotes the jth Stage, and i denotes the 7th block. The supernet can be illustrated by replacing MB in Fig. with MP.
As in Eqn. 7 in S and Sy of the supernet, each block can select from MD or I, while in Sy and Sa, the block can be MB3 MB4D) I. We only enable MB3D in the last two Stages for two reasons. First, since the computation of MHSA grows quadratically with respect to token length, integrating in early Stages would largely increase the computation cost. Second, applying the global MHSA A to the last Stages aligns with the intuition that early stages in the networks capture low-level features, while late layers learn long-term dependencies.

---

Searching Space. Our searching space includes Cj (the width of each Stage), N, (the number of blocks in each Stage, i.e., depth), and last N blocks to apply MB3D.
Searching Algorithm. Previous hardware-aware network searching methods generally rely on hardware deployment of each candidate in search space to obtain the latency, which is time consuming 77. In this work, we propose simple, fast yet effective gradient-based search algorithm to obtain candidate network that just needs to train the supernet for once. The algorithm has three major steps.
First, we train the supernet with Gumbel Softmax sampling blocks within each MP, which can be expressed as
7/2 to get the importance score for the
(007+)) Xi-1 = Σ En C (X.),
(8)
where a evaluates the importance of each block in MP as it represents the probability to select block, the temperature, and
egg., MB4D or MB3D for the 2th block. (O, ensures exploration, represents the type of blocks in MP, i.e. n (4D,1} } for S1 and S2, and n (4D,3D) 1 for S3 and Sq. By using Eqn. [ the derivatives with respect to network weights and a can be computed easily.
The training follows the standard recipe (see Sec. 5.1 to obtain the trained weights and architecture parameter a.
Second, we build a latency lookup table by collecting the on-device latency of MB4D and MB3D with different widths (multiples of 16).
Finally, we perform network slimming on the supernet obtained from the first step through latency evaluation using the lookup table. Note that typical gradient-basee searching algorithm simply select the block with largest a [72), which does not fit our scope as it cannot search the width Cj. In fact, constructing multiple-widhh supernet is coonorrycnnuuiing and even unrealistic given that each MP has several branches in our design. Instead of directly searching on the complex searching space, we perform gradual slimming the single-width supernet follows.
ai and oP for S1,2 and S3,4, respectively. We first define the importance score for MP; as of
Similarly, the importance score for each Stage can be obtained by summing up the scores for all MP within the Stage. With the importance score, we define the action space that includes three options: 1) select I for the least important MP, 2) remove the first MD and 3) reduce the width of the least important Stage (by multiples of 16). Then, we calculate the resulting latency of each action through lookup table, and evaluate the accuracy drop of each action. Lastly, we choose the action based on per-latency accuracy drop (%) This process performed iteratively until target latency is achieved.
We show more details of the algorithm in Appendix.
Experiments and Discussion
We implement EfficientFormer through Py 1.11 [K] and Timm library
72. which is the common practice in recent arts 18 3 Our models are trained on a cluster with NVIDIA A100 and V100 GPUs. The inference speed on iPhone 12 (A14 bionic chip) is measured with iOS version 15 and averaged over 1, 000 runs, with all available computing resources (NPU), or CPU only. CorcMLTools is used to deploy the run-time model. In addition, we provide latency analysis on Nvidia A 100 GPU with batch size 64 to exploit hardware roofline. The trained PyTorch models are deployed in ONNX format and are compiled with TensorRT. We report GPU runtime that excludes preprocessing. We provide the detailed network architecture and more ablation studies in Appendix
5.1 Image Classification
All EfficientFormer models are trained from scratch on ImageNet- 1K dataset 14 to perform the
image classification task. We employ standard image size (224 x 224) for both training and testing. We follow the training recipe from DeiT 3 but mainly report results with 300 training epochs to have the comparison with other ViT-based models. We use AdamW optimizer 75. 76 warm-up training with 5 epochs, and a cosine annealing learning rate schedule. The initial learning rate is set as 10-3 x (batch size/ 10244) and the minimum learning rate is 10-5 The teacher model for distillation
7

---

Table 1: Comparison results on ImgeNett11K, The latency results are tested on iPhone Neural Engine (NPU), iPhone CPU and Nvidia A100 GPU correspondingly. Note that for mobile speed, we report latency per frame, while on A A100 GPU, we report latency per batch size 64 to maximum resource utilization. Hybrid refers unrevealed or unsupported models. fLatency measured with GeLU activation for fair comparison, the original Le ViT-256 model with HardSwish activations runs at 44.5 ms. Different training seeds lead to less than +0.2% fluctuation in accuracy for EfficientFormer, and the enror for latency benchmark is less than +0.1 ms.
mixture MobileNet blocks and ViT blocks. (-) refers to
Latency (ms)
Model
Type
Params(M)
GMACs
Train. Epoch
Top-1(5%)
NPU
CPU
A
MohileNetV2xL.0
CONV
3.5
0.3
300
71.8
1.3
8.0
5.0
MobileNetV2x1.44
CONV
6.1
0.6
300
74.7
1.6
10.7
7.3
ResNet50
CONV
25.5
4.1
300
78.5
3.0
29.4
9.0
EfficientNet-BD
CONV
5,3
0.4
350
77.1
2.7
14.5
10.0
EfficientNet-B3
CONV
12.0
1.8
350
81.6
6.6
52.6
35.0
EfficientNet-B3
CONV
9.9
350
83.6
23.0
258.8
141.0
DeiT-T
Attention
1.2
300/1000
74.5/76.6
9.2
16.7
7.1
DeiT-S
Atlention
22.5
4.5
300/1000
81,2/82.6
11.8
15.5
PVT-Small
Attention
24.5
3.8
300
79.8
24.4
89.5
23.8
12T-ViT-14
Attention
21.5
4.8
310
81.5
21.0
Swin-Tiny
Atlention
29
4.5
300
81.3
22.0
CSwin-T
Attention
23
4.3
300
82.7
28.7
PoolFormer-s12
Pool
12
2.0
300
77.2
6.1
59.0
14.5
PoolFormen-224
Pool
21
3.6
300
80.3
6.2
126.7
28.2
PoolForner-336
Pool
31
5.2
300
81.4
6.7
192.6
41.2
ResMLP-S24
SMLP
30
6.0
300
79.4
7.6
40.2
17.4
Convmixer-768
Hybrid
21.1
20.7
300
80.2
11.6
29.3
LeViT-256
Hybrid
18.9
1.1
1000
81.6
11.9
13.5
4.5
NASVIT-A5
Hybrid
0.76
360
81.8
MobileViT-XS
Iyybid
0.7
74.8
7.2
11.7
MobilcFormer-SSeee
Hybrid
14.0
0.51
450
79.3
13.2
22.2
14.6
EffhcienFormer-L.l
MetaBlock
12.3
1.3
300/1000
79.2/80.2
1.6
11.5
6.2
EffcientFFormrr-Le -3
MetaBlock
31.3
3.9
300
82.4
3.0
28.2
13,9
EffhcientFormer-L7
MetaBlock
82.1
10.2
300
83.3
7.0
67.7
30.7
RegNetY-16GF ir pretrained on ImageNet with 82.9% top-1 accuracy. Results demonstrated in Tab. and Fig.
Comparison to CNNs. Compared with the widely used CNN-based models, EfficientFormer achieves a better trade-off between accuracy and latency. iPhone Neural Engine, EfficientFormer- LI runs at MobileNetV2x11. 1.4 speed while achieving 4.5% higher top-1 accuracy. In addition, EfficientFormer-L33 runs at similar speed EffficientNer-B while achieving relative 5.3% higher top-1 accuracy. For the models with high performance (> 83% l0p-1), EfficientFormer- L7 runs more than 3x faster than EfficientNet-Be B5, demonstrating the advantageous performance of our models. Moreover desktop GPU (A100), EfficientFormer-LI runs 38% faster than EfficientNet-BD while achieving 2.1% higher top-1 accuracy. Efficient Former-L7 runs 4.6x faster than EfficientNet-Bs. These results allow to answer the central question raised earlier; VIT's do not need to sacrifice latency to achieve good performance, and an accurate ViT can still have ultra-fast inference speed as lightweight CNNs do.
Comparison ViTs. Conventional ViTs are still under-performing CNNs terms of latency. For instance, DeiT-Tiny achieves similar accuracy to Efficienttet-BOO while it runs 3.4x slower. However, EfficientFormer performs like other transformer models while running times faster. EfficientFormer. L3 achieves higher accuracy than DeiT-Small (82.4% 81. 81.2%) while being 4x faster. It is notable that though the recent transformer variant, PoolFormer [ naturally has a consistent 4D architecture and runs faster compared to typical ViTs, the absence of global MHSA greatly limits the performance upper-bound. bound. EfficientFormer-L3 achieves 1% higher top-1 accuracy than PoolFormer-Se while being 3x faster Nvidia A100 GPU, 2.2x faster iPhone NPU and 6.8x faster iPhone CPU.
Comparison to Hybrid Designs. Existing hybrid designs, e.g. LeViT-256 and Mobile still struggle with the latency bottleneck of ViTs and can hardly outperform lightweight CNNs. For example, LeViT-256 runs slower than DeiT-Small while having 1% lower top-1 accuracy. For Mobile which is hybrid model with both MHSA and MobileNet blocks, observe that

---

Table 2: Comparison results using EfficientFormer as backbone. Results on object detection & instance segmentation are obtained from COCO 2017. Results on semantic segmentation are obtained from ADE20K.
Detection & Instance Segmentation
Semantic mloU(%)
Backbone
Appoz
AP
AP7a
Apmask
AP 50
AP76
Apmask 76
88
34.0
54.0
36.7
31.2
51.0
32.7
32.9
PoolFormer-S12 11
37.3
59.0
40.1
34.6
55.8
36.9
37.2
EfficienFFormer-Ll
37.9
60.3
41.0
35.4
57.3
37.3
38.9
ResNet50
38.0
58.6
41.4
34.4
53.1
36.7
36.7
PoolFormer- S24
40.1
62.2
43.4
37.0
59.1
39.6
40.3
EfficientFormer--.33
41.4
63.9
44.7
38.1
61.0
40.4
43.5
ResNet101
40.4
61.1
44.2
36.4
57.7
38.8
38.8
PoolFormer- -S36
41.0
63.1
44.8
37.7
60.1
40.0
42.0
FlficientFormer-1.7
42.6
65.1
46.1
39.0
62.2
41.7
45.1
significantly slower than CNN counterparts, e.g., MobileNerV2 and EfficientNer-BO, while the accuracy is not satisfactory either (2.3% lower than EfficientNet-BOX, Thus, simply trading-off MHSA with MobileNet blocks can hardly push forward the Pareto curve, as in Fig. In contrast, EfficientFormer, as pure transformer-based model, can maintain high performance while achieving ultra-fast inference speed. EfficientFormer-Ll has 4.4% higher top-1 accuracy than MobileViT-XS and runs much faster across different hardware and compilers (1.9x faster on Nvidia A100 GPU Computing, 2.3x faster on iPhone CPU, and 4.5x faster on iPhone NPU). At similar inference time, EfficiendFormer--1 outperforms MobileViT-XX by 8.5% top-1 accuracy on ImageNet, demonstrating the superiority of our design.
5.2
ElticientFormer as Backbone
Object Detection and Instance Segmentation. We follow the implementation of Mask-RCNN 77 to integrate EfficientFormer as the backbone and verify performance. We experiment over COCO- 2017 79 which contains training and validations sets of 118K and 5K images, respectively. The EfficientFormer backbone is initialized with ImageNet-IK pretrained weights. Similar to prior work 6
AdamW optimizer 75 76 with initial learning rate of 2 10 and train the model
for 12 epochs. We set the input size as 1333 x 800.
The results for detection and instance segmentation are shown in Tab. FfficientFormers consistently outperform CNN (ResNet) and transformer (PoolFormer) backbones. With similar computation cost, EfficientFormer-13 outperforms ResNet50 backhone by 3.4 box AP and 3.7 mask AP, and outper- forms PoolFormer-S22 backbone with box AP and mask AP, proving that EfficientFormer generalizes well strong backbone in vision tasks.
Semantic Segmentation. We further validate the performance of EfficientFFommer the semantic segmentation task. We use the challenging scene parsing dataset, ADE20K [8] 81 which contains 20K training images and 2K validation ones covering 150 class categories. Similar existing work [ we build EfficientFormer decoder for fair comparison. The backbone is initialized with pretrained weights on ImageNet- K and the model is trained for 40K iterations with a total batch size of 32 over 8 GPUs. We follow the common practice segmentation G [R] use Adam optimizer (75 76 and apply a poly learning rate schedule with power 0.9, starting from a initial learning rate 2 x 10- We resize and crop input images to 512 x 512 for training and shorter side as 512 for testing (on validation set).
backbone along with Semantic FPN [2] as segmentation
As shown in Tab, EfficientFormer consistently outperforms CNN- and transformer-basee backbones by a large margin under a similar computation budget. For example, EfficientFormer-L. outperforms PoolFormer-S22 by 3.2 mloU, We show that with global attention. EfficientFormer learns better long-term dependencies, which beneficial in high-resolution dense prediction tasks.
5.3 Discussion
Relations MetaFormer. The design of FfficientFormer is partly inspired by the MetaFormer concept [6. Compared to PoolFormer, EfficientFormer addresses the dimension mismatch problem,

---

which is root cause of inefficient edge inference, thus being capable of utilizing global MHSA with- out sacrificing speed. Consequently, EfficientFormer exhibits advantageous accuracy performance over PoolFormer. In spite of its fully 4D design, PoolFormer employs inefficient patch embedding and group normalization (Fig. leading to increased latency. Instead, our redesigned 4D partition of EfficientFormer (Fig. is more hardware friendly and exhibits better performance across several tasks.
Limitations. (i) Though most designs in EfficientFormer are seneral-purposed. e.g. dimension- consistent design and 4D block with CONV-BN fusion, the actual speed of EfficientFormer may vary on other platforms. For instance, if GeLU is not well supported while HardSwish is efficiently implemented on specific hardware and compiler. the operator may need to modified accordingly. (ii) The proposed batency-driven slimming is simple and fast. However, better results may be achieved search cost not a concern and an enumeration- based brute search performed.
Conclusion
In this work, we show that Vision Transformer can operate at MobileNet speed on mobile devices. Starting from comprehensive latency analysis, identify inefficient operators in a series of ViT- based architectures, whereby we draw important observations that guide our new design paradigm. The proposed EfficientFormer complies with dimension consistent design that smoothly leverages hardware-fric frieeffriedlly 4D MetaBlocks and powerful 3D MHSA blocks. We further propose a fast latency- driven slimming method to derive optimized configurations based our design space. Extensive experiments on image classification, object detection, and segmentation tasks show that Efficient- Former models outperform existing transformer models while being faster than most competitive CNNs. The latency-driven analysis of ViT architecture and the experimental results validate our claim: powerful vision transformers can achieve ultra-fast inference speed on the edge. Future research will further explore the potential of EfficientFormer on several resource-constrained devices.
Acknowledgment
This work is supported in part by National Science Foundation CCF-1937500.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in newral information processing systems, 30, 2017.
[2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogmition at scale. ICLR, 2021.
[3] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[4] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. arXiv preprint arXiv:2204.00 071111.... 2022.
[5] Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou. Three things everyone should know about vision transformers. arXiv preprint arXiv:2203.0977 2022.
[6] Weihao Yu. Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. arXiv preprint arXiv:2111.11418, 2021.
[7] Lingchen Meng. Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. arXiv preprint 1111:221 2021.
10

---

[8] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[9] Sebastian Jaszezur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojciech Gajew- ski, Henryk Michalewski, and Jonni Kanerva. Sparse is enough in scaling transformers. Advances in Neural Information Processing Systems, 34:9895-9907, 2021.
[10] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint S.Ki::2111.0888, 2021.
[11] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu, Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021.
[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[13] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. arXiv preprint arXiv: 15203, 2021.
[14] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Gird- har. Masked Masked-attention attention mask transformer for universal image segmentation. arXiv preprint arXiv:2112.015277 2021.
[15] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229 Springer, 2020.
[16] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer: Improved multiscale vision transformers for classification and detection. arXiv preprint arXiv:2112.01522 2021.
[17] Xudong Wang, Li Lyna Zhang, Yang Wang, and Mao Yang. Towards efficient vision transformer inference: a first study of transformers mobile devices. In Proceedings of the 23rd Annual International Workshop on Mobile Computing Systems and Applications, pages 1-7, 2022.
[18] Sachin Mchta and Mohammad Rastegari. Mobilevit: Light-weight, general-purposse and mobile-friendly vision transformer. arXiv preprint arX01:2110.02178, 2021.
[19] Xingyu Liu, Jeff Pool, Song Han, and William J Dally. Efficient winose- convolutional
neural networks. arXiv preprint / 06367, 2018
[20] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint /704 2017.
[21] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018
[22] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1314-1324, 2019.
[23] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Herve Jegou, and Matthijs Douze. Levit: A vision transformer in convnet's 's clothing for faster inference. In Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV), pages 12259-12269, October 2021.
[24] Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. arXiv preprint arXiv:2108.05895, 2021.
[25] Chuhan Wu, Fangzhao Wu, Tao Qi, Binxing Jiao, Daxin Jiang, Yongfeng Huang, and Xing Xie. Smart bird: Learnable sparse attention for efficient and effective transformer. arXiv preprint 20Xiv:2108.09193 2021.
11

---

[26] Byungseok Roh, JaoWoong Shin, Wuhyun Shin, and Sachoon Kim. Sparse detr: Efficient end-to-end object detection with learnable sparsity. arXiv preprint mxxi:2111.143300 2021.
[27] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
[28] Chengyue Gong, Dilin Wang, Meng Li, Xinlei Chen, Zhicheng Yan, Yuandong Tian, qiang liu, and Vikas Chandra. NASVit: Neural architecture search for efficient vision transform- ers with gradient conflict aware supemet training. In International Conference on Learning Representations, 2022.
[29] Arnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric Xing. Vision transformer slimming: Multi-dimension searching in continuous optimization space. 2022.
[30] CoreMLTools. Use coremltools to convert models from third-party libraries to core ml., 2021.
[31] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116-131, 2018.
[32] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-awe aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2820-28288 2019.
[33] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-awee aware transformers for efficient natural language processing, arXiv preprint arXiv:2005. 14187, 2020.
[34] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 Icce, 2009.
[35] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 558-567, 2021.
[36] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568-578, 2021.
[37] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Go- ing deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 32-42, 2021.
[38] Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolutional neural networks meet vision transformers. arXiv preprint arXiv:2107.06265 2021.
[39] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-3977, 2021.
[40] Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. On the connection between local attention and dynamic depth-wise convolution. In International Conference on Learning Representations, 2021.
[41] Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, Sercan Arik, and Tomas Pfister. Nested hicrarchical transformer: Towards accurate, data-cfficient and interpretable visual understanding. 2022.
[42] Wenqiang Zhang, Zilong Huang, Guozhong Luo, Tao Chen, Xinggang Wang, Wenyu Liu, Gang Yu, and Chunhua Shen. Topformer: Token pyramid transformer for mobile semantic segmentation, 2022.
[43] Seung Hoon Lee, Seunghyun Lee, and Byung Cheol Song. Vision transformer for small-size datasets. arXiv preprint arXiv:2112. 2021.
12

---

[44] Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. arXiv preprint arXiv:2107.04588 2021.
[45] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
[46] Yanhong Zeng, Huan Yang. Hongyang Chao, Jianbo Wang. and Jianlong Fu, Improving visual quality of image synthesis by a token-based generator with transformers. Advances in Newral Information Processing Systems, 34, 2021.
[47] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin EI-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2105.03444 2021.
[48] Iyya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint
arXiv:2105.01601, 2021.
[49] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114 14. PMLR, 2019.
[50] Uyy O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems, 34, 2021.
[51] Shoufa Chen, Enze Xie, Chongjian Ge, Ding Liang, and Ping Luo. Cyclemlp: A mlp-like architecture for dense prediction. arXiv preprint arXiv:2107.1022: 2021.
[52] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103./1888 2021.
[53] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR. OpenReview.net, 2020.
[54] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi- scale vision transformer for image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 357-366, 2021.
[55] Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili. Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.05704, 2021.
[56] Mohsen Fayyaz, Soroush Abbasi Kouhpayegani, Farnoush Rezaci Jafari, Eric Sommerlade, Hamid Reza Vaezi Joze, Hamed Pirsiavash, and Juergen Gall. Ats: Adaptive token sampling for efficient vision transformers. arXiv preprint 1.Xiv:2111./560 2021.
[57] Wei Li, Xing Wang, Xin Xia, Jie Wu, Xuefeng Xiao, Min Zheng, and Shiping Wen. Sepvit: Separable vision transformer. CORR, abs/2203. 15380, 2022.
[58] Cédric Renggli, André Susano Pinto, Neil Houlsby, Basil Mustafa, Joan Puigcerver, and Carlos Riquelme. Learning to merge tokens in vision transformers. CoRR, abs/2202.1 12015, 2022.
[59] Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Cross- former: A versatile vision transformer hinging on cross-scale attention. arXiv preprint arXiv:2108.00153 2021.
[60] Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon
Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 11936-1 1945, 2021.
[61] Chun-Fu Chen, Rameswar Panda, and Quanfu Fan. Regionvit: Regional-to-local attention for vision transformers. arXiv preprint wcXiv:2/06.02666 2021.
[62] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Lue Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint 2rX70 2021.
13

---

[63] Xiangxiang Chu. Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren. Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting spatial attention design in vision transformers. arXiv e-prints, pages arXiv-2104, 2021.
[64] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynam- icvit: Efficient vision transformers with dynamic token sparsification. In Advances Neural Information Processing Systems (NeurIPS), 2021.
[65]
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and
Yinxiao Li. Maxvit: Multi-axis vision transformer. CoRR, abs/2204.01697, 2022.
[66] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching trans- formers for visual recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12270-12280, 2021.
[67] Qinqin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Xing Sun, Yonghong Tian, Jie Chen, and Rongrong Ji. Training-free free transformer architecture search. arXiv preprint arXiv: 120. 2022.
[68] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22-31, 2021.
[69] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 579-588, 2021.
[70] Asher Trockman and J Zico Kolter. Patches are all you need? arXiv preprint XXi:22001009792 2022.
[71] Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platforn-aware neural network adaptation for mobile applications. In Proceedings of the European Conference on Computer Vision (ECCV), pages 285-300, 2018.
[72] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint DrXiv:I8006 090550 2018
[73] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[74] Ross Wightman. Pytorch image models. pytorch- iea---maaeeeen models 2019.
[75] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6982) 2014.
[76] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[77] Ilija Radosavovic, Raj Prateck Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428- 10436, 2020.
[78] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2966 2017.
[79] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[80] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2017.
[81] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):00--321, 2019.
14

---

[82] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (499,6408,10 2019.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "wgz-vCdFQCG", "reviewer": "Reviewer_mbmg", "summary": "The paper benchmarks the latency of recent efficient vision transformer designs on iPhone 12 through CoreML support and makes observations on the latency vs. accuracy trade-off of different operators. The authors then proposed a super-net search space that bakes in multiple latency-favorable designs and performs gradient-based architecture search followed by a heuristic latency-driven slimming process to obtain sub-net under given latency constraint. The authors evaluate the searched architectures as the backbone across image classification, object detection, and semantic segmentation tasks.", "questions": "Looking at the comparison in Table 1, at a given latency, EfficientViT tends to have a higher parameter count or GMACs (i.e. EfficientViT-L1 vs. MobileNetV2, EfficientViT-L7 vs. MobileViT-XS). From reading the paper, it seems the improvement could come from two sources: 1) better hardware utilization so that a more complex model can run under the same latency constraint (latency analysis guided design space), and 2) when hardware utilization saturates, finding architecture with better compute distribution over operators for higher accuracy (NAS + latency-driven slimming). Is there a way to perform ablation and quantify the performance gain from each source? Understanding the contribution from 1) would be valuable for practitioners optimizing architecture for specific hardware, while understanding 2) will shed light on the value of the proposed design space and search algorithm.\n", "limitations": "The authors clearly addressed some potential limitations of the work: 1) Some observations and subsequent design decisions might be hardware and software dependent; 2) The NAS procedure, specifically the latency-driven slimming procedure is less involved and could be a direction for future exploration.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- *Originality:* To the best of my knowledge, the paper adequately cites related works such as MetaFormer and outlines the differences. While the authors mention that the design space is partly inspired by MetaFormer, the latency vs. accuracy trade-off improvement is novel and largely driven by new insights through latency benchmark on actual hardware.\n- *Quality*: The submission is technically solid in building the EfficientViT design space based on observations from latency analysis on real hardware. The authors clearly addressed two relevant limitations that come to mind when reading the paper: 1) limited insight on how the observations and designs hold across other hardware, and 2) the latency-driven slimming procedure could invite further investigation. Testing the searched architectures across different tasks is a good practice as well.\n- *Clarity*: The submission is clearly written and well organized. The observations from latency analysis naturally build up into respective considerations for the design space.\n- *Significance:* The paper provides comprehensive empirical insights on optimizing vision transformer design space for a given hardware, and achieving MobileNetV2 level latency is meaningful for practitioners. While the latency analysis workflow is general and could serve as a good example of devising design decisions for other hardware, it is not immediately clear how well the insights and the design space will transfer.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MWwxIl5vHOG", "reviewer": "Reviewer_X2kc", "summary": "In this paper, the authors propose a new ViT-based architecture named EfficientViT. In detail, this paper tries to build hybrid designs for MobileNet block and ViT architectures. Finally, this algorithm performs a latency-driven slimming for a series of final models.", "questions": "As shown in the Strengths And Weaknesses.", "limitations": "I do not find any limitations and potential negative societal impact of this work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is easy to read.\n2. The proposed observations in this paper are interesting and useful for the designation of architectures. \n\nWeaknesses:\n1. Many papers focus on the combination of CNNs and Transformers.\n2. The proposed MB blocks have similar architectures to some existing papers.\n3. This paper does not compare with some baseline methods[1][2], and these methods perform better than this paper.\n\n[1] Mobile-Former: Bridging MobileNet and Transformer\n[2] CSWin Transformer: A General Vision Transformer Backbone with Cross-ShapedWindows", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CpBcoV_LZFZ", "reviewer": "Reviewer_RfgL", "summary": "This paper proposes a combination of several implementation details for vision transformers that can be evaluated efficiently (with low latency) on iphones. These details include: \n- Clever management of the shape of the tensors to avoid expensive reshape operations.\n- Using BN instead of LN to save latency by folding in BN parameters into the final model.\n- Using GELU nonlinearity (eg. instead of swish)\nand most importantly a pruning methodology to prune the least essential blocks by training a categorical gating mechanism via gradient descent. \nThe resulting network vastly outperforms previous solutions that have similar latency on both ImageNet classification and semantic segmentation.", "questions": "- Are the dimension-consistent layers relevant in other libraries/hardware, like pytorch, TF or jax on nvidia GPUs? (As the model was trained on  pytorch/GPU), this should not be hard to answer.", "limitations": "Obviously, the paper focuses on a very specific proprietary hardware/software stack, and therefore the applicability of the methods is somewhat limited. The paper does not reference very similar pruning methods presented in 2018 MorphNet paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: Weak-Medium.\nThe paper presents a combination of methods, most of which is well-studied and understood in different contexts:\n- Shape management and alignment has been one of the most optimized aspect of highly tuned numerical linear algebra software for many decades. For example, this has been the focus of optimizing BLAS libraries over many decades.\n- It is well-known that BN has a slight edge over other normalization methods when it comes to latency of the final model, due to the possibility of folding in the normalization parameters.\n- It is surprising that swish is so inefficient on CoreML, but it seems like a special artifact of the current state of the library.\n- While the pruning methodology seems very efficient, it resembles very much to MorphNet [Gordon, Ariel, et al. \"Morphnet: Fast & simple resource-constrained structure learning of deep networks.\" CVPR. 2018], while failing to cite that prior work.\n\nQuality: Medium\nWhile it is impressive that 79% top-1 accuracy on ImageNet is possible within 1.2ms on an iphone, the papers sole focus on this particular library and hardware raises the question of the general applicability of the methods. Also several of the methods just circumvent the special deficiencies of the library (esp swish vs GELU accounts for a large parts of improvements over LeViT 256, which might become competitive with this single change. \n\nClarity: High\nThe paper is well-written with clear ablation analyses and explaining the motivation of all the applied techniques. The methods are evaluated on reasonable benchmarks on both classification and segmentation tasks.\n\nSignificance: Medium\nThis work gives a strong SoTA baseline for vision models on current IPhone hardware. However, it is unclear how/whether these methods generalize to other types of hardware/libraries. It seems unlikely that similar gains could be achieved on android systems, given the difference and maturity of the employed machine learning infrastructure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept