Separable Self-attention for Mobile Vision Transformers
Authors: Sachin Mehta
Mohammad Rastegari
Venue: Accepted by TMLR ()

Abstract: Mobile vision transformers (MobileViT) can achieve state-of-the-art performance across several mobile vision tasks, including classification and detection. Though these models have fewer parameters, they have high latency as compared to convolutional neural network-based models. The main efficiency bottleneck in MobileViT is the multi-headed self-attention (MHA) in transformers, which requires $O(k^2)$ time complexity with respect to the number of tokens (or patches) $k$. Moreover, MHA requires costly operations (e.g., batch-wise matrix multiplication) for computing self-attention, impacting latency on resource-constrained devices. This paper introduces a separable self-attention method with linear complexity, i.e. $O(k)$. A simple yet effective characteristic of the proposed method is that it uses element-wise operations for computing self-attention, making it a good choice for resource-constrained devices. The improved model, MobileViTv2, is state-of-the-art on several mobile vision tasks, including ImageNet object classification and MS-COCO object detection. With about three million parameters, MobileViTv2 achieves a top-1 accuracy of 75.6% on the ImageNet dataset, outperforming MobileViT by about 1% while running $3.2\times$ faster on a mobile device. Our source code is available at: https://github.com/apple/ml-cvnets

Published in Transactions on Machine Learning Research (01/2023)
Separable Self-attention for Mobile Vision Transformers
Sachin Mchta Apple Inc.
Mohammad Rastegari
Apple Inc.
Reviewed OpenReview:
https: https://openrevieu
openreview. net/ forum? id= tBL4yBEyks
Abstract
Mobile vision transformers (MobileViT) can achieve state-of-the-art performance across several mobile vision tasks, including classification and detection. Though these models have fewer parameters, they have high latency as compared to convolutional neural network- models. The main efficiency bottleneck in MobileViT is the multi-headed self-attention (MHA) in transformers, which requires (A2) time complexity with respect to the number of tokens (or patches) k. Moreover, MHA requires costly operations (e.g., batch-wise matrix multiplication) for computing self-altention ion, impacting latency on resorcce-onsttrainee devices. This paper introduces a separable self-attention method with linear complexity, i.e. Okk) simple yet effective charactoristic of the proposed method is that uses element-wise operations for computing self-altention, making good choice for resource-constraie rained devices. The improved model, MobileViTv2, is state-of-the-ar on several mobile vision tasks, including ImageNet object classification and MS-COCO object detection. With about three million parameters, MobileViTv2 achieves a top-1 accuracy of 75.6% on the ImageNet dataset, outperforming MobileVIT by about 1% while running 3.2x faster a mobile device.
Our source code is available at: attps://github. wwt.p//...coaapppp/m/mmaaaaaaaaa
1 Introduction
Vision transformers (ViTs) of Dosovitskiy et al. (2021) have become ubiquitous for a wide variety of visual recognition tasks [Tuvronn et al. 2021 Liu et al. 2021 b, including mobile vision tasks (Mehta & Rastegari 2022]. At the heart of the biseb models, including mobile vision transformers, is the transformer block (Vaswani et al. 20017). The main efficiency bottleneck in ViT-based models, especially for inference on resource-constrained devices, is the multi-headed self-attention (MHA). MHA allows the tokens (or patches) to interact with each other, and is key for learning global representations. However, the complexity of self-attention in transformer block is O(k2), i.e., it is quadratic with respect to the number of tokens (or patches) k. Besides this, computationally expensive operations (e.g., batch-wise matrix multiplication; see Fig. are required to compute attention matrix in MHA. This, in particular, is concerning for deploying ViT-based based models on resource-constrained devices, as these devices have reduced computational capabilities, restrictive memory constraints, and a limited power budget. Therefore, this paper socks to answer this question: can self-allention in tronsformer block be oplimized for resource-conslruined devices?
Several methods (e.g., Child al. 2019 Kitaev et al. 2020 Beltagy et al. 2020 Wang et al. 2020 have been proposed for optimizing the self-attention operation in transformers (not necessarily for VITs). Among these, a widely studied approach in sequence modeling tasks is to introduce sparsity in self-attention layers, wherein each token attends to a subset of tokens in an input sequence CChild et al. 2019 Beltagy et al. 2020). Though these approaches reduces the time complexity from O(k2) to O(kyk) k) or O log k), the cost is a performance drop. Another popular approach for approximating self attention via low-rank approximation. Linformer (Wang et al. 2020) decomposes the self-attention operation into multiple smaller self-attention

---

Published in Transactions on Machine Learning Research (01/2023)
Top-5 operations
Time
Memory
Model
Complexity
atencamm
8.0
2.0 Mb
aten:bmu
1.9
2.5 Mb
Transformer
O(k2)
aten:ssoftinx
354.0 us
2.0 Mb
Linformer
O(k)
aten:copy-
305.0 us
0.0 Mb
Ours
O
aten:add
258.0 us
2.0 MB
Transformer (Total time 12.3 ms)
Theoretical time complexity
Top-5 operations
Time
Memory
aten:mm
7.8
2.0 Mb
aten:bbmm
3.6
3.5 Mb
160
Transformer
atem: aten::soofmex softinax
300.0 us
2.0 Mb
CPU Latency (in ms!
140
Linformer
aten:anul
296.0 us
2.0 Mb
Ours
aten:ciinsu
163.0 us
0.0 Mb
120
Linformer (Total time = 13.8 ms)
100
80
Top-5 operations
Time
Memory
60
aten:amm
6.3 ms
1.5
aten:add
252.0 us
40
fused_relu_me
113.0 us
aten:sum
91.0 us
Kb
20
aten:roul
86.0
512.0 Kb
256
512
768
1024 1280 1536 1792 2048 Number of tokens
Ours (Total time 7.7 ms)
Top-5 operations each layer
CPU latency vs. Tokens
Figure 1: Comparison between different attention units. Transformer and Linformer use costly operations (batch-wire wice matrix multiplication) for computing self-attention. With increase in number of tokens, the cost of computing self-attention (aten: bmm operation) outweighs the cost of linear layers; making it the main bottleneck (as also noted in previous works such as Linformer) for efficient inference on resource- constrained devices. The proposed method does not use such operations, thus accelerating inference on snoource-onssrained devices. Left compares top-5 operations (sorted by CPU time) in a single layer of different attention units for & = 256 tokens. Top Right compares complexity of differont attention units. Bottom Right compares the latency of different atlention units as a function of the number tokens k. These results are computed on single CPU core machine with 2.4 GHz 8- B-Core Intel Core i9 processor, d 512 (token dimensionalityy, h 8 (number of heads; for Transformer and Linformer), and 256 (projected tokens in Linformer) using publicly available profiler in PyTorch Paszke et al. (019)).
82
81
30
Mean Interseccion ever Union
80
Mean Average Precision
6.4%
5.4%
Top-1 accuracy
78
99
3.2 x faster
02
26
(n 9%)
6.7% better
(16
2.4
76
72
22-
70
MobileViTV2
20
MobilcViiV2
Tastor
9ub 9ubkssi
68
14
16
20
80
time ms)
Inference
(a) ImageNet-1k classification
(b) MS-COCO object detection
(c) PASCAL VOC segmentation
Figure 2: MobileViTv2 models are faster and better than MobileViTv1 models of (Mehta
Rastegari 2022) across different tasks. MobileViTv2 models are constructed by replacing multi-headed self-altention in MobileViTvl with the proposed separable self-allention (Section 3.2 Here, inference time is measured on iPhone12 for input resolution of 256 256, 512 512, and 320 320 for classification, segmentation, and detection respectively.
2

---

Published in Transactions on Machine Learning Research (01/2023)
operations via linear projections, and reduces the complexity of self-attention from O(k2) to O(k). However, Linformer still uses costly operations (e-g., batch-wise matrix multiplication; Fig. D for learning global representations in MHA, which may hinder the deployment of these models on resource-conssrained devices.
This paper introduces a novel method, separable self- eeff-attention, with O(k) complexity for addressing the bottlenecks in MHA in transformers. For efficient inference, the proposed self-attention method also replaces the computationally expensive operations (e.g., batch-wise matrix multiplication) in MHA with element-wise operations (e.g., summation and multiplication). Experimental results on standard vision datasets and tasks demonstrates the effectiveness of the proposed method (Fig.
2 Related work
Improving self-attention Improving the efficiency of MHA in transformers is an active area of research. The first. line of research introduces locality to address the computational bottleneck MHA (e.g., Child et al.) 2019 [Beltagy et al.) 2020 Parmar al.) 2018 Qiu et al. 2019 Bello 2021). Instead of attending all & tokens, these methods use predefined patterns to limit the receptive field of self-attention from all k tokens to a subset of tokens, reducing the time complexity from O(k²) to O(kvk) O(k log k). However, such methods suffer from large performance degradation with moderate training/ /aaiii/g///feenceeee speed-up over the standard MHA in transformers. To improve the efficiency of MHA, the second line of research uses similarity measures to group lokens (Kilaev et al. 2020 Vyas el al. 2020 Wang el al. 2021a). For instance, Reformer (Kitaev et al. 2020 uses locality--ensitive hashing to group the tokens and reduces the theoretical self-attention cost from O to O(k log k). However, the efficiency gains over standard MHA are noticoable only for large sequences (k 2048) Kilaev et al., 2020). Because k 1024 in ViTs, these approaches are not suitable for ViTs. The third line of research improves the efficiency of MHA via low-rank approximation (Wang et al.) 2020 Choromanski et al.) 2020)). The main idea is to approximate the self-attention matrix with a low-rank matrix, reducing the computational cost from O(k2) to O(k). Even though these methods speed-up the sell-attention operation significantly, ly, they still use expensive operations for computing attention, which may hinder the deployment of these models on resource-constraine devices (Fig.
In summary, existing methods for improving MHA are limited in their reduction of inference time and memory consumption, especially for resource-constraained devices. This work introduces separable sell-attention method that is fast and memory-efficien (see Fig. which is desirable for resource-constrained devices.
Improving transformer-basee transformers (Liu et al. 2021 Mchta & Rastegari] 2022 Mehta et al., 2021 Wu et al.) 2021 Heo et al. 2021). The majority of these approaches reduce the number of lokens in the translormer block using different
models There has been significant work on improving the efficiency of
methods, including down-sampling (Ryoo et al. 2021 Heo et al.) 2021 and pyramidal structure (Liu et al. 2021 Wang et al. 2021b; MMeta & Rastegari 2022) Because the proposed separable self-attention module is a drop-in replacement to MHA, it can be casily integrated with any transformer-basee model to further improve its efficiency.
Other methods Transformer-basee models performance can be improved using different, methods, including mixed-precision training (Micikevicius et al. 2018), efficient optimizers (Dettmers et al. 2022 Zhai et al.)
2021a), and knowledge distillation (Touvron et al. 2021 Kundu & Sundaresan| (021). These methods are hoggogl to our work, and by default, we use mixedl-precision during training.
3 MobileViTv2
MobileViT of Mehta & Rastegari (2022) is a hybrid network that combines the strengths of CNNs and ViTs. MobileViT views transformers as convolutions, which allows it to leverage the merits of both convolutions (e.g., inductive biases) and transformers (e.g., long-range dependencies) to build a light-weight network for mobile devices. Though MobileViT networks have significantly fewer parameters and deliver better performance as compared to wiiht CNNs (e.g., MobileNets (Sandler et al. 2018 Howard et al. (019)), they have high latency. The main efficiency bottleneck in MobileViT is the multi-headed sell-attention (MHA; Fig. 3a).
3

---

Published in Transactions on Machine Learning Research (01/2023)
Broadccsste element--wise mullipliaaiion
Soltmax
Element-wwise
Dot-preduct
Concaterstioo
d
I
Linear
Linner
ReLL
Sceres
Condect sector
Linwr
(a) MHA in Transformers
(b) MHA in Linformer
(c) Separable self-attention (ours)
Figure 3: Different self-attention units. (a) is a standard multi-headed self-attention (MHA) in transformers. (b) extends MHA in (a) by introducing token projection layers, which project & tokens to a pre-defined number of tokens p, thus reducing the complexity from O(k²) to O(k). However, it still uses costly operations (e.g., batch-wise matrix multiplication) for computing self-attention, impacting latency on resource-constrainne devices (Fig. 1 (c) is the proposed separable self-attention layer that is linear in complexity, i.e. Okt), and uses element-win wise operations for faster inference.
MHA uses scaled dot-product attention to capture the contextual relationships between l tokens (or patches). However, MHA is expensive as it has O(k2) time complexity. This quadratic cost is bottleneck for transformers with large nunber of tokens A (Fig. []. Moreover, MHA uses computationally- and memory- intensive operations (e.g., batch-wise matrix multiplication and softmax for computing attention matrix; Fig. D which could be a bottleneck on resouree-constrained devices. To address the limitations of MHA for efficient inference on resource-constrained devices, this paper introduces sepurable self-atlention with linear complexity (Fig. 3ch.
The main idea of our separable self-attention approach, shown in Fig. 4b is compute context scores with respect to latent token L. These scores are then used to re-weight the input tokens and produce a context vector, which encodes the global information. Because the self-attention is computed with respect to a latent token, the proposed method can reduce the complexity of self- in the transformer by a factor k. A simple yet effective characteristic of the proposed method is that it uses wementtwise operations (c.g., summation and multiplication) for its implementation, making it a good choice for resource- cowurre--onstrainneed devices. We call the proposed attention method separable self-attention because it allows us to encode global information by replacing the quadratic MHA with two separate linear computations. The improved model, MobileViTv2, is obtained by replacing MHA with separable self-attention in MobileViT.
In the rest of this section, we first briefly describe MHA (Section 3.1 and then claborate on the details of separable self-attention (Section 3.2 and MobileViTv2 architecture (Section 3.3
3.1 Overview of multi-headed self-attention
MHA (Fig, 3a) allows transformer to encode inter-token relationships. Specifically, MHA takes an input E k comprising of i d-dimensional token (or patch) embeddings. The input x is then fed to three branches, namely query Q, key XG, and value V. Each branch (Q, KC, and V) is comprised of h linear layers (or heads), which enables the transformer to learn multiple views of the input. The dot-product between the output of linear layers in @ and IC is then compuled simultaneously for all h heads, and is followed by a softmax operation a to produce an attention (or context-mapping matrix a e g Another dot-product is then computed between a and the output of lincar layers in to produce weighted sum output yw E R xh, xh where d, & is the head dimension The outputs of h heads are concatenated to produce a tensor with lt d-dimensional tokens, which is then fed to another linear layer with weights Wo E Radd to
4

---

Published in Transactions on Machine Learning Research (01/2023)
Inner-product between token and j
Token j's values are scaled by context score c,
Input tokoma
Kay tokene
Cv
AL
Compute
Compute
Compuste context
(a) Self-attention in transformers
(b) Proposed separable self-attention method
Figure 4: Example illustrating the interaction between tokens to learn global representations in different attention layers. In (a), each query token computes the distance with all key lokens via dot-product. These distances are then normalized using softmax to produce an attention matrix a, which encodes contextual relationships. In (b), the inner product between input tokens and latent token is computed. The resultant vector is normalized using softmax produce context scores Cs- These context. scores are used to weight key tokens and produce a context vector Cv; which encodes contextual information.
produce the output of MHA E Red, Mathematically, this operation can be described as:
y = Concat
(x xWv ), , (xWa"xxxx') .xvv) Wo
(1)
a"ERExE
where Wq Rdxds, Wk E Rdxds, and Wv' E Rdxds are the weights of the i-th linear layer (or head) in Q, K, and V branches respectively. The symbol (.--) denotes the dot-product operation.
3.2 Separable self-attention
The structure of separable self-attention is inspired by MHA. Similar MHA, the input is processed using three branches, i.e., input I, key K, and value V. The input branch I maps each d-dimensional token in x to scalar using a linear layer with weights Wi E Rd. The weights Wi serves as the latent node L. in Fig. 4b This linear projection is an inner-product operation and computes the distance between latent token L and x, resulting a k-dimensional vector. softmax operation then applied to this k-dimensional vector to produce context scores C R Unlike transformers that compule the atiention (or context) score for each token with respect all tokens, the proposed method only computes the context score with respect to a latent token L. This reduces the cost of computing attention (or context) scores from O(k2) to O
The context scores C are used to compute a context vector Cy- Specifically, the input is linearly projected to d-dimensional space using key branch K. with weights WK E Rdxd to produce an output Xk E R The context vector C2 E Rd is then computed as a weighted sum of Xι as:
C, = (()xx(()
(2)
The context vector cy is analogous to the attention matrix a in Eq. 1 in a sense that it also encodes the information from all tokens in the input x, but is cheap to compute.
The contextual information encoded in cy is shared with all tokens x. To do so, the input x is lincarly projected to d-dimensional space using a value branch y with weights Wy Raddd followed by ReLU
5

---

Published in Transactions on Machine Learning Research (01/2023)
Table Effect of different self-attention methods on the performance of MobileViT on the ImageNet- dataset. Here, all models have similar number of parameters and FLOPs, and latency measured on iPhonc12 (CPU and neural engine (NE)).
Latency
Attention unit
Top-1 1
NE
CPU
Self-attention in Transformer (Fig. 3a.
9.9 ms
78.6 ms
78.4
Self-attention in Linformer (Fig.
3b
10.2 ms
83.2
78.2
Separable self-attention (Ours: gg.
3e
3.4 ms
38.6 ms
78.1
activation to produce an output xv R The contextual information in Cv then propagated to xv via broadcasted dlement-wise multiplication operation. The resultant output is then fed to another linear layer with weights Wo E Raxd to produce the final output E Rexd, Mathematically, separable self-attention can be defined as:
CCER*
y
Σ
(xW1 #X
ReLUU(XW)
Wo
(3)
CvERd
where * and Σ are broadcastable element wise multiplication and summation operations, respectively,
Comparison with self-attention methods Fig. compares the proposed method with Transformer and Linformet"] Because time complexity of self-attention methods do not account for the cost of operations that are used to implement these methods, some of the operations ions may become bottleneck on resource-consl devices. For holistic understanding, module-level latency on single CPU core with varying k is also measured in addition to thcoretical metrics. The proposed separable self-attention is fast and efficient as compared to MHA in Transformer and Linformer.
Besides these module-level results, when we replaced the MHA in the transformer with the proposed self- separable attention in the MobileViT architecture, we observe 3x improvement in inference speed with similar performance on the ImageNet- 1k dataset (Table 1 These results show the efficacy of the proposed separable Transformer and Linformer yields similar
self-attention at the architecture-level. Note that self-attention results for MobileViT. This is because the number of tokens k in MobileViT fewer (k < 1024) as compared to language models, where Linformer is significantly faster than the transformer.
Relationship with additive addition The proposed approach resembles the attention mechanism of Bahdanau et al. (2014), which also encodes the global information by taking a weighted-sum of LSTM outputs at each time step. Unlike Bahdanau al. (2014), where input tokens interact via recurrence, the input tokens in the proposed method interact only with latent token.
3.3 MobileViTv2 architecture
To demonstrate the effectiveness of the proposed separable self-attention on resource-constrainee devices, we integrate separable self-attention with a recent ViT-basod model, MobileViT (Mehta & Rastegari 2022). MobileViT a liigt, mobile-friendly hybrid network that delivers significantly better performance than other competitive CNN-based, transformer-based or hybrid models, including MobileNets (Howard al.)
IIsppired by sampling- based methods (e.g., DynamieViI [Rao al. 2021)) for reducing the computationsl cost of self- attention recent works, also studied the proposed method with different sampling methods (random, uniform, top-k). We observe that these approaches effective in reducing the PLOPs of the model with little drop in performance. However, these sampling-based methods have high lateucy mobile devices compared to the modcls without these methods (sce Appendix P. This likely because these approaches have high memory access cost (because tensor memory re-ordering) and constitutes large portion latency.

---

Published in Transactions on Machine Learning Research (01/2023)
2017 Sandler et al. 2018 Howard et al. in the rest of the paper.
2019 To avoid ambiguity, we refer to MobileViT as MobileViTvi
Specifically, we replace MHA in the transformer block in the MobileViTv1 with the proposed separable
self-attention method. We call the resultant architecture MobileViTv2. We also do not use the soip-ccnnection and fusion block in the MobileViT block (Fig. 1b (Mehta & Rastegari 2022) as it improves the performance
marginally (Fig. 12 in (Mehta & Rastegari 2022 ). Furthermore, create MobileViTv? models at different complexities, we uniformly scale the width of MobileViTv2 network using a width multiplier a E {0.5, 2.0).
This is in contrast to MobileViTvi which trains three specific architectures (XXS, XS, and S) for mobile devices. More details about MobileViTv2's architecture are given in Appendix A
4
Experimental results
4.1 Object classification on the ImageNet dataset
Training on ImageNet-1k from scratch We train MobileViTv2 for 300 epochs with an effective batch size of 1024 images (128 images per GPU x 8 GPUs) using AdamW of Loshchilov & Hutter (2019) on the ImageNet- dataset (Russakovsky et al. 2015) with 1.28 million and 50 thousand training and validation images respectively. We linearly increase the learning rate from 10-6 to 0.002 for the first 20k iterations.
After that, the learning rate is decayed using cosine annealing policy (Loshchilov & Hutter 2017). To
reduce stochastic noise during training, we use exponential moving average
(EMA) (Polyak & Juditsky 1992
as we find it helps larger models. We implement our models using CVNets (Mchta & Rastegari 2022 Mchta
et al. 2022) and use their provided scripts for data processing, training, and evaluation.
Pre-training on ImageNeet2Ik-P and finetuning on ImageNet--lk We train on the Image (winter'21 '21 relcase) that contains about 13 million images across 19k classes. Specifically, we follow Ridnik et al. (2021) to pre-process (e-g., remove classes with fewer samples) the dataset and split it into about 11 million and 522 thousand training and validation images spanning over 10,450 classes, respectively. Following Ridnik et al. (2021). ) we refer to this pre-processed dataset ImageNee-22kkPP. Note that the ImageNet-21k-P validation set does not overlap with the validation and test sets of ImageNet-lk.
We follow Ridnik et al. (2021) for pre-training MobileViTv2 on ImageNot-21k-P. For faster convergence, we initialize MobileViTv2 models with ImageNet- 1k weights and finetune on ImageNNet-21k-) for 80 epochs with an effective batch size of 4096 images (128 images per GPU x 32 GPUs). We do not use any linear warm-up. Other scttings follow ImageNet-1k training.
We finetune ImageNet-21k-P pre-trained models on ImageNet-Ik for 50 epochs using SGD with momentum (0.9) and cosine annealing policy with an effective batch size of 256 images (128 images per GPU x 2 GPUs).
Finetuning at higher resolution MobileViTv2 is a hybrid architecture that combines convolutto and separable self-attention to learn visual representations. Unlike many ViT-based models (e.g. DeiTT), MobileViTv2 does not require adjustment patch embeddings or positional biases for different input resolutions and is simple to finetune. We finetune MobileViTv2 models at higher resolution (i.e., 384x384) for 10 epochs with fixed learning rate of 10 using SGD.
Comparison with existing methods Table 2 and Fig. recent methods. We make following observations:
2 compares MobileViTv2's performance with
When MHA in MobileViTvl is replaced with separable self-attention, the resultant model, MobileViTv2, is faster and better (Fig.) 2 validating the effectiveness of the proposed separable self-attention method for mobile ViTs.
Compared to transformer- based (including hybrid) models, MobileViTv2 models are fast on mobile devices. For example, MobileViTv2 is about 8x faster on a mobile device and delivers 2.5% better performance on the ImageNet- 1k dataset than MobileFormer (Chen et al. (021b), even though MobileFormer is FT.P efficient (R9 vs. R10). However, on GPU, both Fobmee and MobileViTv2 run at similar
7

---

Published in Transactions on Machine Learning Research (01/2023)
Table Classification performance on the ImageNet-1k validation set. Here, NS means that we are not able measure the latency on mobile device as some operations (e.g., cyclic shifts) are not supported on mobile devices. Following Mehta & Rastegari 20222,, latency measured on iPhone12 with a batch size of Similar to Liu et al. (2021) and Liu et al. (2022), throughput is measured NVIDIA V100 GPUs with batch size of 128. The rows are grouped by network parameters.
Neural
Image sizte
Latency (in ms)
Throughput (images/ sec)
Top-1 (in %)
Row #
Model
Type
# Params
FLOPs
search?
data
R1
MobibWIT-XXS
Hybeid
Nome
206
1.3 M
0.4 G
4.8
42225
69.0
KobileVitv2-00.
Hybeid
Nome
256
G
1595
R3
MobeleFormee-522
Hybrid
Nome
224
3.6 M
52 M
7.1
4445
68.7
R4
Kob1leVilV2-100
Hybrid
Nome
256
4.9 M
1.8 G
3.1
2351
78.1
I15
LffiiientNet-h0
CNN
Nome
224
5.3
422
1.6
4619
T7.1
R6
De/T-Thy
Transformce
X
Nome
2249
5.5
1.3 G
3.4
4541
72.2
MobileViT-S
Hybrid
Nome
256
78.4
R8
EffiiennNet-b22
CNN
Nome
288
9.1 M
1.2 G
3.8
2032
80.1
R9
KobileViIv2-1.5
Hybeid
Nome
2562
10.6 M
4.0 G
5.1
11118
80.4
R10
MobllaFormeer224
Hybrid
Nome
224
M
40.7
1402
77.9
R11
Kob1leV1lV2-2.0
Hybrid
X
Nome
256
18.5 M
7.5 G
7.5
1105
81.2
Ilybrid
Nome
R13
ConsNext-T
CNN
None
224
28.6 M
4.5 C
3.7
1800
82.1
R14
DeTT-Base
Tranffrmer
Nome
2242
86.6 M
17.8
13.2
958
81.8
R15
KabileVitv2-20
Hybrid
ImagoNNt-2214
206
18.5
7.5 G
1105
82.4
R16
Con-Nexi-T
CNN
X
ImageNet-21k
2242
28.6 M
4.5 G
3.7
1800
82.9
R17
KobileViTv2-2.0
Hybeid
ImagooNtt22144
18.5
16.1
17.0
488
$3.4
R18
ConvNexi-T
CNN
x
ImageNNt-221
38-2
28.6 M
13.1 G
8.6
645
$1.1
speed. The discrepancy in FLOPs and speed of MobileFormer across devices primarily because of its architectural design. MobileFormeer has conditional operations between mobile and former blocks. Such conditional operations, especially on resource-consee constrained devices, have low degree of parallelism and create memory bottlenecks, resulting in high latency network. observation for CNN-based architectures.
Ma et al. (2018) also makes similar
MobileViTv2 bridges the latency gap between CNN- and ViT-based models on mobile devices while maintaining performance with similar fewer parameters. For example, (Liu et al. 2022) (CNN-based model) is 2x and 3.6x faster than MobileViTv2 (hybrid model) and DeiT trransformer-baseee model) for similar performance respectively (sce R11, R13, and R14). The low latency of fully CNN-based models on mobile devices can be altributed to several device-level optimizations that have been done for CNN-based models over the past few years (e.g., dedicated hardware implementations for convolutions and folding batch normalization with convolutions). ViT-based models still lack such optimizations and therefore, the resultant inference graphs are sub-optimal. Though MobileViTv2 bridges the latency gap between CNNs and ViTs, we believe the latency of ViT-based models will improve in the future with similar optimizations.
mobile device, ConyNexT
The delta. in speed (on GPU) between ConvNext and MobileViTv2 (R15-R18) at higher model complexities reduces from 1.6x to 1.3x when input resolution increased from 224 224 (or 256 256) 384 384, suggesting ViT- (including hybrid) models exhibit better scaling properties as compared to CNNs. This is because of higher degree of parallelism that ViT-based models offer large scale (Dosovitskiy ct al. 2021: Brown et al. 2020). Our results on down-stream tasks in Section 4.2 and previous work on scaling ViT's (Dosovitskiy et al. 2021 Zhai et al. 2021bb further supports this observation.
See Appendix for more results on the task image classification (e.g., Appendix B for MobileViTv2's classification performance on ImageNet-lk/21k-P Appendix C for comparison with light- models, and Appendix E for ablations, including the effect of augmentation, loss functions, multiple latent tokens, and locality-based sampling methods FLOP efficiency).
8

---

Published in Transactions on Machine Learning Research (01/2023)
Table 3: Semantic segmentation results on the ADE20k and the PASCAL VOC 2012 datasets. Here, throughput, network parameters, and FLOPs are measured on the ADE20k dataset for an imput with a spatial resolution of 512 512. mloU (mean intersection over union) score calculated for a single scale only. Throughput is calculated using batch size of 32 images single NVIDIA V100 GPU with 32 GB memory and is an average of over 50 iterations (excluding 10 iterations for warmup). We do not report latency a mobile device some of the operations (e.g., pyramid pooling in PSPNet) are not optimally implemented for mobile devices. The baseline results are from MMSegmentation (2020) Rows are grouped by network parameters.
Seg-
ImageNet-1k
Image
Throughput, f
# Params
FLOPs
mIoU
Model
Backbone
Size
(images/sec)
(in millions)
(in billions)
ADE20k
PASCAL VOC
NobileViTv2-0.5 (Ours)
512²
439
3.6 M
15.4 G
31.8
74.6
PSPNet
MobileNetv2
512
276
13.7 M
53.1 G
29.7
MobileV11v2-1.75 (Ours)
512a
114
22.5
95.9
30.8
80.2
PSPNet
ResNet-50
5122
119
49.1 M
179.1 G
41.1
76.8
MobileV1Tv2-0.77 (Ours)
512²
241
9.6 M
40.0 G
34.7
75.1 (a=0.5)
DeepLabr3
MobileNetv2
512"
246
18.7 M
75.4 G
34.1
Mob1leV1Tv2-2.0 (Ours)
512
90
34.0 M
147.0 G
40.9
80.3 (a=1.5)
DocpLabe3
ResNet-50
512?
103
68.2 M
270.3 G
42.4
79.1
4.2 Evaluation on down-stream tasks
Semantic segmentation We integrate MobileViTve with Iwo standard segmentation architectures, PSPNet. [Zhao et al. 2017) and DeepLabv3 (Chen et al. 2017j, and study on two standard semantic segmentation datascts, ADE20k (Zhou et al. 2017) and PASCAL VOC 2012 (Everingham et al. 2015). For training details supplementary material.
including hyper-parameters,
Table 3 and Fig. 2c compares the segmentation performance in terms of validation mean intersection over union (mIOU) of MobileViTv2 with different segmentation methods. MobileViTv2 delivers competitive performance at different complexities while having significantly fewer parameters and FLOPs. Interestingly, the inference speed of MobileViTv2 models comparable to CNN-based models, including light-weight MobileNetv2 and heivy-weight ResNet-50 (He et al. 2016) model. This is consistent with our observation in Section 4.1 (R17 vs. R18; Table 2) where we also observe that ViT-based models scale better than CNN's higher input resolutions and model complexities.
Object detection We integrate MobileViTv2 with SSDLite of Sandler et al. 2018 (SSD head (Liu et al. 2016) with separable convolutions) for mobile object detection, and study its performance on MS-COCO dataset (Lin et al. 2014). We follow Mehta & Rastegari (2022) for training detection models. Table g and Fig. 2b compares SSDLite's detection performance in terms of validation mean average precision (mAP) using different ImageNet-1k backbones. MobileViTv2 delivers compelitive performance models with different. capacitios, further validating the effectiveness of the proposed self-scparable attention method.
5 Visualizations of self-separable attention scores
Fig. visualizes what the context scores learn different output strides of MobileViTv2 network. We found that separable self-attention layers pay attention to low-, mid-, and high-level features, and allow MobileViTv2 to learn representations from semantically relevant image regions.
6 Robustness analysis of self-separable attention layer
Previous works have shown that vision transformer (ViTs) models generalizes better and robust as compared to CNNs. A natural question arises is: Do separable self-attention layer exhibit similar properties? To study the robustness of MobileViTv2, we evaluate the performance of MobileViTv2 on the ImageNetv2 dataset.
2Qutput stride is the ratio of the spatial dimension of the imput to the feature map.

---

Published in Transactions on Machine Learning Research (01/2023)
Table 4: Object detection using SSDLite on the MS-COCO dataset. Here, throughput measured with a batch of 128 images on the NVIDIA V100 GPU, and an average over 50 iterations (excluding 10 iterations for warmup). Latency on mobile device not reported as some operations (e.g., hard swish) are not optimally implemented for such devices. Rows are grouped by network parameters.
ImageNet--k
Image Size
Throughput (images/ser sse))
# Params (in millions)
FLOPs + (in billions)
mAP
backbone
MobileViTv1-XXS
320
2246
1.7 M
0.9 G
19.9
MobileViTv2-0.5 (Our)
320
2782
G
MobileViTv2-0.75 (Ours)
320
1876
3.6 M
1.8 G
24.6
Mobilenetv2
320
3052
4.3 M
0.8 G
22.1
MobileNetv3
320
3884
0.6 G
MobileNetvl
320
4330
5.1 M
1.3 G
22.2
MobileViTv2-1.75
320
780
14.9 M
9.0 G
29.5
ResNet-50
300
744
22.9 M
20.2
Figure 5: Context score maps at different output strides (OS) of MobileViTv2 model. Observe how context scores pay attention to semantically relevant image regions. (Left to right: input image, context scores at OS=8, context scores O$=16, and context scores OS=32). For more examples and details about context score map generation, see Appendix
Model
# Params +
ImageNet Top-1 t
ImageNetv2 Top-1 f
DeiT-T
5 M
72.2
60.4
MobileViT-S
5.5
78.4
66.5
MobileVFTv2-1.5
4.9 M
78.1
66.4
DeiT-S
22 M
79.8
68.5
MobileVrTv2-2.0
18.5
81.2
70.0
Table 5: MobileViTv2 models are robust and generalizes to harder images.
(matched frequency split) (Recht et al. 2019). Table shows that MobileViTv2 models exhibit similar robustness properties to transformer- based models (DeiTT and MobileViT) and generalizes harder images.
7 Conclusions
Transformer-basee vision models are slow on mobile devices as compared CNN-basod models because multi- heati- self-attention is expensive on resource-constrained devices. In this paper, we introduce a separable self-allention method that has linear complexity and can be implemented using hardware-friendll
10

---

Published in Transactions on Machine Learning Research (01/2023)
element-wise operations. Experimental results standard datasets and tasks demonstrate the effectiveness of the proposed method over multi-headed self-attention.
In this paper, we do not use methods (e.g,, distillation and neural architecture search), which have found to be effective in improving the accuracy and efficiency of recent efficient vision transformer models (Li et al. 2022). In the future, we plan to study such methods further improve the accuracy and efficiency of MobileViTv2 models.
Acknowledgements
We are grateful Ali Farhadi, Peter Zatloukal, Oncel Tuzel, Rick Chang, Fartash Faghri, Farzad Abdolhosseini, Lailin Chen, and Max Horton for their helpful comments. We are also thankful to Apple's infrastructure and open-source teams for their help with training infrastructure and open-source release of the code and pre-trained models.
References
Damitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.04 04..0,77, 2014.
Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. T5w:2102.08603 2021.
arXiv preprint
Tz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 05515:2004001150, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 18:1877-1901, 2020.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Cross Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021a.
Liang-Cuun Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image sogmentation. arXiv preprint Xrv: 10X.:
706. 055807 2017.
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. ar preprinl ar Xrxi2:2008. 05895, 2021b.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, arXiv preprint arXie:1904.10500 2019.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. Xiv preprint (ete22009.4499, 2020.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with a reduced scarch space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognilion Workshops, pp- 702-703, 2020.
Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent. Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint (rXim:2103.10699 2021.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. In International Conference on Learning Representattons, 2022. URL http::///penreviiennntt/// id=shpkpVXzo3h
11

---

Published in Transactions on Machine Learning Research (01/2023)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Usækoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL httpss//opeereviien http://wwww..ooorooooooooaaaaaaa
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- linear units for neural network function approximation in reinforcement learning. Newral Networks, 107:3-11, 2018.
Mark Everingham, SM Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer viston, 111 (1):98-136) 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognátion, pp. 770-778, 2016.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 11936 11945, 2021.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenetv3. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 1314-1324, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arxiv preprint or Xiv:1704.04880 2017.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint
arX4v:2001.04451, 2020.
Souvik Kundu and Sairam Sundaresan. Attentionlite: Towards efficient self-attention models for vision. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2225-2220 IEEE, 2021.
Yanyu Li, Geng Yuan, Yang Wen, Eric Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. ar preprint arXiv:2206.0119) 2022.
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool, Localvit: Bringing locality vision transformers. arXiv preprint arXiv:2104.05700 0570077 2021.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Europeon conference on computer vision, pp. 740-755. Springer, 2014.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexan- der C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012 10022, 2021.
Zhuang Lin, Hanzi Mao, Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnct for the 2020s. orXiv preprinl arxim:2201.020 0.3545, 2022.
Iya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httppp///ppppaeeeee.eee////r//ts
12

---

Published in Transactions on Machine Learning Research (01/2023)
Ningning Ma, Xiangyu Zhang. Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the Puropean conference on computer vision (ECCV), pp- 116-131, 2018.
Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer. In International Conference Learning Representations, 2022. URL https:/
bet/forumiid OstUUUS
Sachin Mehta, Mohammad Rastegari, Linda Shapiro, and Hannaneh Hajishirzi. Espnetv2: A light-weight, power efficient, and general purpose convolutional neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 9190-9200, 2019.
Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Delight:
Deep and light- transformer. In International Conference on Learning Representations, 2021. URL
Sachin Mchta, Farzad Abdolhosscini, and Mohammad Rastegari. Cynots: High performance library for computer vision. CoRR, 2022.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston; Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training.
In International Conference on Learning Representations, 2018. URL https://openreeveer. htt.:/nrrreeeee..e//ruurt.t id=rlga9JgRZ
Contributors MMSSegmenattio.. MMSegmennattin:: Openmmlab semantic segmentation toolbox and bench- mark. https: h/ap:///oogaaammaaa///aaaaaaaeea
2020.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pp. 4055 4064. PMLR, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Boris T Polyak and Anatoli B Juditsky, Acceleration of stochastic approximation by averaging. SIAM journal control and optimization, 30(4):838 855, 1992.
Jiezhong Qiu, Hao Ma, Omer Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang. Blockwise self-attention for long document understanding. ar preprint Xiw:1191100297. 2019.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt,, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 1389-5400. PMLR, 2019.
Tal Ridnik, Emanuel Ben Ben-Bar Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.11997 2021.
Olga Russakovsky, Jía Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-22.. 2015.
Michael Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: Adaptive spaco-time tokenization for videos. Advances in Neural Information Processing Systems, 34, 2021.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenctv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 4510-4520, 45220 2018.
13

---

Published in Transactions on Machine Learning Research (01/2023)
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedinggs of the IEFF conference on computer vision and pattern recognition, pp. 1 9, 2015.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficicnt image transformers & distillation through attention. In International Conference Muchine Learning, pp. 00347-10357, PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in nearol information processing systems, 30, 2017.
Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. Advances Newral Information Processing Systems, 33:21665-21674, 2020.
Shuohang Wang, Luowei Zhou, Zhhe Gan, Yen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng, and Jingjing Liu. Cluster-former: Clustering- based sparse transformer for question answering. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp.
2021a.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXie preprint Xrx:220006 04768, 2020.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 568 578, 2021b.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arxiv preprint arXiv:2110.000 004006000.76.0 2021.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang, Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22-31, 2021.
Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-60000000 00036603,2222 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2021a. URL h/RL://ttg///arr...rr///////..//
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. CoRR, 2021b.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. orXiv preprint arXio: 1rXiv::111000441,, 2017.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference compuler vision and pallern recognition, pp. 2881 2890, 2017.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. Proceedings of the AAAI conference on artificial intelligence, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633 641, 2017.
14

---

Published in Transactions on Machine Learning Research (01/2023)
A Detailed architecture of MobileViTv2
MobileViTv2's architecture follows MobileViTv1 (Mehta & Rastegari 2022) and is given in Table G
MobileViTv2 block, shown in Fig. makes two changes to the MobileViTvi block: (1) it replaces the
multi-headed self-attention with the proposed separable self-attention to learn global representations and (2)
it does use fusion block and skip-connection (see Fig. 1b in Mehta & Rastegari
(2022)) as they improve the performance marginally (see Fig. 12 in Mehta & Rastegari 2022 ). The expansion factor in MobileNetv2
(Sandler et al. 2018 blocks and feed-forward layers is two. Similar to Mehta & Rastegari (2022), use Swish [llwing el al. 2018) as a non-linear activation function. Unlike MobileViTvl that creates three specific architectures (XXS, XS, and S) for mobile devices, we uniformly scale the width of MobileViTv2 network using a width multiplior a E 0.5, 2.0 to create models at different complexitics.
B MobileViTv2's classification performance
ImageNet-1k Table 7 shows the results of MobileViTv2 on the ImageNet-1k dataset. Finetuming MobileViTv2 models at higher resolution (384 x 384) shows improvement across the board. For exam- ple, the performance of MobileViTv2-0.50 with 1.4 million parameters improves by about 2% when finetuned at higher resolution (R1 vs. R2). Similarly, pre-training on the ImageNet-211s dataset helps improve the performance of MobileViTv2 models. For example, ImageNet-21k-P pretraining improves the performance of MobileViTv2-2.0 improves by 1.2% (R17 vs. R18). Notably, MobileViTv2 models pretrained on the
Bx
X Depth-wise
Point-wise
Unfold
Separuble sel--ateentio
Feed-forwat network
Fold
Point-wise
y
Figure 6: MobileViTv2 block. Here, depth-wise convolution uses a kernel size of 3 x 3 to encode local representations. Similar to MobileViTvl, unfolding and folding operations uses patch height and width of two respectively. The separable sell-attention and feed-forward layers repeated Bx before applying the folding operation.
Table 6: MobileViTv2 architecture. Here, d represents dimensionality of the input to the separable self-attention layer, B denotes the repetition of transformer block with separable self-attention inside the MobileViTv2 block (Fig. 6), and MV2 indicates MobileNetv2 block. Similar to MobileViTvi block, we set kernel size as three and spatial dimensions patch (height h and width w) as two the MobileViTv2 block.
Layer
Output size
Output stride
Repeat
Output channels
Image
256 256
Conv-3 x 3, 422 MV2
1
32a
128 x 128
2
1
64a
MV2, 22 MV2
1
128a
64 x 64
4
2
128a
MV2, 22 MobileViTv2 block (Fig.
1
256a
32 x 32
8
2)
1
256 (d = 128a)
MV2, 12
1
384a
16 x 16
16
MobileViTv2 block (Fig.
1
384a (d 192a)
MV2, 12
1
512a
MobileViTv2 block (Fig.
3)
1
512a (d 256a)
Global pool
512x
1 x
256
Linear
1000
15

---

Published in Transactions on Machine Learning Research (01/2023)
Table 7: Classification performance of MobileViTv2 on the ImageNNet-1k dataset. Here,
indicates
finetuning at higher resolution.
Row #
Model
Image size
Extra data
# Params
FLOPs
Top-1
R1
Nob1leW11Vv-0050
256
None
14 M
0.5 G
70.2
R2
Mob1leW1Tv2-0.60
384
None
141 M
1.0 G
72.1
R3
HobileViTv2-0.75
256
Nome
2.9 M
1.0 G
75.6
R4
MobileV11VV.00771
3842
None
2.9 M
2.3 G
77.0
R5
Mob:leWiTTv-11.0
256
None
4.9 M
1.8
78.1
R6
MobileV1TVV-LL00
3841
None
4.9 M
4.1 G
79.7
R7
MobileViTvv-1122
256
None
7.5 M
2.8 G
79.6
HobileViTvV-11.
3842
None
7.5 M
6.3
80.9
MobileViIV2-11.6
Noue
10.6 M
4.0
80.4
RIO
Mob1leV1Tv2-1.33
256
ImageeNet-2Ik-P
M
81.5
RI1
Hob1leV1TVV-1..2 550
3841
None
10.6 M
9.1
81.5
I112
Mob:leV::Vv.1155/
384
ImageNeet-21k.P
10.6 M
9.1 G
82.6
R13
MobileViTv2-1ne 1.75
256
Nome
14.3 M
5.5 G
80.8
R14
Mob:leWiTv2:1.7
256
ImageNet-211a.
14.3 M
5.5 G
81.9
R15
MobileWi1vv211.77
3841
Nome
14.3 M
12.3 G
82.0
R16
MobileViTv2 1.75
ImageNNt-2114 Nek-2
14.3 M
12.3
82.9
117
Mob:leWi1Vv22.000
256
None
18.5 M
7.2 G
81.2
RIS
Mob1leV11V2-1.77
ImageNNet-2Ik-P
M
7.2
82.4
R19
Mob1leV1Tv2-2200 DOT
3841
None
18.5 M
16.1
82.2
R20
Mob:le/1:Vv.11.00
384
ImageNeet-21..P
18.5 M
16.1
83.4
Table 8: Performance of MobileViTv2 on the ImageNet-2ll4 21k- validation set.
Width factor
Params
PLOPs
TTpp- f Top-5
1.50
17.9 31
4...
41.5
1.25
55.
45.8
2,00
28.1 M
720
46.
76.6
ImageNNe-21k-P are able to achieve the similar performance with fewer FLOPs to models finetuned on ImageNet-1k 1k with a higher resolution (e.g., R10 vs. R11; R14 vs. R15; R18 vs. R19 Table
ImageNNet-21k-P Table Table@shoree shows the results on the ImageNet-21k-PP validation datasot. The performance of MobileViTvv improves with increase in model size.
c Comparisons with light- networks the ImageNet-1k dataset
Comparison with lighh- CNNs. Fig.
7a shows that MobileViTv2 outperforms light-weight CNNs
across different network sizes (MobileNetvi HHoward et al. 2017). MobilcNetv2 (Sandler et al.) 2018
ShuffleNetv2 (Ma et al. 2018 ESPNetv2 (eehta et al. 2019 and MobileNetv3 (Howard et al. 2019))
Comparison with lighh- VITs.
Fig. 7b shows that MobileViTv2 achieves better performance
than previous light-weight ViT-basod models acorss different network sizes (DeIT (Touvron al. 2021
T2T Yuan et al. (021), CiossViT (Chen al. 2021a LocalViT (Li et al. 2021 ConViT (d'Ascoli el. al.
2021
and Mobile-former (Chen al. 2021b)
D
Visualizations of separable self-attention scores
The MobileViTv2 block, Fig. 6 unfolds the input x E Rd to obtain E RdxMXx where N Hw are the number of patches, each patch with width w and height h (M- hw pixels per patch). This unfolded feature map is fed to separable self-attention module to learn non-local representations. To better understand how separable self-attention processes Xu, we visualize context scores Cs.
kw
The separable self-attention in MobileViTv2 block computes context scores C2 for M pixels simultancously across N patches. Therefore, Cy has dimensions of M x N. To visualize context scores, fold cx E M to
16

---

Published in Transactions on Machine Learning Research (01/2023)
80-
78
75
Top-1 accuracy (96)
arcuracy
74
72
Mobllenebyl
70
Mobiishettz
Mobte-forerr
Noblichette
CrossVI
Shulllelteer
ESPNett2
66
121
LOCaMT
10ur))
64
MobleviTVV
2
10
Parameters (in million)
Parameters (in million)
(a) Comparison with light-weight CNNs
(b) Comparison with light-weight ViTs
Figure 7: Comparison with light-weight CNN- and ViT-based models. MobileViTv2 is smaller and better, which desirable for mobile devices.
the same spatial dimensions as the input and obtain context score map Cm E HWxW For ease of visualization, we scale Cm using min-max normalization.
The context score maps for different input images at different output strides of MobileViTv2 model are shown in Fig. S These visualizations show that the proposed separable self-attention method is able to (1) aggregate information from entire image under different scttings, including complex backgrounds, illumination & view-point changes, and different objects, and (2) learn high-, mid-, and low-level representations.
E MobileWiTv2's ablation studies on the ImageNet-1k dataset
In this section, we study the effect on different methods on the performance of MobileViTv2 models, including augmentation methods.
Standard advanced augmentation We study two different augmentation methods: (1) standard augmentation that uses Inception-style augmentation Szegedy al. (2015) i.e., random resized cropping and horizontal flipping and (2) advanced augmentation that uses Rand (Cubuk et al. 2020), CutMix Yun et. al. 2019), MixUp (Zhang et al. 2017). and RandomErase (Zhong et al. 2020) along with standard augmentation methods. The effect these augmentations on the performance of MobileViTv2 is shown in Figure g Smaller models (< 4.5 million paramcters) benefit from standard augmentation while larger models (2 4.5 million parameters) benefit from advanced angmentation. For simplicity, we use advanced augmentation for all variants of MobileViTv2 in this paper.
Loss functions CutMix and Mixup augmentations mixes the samples in a batch. As a result, each sample has multiple labels. Therefore, in presence of these augmentations, ImageNet classification can be thought multi-label classification task. Similar to Wightman el. al. (2021), we trained MobileViTv2 by minimizing binary cross-entropy loss. Unlike Wightman al. (2021), we did not observe any improvements in the performance when cross-entropy loss with label smoothing is replaced with binary cross-entropy loss. Therefore, we use cross-entropy with label smoothing for training MobileViTv2 models.
Effect of multiple latent tokens Similar to multi-head attention in transformers, the proposed separable self-attention can have multiple latent tokens. When we changed the number of latent tokens from 1 to 8, the performance improvements on the ImageNet-1k dataset were negligible (within +0.1 top-1 accuracy). Therefore, we use only one latent token in our experiments.
We note that changing the number heads from to in multi headed self-attention in the transformer block of the MobileV/Tv1-S architecture dropped the top-1 accuracy by 0.7%. This observation similar Vaswani al. (2017), who also found that multiple heads in multi-headed self-attention improve transformers performance the task of neural machine translation.
17

---

Published in Transactions on Machine Learning Research (01/2023)
Improving FLOP-efficieccy via pixel- and patch-sampling The MobileViTvl model Mchta & Rastegari (2022) unfolds an input feature map into N patches, each patch with M = hw pixels and applies a transformer block for each pixel in a patch independently, where h and w are patch's 's height and width respectively. Because pixcls in a patch are spatially correlated, one can sub-sample m pixels from M pixels and learn non-local representations by applying sell-altention layers on m pixels only. Such sub-sampling methods should help reducing model FL.OPs.
We tried following sampling methods at pixel- well as patch-level:
Random sampling, wherein m pixels (or n patches) from M pixels (or N patches) are randomly selected during training and uniformly during validation.
Top-m (or top-n) sampling, wherein top-m pixels (or top-n patches) are selected based on their magnitude computed using L2 norm.
Uniform sampling, wherein pixels (or patches) are sampled uniformly from M pixels (or N patches).
We found that these methods can reduce the FLOPSs by 1.2x to 1.6x with little or no drop top-1 accuracy on the ImageNet-1k dataset for both MobileViTvl (with multi-headed self-attention) and MobileViTv2 (with the proposed separable self-attention) models. However, these improvements in FLOPs did not translate to mobile device. In fact, models with these sampling methods were significantly
latency improvements slower than the models without these methods. The high-latency of models with these sampling methods on mobile devices can be attributed to their high memory access cost, as these methods change the memory order of lensor. Because their high-latency on mobile devices, we did not use these methods in the MobileViTv2 model.
F MobileViTv2 training configurations
Configurations for training and finctuning MobileViTv2-2.0 on the ImageNct-1k and ImageNet-21k-P datasots are given in Table and Table I0 respectively while configurations for finetuning MobileViTv2 on downstream tasks are given in Table [1
18

---

Published in Transactions on Machine Learning Research (01/2023)
Training config
MobileViTvv-2.0
Dataset
ImageNet-1lk
ImageNet-21k-P
# Training samples
1.28 M
11 M
# Validation samples
50 k
523 k
Train resolution
256 x 256
256 x 256
Val resolution
256 x 256
256 256
RandAug
CutMix
MixUp
Random resized crop
Random horizontal flip
Random crase
Stochastic depth
X
X
Label smoothing
/
V
Loss
CE
CR
Optimizer
AdamW
AdamW
Weight decay
0.05
0.05
Schoduler
Cosine
Cosine
Warm-up iterations
20 k le
None
Warm-up init LRR
None
Warm-up scheduler
Linear
None
Base LR
0.002
0.0003
Epochs
300
80
Batch size
1024 X
4096
Layer-wise LR decay
X
Grad. clip
10
Exp. moving average
/
/
Weight init
Random
ImageNet-1k
Table 9: Configuration for training Mob1leViTv2-220 on the ImageNet-1ke 1k/22kk datascts. The validation
set in ImageNel-21k- does not overlap with ImageNet-1k validation set, and is created following Ridnik et al. 2021
19

---

Published in Transactions on Machine Learning Research (01/2023)
Input Image
Input Image
Context score map an output stride of
Context score map at an output stride of
Context score at output stride of 16
Concext score
output stride of 16
f
Context map oulput stride 32
Context map Cm output stride of
Input Image
Input Image
Context map Cm output stride of
Context map output stride of
Context score map e. oulput stride
map ep output stride
Context score map C. at am output stride
Context acore map em at an output stride
(d)
Figure 8: Layer-wise visualization of context score maps Cm different output strides. Recall that MobileViTv2 (Fig. 0 and Table 6) applies B = B = 4, and B 3 separable self-attention layers at an oulput strides of 8, 16, and 32 respectively. Therefore, we have 2, 4, and 8 context score maps al an outpul stride of 8, 16, and 32 respectively
20

---

Published in Transactions on Machine Learning Research (01/2023)
82
80
Top-1 accuracy (In %)
78-
76
72
Standard
70
0.5
2.5
4.5 6.5
12.5 14.5 16.5 18.5 Parameters (in millions)
8.5
10.5
Figure 9: Impact of data augmentation on the performance of MobileViTv2 models on the ImageNet-1k dataset. For smaller models (< 4.5 million parameters), standard augmentation works best while larger models (2 4.5 million parameters) benefit from advanced augmentation.
Training config
MobileViTV2-2.00
Dataset
ImageNet-lk
ImageNet-1k
ImageNet-1k
4 Training samples
1.28 M
1.28 M
1.28 M
# Validation samples
50 k
50 k
Train resolution
384 x 384
256 x 256
384 x 384
Val resolution
384 384
256 256
384 x 384
Weight init
ImageNet-1k
ImageNet-21kkP
ImmgeNNet-21k-P-1e
RandAug
CutMix
MixUp
Random resized crop
Random horizontal flip
Random erase
Stochastic depth
X
X
X
Label smoothing
/
/
/
Loss
CE
CE
CE
Optimizer
SGD
SGD
SGD
Weight decay
-5
4e
-5
Scheduler
Fixed
Cosine
Fixed
Warm-up iterations
None
None
None
Warm-up init LRR
None
None
None
Warm-up scheduler
None
None
None
Base LR
0.001
0.01
0.001
Epochs
10
50
10
Batch size
256
128
Layer-wise LR decay
X
X
X
Grad. clip
10
10
10
Exp. moving average
/
/
Table 10: Configuration for finetuning MobileViTv2-2.0 on the ImageNet- 1k dataset. Here, denotes that the ImageNet-21ke. model finetuned on the ImageNet-1k 1k dataset at 256 256 image resolution is used for initializing the weights.
21

---

Published in Transactions on Machine Learning Research (01/2023)
Training config
SSDLLit-Nob:11VVIIV2..777
DeepLabx3-Moob1oottv22.....
DeppLabs3HtooieevsTv222
Dataset
MS-COCO
ADR20k
PASCAL VOC 2012
Extra Data
None
Noue Segmentation
COCO Segmentation 128 k
Task
Detection
Training samples
117 k
20 k
Validation samples
1.45 k 512 x 512
Train resolution
320 x 320
512 512
Val resolution
320 320
Shortest side 512
Shortest side 512 ImageNet-1k
Weight init
ImageNNt-111
ImageNet-11
SSD Cropping
Photometric distortion
Random horizontal flip
Resize
Random short size resize
Random Crop
Random Gaussian blur
Random rotation
Loss
Smooth L1 + CE
CE
CE
Optimizer
AdamW
SGD
AdamW
Weight decay
0.05
1e-4
0.05
Scheduler
Cosine
Cosine
Cosine
Warm-up iterations
500
None
500 5e-5 5
Warm-up init LR
9e-5
None
Warm-up schoduler
Lincar
None
Linear
Base LR
0.0009
0.02
0.0005
Epochs
200
120
50
Batch size
128
128
Layer-wise LR decay
X
X
X
Grad. clip
10
10
10
Exp. moving average
/
Table 11: Configuration for finetuning MobileViTv2 on downstream tasks. For Ade20k, we found that SGD was more stable as compared to AdamW across different MobileViTv2 configurations, and therefore, we used SGD for finetuning on Ade20k dataset. The configurations for MobileViTv2 with PSPNet are the same as Deeplabv3 on both PASCAL VOC and Ade20k datasets.
22


Reviews:
[{"review_id": "KwLtU67N7j", "reviewer": "Reviewer_AaHu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-----------------\n* The paper is well motivated with impressively written introduction and updated related works.\n* The writing is good with good and thorough empirical evaluations to defend the claims made in the paper.\n* Highlight of the latency issue of Linformer with supporting results.\n* The Fig. 3 and 4 are well explanatory.\n* Results with downstream tasks are inspiring.\n\nWeaknesses:\n-----------------\n* Please provide additional comparisons with EfficientFormer [1].\n* Please add CPU latency comparison with MobileViT in the Fig. 1 bottom right.\n* The mention of optimization (pruning) and distillation is provided as two separate strategy, which is the general case. However, recent research has provided joint optimization to perform both pruning and distillation via a single training loop [2]. Thus, this part needs to be updated with mention of such methods to provide a wholesome scope of the possible options. \n* Also, the authors should touch upon various NAS ViT methods that can yield accuracy-complexity pareto front suitable for target hardware. In particular, it would be interesting to see a discussion (and possibly results) of the current model in the context of NAS.\n*  The code and/or training script is not provided for reproducibility check. As EfficientFormer can be treated as aform of NAS, a detailed comparison with that would do as well (in understanding the value of the current separable self-attention method).\n* Please provide comparison with other ViT variants on downstream tasks apart from providing comparison with only MobileViT-1.\n* Please show the default robustness of this model variants. Generally, ViTs are inherently robust against OOD tasks. **It is important to understand whether such separable self-attention modules impact on that or not**.\n\n[1] EfficientFormer: Vision Transformers at MobileNet Speed, NeurIPS 2022.\n\n[2] AttentionLite: Towards Efficient Self-Attention Models for Vision, ICASSP 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an efficient self-attention mechanism namely separable self attention to improve the latency of transformer models for resource constrained applications. In particular, the authors have proposed to replace batch-wise matmul with context aware element-wise operations. Results on various FLOPS/Param budget show the efficacy of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness.", "broader_impact_concerns": "The broader impact section is not provided in the current draft.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F6KW5a6WXF", "reviewer": "Reviewer_uYAm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper analyzed the latency of few represented models and pointed out that some specific operation is not suitable for mobile device and then improve based on that.\n2. The proposed module is simple and effective and the authors evaluated on many visual tasks. \n\nWeaknesses:\n1. When comparing the ConvNeXt, the proposed MobileViTv2 only outperform it in terms of parameters. Even though the authors mentioned the ViT-based models exhibit better scaling properties but it seems that it conflicts with the goal of the paper, which aims for resource-constrained device. Moreover, if we compare R4 and R13, the ConvNeXt-T provided similar throughput and latency but much better accuracy. I think the authors should elaborate more on this.\n\n2. Figure 2 shows the comparison with MobileViTv1 but why there is no comparison to MobileViTv1 in Table 3 for the ADE20k dataset and there is no latency comparison for both Table 3 and 4.\n\n3. In Table 3 and 4, why not compare to ConvNeXt as well as MobileViTv2-2.0 also listed in the table?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper improved the existed mobile vit by replacing the multi-head self-attention by the proposed separable self-attention. The paper show the results in many visual tasks.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the pointed listed in weakness sections.\n\nAnd there are a few more points:\n\n1. In the visualization, the authors should also show the result of MobileViTv1 for the comparison.\n\n2. In Fig. 1, I think using the analysis of mobile device, e.g. iPhone 12, will make the story more consistent. As the architecture of CPU is also different from mobile device.\n\n3. Missing citation, as I think the concept of the paper is similar to LambdaNet [1], which also model the global context directly instead of multi-head self-attention.\n\n[1] LAMBDANETWORKS: MODELING LONG-RANGE INTERACTIONS WITHOUT ATTENTION, ICLR 2021\n", "broader_impact_concerns": "This paper proposed new building block for a neural network, as others did, it shares many of the pitfalls and the benefits associated with deep learning models. \n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zIEdjZAwmX", "reviewer": "Reviewer_fE6c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n> + Clear logic flow: the paper is well-written. I really like their writing style of starting from analysis on the real devices, which provides solid motivation to the proposed separable self-attention.\n> + Discuss the failure cases: although many papers are trying to hide unsatisfying, this work directly shows that there is still a gap between ViT and Conv-based models in terms of the accuracy vs. mobile device latency even for their proposed MobileViTv2. It is also a good point to show to the community and motivate more works on optimization ViT latency on mobile devices.\n> + Comprehensive experiments: the benefit of the proposed MobileViTv2 is supported by experiments on different datasets and applications, include image classification, semantic segmentation, and object detection.\n\n## Weaknesses\n> + Not cover enough baseline for comparison: The proposed separable self-attention still belongs to the linear attention category but only Linformer is included in the comparison, more linear self-attention works should be included in the comparison. \n> + Need more evidence for \"The main efficiency bottleneck in MobileViT is the multi-headed self-attention\n(MHA) in transformers\": as shown in Figure 1, BMM corresponding to self-attention is not the most costly operator but the MM corresponding to linear layers is. So the claim may need modifications or more evidences.\n> + Need more devices to support the claimed improvement on mobile devices: In experiments, only iPhone 12 is used as the devices to run benchmark.  However, it is very powerful smartphone with mobile GPU. It would be better to add the performance on some other devices, especially those phones without any specific accelerator. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work propose separable self-attention for mobile vision transformers. Specifically, it first profile different transformers model on mobile devices and detect multi-head self-attention as the one the bottleneck. Then it proposes a separable self-attention scheme to convert the batched matrix multiplication in standard multi-head self-attention into element-wise operations, thus the complexity decreases from O(k^2) to O(k), with k as the sequence length. The experiments on both ImageNet and MS-COCO verify that the new MobileViTv2 equipped with the proposed separable self-attention can achieve better accuracy vs. mobile device latency trade-offs as compared to MobileViT and some other efficient transformer baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be better if the authors can address the Weaknesses mentioned above, the three points are critical to securing my recommendation for acceptance.", "broader_impact_concerns": "No concern on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have all found the paper to be well written, the experiments strong and well motivated. All three reviewers have recommended acceptance, subject to a number of relatively minor changes that I believe the authors should be able to achieve: such as updating the related works with the citations pointed out by reviewers in their initial reviews, adding the rebuttal & discussion results into the main paper as well as making a bona-fide effort for open-sourcing this work. If the latter is there, I'd vote for having this certified as reproducible.

Some specific recommendations also include having (1) more evidence and discussion for "The main efficiency bottleneck in MobileViT is the multi-headed self-attention (MHA) in transformers" in the paper itself and (2) more baseline comparisons in addition to Linformer (+discussion for why some comparisons may be apples to oranges).
Decision: Accept (poster)