MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
Authors: Sachin Mehta
Mohammad Rastegari
Venue: ICLR 2022 Poster (2022)
Keywords: Vision transformer
Mobile
Edge Devices
Transformer
CNN
Efficient Network
Detection
Segmentation
ImageNet

Abstract: Light-weight convolutional neural networks (CNNs) are the de-facto for mobile vision tasks. Their spatial inductive biases allow them to learn representations with fewer parameters across different vision tasks. However, these networks are spatially local. To learn global representations, self-attention-based vision trans-formers (ViTs) have been adopted. Unlike CNNs, ViTs are heavy-weight. In this paper, we ask the following question: is it possible to combine the strengths of CNNs and ViTs to build a light-weight and low latency network for mobile vision tasks? Towards this end, we introduce MobileViT, a light-weight and general-purpose vision transformer for mobile devices. MobileViT presents a different perspective for the global processing of information with transformers, i.e., transformers as convolutions. Our results show that MobileViT significantly outperforms CNN- and ViT-based networks across different tasks and datasets. On the ImageNet-1k dataset, MobileViT achieves top-1 accuracy of 78.4% with about 6 million parameters, which is 3.2% and 6.2% more accurate than MobileNetv3 (CNN-based) and DeIT (ViT-based) for a similar number of parameters. On the MS-COCO object detection task, MobileViT is 5.7% more accurate than MobileNetv3 for a similar number of parameters. 

Our source code is open-source and available at: https://github.com/apple/ml-cvnets

Published as a conference paper at ICLR 2022
MOBILEVIT: LIGHT-WEIGHT, GENERAL-PURPOSE, AND MOBILE-FRIENDLY VISION TRANSFORMER
Sachin Mehta Apple
Mohammad Rastegari
Apple
ABSTRACT
Light-weight convolutional neural networks (CNNs) are the de-facto for mo- bile vision tasks. Their spatial inductive biases allow them to learn representa- tions with fewer parameters across different vision tasks. However, these net- works are spatially local. To learn global representations, self-attention-bass vision transformers (ViTs) have been adopted. Unlike CNNs, ViTs are heavy- weight. In this paper, we ask the following question: is it possible to com- bine the strengths of CNNs and ViTs to build a light-weight and low latency network for mobile vision tasks? Towards this end, we introduce VibileVIT, light-weight and general-purpont vision transformer for mobile devices. Mobile- ViT presents a different perspective for the global processing of information with transformers. Our results show that Mobile ViT significantly outperforms CNN- and ViT-based networks across different tasks and datasets. On the ImageNet-Ik dataset, MobileViT achieves top-1 accuracy of 78.4% with about 6 million pa- rameters, which is 3.2% and 6.2% more accurate than MobileNetv3 (CNN-based) and DeIT (ViT-based) for a similar number of parameters. On the MS-COCO object detection task, MobileViT is 5.7% more accurate than MobileNetv3 for a similar number of parameters. Our source code is open-source and available at: https: //t::///gtttii httpgttthhuu.to///ggg/g com/ apple/mll cvnets.
INTRODUCTION
Self-attention-baseee models, especially vision transformers (ViTs; Figure la; Dosovitskiy et al., 2021), are an alternative to convolutional neural networks (CNNs) to learn visual representations. Briefly, ViT divides an image into a sequence of non-overlapping patches and then learns inter- patch representations using multi-headed self-attentin in transformers (Vaswani et al., 2017). The general trend increase the number of parameters (e.g., Touvron et al., 2021a; Graham et al., 2021; Wu et al., 2021). However, these performance improvements come at the cost model size (network parameters) and latency. Many real-world applications (e.g., augmented reality and autonomous wheelchairs) require visual recognition tasks (e.g., object detection and semantic segmentation) to run on resource-comstrainge mobile devices in timely fashion. To be effective, ViT models for such tasks should be light-weight and fast. Even if the model size of VIT models is reduced to match the resource constraints of mobile devices, their performance is significantly worse than light-weight CNNs. For instance, for a parameter budget of about 5-6 million, DeIT (Touvron et al., 2021a) is 3% less accurate than MobileNetv3 (Howard al., 2019). Therefore, the need to design weihtt ViT models imperative.
VIT networks improve the performance
Light-weight CNNs have powered many mobile vision tasks. However, ViT-based networks are still far from being used on such devices. Unlike light-weight CNNs that are easy to optimize and integrate with task-specific networks, ViTs are heavy- (c.g.. ViT-B/16 vs. MobileNetv3: 86 vs. 7.5 million parameters), harder to optimize (Xiao al., 2021), need extensive data augmen- tation and L2 regularization to prevent over-fitting (Touvron et al., 2021a; Wang et al., 2021). and require expensive decoders for down-stream tasks, especially for dense prediction tasks. For in- stance, ViT-based segmentation network (Ranftl et al., 2021) learns about 345 million parameters and achieves similar performance as the CNN- based network, DeepLabv3 (Chen et al., 2017), with 59 million parameters. The need for more parameters in ViT-based models is likely because they lack image-specifie inductive bias, which is inherent in CNNs (Xiao et al., 2021). To build robust and high-performing ViT models, hybrid approaches that combine convolutions and transformers

---

Published as a conference paper at ICLR 2022
Lx
Lincar
Transformer
Linear
Logits
Flatten image putches
Positional encoding
(a) Standard visual transformer (VIT)
Transfurere Comolatiion Opktal rpprosatation
Local CI
Condial
Finis
2x
Comedt x
MY2
MVV2
MV2
MV2 12
MibdeMT Block
MY2
MobileVIT bock
MYV
MonheViT Neck
Crese x
Giobal pool
-e Lince
Loais
=
A -
Output spotial dimetssions
(b) VobileViT. Here,
Conv-n n in the MibileViT block represents a standard convolution and
MV2 refers to MobileNetv2 block. Blocks that perform down-sampling are marked with 2.
Figure Visual transformers vs. Mobile
Figure 2: MobileViT shows better task-level gener- alization properties as compared to light-weight CNN models. The network parameters are listed for SSDLite network with different feature extractors (MobileNetvl (Howard al., 2017), MobileNetv2 (Sandler al., 2018), MobileNetv3 (Howard et al., 2019). MNASNet (Tan et al., 2019), MixNet (Tan & Le, 2019b), and Mo- bileVIT (Ours) the MS-COCO dataset.
Benkioone
Hubi lahes2
Better 11.8%
Smaller:
mAp
S.M
22
045M
Top accuracy (ImageMet)
are gaining interest (Xiao et al., 2021; d' Ascoli et al., 2021; Chen al., 2021b). However, these hybrid models are still heavy- and are sensitive to data augmentation. For example, remov- ing CutMix (Zhong et al., 2020) and DelT-style (Touvron et al., 2021a) data augmentation causes a significant drop in ImageNet accuracy (78.1% to 72.4%) of Heo et al. (2021).
remains an open question to combine the strengths of CNNs and transformers to build ViT models for mobile vision tasks. Mobile vision tasks require lightwweight low latency, and accurate models that satisfy the device's 'ss resource constraints, and are seneral-purposs so that they can be applied to different tasks (e.g., segmentation and detection). Note that floating-poin operations (FLOPs) not sufficient for low latency on mobile devices because FLOPs ignore several important inference- related factors such memory access, degree of parallelism, and platform characteristics (Ma al., 2018). For example. the ViT-based method of Heo et al. (2021), PIT. has 3x fewer FLOPs than DeIT (Touvron al., 2021a) but has similar inference speed on mobile device (DelT vs. PiT on iPhone-13 12: 10.99 ms vs. 10.56 ms). Therefore, instead of optimizing for FLOPs!. this paper focuses on designing light- ($3), general-purpose ($4.1 & $4.2), and low lateney ($4.3) network for mobile vision tasks. We achieve this goal with Mobile that combines the benefits of CNNs (e.g., spatial inductive biases and less sensitivity to data augmentation) and ViTs (e.g., input-adaptive weighting and global processing). Specifically, we introduce the Mobile block that encodes both local and global information in a tensor effectively (Figure 1b). Unlike ViT and its variants (with and without convolutions). Vib presents different perspective to learn global representations. Standard convolution involves three operations: unfolding, local processing, and
ViT FL.OPs be further reduced using existing methods (e.g., Dynamic ViT (Rao al., 2021)).
2

---

Published as a conference paper at ICLR 2022
folding. ViT block replaces local processing in convolutions with global processing using transformers. This allows MibileViT block to have CNN- and ViT-like properties, which helps it learn better representations with fewer parameters and simple training recipes (e.g., basic augmen- tation). To the best of our knowledge, this is the first work that shows that light-weight ViT's can achieve light-weight CNN-level performance with simple training recipes across different mobile vision tasks, For a parameter budget of about 5-6 million, Mobile achieves a top-1 accuracy of 78.4% on the ImageNet-IK dataset (Russakovsky et al., 2015), which is 3.2% more accurate than MobileNetv3 and has a simple training recipe (MobileViT vs. MobileNetv3: 300 vs. 600 epochs; 1024 vs. 4096 batch size). We also observe significant gains in performance when MobileViT is used as feature backbone in highly optimized mobile vision task-specific architectures. Replacing MNASNer (Tan et al., 2019) with Mobile VīT as a feature backbone in SSDLite (Sandler et al., 2018) resulted in better (+ (+1.8% mAP) and smaller (1.8x) detection network (Figure 2).
2 RELATED WORK
Light- CNNs. The basic building layer in CNNs is standard convolutional layer. Because this layer is computationally expensive, several dactorization-base methods have been proposed to make it fight-weight and mobile-friendly (e.g., Jin et al., 2014; Chollet, 2017: Mehta et al., 2020). Of these, separable convolutions of Chollet (2017) have gained interest, and are widely used across state-of-the-art light-weight CNNs for mobile vision tasks, including MobileNets (Howard et al., 2017: Sandler et al., 2018; Howard et al., 2019). ShuffleNetv2 (Ma et al., 2018), ESPNetv2 (Mehta et al., 2019), MixNet (Tan & Le, 2019b), and MNASNet (Tan et al., 2019). These light-weight CNNs are versatile and easy to train. For example, these networks can easily replace the heavy- weight backbones (e.g.. ResNet (He et al., 2016)) in existing task-specific models (e.g.. DeepLabv3) to reduce the network size and improve latency. Despite these benefits, one major drawback of these methods is that they are spatially local. This work views transformers as convolutions; allowing to leverage the merits of both convolutions (e.g.. versatile and simple training) and transformers (c.g., global processing) to build light-weight ($3) and general-purpose ($4.1 and $4.2) ViTs.
Vision transformers. Dosovitskiy et al. (2021) apply transformers of Vaswani et al. (2017) for large-scale image recogmition and showed that with extremely large-scale datasets (e.g.. JFT-300M), ViTs can achieve CNN-level accuracy without image-specific inductive bias. With extensive data augmentation, heavy L.2 regularization, and distillation, ViTs can be trained on the ImageNet dataset to achieve CNN-level performance (Touvron et al., 2021a;b; Zhou et al., 2021). However, unlike CNNs. ViTs show substandard optimizability and are difficult to train. Subsequent works (c.g., Graham et al., 2021; Dai et al., 2021; Liu al., 2021; Wang et al., 2021; Yuan et al., 2021b; Chen et al., 2021b) shows that this substandard optimizability is due to the lack of spatial inductive biases in ViTs. Incorporating such biases using convolutions in ViTs improves their stability and perfor- mance. Different designs have been explored to reap the benefits of convolutions and transformers. For instance, ViT-C of Xiao et al. (2021) adds an early convolutional stem to ViT. CvT (Wu et al., 2021) modifies the muati-headdaee attention in transformers and uses depth-wise separable convolutions instead of linear projections. BNet (Srinivas et al., 2021) replaces the standard 3 convolution in the bottleneck unit of ResNet with multi-head attention. ViT (d' Ascoli et al., 2021) incorporates soft convolutional inductive biases using a gated positional self-attention. PiT (Heo et al., 2021) extends ViT with depth-wise convolution-based pooling layer. Though these models can achieve competitive performance to CNNs with extensive augmentation, the majority of these models are heivy- For instance, PiT and CvT learns 6.1x and 1.7x more parameters than EfficientNet (Tan & Le, 2019a) and achieves similar performance (top-1 1 accuracy of about 81.6%) on ImageNet- 1k dataset, respectively. Also, when these models are scaled down to build wight ViT models, their performance is significantly worse than fight-weigh CNNs. For a parameter budget of about 6 million, ImageNet- kk accuracy of PiT is 2.2% less than MobileNetv3.
Discussion. Combining convolutions and transformers results in robust and figh-performing ViTs as compared to vanilla ViTs. However, an open question here is: how to combine the strengths of convolutions and transformers to build light- networks for mobile vision tasks? This paper focuses on designing light liigh ViT models that outperform state-of of-the-art models with simple training recipes. Towards this end, we introduce Mobile that combines the strengths of CNNs and ViT's to build a light- weight, general-purper and mobile- friendl network. Mobile brings several novel observations. (i) Better performance: For a given parameter budget, MobileVi1 mod-
3

---

Published as a conference paper at ICLR 2022
60-
Tep-1 1op-5
MobileVIT-XXS
MobileVIT-XXS
MENAATTAN MARKATS
596
Mobilevi
MobileViT-XS MobilaMIT-S
50
Vo dolldottonn orror 1883
36M
75.4
Traiting eror
45-
45
(c) Validation accuracy
40-
2.5
30
30
Paramettrr
[nn maltoo)
20
25
25
150 Epochs
200
250
Epochs
150
200
250
300
0.0
1231
Output spatial dimensions
3,2
15
Training error
(b) Validation emror
(d) Parameter distribution
Figure 3:
MobileViT shows similar generalization capabilities as CNNs. Final training and vali- dation errors of MobileNetv2 and ResNet-50 are marked with * and o, respectively ($B).
els achieve better performance compared to existing liggh- CNNs across different mobile vision tasks ($4.1 and $4.2). (ii) Generalization capability: Generalization capability refers to the gap between training and evaluation metrics. For models with similar training metries, the model with better evaluation metrics is more generalizable because can predict better on an unseen dataset. Unlike previous ViT variants (with and without convolutions) which show poor generalization capability even with extensive data augmentation as compared to CNNs (Dai et al., 2021), Mobile shows better generalization capability (Figure 3). (iii) Robust: A good model should be robust to hyper-paramese parameters (e.g., data augmentation and L2 regularization) because tuning these syper-parameters time- and resource-consuming. Unlike most ViT-based models, Mobile- VIT models train with basic augmentation and are less sensitive to 1.2 regularization ($C).
MOBILEVIT: A LIGHT-WEIGHT TRANSFORMER
A standard ViT model, shown in Figure la, reshapes the input X #X into sequence of flattened patches X, E PPC projects it into a fixed d-dimensional space X, E NRdx and then learn inter-patch representations using stack of L transformer blocks. The computational cost of self-attention vision transformers is (NN2). Here, C, H, and W represent the channels, height, and width of the tensor respectively, and P wh is number of pixels in the patch with height /t and width w, and N is the number of patches. Because these models ignore the spatial inductive bias that is inherent CNNs, they require more parameters to learn visual representations. For instance, DPT (Dosovitskiy et al., 2021). ViT-based network, learns 6x more parameters as compared to DeepLabv3s (Chen et al., 2017), a CNN-based network, to deliver similar segmentation performance (DPT vs. Lab33: 345 M vs. 59 M). Also, in comparison to CNNs, these models exhibit sub- standard optimizability. These models are sensitive to L2 regularization and require extensive data augmentation to prevent overfitting (Touvron et al., 202 Xiao et al., 2021).
This paper introduces a light-weight ViT model, Vobile The core idea is to learn global repre- sentations with transformers convolutions. This allows us to implicitly incorporate convolution- like properties (c.g., spatial bias) in the network, learn representations with simple training recipes (c.g.. basic augmentation), and casily integrate Mobile with downstream architectures (c.g., Deeplaabv3 for segmentation).
3.1 MOBILEVIT ARCHITECTURE
Mobile block, The ViT block, shown in Figure 1b, aims model the local and global information in input tensor with fewer parameters. Formally, for given input tensor X # XC Mobile applies a n X n standard convolutional layer followed by a point-wise (or convolutional layer encodes local spatial information while the point-wise convolution projects the tensor to high-dimensionss space (or d-dimensional, where d C) by learning linear combinations of the input channels.
1 1x 1) convolutional layer produce X H xd The
With Mobile we want to model long-range non-local dependencies while having an effective receptive field of H W. One of the widely studied methods to model long-range dependencies
4

---

Published as a conference paper at ICLR 2022
Figure 4: Every pixel every other pixel in the MobileViT block. In this example, the red pixel attends to blue pixels (pixels at the corresponding location in other patches) using transformers. Because blue pixels have already encoded information about the neighboring pixels using convolu- tions, this allows the red pixel to encode information from all pixels in an image. Here, each cell in black and gray grids represents a patch and pixel, respectively.
is dilated convolutions. However, such approaches require careful selection of dilation rates. Oth- erwise, weights are applied to padded zeros instead of the valid spatial region (Yu & Koltun, 2016; Chen al., 2017; Mehta al., 2018). Another promising solution is self-attention (Wang al., 2018; Ramachandran et al., 2019; Bello et al., 2019; Dosovitskiy et al., 2021). Among self-attention methods, vision transformers (ViTs) with multi-head self-attention are shown be effective for visual recognition tasks. However, ViTs are heavy-weight and exhibit sub-standard optimizability. This is because ViTs lack spatial inductive bias (Xiao al., 2021; Graham al., 2021).
To enable Mobile VIT to learn global representations with spatial inductive bias, we unfold X P
into N non-overlapping flattened patches X E Pxxxx Here, P wh, N HW is the number of patches, and h < π and S are height and width of patch respectively. For each p E (I...., P). inter-patch relationships are encoded by applying transformers to obtain XG E R PxNxd as:
(a(p) = (x
(1)
Unlike ViTs that lose the spatial order of pixels, Mobile neither loses the patch order nor the spatial order of pixels within each patch (Figure 1b). Therefore, obtain XF E # xW xd XF is then projected to low C-dimensional space using a point-wise convolution and combined with X via concatenation operation. Another n n convolutional layer is then used to fuse these concatenated features. Note that because Xy (p) encodes local information from n region using convolutions and (p) encodes global information across P patches for the p-th location, each pixel in X can encode information from all pixels in X, as shown in Figure Thus, the overall effective receptive field of Mobile is H W.
fold XG R Pxxxxx to
Relationship to convolutions. Standard convolutions can be viewed as stack of three sequential operations: (1) unfolding,
(2) matrix multiplication (to learn local representations), and (3) fold- ing. Mobile block is similar to convolutions in the sense that it also leverages the same build- ing blocks. VibileViT block replaces the local processing (matrix multiplication) in convolutions with deeper global processing (a stack of transformer layers). As consequence, MobileVIT has convolution- properties (c.g., spatial bias). Hence, the Mobile ViT block can be viewed as trans- formers convolations. An advantage of our intentionally simple design is that low-level efficient implementations of convolutions and transformers can be used out-of-the-box; allowing us to use Mobile on different devices without any extra effort.
Light-weight. MobileViT block uses standard convolutions and transformers to learn local and global representations respectively. Because previous works (e.g.. Howard et al., 2017: Mehta al., 2021a) have shown that networks designed using these layers are heavy-weight, a natural question arises: Why Mibile is light-weight? We believe that the issues lie primarily learning global representations with transformers. For given patch, previous works (c.g.. Touvron et al., 2021a; Graham et al., 2021) convert the spatial information into latent by learning a linear combination of pixels (Figure la). The global information is then encoded by learning inter-patch information using transformers. As a result, these models lose image-specific inductive bias, which is inherent in CNNs. Therefore, they require more capacity to learn visual representations. Hence, they deep and wide. Unlike these models, Mobile ViT uses convolutions and transformers in a way that the resultant Mobile ViT block has convolution- like properties while simultaneously allowing for global processing. This modeling capability allows us to design shallow and narrow VobileViT models, which in turn light- Compared to the ViT-based model DelT that uses 1.=12 and d=192,
5

---

Published as a conference paper at ICLR 2022
Standard sampler
Mialti-ccale sampler
GPU-
GIU-1
Teeacs
Sumpler
Upddates
Epoch
GM3-2
Standard
375
scr
Multi-scale (ours)
232
270 sec
GROS
GM34
GPU-
Ciuthers syne
(h) Training efficiency. Here, standard sampler refers to Py PyTorch's Distributed- DataParallel sampler.
lenc
(a) Standard vs, multi-scale sampler illustration
Figure 5: Multi-scale vs. standard sampler.
VibileViT model uses I= (2, 4, 3} and d=(96, 120, 144) at spatial levels 32 x 32, 16 x 16, and respectively. The resulting MobileViT network is faster (1.85x), smaller (2x), and better (+1.8%) than DeIT network (Table 3; $4.3).
Computational cost. The computational cost of multi-headed self-attention in MibileViT and ViTs (Figure 1a) is O(N2Pd) and O(N²d), respectively. In theory, MobileViT is inefficient as compared to ViTs. However, FLOPs and delivers 1.8% better accuracy than Del'T on the ImageNet- K dataset (Table 3; $4.3). We believe that this is because of similar reasons as for the light-weight design (discussed above).
practice, MobileViT is more efficient than ViTs. Mobile has 2x fewer
Mobile VIT architecture. Our networks are inspired by the philosophy of light-weight CNNs. We train ViT models at three different network sizes (S: small, XS: extra small, and XXS: extra extra small) that are typically used for mobile vision tasks (Figure 3c). The initial layer in Mo- ViT is a strided 3 x 3 standard convolution, followed by MobileNetv2 (or MV2) blocks and ViT blocks (Figure lb and ŞA). We use Swish (Elfwing et al., 2018) tion. Following CNN models, we use n = 3 in the ViT block. The spatial dimensions of feature maps are usually multiples of and h, S n. Therefore, we set h. w 2 at all spatial levels (see &C for more results). The MV2 blocks in Mobile network are mainly responsible for down-sampling. Therefore, these blocks are shallow and narrow in Mobile network. Spatial- level-wise parameter distribution of Mobile ViT in Figure 3d further shows that the contribution of MV2 blocks towards total network parameters very small across different network configurations.
activation fune-
3.2 MULTI-SCALE SAMPLER FOR TRAINING EFFICIENCY
A standard approach in ViT-based models to learn multi-scale representations is fine-tuning. For in- stance, Touvron et al. (2021a) fine-tunes the DeIT model trained at spatial resolution of 224 x 224 on varying sizes independently. Such approach for learning multi-scale representations is prefer- able for ViTs because positional embeddings need to be interpolated based on the input size, and the network's performance is subjective to interpolation methods. Similar to CNNs, MobileViT does not require any positional embeddings and it may benefit from multi-scale inputs during training.
Previous CNN-based works (e.g., Redmon & Farhadi, 2017; Mehta al., 2021b) have shown that multi-scale training is effective. However, most of these works sample a new spatial resolution after fixed number of iterations. For example, YOLOv2 (Redmon & Farhadi, 2017) samples new spatial resolution from a pre-defined set at every 10-th iteration and uses the same resolution across different GPUs during training. This leads to GPU under- under-utilization and slower training because the same batch size (determined using the maximum spatial resolution in the pre-defined set) is used across all resolutions. To facilitate Mobile learn multi-scale representations without fine- tuning and to further improve training efficiency (i.e., fewer optimization updates), we extend the multi-scale training method to variably-sized batch sizes. Given a sorted set of spatial resolutions W,)} and batch size 6 for a maximum spatial resolution of (f1,, Wx), we randomly sample a spatial resolution (He W.) E S at t-th training iteration on each GPU and H.WW compute the batch size for f-th iteration as: b, = H. H.W. As a result, larger batch sizes are used for smaller spatial resolutions. This reduces optimizer updates per epoch and helps in faster training.
S ((11) wi),... (Hi,,
Figure 5 compares standard and multi-scale samplers. Here, refer to DistributeDDttaParalte in PyTorch as the standard sampler. Overall, the multi-scale sampler (i) reduces the training time as requires fewer optimizer updates with variably-sizer batches (Figure 5b), (ii) improves performance by about 0.5% (Figure 10; SB), and (iii forces the network to learn better multi-scale representations (SB), i.e., the same network when evaluated at different spatial resolutions yields better performance

---

Published as a conference paper at ICLR 2022
80
Model
Params.
Top-1
MobilaNete]
2.6.
68.4
MobileNatv2
2.6
Top-1 accuracy (%6)
75
MobileNere3
2.5 M
67.4
ShufflaNetv2
2,3 M
69.4
ESPNav2
MobileNiT.XX (Ouns)
2.3 M
74,8
Metelmeed
(b) Comparison with light-weight CNNs (similar parameters)
MotaleNea2
WoNileenn3
Mudel
Purums.
Top-1 f
60
Shufflches22
DonseNet-699
14
76.2
ESPNe122
FMiicienNet-B0
76.3
55
Potsiell (Dus)
ResNet. 101
44,5 M
77.4
ResNet-101-SE
49.3
77.6
Parameters (In million)
MobileVVT-S (Ours)
5.6
78.4
(a) Comparison with light-weight CNNs
(c) Comparison with heavy-weight CNNs
Figure 6: VibileViT vs. CNNs on ImageNet-IK Ik validation set. All models use basic augmentation.
as compared to the one trained with the standard sampler. In $B, we also show that the multi-scale sampler generic and improves the performance of CNNs (e.g., MobileNerv2).
4 EXPERIMENTAL RESULTS
In this section, we first evaluate Mobile performance on the ImageNet- Ik dataset and show that Mobile VIT delivers better performance than otate-of-the-art networks ($4.1). In $4.2 and $4.3, show ViTs are
and frienddf respectively.
4.1 IMAGE CLASSIFICATION ON THE IMAGENET-IK DATASET
Implementation details. We train ViT models from scratch on the ImageNet- Ik classifica- tion dataset (Russakovsky et al., 2015). The dataset provides 1.28 million and 50 thousand images for training and validation, respectively. The Mobile ViT networks are trained using Py PyTorch for 300 epochs on 8 NVIDIA GPUs with an effective batch size of 1 1,024 images using Adam optimizer (Loshchilov & Hutter, 2019), label smoothing cross-entropy loss (smoothing=O.1), and multi-scale sampler (S = {(160, 160), (192, 192), (256, 256), (288, 288), (320, 320) }). The learning rate is in- creased from 0.0002 to 0.002 for the first 3k iterations and then annealed to 0.0002 using a cosine schedule (Loshchilov & Hutter, 2017). We use L2 weight decay of 0.01. We use basic data augmen- tation (i.e.. random resized cropping and horizontal flipping) and evaluate the performance using a single crop top-1 accuracy. For inference, exponential moving average of model weights is used.
Comparison with CNNs. Figure 6a shows that Mobile outperforms light-weight CNNs across different network sizes (MobileNetvl (Howard al., 2017), MobileNetv2 (Sandler et al., 2018), ShuffleNetv2 (Ma et al., 2018), ESPNetv2 (Mehta et al., 2019), and MobileNetv3 (Howard et al., 2019)). For instance, for model size of about 2.5 million parameters (Figure 6b), Mobile out- performs MobileNetv2 by 5%, ShuffleNetv2 by 5.4%, and MobileNerv3 by 7.4% on the ImageNet- 1K validation set. Figure 6c further shows that Mobile delivers better performance than heavy- weight CNNs (ResNet (He et al., 2016). DenseNet (Huang et al., 2017). ResNet-SE (Hu et al., 2018), and EfficientNet (Tan & Le, 2019a)). For instance, Mobile is 2.1% more accurate than EfficentNet for similar number of parameters.
Comparison with ViTs. Figure compares Mobile with ViT variants that are trained from seratch on the ImageNet-IK 1k dataset without distillation (DeIT (Touvron al., 2021a), T2T (Yuan et al., 2021b), PVT (Wang et al., 2021), CAIT (Touvron et al., 2021b), Deep (Zhou et al., 2021), CeiT (Yuan et al., 2021a), Cross VIT (Chen et al., 2021a), LocalViT (Li et al., 2021), PiT (Heo al., 2021), ViT (d' Ascoli et al., 2021), ViL (Zhang al., 2021), BoTNet (Srinivas et al., 2021), and Mobile-former (Chen et al., 2021b)). Unlike VIT variants that benefit significantly from ad- vanced augmentation (e.g., PiT w/ basic vs. advanced: 72.4 (R4) vs. 78.1 (R17): Figure 7b), Mo- ViT achieves better performance with fewer parameters and basic augmentation. For instance, Mobile is 2.5x smaller and 2.6% better than DelT (R3 vs. R8 in Figure 7b).
Overall, these results show that, similar to CNNs, MobileVITs are easy and robust to optimize. Therefore, they can be easily applied to new tasks and datasets.

---

Published as a conference paper at ICLR 2022
Row
Model
Augmentation Params.
Top-1
R1
Deff
Basic
5.7M
08.7
K2
121
Advanced
4.3.M
71.7
R3
Deff
Advanced
5.7
72.2
00.0
Basic
10.6
72.4
77.5
R6
PIT
Advanced
4.9 M
73.0
72.8
75.0
R7
CrsaViT
Advanced
6.9 M
73.4
888
MobuRVIE-S (Ours
Basic
2.3.M
74.8
Top-1 accuracy
Advanced
DelT
CONMT
R10
R11
T2T
Deff
Advaneed
Advaneed
6.9 M
10 M
75.9
76.5
Mo former
R12
VIT
Advanced
6.7 M
76.7
Crossvil
LocalVit
Advaneed
DespVIT
Mobule
Advanced
LocaMIT
R15
Advanced
13.2
127
R16
ConVIT
Advaneed
10 M
76.7
MobietiT
817
PIT
Advanced
10.5 M
78.1
R18
BoTNe
Basic
20.8 M
77.0
8
10
BoTNe
Advanced
# Parameters (in million)
MoNIeVIT-S (Ours)
36.M
(a)
(b)
Figure 7: Vob vs. ViTs on ImageNet- Ik validation set. Here, basic means ResNet-style aug- mentation while advanced means a combination augmentation methods with basic (e.g.. Mix (Zhang et al., 2018), RandAugmentation (Cubuk et al., 2019), and CutMix (Zhong et al., 2020)).
4.2
MOBILEVIT AS A GENERAL-PPRPOSE BACKBONE
To evaluate the general-purpose purpose nature of VobileViT, we benchmark Mobile ViT on two widely studied mobile vision tasks: (1) object detection ($4.2.1) and (2) semantic segmentation ($4.2.2).
4.2.1
MOBILE OBJECT DETECTION
Implementation details. We integrate MibileVIT with a single shot object detection backbone (SSD; Liu et al., 2016). Following light- CNNs (C.... Mobileeets,, we replace standard convolutions in the SSD head with separa- ble convolutions and call the resultant network as SSDLite. We finetune ViT, pre-trained on the ImageNet- 1k dataset, an input resolution of 320 320 using Adam W on the MS-COCO dataset (Lin et al., 2014) that contains 117k training and 5k validation images. We use smooth L1 and cross-entropy losses for object localization and classi- fication, respectively. The performance is evaluated on the validation set using mAP@ MAP@@IOU IoU of 0.50:0.05:0.95. For other hyper-parameters, see $D.
backbone
||
mAP π
MobileNerva
4,9 M
22.0
MobileNetv2
4.3 M
22.1
MobileNetvi
5.1 M
22.2
4.5
MNASNet
4,9 M
23.0
MobileViT-XS (Ours)
27M
24.8
MobileVi-SS (Ours)
5.7
27.7
(a) Comparison light-weight CNNs
Feature backhone Params.
mAP
VGG
35.6
25.
ResNet50
22.9 M
25.2
MobileViT-S (Ours)
5.7 M
27.7
(b) Comparison w/ heavy-weight CNNs
Results. Table Ia shows that, for the same input resolu- tion of 320 x 320, SSDLite with MobileViT outperforms SSDLite with other light-weight CNN models SobbileNetv1/v2//3 MNASNet, and MixNet). For instance, SSDLite's s performance improves by 1.8%. and its model size reduces by 1.8x when Mo- Vile is used as backbone instead of MNASNet. Further, SSDLite with Mobile outperforms standard SSD-300 with heavy-weight backbones while learning significantly fewer parameters (Ta- ble 1b). Also, qualitative results in SF confirms MobileViT's ability detect variety of objects.
Table 1: Detection SSDLite.
4.2.2 MOBILE SEMANTIC SEGMENTATION
Implementation details. We integrate Mobile- ViT with DeepLabv3 (Chen et al., 2017). We fine- tune Mobile ViT using AdamW with cross-entropy loss on the PASCAL VOC 2012 dataset (Ever- ingham et al., 2015). Following a standard train- ing practice (e.g., Chen et al., 2017; Mehta al., 2019), we also use extra annotations and data from Hariharan al. (2011) and Lin et al. (2014), spectively. The performance is evaluated on the validation set using mean intersection over union (mIOU). For other
Feature backbone
Paramss.
miOU
11.2
MotsleNetv2
4.5 M
75.7
MobileViEXXXS (Ours)
MobsleVIT-X (Ours)
2.9 M
77.1
ResNet-
58.2
MobsleVIT-S (Ours)
6.4 M
79.1
Table 2: Segmentation w/ DeepLabv3.
see $D.
8

---

Published as a conference paper at ICLR 2022
Patic sizes
Patth sizes
84.
2,2.3
m13R 12
165
+XX5
8000
90005
Real- -ilm-itmme
X055
Hime
Inference
time
(a) Classification 256 x 256
(b) Detection @ 320 x 320
(c) Segmentation 512 x 512
Figure 8: Inference time of Mobile models on different tasks. Here, dots in green color region represents that these models runs in real-time (inference time < 33 ms).
Results. Table 2 shows that DeepLabv3 with Vob is smaller and better. The performance of DeepLabv3 is improved by 1.4%, and its size reduced by 1.6x when MobileViT used backbone instead of MobileNetv2. Also, VibileViT gives competitive performance to model with ResNet- 101 while requiring 9x fewer parameters; suggesting Mobile is a powerful backbone. Also, results in $G shows that Mobile VIT learns generalizable representations of the objects and perform well on an unseen dataset.
4.3 PERFORMANCE ON MOBILE DEVICES
Light-weight and low latency networks are important for enabling mobile vision applications. To demonstrate the effectiveness of Mobile for such applications, pre-trained full-precision Mobile- VIT models are converted to CoreML using publicly available CoreMLTools (2021). Their inference time is then measured (average over 100 iterations)
mobile device, i.e., iPhone 12.
Mobinl-friendly. Figure 8 shows the inference time of Mobile networks with two patch size settings (Config-A: 2, 2 and Config-B: 8, 4, on three different tasks. Here P1-P2, Ps in Config- X denotes the height h (width w = h) of a patch at an output stride² of 8, 16, and 32, respectively. The models with smaller patch sizes (Config-A) are more accurate compared to larger patches (Config-B). This is because, unlike Config-A models, Config-B models are not able to encode the information from all pixels (Figure 13 and $C). On the other hand, for given parameter budget, Config-B B models are faster than Config- A even though the theoretical complexity of self-attention in both configurations have fewer number of patches N as compared to smaller patch sizes (e.g., P=22=4). As result, the computation cost of self-attention is relatively less. Also, Config-B models offer higher degree of parallelism compared Config-A A because self-attention can be computed simultancously for more pixels in a larger patch (P=64) as compared to a smaller patch (P=4). Hence, Contig-B B models are faster than Config-A. To further improve Mobile latency, linear self-attention (Wang al., 2020) can be used. Regardless, all models in both configurations run in real-time (inference speed 30 FPS)
the same, i.e., O(N With larger patch sizes (c.g. P=82=64), we
mobile device except for Mobile models for the segmentation task. This is expected as these models process larger inputs (512 x 512) as compared to classification (256 x 256) and detection (320 320) networks.
Discussion. We observe that MobileViT and other ViT- based networks (e.g., DeIT and PiT) are slower as com- pared to MobileNetv2 on mobile devices (Table 3). This observation contradicts previous works which show that ViTs are more scalable as compared to CNNs (Dosovit- skiy et al., 2021). This difference is primarily because of two reasons. First, dedicated CUDA kernels exist for transformers on GPUs, which are used out-of-the-box VITs to improve their scalability and efficiency GPUs (e.g.. Shoeybi et al., 2019; Lepikhin al., 2021). Second, CNNs benefit from several device-level optimizations, including batch normaliza- tion fusion with convolutional layers (Jacob et al., 2018). These optimizations improve latency and memory access. However, such dedicated and optimized operations for transformers are currently not available for mobile devices. Hence, the resultant inference graph of Mobile and ViT-based networks for mobile devices is sub-optimal. We believe that similar to CNNs, the inference speed of ViT and ViTs will further improve with dedicated device-level operations in the future.
Model
# Parums.
FLOPs
Time |]
Top-1
MobilNNew2
3.5
0.3
0.92 ms
23.3
DefT
5.7 M
1.5G
10:99 ms
72.2
PIT MONWVVT (Ours)
4.
0.7
10.50 ms
73.0
2
077
7.28 ms
74.8
Table 3: ViTs are slower than CNNs. Results with multi-scale sampler ($B).
Output stride: Ratio of the spatial dimension of the input the feature map.

---

Published as a conference paper at ICLR 2022
5 ACKNOWLEDGEMENTS
We are grateful to Ali Farhadi, Peter Zatloukal, Oncel Tuzel, Ashish Shrivastava, Frank Sun, Max Horton, Anurag Ranjan, and anonymous reviewers for their helpful comments. We are also thankful to Apple's infrastructure and open-source teams for their help with training infrastructure and open- source release of the code and pre-trained models.
REFERENCES
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3286-32955 2019.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. CrossVit: Cross-attention multi-scale vision transformer for image classification. Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021a.
Liang-Chies Soig--Chihh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint urXiv:1706.05530 05587, 2017.
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengehen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. arXiv preprint trXiv:2108.05895 2021b.
François Chollet. Xeeption:
Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017. 2017.
CoolMLTools. Use coremitools to convert models from third-party haird-pprtt libraries to CoreMI. https: //coremltool readme. a docs, 2021. [Online; accessed 2-September-20211
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. arXiv preprint arXiv:2106.0es 048003, 2021.
Stéphane d' d'Acooli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103. 100:29003.0097.. 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Siggoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, (11(())98-11))) 2015.
Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2104.0/1 00.300....1........ 2021.
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. In 2 International Conference on Computer Vision, pp. 991-998. IEFE, 2011.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Andrew Howard. Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314 1324, 2019.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: /70..o/444...111 2017.
Jie Hu, Li Shen, and Gang Sun. Squeere-and-exciattion networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7700-4708, 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient intger-arithmeticoonlyy inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018.
Jonghoon Jin, Aysegul Dundar, and Eugenio Culurciello. Flattened convolutional neural networks for feedforward acceleration. arXiv preprint 1rX1v:1412.5444, 2014.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang. Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {GS) Hard: Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Represen- tutions, 2021.
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint FrXi:22004.05000 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yar Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint 0rXiv:2103.14033 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Ilya Loshchilov and Frank Hutter. Sgdr:
Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tar Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV). pp. 116-131, 2018.
11

---

Published as a conference paper at ICLR 2022
Sachin Mehta, Mohammad Rastegari, Anat Caspi, Linda Shapiro, and Hannaneh Hajishirzi. Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation. Proceedings of the european conference computer vision (ECCV), pp. 552-568, 2018.
Sachin Mehta, Mohammad Rastegari, Linda Shapiro, and Hannaneh Hajishirzi. Espnetv2: light- weight, power efficient, and general purpose convolutional neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9190-9200, 2019.
Sachin Mehta, Hannaneh Hajishirzi, and Mohammad Rastegari. Dicenet: Dimension-wise convo- lutions for efficient networks. IEEE Transactions on Paltern Analysis and Machine Intelligence, 2020.
Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Delight: Deep and light- weight transformer. In International Conference on Learning Represen- tations, 2021a.
Sachin Mehta, Amit Kumar, Fitsum Reda, Varun Nasery, Vikram Mulukutla, Rakesh Ranjan, and Vikas Chandra. Evrnet: Efficient video restoration on edge devices. In Proceedings of the ACM Multimedia, 2021b.
PyTorch. Torchvision semantic segmentation.
http:pytorc....rr stable/ models html semantic- seaetiiitssgeeeaaiiiinn. 2021. Online; accessed 15 November 2021.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' FFAché--Bu, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. arXiv preprint 3rXiv:2103.134411 2021.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. arXiv preprint arxiv:220060020:: 2021.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1153))211-255, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich -hhich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520, 2018
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training billio parameter language models using model par- allelism. arXiv preprint arXiv: 1XXiv:999900005: 2019.
Aravind Srinivas, Tsung-Yt Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16519-16529, 2021.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114, PMLR, 2019a.
Mingxing Tan and Quoe V Le. Mixconv: Mixed depthwise convolutional kernels. In Proceedings of the British Machine Vision Conference (BMVC), 2019b.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 2820-2823 2019.
12

---

Published as a conference paper at ICLR 2022
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103. 2021b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. 2020.
Wenhai Wang. Enze Xie, Xiang Li, Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7794-7803, 2018.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2 103. /5808, 2021.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. arXiv preprint arXiv:2106.1 14881, 2021.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In Interna- tional Conference on Learning Representations, 2016.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating con- volution designs into visual transformers. arXiv preprint /8116 2021a.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang. Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, 2021b.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In International Conference Learning Representations, 2018.
Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. arXiv preprint a5335:2103./5358, 2021.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data aug- mentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp- 13001-13008, 2020.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint 2rXiv:2103.118888 2021.
13

---

Published as a conference paper at ICLR 2022
A MOBILEVIT ARCHITECTURE
VobileViT's are inspired by the philosophy of light-weight CNNs and the overall architecture of ViT at different parameter budgets is given Table 4. The initial layer in VobileViT is strided 3 x 3 standard convolution, followed by MobileNerv2 (or MV2) blocks and Mobile- VIT blocks. We use Swish (Elfwing et al., 2018) as an activation function. Following CNN models, = 3 in the Vob block. The spatial dimensions of feature maps are usually mul- w at all spatial levels. The MV2 blocks
tiples of 2 and h, w S. n. Therefore, we set in ViT network are mainly responsible for down-sampling Therefore, in these blocks, use an expansion factor of four, except for MobileViT-XXX where we use an expansion factor of 2. The transformer layer in Mobile VTT takes d-dimensional input, shown in Figure 1b. We set the output dimension of the first feed-forward layer in a transformer layer as 2d instead of 4d, a default value in the standard transformer block of Vaswani et al. (2017).
B MULTI-SCALE SAMPLER
Multi-scale sampler reduces generalization gap. Generalization capability refers the gap be- tween training and evaluation metrics. For two models with similar training metrics,the model with better evaluation metrics more generalizable because it can predict better on an unseen dataset. Figure 9a and Figure 9b compares the training and validation error of the Mobile model trained with standard and multi-scale samplers. The training error of MobileViT-S with multi-scall sampler is higher than standard sampler while validation error is lower. Also, the gap between training error and validation error of MobileViT-S with multi-scale sampler is close to zero. This suggests that a multi-scale sampler improves generalization capability. Also, when Mobile trained indepen- dently with standard and multi-scale sampler evaluated at different input resolutions (Figure 9c), observe that Mobile trained with multi-scale sampler more robust compared to the one trained with the standard sampler. We also observe that multi-scale sampler improves the perfor- mance of Mobile models at different model sizes by about 0.5% (Figure 10). These observations in conjunction with impact on training efficiency (Figure 5b) suggests that multi-scale sampler is effective. Pytorch implementation of multi-scale sampler is provided in Listing
Multi-scale sampler is generic. We train a heavy-weight (ResNet-50) and a light-weight (MobileNetv2-11O)) CNN with the multi-scale sampler to demonstrate its generic nature. Results in Table show that a multi-scale sampler improves the performance as well as training efficiency.
Output channels
Layer
Output size
Output stride
Repeat
XXS
XS
S
Image
256 256
1
Conv-3 3.42 MV2
I
16
16
16
128 128
2
1
16
32
32
MV2, 12 MV2
1
24
48
64
64 64
4
2
24
48
64
MV2, 2 MobileVTT block (I. =
1
48
64
96
32 32
8
48 (d 64)
64
96)
96 (d 144)
MV2, 22 MobileVIT block (I. 4)
1
64
80
128
16 x 16
16
1
64 (d 80)
80 (d 120)
128 (d 192)
MV2, 2
80
96
160
MobileVTT block (I = 3)
x
32
1
80 (d 96)
96 (dl 144)
160 (d = 240)
Conv-1
320
384
640
Global pool
Linear
x
256
1 1000
1000
1000
Network Parameters
1.3 M
2.3 M
5.6 M
Table 4: Mib architecture. Here, d represents dimensionality of the input to the transformer layer in Mobile ViT block (Figure lb). By default, in Mobile ViT block, we set kernel size n as three and spatial dimensions of patch (height h and width w) in Mobile ViT block as two.
14

---

Published as a conference paper at ICLR 2022
60-
Standard
Standard
(Qun))
(Our))
Standard
68
Coochs
Specta
Irput resdtution
(a) Training crror
(b) Validation error
(c) Validation acc. vs. input resolution
Figure 9: Mobile ViT-S learns better representations with multi-scale sampler ImageNet- 1k.
Top-1. accuracy (%6
78
Mull-scaa (Durs)
Staridand
72-
70
MobileVIT models
Figure 10: MobileViT's performance on ImageNet-Ik with standard and multi-scale sampler.
Model
A Params
A Epochs
# Updates ||
Top-I accuracy
Training time
ResNet-50 w/ standard sampler (Py Torch)
25 M
76.2
(0.0%)
ResNet-50 w/ standard sampler (our repro.)
25 M
150
187 k
77.1
(+0.9%)
k sec. sec.
(L.35x) (1x)
ResNet-5t w/ multi-scale sampler (Ours)*
25
150
116k
78.6
(+2.4%)
MobileNenv2-100 w/ standard sampler (PyTorch)
3.5
71.9
(0.0%)
MobileNewv2-10 2-1.0 w/ standard sampler (our repro.)
3.5 M
300
375 k
72.1
(+0.29)
78 k sec.
(L.16)
MohileNNtv
sampler (Qurs)
3.5
(+1.49)
Table 5: Multi-scale sampler is generic. All models trained with basic augmentation the ImageNet-1k Ik. Results are with exponential moving average.
For instance, a multi-scale sampler improves the performance of MobileNetv2-110 by about 1.4% while decreasing the training time by 14%.
C ABLATIONS
Impact of weight decay. A good model should be insensitive or less sensitive to L2 regularization (or weight decay) because tuning it for each task and dataset is time- and resource-consuming Unlike CNNs, ViT models 2021a; Xiao et al., 2021). To study if Mobile models are sensitive to weight decay or not, we train the MobileViT-S model by varying the value weight decay from 0.1 to 0.0001. Results are shown in Figure 11. With an exception to the Mobile model trained with weight decay of 0.1, all other models converged to a similar solution. This shows that ViT models are robust to weight decay. In our experiments, we use the value of weight decay as 0.01. Note that 0.0001 is the widely used value of weight decay in most CNN-based models, such as ResNet and DenseNet. Even at this value of weight decay, Mib outperforms CNNs the ImageNet- Ik dataset (e.g.. DenseNet vs. Mobile VIT: 76.2 with 14 M parameters vs. 77.4 with 5.7 M parameters).
sensitive weight decay (Dosovitskiy et al., 2021; Touvron al.,
Impact of skip-connection. Figure 12 studies the impact of akip-connection in the Mobile block (red arrow in Figure lb). With this connection, the performance of Mobile ViT-S improves by 0.5% on the ImageNet dataset. Note that even without this skip-connection, MobileViT-S delivers similar or better performance than state-of-the-an CNN- (Figure 6) and ViT-based (Figure 7b) models, that too with basic data augmentation.
Impact of patch sizes. Vib combines convolutions and transformers to learn local and global representations effectively. Because convolutions are applied x regions and self-attention
15

---

Published as a conference paper at ICLR 2022
35
Weight dec
Weight dec
50
50
45
0.01
Valddeiion 1%0
0.01
0.001
0.001
0.0001
20
Weight decay Top-1 f
20
20-
0.1
76.5
0.01
78.4
0.001
77.6
1.50
0.0001
77.4
Training
(b) Validation
(c) Validation accuracy
Figure 11: Impact of weight decay. Here, results are shown for MobileViT' model (5.7 M param- eters) on the ImageNet-IK dataset. Results in (c) are with exponential moving average.
45
wio skip skip
WS skip skip
40-
maiiing eror (S)
Veliaation ermer
35-
25
Skip-connection Top-1
20
1.0
Epochs
1.50
200
250
100
20
50
1.00
Epocts
200
250
77.9
78.4
(a) Training error
(b) Validation error
(c) Validation accuracy
Figure 12: Impact of skip connection. Here, results are shown for Mobile model (5.7 M parameters) on the ImageNet- 1k dataset. Results in (c) are with exponential moving average.
Patch sizes
Params.
Time
Top-1
2,2,2
5.7 M
9.85
78.4
3,3,3
5.7 M
14.69
78.5
4,4.4
5.7 M
8.23 ms
77.6
8,4.2
5.7 M
8.20 ms
77.3
Table Impact of patch sizes. Here, the patch sizes are for spatial levels at 32 x 32, x 16, and 8 x 8, respectively. Also, results are shown for MobileViT-S model on the ImageNet-1k dataset. Results are with exponential moving average. Spatial dimensions feature map are not multiple of patch dimensions. Therefore, use bilinear interpolation in folding and unfolding operations to resize the feature map.
computed over patches with spatial dimensions of and w, is essential to establish a good relationship between n, h, and Following previous works CNN designs, and then vary and w. Specifically, we study four configurations: (i) = w = 2 at all spatial and would allow each pixel to encode information from
levels (Figure 13a). In this case, every other pixel using MobileViT. (ii) h = w = 3 at all spatial levels (Figure 13b). In this case, n. Similar to (i), this configuration would also allow each pixel to encode information from every other pixel using ViT. (iii) h w = 4 at all spatial levels (Figure 13c). In this allow each pixel aggregate information from other pixels in the tensor. (iv) h = = 8, = = 4, and h = respectively. Unlike (i), (ii), and (iii), the number of patches N is the same across different spatial 2. Note that
case, h, > and would
2 at spatial level of 32 x 32, 16 x 16, and 8 x 8,
resolutions in (iv). Also. h, < only for spatial level of 8 x 8 where h all these models have the same number of network parameters and the same computational cost of self-atte atff-tttenton, i.e., 2 Pd). Here, N is the number of patches, P = hw is the number of pixels in patch with height h and width w. and d is the model dimension.
Results shown in Table 6. We can that when h, n, Mobile can aggregate information more effectively, which helps improve performance. In our experiments, we used h = w = instead of h 3 because spatial dimensions of feature maps multiples of 2, and using
16

---

Published as a conference paper at ICLR 2022
=
h == 7== 3
(c) n
Relationship between kernel size x n) for convolutions and patch size (h w) for folding and unfolding in MobileViT. In and b, the red pixel is able to aggregate information from all pixels using local (cyan colored arrows) and global (orange colored arrows) information while in (c), every pixel is not able to aggregate local information using convolutions with kernel size of x from 4 4 patch region. Here, each cell in black and gray grids represents patch and pixel, respectively.
Figure 13:
LS
EMA
Top-1
X
78.0
X
78.3
78.4
Table 7: Effect of label smoothing (LSS) and exponential moving average (EMA) on the perfor- mance MobileViT-S on the ImageNet-Ik dataset. First row results are with cross-entropy.
h = w-- w= 3 requires additional operations. For folding and unfolding, we need to either pad or resize. In the case of padding, need mask the padded pixels in self-attention in transformers. These additional operations result in latency, as shown in Table 6. To avoid these extra operations, we choose h w 2 in our experiments, which also provides a good trade-off between latency and accuracy.
Impact exponential moving average and label smoothing. Exponential moving average (EMA) and label smoothing (LSS) are two standard training methods that are used to improve CNN- and Transformer- models performance (Sandler et al., 2018; Howard et al., 2019; Tan et al., 2019; Touvron et al., 2021a; Dai et al., 2021; Xiao et al., 2021). Table 7 shows that LS marginally improves the performance of MobileViT-S while EMA has little or no effect on model's performance on the ImageNet- Ik dataset. Because previous works have shown these methods be effective in reducing stochastic noise and prevent network from becoming over-confident, we use these methods to train MobileViT models.
D TRAINING DETAILS FOR SSDLITE AND DEEPLABV3
All SSDLite- MiLitee-Mobieeiiii and DeepLabv3-Mobieeii networks are trained for 200 and 50 epochs with a standard sampler on 4 NVIDIA GPUs and with an effective batch size of 128 images, re- spectively. The learning rate is increased from 0.00009 to 0.0009 in the first 500 iterations and then annealed 0.00009 using a cosine learning rate scheduler. We use weight decay of 0.01. We change the stride of MV2 block from two one an output stride of 32 in Table 4 to obtain DeepLaby3-Mobileen models at an output stride of 16.
For these models, we do not use multi-scale sampler. This is because these task-specific networks are resolution-dependere
For example, DeepLabv3 uses an atrous (or dilation) rate of 6, 12, and 18 at output stride of 16 learn multi-scale representations. If we lower resolution (say 256 x 256) than 512 x 512, then the atrous kernel weights will be applied to padded zeros; making multi-scale learning ineffective.
17

---

Published as a conference paper at ICLR 2022
E EXTENDED DISCUSSION
Memory footprint. A light network running on mobile devices should be memory effi- cient. Similar to MobileNetv2, we measure the memory that needs to be materialized each spatial level (Table 8). At lower spatial levels (i.e., an output stride of 8, 16, and 32) where MobileViT blocks are employed, required memory is lesser or comparable light-weight CNNs. Therefore, similar to light- CNNs, VibileViT networks are also memory efficient.
FLOPs. Floating point operations (FLOPs) another metric that is widely used to measure the efficiency of neural network. Table 9 compare FLOPs of MobileViT with different ViT-based networks on the ImageNet-IK dataset. For similar number of FLOPs, Mobile is faster, smaller, and better. For instance, PiT and Mobile has the same number of FLOPs, but MobileViT is 1.45x faster, 2.1x smaller, and 1.8% better (R2 vs. R4 in Table 9). It important to note that FLOPs for networks R2-R4 are the same, but their latency and performance are different. This shows that FLOPs is not sufficient metric for network efficiency as it does not account for inference-related factors such as memory access, degree of parallelism, and platform characteristics.
The ImageNet-1k 1k pre-training helps in performance improvement in down-stream tasks such as object detection and semantic segmentation (Long et al., 2015; Chen et al., 2017; Redmon & Farhadi, 2017). Because such tasks are used in real-world applications and often uses higher image inputs as compared to the ImageNet- Ik classification task, it is important to compare the FLOPs of network on down-stream tasks. Towards this end, we compare the FLOPs of ViT with MobileNetv2 on three tasks, i.c., classification, detection, and segmentation. Results are shown in Table 10. We observe that (1) the gap between MobileNetv2 and MobileViT FLOPs reduces as the input resolution increases. For instance, MobileNetv2 has 2 fewer FLOPs as compared to Mobile the ImageNet-Il Ik classification task, but on the semantic segmentation, they have similar FLOPs (Table 10a vs. Table 10c) and (2) MobileNetv2 models are significantly faster but less accurate than ViT models across different tasks. The low-latency of MobileNetv2 models is likely because of dedicated and optimized hardware-leve. operations on iPhone. We believe that (1) the inference speed of ViT will further improve with such dedicated operations and (2) our results will inspire future research in the area of hardware design and optimization.
Inference time on different devices. Table 11 compares the inference time of different models on three different devices, i.e., iPhone12 CPU, iPhone12 neural engine, and NVIDIA V100 GPU. MobileNetv2 is the fastest network across all devices. On iPhone (both CPU and neural engine), DeIT and PİT. However, on GPU, DelT
ViT delivers better performance as compared and PiT are faster than MobileViT. This is likely because MibileVIT models (1) are shallow and narrow, (2) run at higher spatial resolution (256 x 256 instead of 224 224), and (2) did not use
OS
MobileNeev2--100
MobileViT-XS
784
200
294
100
98
16
62
31
32
32
37
Top-1
73.3
74.8
Table 8: Comparison between MobileNetv2 and MobileViT in terms of maximum memory (in kb) that needs to be materialized at each spatial resolution in the network. The top-1 accuracy is measured on the ImageNet- validation set. Here, OS (output stride) is the ratio of spatial dimensions of the input to the feature map.
Model
Params. J.
FLOPSs |
Time J.
Top-1 f
(R1) DefT
5.7 M
1.3G
10.99 ms
72.2
(R2) PIT
4.9 M
G
10.56
73.0
(R3) MobileViT-XS (Ours; 8.4.2)
2.3
5.93 ms
73.8
(R4) Mobil-VT-XS (Ours; 2,2.2)
2.3
0.7 G
7.28 ms
74.8
Table 9: Comparison of different ViT-based networks. The performance of MobilcViT-XX model is reported at two different patch-size settings. See $A for details.
18

---

Published as a conference paper at ICLR 2022
Model
Params,
FLOPs (
Time
Togg1
Backbone
FL.P [
Time I.
mAP [
MobileNmm2
MoNiiNNev22
23
8.4.)
0.0
MaNileNI--S (Ouns 84.2)
Mi6ileVIEXS (Ours; 2,2.2)
MoNicNNT-XS (Ounc2,22)
1.6G
m2.6m
(a) ImageNet- Ik classification
(b) Object detection w/ SSDLite.
Backbone
Params.
FL.O%s
Time ||
mt0U
MobuleNNet2
4.5
3.8 G
6.5 ms
75.7
MobuleVVL XS (Ours)
29 M
5,7 G
75.4 25.1 ms
MobileViT-XS (Churs)
299 M
6.7
32.
77.1
(c) Semantic segmentation w/ DeepLabv3.
Table 10: MobileViT vs. MobileNetv2 on different tasks. The FLOPs and inference time in (a), (b) and (c) are measured at 224 224, 320 320, and 512 512 respectively with exception to MobileViT-XS model in (a) which uses 256 x 256 as an input resolution for measuring inference time on iPhone 12 neural engine. Here, the performance of Mobile ViT-XS models is reported two different patch-size settings. See SA for details.
Inference time ||
Model
# Params
FLOPs ||
Top-1 ||
iPhone12-
iPhone12- Neural Engine
NVIDIA V100 GPU
MobileNetv22
3.5 M
0.3
73.3
7,50 ms
0.92 ms
0.31 ms
DeIT
5.7 M
G
72.2
28.15
10.99 ms
0.43
ms
VobileViT (Ours)
2,3 M
0.7
74.8
17.86 ms
7.28 ms
0.62 ms/0.47 ms!
Inference time on different devices. The run time of MobileViT's is measured at 256 x 256 while for other networks, it is measured 224 224. For GPU, inference time is measured for batch of 32 images while for other devices, we use batch size of one. Here, represents that Mo- ViT model uses PyTorch's Unfold and Fold operations. Also, patch sizes for MobileViT model at an output stride of 8, 16, and 32 are set to twO.
Table 11:
GPU- accelerated operations for folding and unfolding as they are not supported on mobile devices. However, when replaced unoptimized fold and unfold operations with PyTorch's Unfold and Fold operations, the latency of Mobile VIT model improved from 0.62 ms to 0.47 ms.
Overall, findings suggest that they opportunities for optimizing ViT-based models, includ- ing MobileViT, for different accelerators. We believe that our work will inspire future research in building more efficient networks.
F QUALITATIVE RESULTS ON THE TASK OF OBJECT DETECTION
Figures 15, 14, and 16 shows that SSDLite with MobileViT-S detect different objects under different settings, including changes in illumination and viewpoint, different backgrounds, and non- rigid deformations.
19

---

Published as a conference paper at ICLR 2022
backpack
orange
d6r
Figure 14: Object detection results of SSDLite-MobileeiTs on the MS-COCO validation set.
20

---

Published as a conference paper at ICLR 2022
dining table cup
BAs take Caver Bold
Netrosentre
via gren 96
carrol
brocco
Tyme
carrot
vine glass
dining
Kaaebbordd
person
Stning table!
PIZZ
baseball glove
SCV
Figure 15: Object detection results of SSDLLite-MoblleWT-S on the MS-COCO validation set.
21

---

Published as a conference paper at ICLR 2022
Iep 1.p
person
deina
tennis racket
p person person
Figure 16: Object detection results of SSDLite-MobileWiTSS the MS-COCO validation set.
22

---

Published as a conference paper at ICLR 2022
G SEMANTIC SEGMENTATION RESULTS ON AN UNSEEN DATASET
To demonstrate that Mobile learns good generalizable representations objects, we evaluate the DeepLaby3-MobileVii model in Section 4.2.2 on the MS-COCO validation set that contains 5k images. Following official torchvision segmentation models (PyTorch, 2021), object classes in the MS-COCO dataset are mapped to the object classes in the PASCAL VOC dataset and models are evaluated in terms of mIOU. Note that the MS-COCO validation set is an unseen test set for DeepLaby3-Mobieeiit models because these images are neither part of the training nor the valida- tion set used for training Deepl abw3-Mobilee models.
Table compares the performance of Deeplaaby3-Mobilevii models with MobileNetv3- that was trained with three different segmentation backbones (LR-ASPP (Howard et al., 2019), DeepLabv3, and FCN (Long et al., 2015)). For the same segmentation model, i.e. DeepLabv3, Vibi is more effective backbone than MobileNetv3. D ViT-- model is 1.7x smaller and 5.1% more accurate than MyppLaaby--obileNettee qualitative results in Figure 17 and Figure 18 further demonstrates that Mobile VIT learns good gen- eralizable representations of the objects and perform well in the wild.
model. Furthermore,
Model
Params
mIOU
LR-ASPP w/ V3baae
3.2
57.9
FCN MobileNatV3-Lagge
57.8
DeepLabbv3 MobiieeeeVV-LLagge
11.0 M
60.3
DeepLaby3 MobileVIT-XXX (Ours)
1.9
46.7
DecpLaby3 MobikcViI-XS VTI-XS (Our))
2.9 M
57.4
DeepLabvv Mobile VIT-S (Ours)
Table 12: Semantic segmentation on the MS-COCO validation set. MobileNenvs-Large results are from official torchvision segmentation models (Py (PyTorch, 2021).
23

---

Published as a conference paper at ICLR 2022
B-ground
Aero plane
Bicycle
Bird
Boat
Bottle
Bus
Car
Cat
Chair
Cow
Dining-Table
Dog
Horse
Motorbike
Person
Patted
Sheep
Sofa
Train
TV/Monitor
Figure 17: Semantic segmentation results of Deeplabw3-MobileWiiTi model on the unseen MS- COCO validation set (left: input RGB image, middle: predicted segmentation mask, and right:
Segmentation mask overlayed on RGB image). Color encoding for different objects in the PASCAL VOC dataset is shown in the last row.
24

---

Published as a conference paper at ICLR 2022
UINNESS
WINNES
DRAUGHT
DRAUGHT
B-ground
Aero plane
Bicycle
Bird
Boat
Bottle
Bus
Car
Cat
Chair
Cow
Dining-Table
Dog
Horse
Motorbike
Person
Potted Plant
Sheep
Sofa
Train
TV/Moritor
Figure 18: Semantic segmentation results of Deeplabv3-Mobile ViT-S model on the unseen MS- COCO validation set (left: input RGB image, middle: predicted segmentation mask, and right: Segmentation mask overlayed on RGB image). Color encoding for different objects the PASCAL. VOC dataset is shown in the last row.
25

---

Published as a conference paper at ICLR 2022
import torch
tarch utils sampler import Sampler
import torch. distributed dist
import math
import random
import numpy op
class MairiSealeSampleeDDD Sampler):
der .init. (self.
base
batch int, samples:
min mult: flaat 0.5, scale mule: fleat 1.5, seatex: 5, training: co Felse] -> None:
11
min. and spatial dimensions
seale (hac min_im_h. max int (base. im_ min_scalle mult), iase .im, max_scale_muit
mult)
15
Get the GPU and snde related information aum_replicos rank dist get rank
war_warld_size(()
17
19
adjust the total samples to avoid batch dropping samples per replica int ceil samples total_size nem.samples..et.repiii img-indice, [ids for idx in rangr<(.dataasunn data_ssmples) img_indices + inggindiees rotal_aize s_ddle, (impp.indices) taral_size
repliens))
num_replicas
23
25
self. shuffle False if is.truining:
27
corresponding width_dims list (op. linspace (minjim_w. max max_im_w, n_seales))
campute spatial
height.dims = list linspace (min.im.h h., max.im.h. h. n.scales)) A ImageNet models doun-sumple dimensions factor of 32, muttiple
31
Fesure that
33
width [(w 32) for w in dimsh
height_dims = [(h // 32) a 32 for in height,dims]
35
img. paich list() base.elements hase_im_w base,im_h base.hatch.size for (h, w) in zip <ip(hiight.dims width:dims): baichasize max(l. (base_elemente (h w))
37
39
iag.batch_pairs. appppp ((h, batch_size))
41
self iig.mmmtch..paie
img.bateh.pairs
self. shuffle = True
43
clse:
batchpathspsirs [(base.im.h, base.im.w. base.batch.size))
45
self. lnniceeices img_iadices
47
self
nwm.sumplo.perrrepiia
epach
49
self.rank = rank
self, num.replicas num.replicas
51
..iiter-(ssell (self))
53
if self. shuffle:
random. seedi epach)
55
random shuffle self img.indices).
img.batch.ppains
37
indices.rank.i self. self. leak:len/sede selff
:ea.::le/islf..iig.idieeee img..iddiess I: 1:
h:m.:::ummmmmmmmmllli........... num.replicas!
indives.rank.i self. img.iindiees[sele len:: (ennself eise:
59
ing. tink:
mem. rum.replivas]
index =
while indet-index < self. ..samppes.per.replia
random.
aadomm.hooieesse choice (boiceesslf.igg.batcc..... img.batch.pairss
65
end_index manssaart.index curr.bsz. self. b.samples.pe..eeliees) batch iadices.rank.il star-iindxx:eddlidds..
67
b.batch samples len(batch-idss n-batch.samples curr.bsz:
69
batch ids te indices rank cfre bsz batch samples)] stdrt.index cuzr.baz
lenchatch ids)
75
batch [(eurr.h, curr.w, baid) for baid in batch.ids] yield batch
75
set-opoch epoch: iot) >> self. epech epoch
77
Listing 1: Py implementation of multi-scale sampler
26


Reviews:
[{"review_id": "mgQgrKU6To9", "reviewer": "Reviewer_cGXj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n-\tThe design of the new MobileViT block, which combines local convolution and global self-attention block, is neat and simple to implement.\n-\tThe paper writing is easy to follow and technical exposition is clear.\n-\tThe experimental results are quite promising.\n\nWeakness\n-\tIf I understand correctly, quite a few factors contribute to the final superior results of MobileViT on three benchmarks, including MobileViT model architecture, multi-scale training, label smoothing, and EMA. The paper claims the model architecture and multi-scale training as the main contributions. Thus, it is important to measure how much label smoothing and EMA improves the final results, and note whether other competing approaches use them during external comparisons.\n-\tThe paper mainly uses number of the parameters to compare model complexity. This is a theoretical metric. I would also like to compare GFLOPS between MobileViT and other competing models (e.g. MobileNet family models, DEIT, MnasNet, PiT).\n-\tTable 1 (b), comparisons with heavy-weight CNN are not quite convincing. More recent backbones should be considered, such as MobileNetV3, NAS-FPN, DET-NAS.\n-\tTable 2, PASCAL VOC 2012 is a small benchmark for segmentation. It would be more convincing to include results on more recent bencmarks such as COCO, LVIS. The competing approaches MobileNetv1, v2 and R-101 are not up-to-date. Please consider more recent backbones such as MobileNetV3, NAS-FPN, DET-NAS.\n-\tTable 2, is there a strong reason to not compare with transformer-based backbone, such as Multi-scale Vision Transformer and Pyramid Vision Transformer.\n-\tTechnical exposition: After Eqn (1), “Another convolutional layer is then used to fuse local and global features in the concatenated tensor”.  This is a bit confusing. According to Figure 1 (b), the concatenation takes the original feature and another feature computed by local conv and global self-attention building block\n", "summary_of_the_paper": "This work proposes a hybrid backbone which combines existing MV2 block and new MobileViT block for efficient classification, detection and segmentation. The core idea with MobileViT block is to use both convolution and self-attention block to aggregate local and global feature, respectively. Experiments on IN-1K, COCO and PASCAL VOC have shown quite promising results.", "summary_of_the_review": "The overall design of MobileViT block is neat, simple and effective. The MobileViT model is a competitive example of combing existing CNN blocks and new transformer-based block to achieve high accuracy and low-latency on mobile CPUs.\n\nThe paper quality can be improved by disentangle the improvement from the claimed contributions (i.e. MobileViT architecture, multi-scale training) from the rest (e.g. label smoothing, EMA, and any other training recipe under the hood). This is important for the main results in Figure 6 and 7.\n\n ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "c9mz5tw5Fkw", "reviewer": "Reviewer_GY8P", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- (+) the paper is well written, the contributions are clearly stated and supported by empirical results.\n- (+) the proposed backbone architecture exhibits improved performance with respect to previous work based on ViT: robustness to hyperparameters, training stability and generalization capability.\n- (+) with the proposed block the authors are able to reduce the number of parameters while preserving or improving performance on classification, object detection and segmentation, suggesting they are indeed providing a good backbone for many downstream tasks.\n- (+) the core technical idea of the paper is simple and convincing. I like the idea that by removing the need for positional encoding, authors enable multi-scale training of ViTs.\n- (+) the experiments are well designed to support most of the authors' claims. They wisely compare against both famous CNNs and ViT-based recent work. Training and design details are provided and the overall method seems reproducible. The model is trained from scratch, hinting to the optimization robustness claimed by the authors.\n\nThe main drawbacks I see in the paper are related to memory and time performance analysis, that should be better clarified given the mobile focus of the work:\n- (-) The worst case analysis presented in Sec. 3.1 is not helpful to the reader, since it does not explain eventual speed improvements. The analysis is i) worst case, ii) asymptotic and, to my understanding, iii) the parameters involved change between MobileViT and standard ViTs. I think the analysis should either be fixed or replaced with a FLOPs count comparison.\n- (-) Together with the number of parameters and time complexity / FLOPs, memory footprint is another important element for embedded computer vision algorithms based on deep learning. This is particularly true given the notoriously poor performance of transformers on this topic.\n- (-) MobileViT is compared against MobileNetV2, DeIT and PiT on an iPhone 12. It would be interesting to see the same comparison on a number of different devices, e.g. a low end mobile device, as well as GPU (for which only training timings are reported) and CPU.\n\nMinor issues (do not contribute to review scoring):\n- at the end of Sec. 4.3 Mobile-friendly, there is a reference to \"larger inputs\" not quantified until later in the appendix. Since the authors are drawing a conclusion here, it would be more appropriate to state it.\n- in the abstract and in Sec. 3 the term \"transformers as convolutions\" is used, but it is only explained later in Sec. 3.1 Relationship to convolutions. At this point the reader cannot make much sense of those words. I would suggest a rephrasing, or dropping the term altogether as I personally don't think it adds anything to the understanding of the contributions... I actually found it confusing.", "summary_of_the_paper": "In this paper the authors introduce a new block that combines the global receptive field of transformers with the positional bias of convolutions. They build on this block to produce an architecture that compares favourably against both well known CNNs and classical ViT solutions in terms of number of parameters and task performance ratio.", "summary_of_the_review": "The contribution is interesting and claims are supported. Unfortunately, the analysis part is a bit lacking as I highlighted in the main review section.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "4dm_X_B6xBl", "reviewer": "Reviewer_XdK7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Weaknesses\n- How to reflect the low latency attribute of this network?\n- Why use unfold operation to get non-overlapping flatten vectors rather than flatten the input feature directly?\n- Could the authors provide the comparison of the top5 accuracy metric in Figure-7? Besides, the author should provide more details about the competitors for the readers' convenience. For example, BoTNet > BoTNet-50.\n- The reviewer observes that the authors use an exponential moving average strategy during inference. Is the comparison fair for other competitors, such as BoTNet in Figure-7? Note that there is only a 0.1% difference between those two models.\n- From Table-3, we can observe that MobileNetV2 (3.5M & 73.3@top1) outperform the performance of DeiT (5.7M & 72.2@top1) and PiT (4.9 & 73.0@top1). This raises the following two concerns: (a) the base model MobileNetV2 can achieve such promising performance. Is the choice of base model contributes to holding the superiority of MobileNetViT? Have you ever tried other base light-weight models? (b) The efficiency-performance ratio of MobileNetV2 is better than the proposed MobileNetViT. Thus, the value in the real-world practice is smaller than the MobileNet families.\n- The PASCAL VOC 2012 dataset is too old-fashioned to verify the actual effectiveness of the proposed method. The authors should conduct the experiments on ADE20K [1*].\n- Lack of visualisation results in object detection and segmentation tasks.\n- Some typos: DeIT > DeiT.\n\n# Reference\n[1*] Scene parsing through ade20k dataset. CVPR2017.", "summary_of_the_paper": "This paper introduces a lightweight and general-purpose vision transformer, termed MobileViT, for mobile devices. It attempts to build a model which can combine the strengths of CNNs and ViTs to build a light-weight and low latency network for mobile vision tasks. MobileViT achieves top-1 accuracy of 78.4% on ImageNet-1k dataset (classification task), 27.7 mAP on MS-COCO dataset (object detection task), and 79.1 mIOU on VOC 2012 dataset (semantic segmentation task).", "summary_of_the_review": "Overall, I think this paper is well-written and -organised. The promising performance impresses me a lot. \n\nHowever, this model stack the mobilenet and transformer models, and thus, the novelty and theory may not be enough for the ICLR community's publication. Besides, according to the reviewer's previous experience, the proposed framework can also work well in most lightweight backbones. The reviewer suggests that the authors should conduct extended experiments to verify this framework is a CNN-agnostic strategy. It can provide general insight to the researcher. Finally, this model's mobile device application is limited (though the authors mention this point in the submission).\n\nOverall, the reviewer votes for the borderline in the first round. The reviewer is also happy to change the initial score according to the quality of the rebuttal content.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a light weight hybrid model using both convolutions and Transformer layers resulting in models with lower computational cost and good performance. Reviewers find the paper interesting and agree that the paper did a good job in presenting convincing experimental results. There were questions about role of different components in the proposed model, which author's addressed in the response with additional ablation studies. One reviewer expressed concerns about lack of theoretical foundations for the proposed approach. However they also agree that the paper presents a good and useful experimental study. I think overall the paper has good contributions that others can build on in the future and recommend acceptance.
Decision: Accept (Poster)