MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features
Authors: Shakti Nagnath Wadekar
Abhishek Chaurasia
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Learning
Vision Transformers
Convolutional Neural Networks
Mobile Vision Transformers
Light-weight neural network

Abstract: MobileViT (MobileViTv1) combines convolutional neural networks (CNNs) and vision transformers (ViTs) to create light-weight models for mobile vision tasks. Though the main MobileViTv1-block helps to achieve competitive state-of-the-art results, the fusion block inside MobileViTv1-block, creates scaling challenges and has a complex learning task. We propose changes to the fusion block that are simple and effective to create MobileViTv3-block, which addresses the scaling and simplifies the learning task. Our proposed MobileViTv3-block used to create MobileViTv3-XXS, XS and S models outperform MobileViTv1 on ImageNet-1k, ADE20K, COCO and PascalVOC2012 datasets. On ImageNet-1K, MobileViTv3-XXS and MobileViTv3-XS surpasses MobileViTv1-XXS and MobileViTv1-XS by 2% and 1.9% respectively. Recently published MobileViTv2 architecture removes fusion block and uses linear complexity transformers to perform better than MobileViTv1. We add our proposed fusion block to MobileViTv2 to create MobileViTv3-0.5, 0.75 and 1.0 models. MobileViTv3-0.5 and MobileViTv3-0.75 outperforms MobileViTv2-0.5 and MobileViTv2-0.75 by 2.1% and 1.0% respectively on ImageNet-1K dataset. For segmentation task, MobileViTv3-1.0 achieves 2.07% and 1.1% better mIOU compared to MobileViTv2-1.0 on ADE20K dataset and PascalVOC2012 dataset respectively.

Under review as a conference paper at ICLR 2023
MOBILEVITV3: MOBILE-FRIENDLY VISION TRANS- FORMER WITH SIMPLE AND EFFECTIVE FUSION OF LOCAL, GLOBAL AND INPUT FEATURES
Anonymous authors Paper under doublc-blind review
ABSTRACT
MobileViT (MobileViTvI) combines convolutional neural networks (CNNs) and vision transformers (ViTs) create light-weight models for mobile vision tasks. Though the main Sobileeiiyl-bloeee helps to achieve competitive state-of-the- art results, the fusion block inside Mobile ViTveeiiilllloo creates scaling challenges and has a complex learning task. We propose changes to the fusion block that are simple and effective to create Mobile and simplifies the learning task. Our proposed ViTyle MobileViTv3-XXSS XS and S models outperform Mobile ViTv on ImageNet-IK, ADE20K, COCO and PascalVOC2012 datasets. On ImageNet-18 K, MobileViTv3- XXS and MobileViTv3-XX surpasses MobileViTvI-XX: and MobileViTvI-XS by 2% and 1.9% respectively. Recently published Mobile architecture re- moves fusion block and uses linear complexity transformers to perform better than ViTvl We add our proposed fusion block to MobileViTv2 to create MobileViTv3-005, 0.75 and 1.0 models. ViTv3-- and ViT outperforms VobileViTv2-4 v-00.5 and Mobile by 2.1% % and 1.0% respec- tively on ImageNet- KK dataset. For segmentation task, Mobile 2.07% and 1.1% better mIOU compared to ViT 1.0 ADE20K dataset and PascalVOC201 dataset respectively. Our code and the trained models will be made available GitHub.
which addresses the scaling used to create
achieves
INTRODUCTION
2016 DenseNet (Huang et al. 2017) and EfficientNet (Tan & Le [019 are widely used for vision tasks such as classification, detection and segmentation, due to their strong performance on the established benchmark datasets such as Imagenet (Russakovsky et al. 2015), COCO (Lin et al., 2014), PascalVOC (Everingham et al.) 2013). ADE20K (Zhou al. 2017) and other similar datasets. When deploying CNNs edge devices like mobile phones which are generally resource constrained, light-weight CNNs suitable for such environments come from family of models of MobileNets (MobileNetvi, MebvleNetv2, MobileNetv3) (Howard et al. 2019), ShuffleNets (ShuffleNetvi and ShuffleNetv2) (Ma et al.) 2018) and light-weight versions of EfficientNet (Tan & Le| 2019) (EffficienNeet-O0 and EfficientNet-BI), These relatively small models lack in accuracy when compared to models with large parameters and FLOPs. Recently, Vision Transformers (ViTs) have emerged an strong alternatives CNNs on these vision tasks. Self-attention mechanism in ViTs interacts with all parts of the image to produce features which have global information embedded them. This has been demonstrated to produce comparable results to CNNs but with large pre-training data and advance data augmentation (oosovitskiy et al.) 2020). Also, this global processing comes at a cost large parameters and FLOPs match the performance of CNNs as seen in ViT (Dosovitskiy et al. 2020), and its different versions such as DeiT (Touvron et al. I02 SwinT (Liu et al. 2021). MVIT (Fan et al. 2021).
Convolutional Neural Networks (CNNs) [ResNet (He et al.
Focal-ViT (Yang al. 2021), PVT (Wang al. 2021 T2T-VIT (Yuan et al. 2021b), XCTT (Ali et al. 2021)
Many recent work have introduced convolutional layers in ViT architecture form hybrid networks to improve performance, achieve sample efficiency and make the models more efficient terms of parameters and FL.OPs like MobileViT's (MobileViTvl (Mehta & Rastegari 2021). MobileViTv2

---

Under review as a conference paper at ICLR 2023
81
Mublekieess
Coaf-Ute Mini
MusteWita--00
XGT-712
ResT-Smal
79
78
McbieViiVLS
WIiv-881
MobieviiV3
Mobile Former 508M
MobleитVV-10
Accuracy
76
MobreviTV2 0.75
CosT
VIL Tiny RPB
Res
75
Mobil657c2-00755
LVT
TNT
Woble-Formeh-S511
ConMil-Ti
74
MobIeNTTV--S
Min-DefF-
PI-TI
Tiny
73
M.
CanVil-T
MobiReVIVV
XCIT-112
70
PVTv2:B8 80
Del--ii
500
3000
1900
2000
M.b
Fobile
MobiekiTV Tit Xes
Mossll-orrme-53M
68
5
7
9
10 11
12
13 14
15
#Parameters (in millions)
Figure 1: Comparing Top-1 accuracies of MobileViTv3, VIT variants and hybrid models on ImageNet- 1K dataset. The area of bubbles correspond to number of FLOPs in the model. The reference FLOP sizes are shown in the bottom right (example, 250M is 250 Aega-FLLPP/MMiliinn FLOPs). Models of our Mobile ViT'3 architecture outperforms other models with similar parameter budget of under 2M, 2-4M and 4-8M. Also, they achieve competitive results when compared to the models greater than 8M parameters.
Mehta & Rastegari (2022) ) CMT Guo et al. 2022), CvT (Wu et al., 2021). PVTv2 Wang et al., 2022). ResT (Zhang & Yang 2021). MobileFormer (Chen et al. 2022), CPVT Chu et al. 2021).
Min (Zhang et al 2022). CoAtNet (Dai et al. 2021). CoaT (Xu et al. 202 Performance of many of these models on ImageNet-IK, with parameters and FLOPs is shown in Figure ! Among these models, only Mobile viTs and MobileFormer
specifically designed for resource constrained environment such as mobile devices. These two models achieve competitive performance compared other hybrid networks with less parameters and FLOPs. Even though these small hybrid models are critical for the vision tasks on mobile devices, there is little work done in this area.
Our work focuses improving one such liigh family of models known Mobile (Mo- ViTvv (Mehta & Rastegari 202I) and MobileViTv2 (Mehta & Rastegari 2022)). When com- pared the models with parameter budget of 6 million(M) or less, Mobile ViTs achieve competitive state-of-the ahe-art results with a simple training recipe (basic data augmentation) on classification task. Also it can be used as an efficient backbone across different vision tasks such as detection and seg- mentation. While focusing on only the models with 6M parameters or less, we pose the question: Is possible to change the model architecture to improve its performance by maintaining similar pa- rameters and FL.OPs? To do so, our work looks into challenges of MobileViT-bloce architecture and proposes simple and effective way to fuse input, local (CNN) and global (ViT) features which lead to Imagenet-IK, ADE20k, PascalVOC and COCO dataset.
significant performance improvements We propose four main changes to Mobile block (three changes w.r.t MobileViTv2 block) as shown in figure 2 Three changes in the fusion block: First, 3x3 convolutional layer is replaced with Ix1 convolutional layer. Second, features of local and global representation blocks are fused to- gether instead of input and global representation blocks. Third, input features added in the fusion block as a final step before generating the output of Mobile block. Fourth change is proposed in local representation block where normal 3x3 convolutional layer replaced by depthwise 3x3 con- volutional layer. These changes result in the reduction of parameters and FLOPs of MobileViTv block and allow scaling (increasing width of the model) create new Mobile ViTv3SS, XS and XXS architecture, which outperforms Mobile ViTvl on classification (Figure D segmentation and detection tasks. For example, Mobile V and MobileViTv3-P3 perform 2% and 1.9% better with similar parameters and FLOPs ImageNet-1K K dataset compared MobileViTv1-XX and Mobile respectively. Mobile fusion block is absent. Our proposed fusion block introduced in Mobile architecture to create MobileViTvv.1.00 0.75 and 0.5 architectures. Mobile V-Tv3-0.5 and ViTv3-0.7 outperforms VobileViTv2-0.5 and VobileViTv2-00 by 2.1 % and 1. 1.0% respectively with similar parameters and FLOPSs ImageNet- KK dataset.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
Vision Transformers: VIT Dosovitskiy et al.
2020) introduced the transformer models used for
Natural Language Processing tasks to vision domain. specifically for image recognition. Later, its different versions such as DeiT (Touvron et al., 2021) improved the performance by introducing novel training technique and reducing the dependency on large pre-training data. Works focusing on
improving oeff-attention mechanism to boost performance include XCIT (Ali et al., (02I), SwinT (Liu et al. 2021), ViL (Zhang et al. 202I) and tranffrranformer (Yang et al.] (021). XCIT in- troduces cross-covariance attention where self-attention is operated feature channels instead of tokens. Swin'T modified the ViT to make it a general purpose architecture for classification, de- tection and segmentation tasks. ViL improves ViT by encoding image multiple scales and uses self-attention mechanism which is a variant of Longformer (Beltagy et al.) 2020). Recent works like T2T-ViT (Yuan et al. b021b) and PVT (PVTv1) (Wang al. (021 also focus on introducing CNN like hierarchical feature learning by reducing spatial resolution or token sizes of output after each layer. Few new architectures like Cross ViT (Chen et al. 2021). MVIT (Fan et al.) (021). MViTv2 (Li et al. 2022) and Focal tonal--tasffommer (Yang et al. [202] learn both local features (features learnt specifically ly from neighbouring pixels/feanrres//acchee and global features (features learnt using all pixels/featurrs/pacches).
CNNs: Models like EfficientNet-B7 (Tan Le 2019). ConwNeXt (Liu al. 2022), EfficientNetV2
(Tan & Le| (02I). YegNetY (Radosavovic et al. 2020) and NFNet-I4+ (Brock et al. 202 achieve high accuracy on ImageNet- IK dataset and also many can be used as a general purpose backbone models for detection and segmentation tasks. But, these CNN models generally high in num- ber of parameters and FLOPs. Light-weight CNNs that achieve competitive performance with less parameters and FLOPs include EfficientNNe-B00 and B1, MobileNetV3 (Howard al. 2019), Shuf- fleNetv2 (Ma et al. 2018) and ESPNety2 (Mehta et al.] (209)). EfficientNer studied model scaling and developed family of efficientnet models which are still one of the most efficient CNNs in terms of parameters and FLOPs. MobileNetV3 belongs to category of models specifically developed for resource constrained environments such as Mobile phones. Building block of MobileNetV3 archi- tecture uses MobileNetv2 (Sandler et al. 2018) block and Squeeze-and aaeee-nnnd-iittte (Hu et al., 2018) network in it. ShuffleNetv2 studies and proposes guidelines for efficient model design and produces 2hufflenetv2 family of models which also performs competitively with other light-weight CNN mod- els. ESPNetv2 uses depth-wer wise dilated separable convolution to create EESP (Extremely Efficient Spatial Pyramid) unit which helps to reduce parameters and FLOPs and achieve competitive results.
Hybrids: Lately, several models are being proposed to combine CNNs and ViTs together in one architecture to capture both long-range dependencies using self-attention mechanism of ViT and local information using local kernels in CNNs to improve performance on vision tasks. Mobile (Mobile VITv1 (Mehta & Rastegari ) Mobile Mehta & Rastegari (2022) and Mobile- Former (Chen et (022) have been specifically designed for constrained environments like mobile devices. CMT (Guo et al. 2022) architecture has convolutional stem and stacks convolutional lay- ers and transformer layers alternatively. CvT (Wu et al., 202T) uses convolutional token embedding instead of linear embedding and a convolutional transformer layer block that leverages these convo- lutional token embeddings to improve performance. CoAtNet (Dai et al. 2021) unifies depthwise convolution and self-attention using simple relative attention and vertically stacks convolutional lay- enhanced-feed-ffeee
layer by using depth-wise wise convolution with other changes to achieve competitive results. VITAE (Xu et al., 202 has convolution layers in parallel to multi-head self-attention module and both are fused and fed to feedforward network, also ViTAE uses convolutional layers to embed inputs to token. RVT (Mao et al., 2022) uses convolutional stem to generate patch embeddings and uses convolutional foed-forward network in transformer to achieve better results.
ers and attention layers. CeiT (Yuan et al.) (221a)) introduces
3 NEW MOBILEVIT ARCHITECTURE
Our work proposes four design changes to the existing MobileViTvI block architecture to build Mobile VobileViTvy3-looo as shown in Figure 2a Section expplains these four changes in ViTileViTv3: block architecture and compares with Mobile ViTvI I and VobileViTv2-blocke Section 3.2 details VibTleVVTT33.. XS and XXS architectures and shows how it is scaled compared to Mobile ViTvl- S, XS and XXS. In recently published Mobile architecture, changes applied to ViT-
3

---

Under review as a conference paper at ICLR 2023
MobileViTvi
block for MobileViTv1 XS
MobileViTv2 block MobileViTv2-005, 0.75 and
Brock
Local ispreannialln
Local rsprcantalkan
Glea
Giebll
Naokk
Pussen Stock
Pusken Slock
Locel reproooration
Locell eeposooaation
Glibal
NN
MobileViTvv block for MobileVITv3-XXS,
MobileViTv3 block MobileViTvv--0.,
(a) MobileViTvl and MobileViTvà blocks
(b) Mobile and Mobile blocks
Figure 2: A comparison between: (a) Mobile and MobileViTvv modules, and (b) Mobile- ViTv2 and MobileViTv3 modules. The proposed architectural changes are highlighted in red.
block are, fusion block is removed, transformer block uses self-attention with linear complexity and depthwise convolutional layer used local representation block. We add back the fusion block to Mobile with our proposed changes to create Mobile ViTv3--o shown in Figure 2b for Vob-leVi 0.75, 1.0 architectures.
3.1 MOBILEVITV3 BLOCK
Replacing 3x3 convolutional layer with 1x1 convolutional layer in fusion block: Two main mo- tivations exist for replacing 3x3 convolutional layer in fusion. First, fuse local and global features independent of other locations in the feature map simplify the fusion block's learning task. Con- ceptually, 3x3 convolutional layer is fusing input features, global features, and other location's input and global features which are present in the receptive field, which complex task. Fusion block's goal can be simplified by allowing it to fuse input and global features, independent of other locations feature map. To do so, we use 1x1 convolutional layer in fusion instead of 3x3 convolutional layer. Second, is to remove one of the major constraints in scaling of MobileViTvi architecture. Scaling ViTil from XXS to done by changing width of the network and keeping depth con- stant. Changing width (number of input and output channels) of Mobile ViTvl block causes large increase in number of parameters and FLOPs. For example, the input and output channels are doubled (2x) in Mobile ViTv1 block, the number of input channels 3x3 convolutional layer inside fusion block increases by 4x and output channels by 2x, because input to the 3x3 convolutional layer concatenation of input and global representation block features. This causes large increase in parameters and FLOPs of MobileViTv block. Using Ixl convolutional layer avoids this large in- crease parameters and FLOPs while scaling. Local and Global features fusion: In fusion layer, features from local and global representation blocks are concatenated in our proposed Mobile ViTv3 block instead of input and global representation features. This because the local representation features are more closely related to the global representation features when compared to the input features. The output channels of the local representation block are slightly higher than the channels in input features. This causes an increase in the number of input feature maps to the fusion block's 's Ix1 convolutional layer, but the total number of parameters and FLOPs are significantly less than the baseline MobileViTv block due to the change of 3x3 convolutional layer to 1xl convolutional layer. Fusing input features: Input features are added to the output of 1xl convolutional layer in the fusion block. The residual connections in models like ResNet and DenseNet have shown to help
4

---

Under review as a conference paper at ICLR 2023
the optimization of deeper layers in the architecture. By adding the imput features the output in the fusion block, we introduce this residual connection in new Mobile ViTv3 architecture. Ablation study results shown in table demonstrates that this residual connection contributes 0.6% accuracy gain. Depthwise convolutional layer in local representation block: To further reduce parameters, 3x3 convolutional layer in local representation block is replaced with depthwise 3x3 convolutional layer. As seen in the ablation study results table 5 this change does not have a large impact on the Top-1 ImageNet-IK K accuracy gain and provides good parameter and accuracy trade-off.
3.2 SCALING BUILDING BLOCKS
Applying the changes proposed in section 3.1 allows scaling of our MobileViTv3 architecture by increasing the width (number of channels) of the layers. Table shows VobileVVTV33-1 XS and XXS architectures with their output channels in each layer, scaling factor, parameters an FLOPs.
Table VobileViTv3-S, XS and XXS architecture details and comparison with MobileViTv1-S, XS and XXS. Values given in brackets '()' represent scaling factor compared Mobile ViTv models.
Layer
Size
Stride
Repeat
XXS
XS
Image
256x256
3x
128x 128
16
MV2
128x 128
32
MV2 MV2
64x64
64x64
4
2
24
48
64
MV2, 2 Mibile block (L=2)
32x32
8
64 (1.3x)
96 (1.5x)
128 (1.3x)
32x32
64 (1.3x)
96 (1.5x)
128 (1.3x)
MV2, 4 Mobile VIT block (L=4)
16x16
16
80 (1.3x)
160 (2.0x)
256 (2.0k)
16x16
16
(1.3x)
160 (2.0x)
256 (2.0k)
MV2. 2
8x8
32
128 (1.6x)
160 (1.7x)
320 (2.0k)
160
(2.0x)
Conv-1xl,
(1.6x)
640 (1.7x)
1280 (2.0x)
Global pool Lincar
256
512
640
1280
256
1000
1000
1000
Parameters (M)
1,25
2.5
5.8
(M)
4 EXPERIMENTAL RESULTS
segmentation task using ADE20K and PASCAL VOC 2012 datasets in section 4.2 detection task using COCO dataset in section 4.3 We also discuss changes to our proposed MobileViT33 architecture for improving
Our work shows results on classification task using ImageNNtt-1E IK in section 4.1
latency and throughput in appendix B
4.1 IMAGE CLASSIFICATION ON IMAGENET- 1K
Implementation details: Except for the batch size, hyperparameters used for MobileViTv3-S, XS and XXS are similar the ViTileViTvI and hyperparameters used for MobileViTv3-1.0. 0.75 and 0.5 are similar to VobileViTv2. Due to resource constraints, we were limited to using a total batch size of 384 (32 images per GPU) for experiments on MobileViTy3-S and XS. To maintain consistency in batch sizes, Vobi is also trained on batch size of 384. Batch size of 1020 (85 images per GPU) used of V0T3---V7 and 1.0 training. More hyperparameter details in appendix C.I Performance is evaluated using single crop top-1 accuracy, for inference an exponential moving average of model weights is used. All the classification models are trained from seratch on the ImageNet-1K K classification dataset. This dataset contains 1.28M and 50K images for training and validation respectively.
Comparison with MobileViTs: Table 2 demonstrates that performance of all the versions of Mo- ViTv3 surpass MobileViTvl and MobileViTv> versions with similar parameters and FLOPs and smaller training batch size. Increasing total batch size from 192 384 improves accuracy of
5

---

Under review as a conference paper at ICLR 2023
Vobile XS and XXS models (Table 2 This indicates the potential for further accuracy gains with batch size of 1024 on MobileViTv3-XXx XXxs, XS and S models. It is also important to note that ViT XS and XXS models trained with basic data augmentation not only outper- forms VibileViTv1-3 XS, XXS, but also surpasses performance of Mobile 0.75 and 0.5 which are trained with advanced data augmentation.
Table ViT VI, V2 and V3 comparison in terms of Top-1 ImageNet- accuracy, parame- ters and operations. Models with similar parameters and operations are grouped together for clear comparison.
Model
Training Batch size
FLOPs (M)
Params. (M)
Top-1 (%+)
MobileViTv- -XXX
1024
364
1.3
69.00
MobileViTva-XX5
192
289
1.2
70.02 (+1 %%)
VITV-XX
384
289
1.2
70.98 (+2%)
MobileViTv2-0055
466
1.4
70.18 72.33 (+2.1%)
Mob-leVITv3-0.5
1020
481
1.4
MobileVVTvi-XS
1024
986
2.3
74.8
MobileViTv3-XS
192
927
2.5
76.3 (+1.5%%
MobileViTv3-XS
927
76.7 (+1.99%)
MobileViTv2...5
1024
1030
2.9
75.56
MobileViTVv-0.75
1020
1064
3.0
76.55 (40.99%)
MibiileiTvv..S
78.4
MobileViTvå-S
192
1841
5.8
78.8 (+0.4%%
VobileViTv3-S
384
1841
79.3 (+0.9%%
MchileVTv2-L.0
1024
1851
4.9
78.09
MobileViTv3-100
1876
(+0.55%))
Comparison with ViTs: Figure [ compares our proposed MobileViTv3 models performance with other VIT variants and hybrid models. Following MobileViTvI, mainly compare our models with parameter budget around 6M or less. Also, when comparing to models greater than 6M parameters, we limit FLOPs budget to ~2 GFLOPs or less because largest model in this work has ~22 GFLOPs. Models under million parameters: To the best our knowledge, only Mobile ViT variants exist this range. VobileViTvv-3 variants. ViT by far achieves the best accuracy of 72.33 %. Models between 2-4 million parameters: Mobile V-T and MobileViTv3-00755 outperform all the models in this range. Top- accuracy of MobileViTv3-XX ViTv3-XS on ImageNet- 1k is 76. 76.7%, which is 3.9% higher than Mini-DeiT- Ti (Zhang et al. 2022) and 4.5 % higher than XCTT-N12 (Ali et al. 2021 Although Mobile- Former-53M (Chen et al. 2022) uses only 53 GFLOPs, it lags in accuracy by a large margin of 12.7% with Mobile vTT Models between 4-8 million parameters: MobileViTv3-S attains the highest accuracy of 79.3% in this parameter range. Mobile ViTv3-S with simple training recipe and 300 training epochs is 0.7% better than XCiT-TI2 trained using distillation, advanced data augmentation and 400 epochs. It is 1.8% and 2.6% better than Coat-Lite-Tie Tiny (Xu et al., 2021al la) and FiL-TinyyRPP (Zhang et al. 2021 respectively, MobileViTv3-SS is 1 1% better with 0.5x FLOPs and similar parameters as compared to CoaT-Tiny Xu al., 2021a). Models greater than 8 million parameters: We also compare our designed models with existing models having more than 8M parameters and around 2 GFLOPs. When compared with Mobile ViT (79.3%) trained with basic data augmentation and 300 epochs, MobileFormer-5082 achieves similar accuracy of 79.3% with 2..5 more parameters, ~3.5x less FLOPs, advance data augmentation and 450 training epochs, CMT-Ti (Guo et al. 2022 achieves 79.1% with 1.66 more parameters, 2.9x less FLOPs (due to input image size of 160x and advanced data augmentation.
and Vobi-e- outperform other Mob
Comparison with CNNs: 3 compares our proposed models with the CNN models which are weight with parameter budget of ~6M or less, similar Mobile (Mehta Rastegari] 2021). Models in 1-2 million parameters range: Mobile ViTv3-0.5 and VibileViTv3-XXS with 72.33% and 70.98% respectively are best accuracies this range. MobileViTv3-0.5 achieves over 2.5% improvement compared to CSPNetv2-123M. MobileNett3-smalll.. and MobileNeeV2(0.5) (Sandler et al., 2018). Models with 2-4 million parameters: Mobile vTTv3- XS achieves over 4% improvement compared to MobileNett3-LLrgeell
(Howard et al.) 2019)
ShuffleNeev2(1155)
2SPNetv2-284M and MobileNetv2(0.75) Models with 4-8 million parameters: MobileViTv3.
S shows more than 2% accuracy gain over EfficientNet-B0 (Tan & Le 2019), ESPNetv2-602M

---

Under review as a conference paper at ICLR 2023
81
Milllesfii-..55
Musieslled
MULRENTIVG
Filr aortee-
Mctrww-TT2-L..
Srceralter33-arg (1:25)
Stuffehha2 [LS)
75
Modree
ESPRiced
Accuracy
Stuffettet22 SMM
ESPNeIv2-CCIK
71
efficssties
Moniiekenel 11:001
67
Medrishutt/ Larae (0.551
Mobéichuttc (0.75)
Motiieictizz (2.50)
63
Motarkenthi aqp (D-35)
Mobi ieteevt S0t
Mobrisher''s Mobiikeev22 (0:5))
201
59
(0:))
1 500
millionns)
Figure 3: Top-1 accuracy comparison between Mobile V3T models and existing light-weight CNN models on ImageNet- 1K dataset. The bubble size corresponds to the number of FLOPs. The refer- ence FLOP sizes are shown the bottom right (example, 50M 50 Mega=FLLPx>Milloon-TT Models of our Mobile architecture outperform other models with similar parameter budget of 1-2M, 2-4M and 4-8M.
and MobileNeew3-LLrree.... (.22) EfficientNet-Bl with 1.3x more parameters and 2.6x less FLOPs achieves competitive accuracy of 79.1 1% compared MobileViTv3-S with accuracy of 79.3%.
4.2 SEGMENTATION
Implementation details: The segmentation performance is evaluated on the validation set and re- ported using mean Intersection over Union (mlOU). PASCAL VOC 2012 dataset: Following Mo- ViTvV. Mobile
is integrated with DeepLabv3 (Chen et al. (017)) for segmentation task on PASCAL VOC 2012 dataset (Everingham et al. 2015). Extra annotations and data used from HHariharan et al. 2011) and (Lin et al. (2014) respectively, which is standard practise for training on PascalVOC201 12 dataset (Chen et al. 2017); (Mehta et al. 2019).. For Vobile XS and XXS, training hyperparameters are similar to Mobile except the batch size. Smaller batch size of 48 (12 images per GPU) is used compared to 128 (32 images per GPU) for Mobile For 0.75 and 0.5, all the hyperparameters are kept same used for MobileViTv2- 1.0, 0.75 and 0.5 training. ADE20K dataset (Zhou et al.) 2019): Contains total 25K images with 150 semantic categories. Out of 25K images, 20K images used for training, 3K images for test and 2K images for validation. Same training hyperparameters are used for MobileViTv2 models as MobileViTv3-1.0 0.75 and 0.5 models.
MobileViTv3-
Results: PASCAL VOC 2012 dataset: Table 3a demonstrates MobileViTv3 models with lower training batch size of 48, outperforming their corresponding counterpart models of MobileViTvl and Mobile which are trained on higher batch size of 128. MobileViTv3- 1.0 achieves 80.04% mIOU, which outperforms Mobile ViTv2-1.0 by 1.1%. MobileViTv3-XS is 1.6% bet- ter than Mobile dataset: Table 3b shows the results of Mobile 0.75 and 0.5 models on ADE20K dataset. Mobile 1.0, 0.75 and 0.5 models outperform MobileViTv2-1 0.75 and 0.5 models by 2.07%, 1.73% and 1.64% respectively.
and MobileViTv3-0.5 surpasses MobileViTv2-0.55 by 1.41%. ADE20K
4.3 OBJECT DETECTION
Implementation details: MS-COCO dataset (Lin et al. 2014) with 117K training and 5K valida- tion images, is used to evaluate the detection performance MobileViTy3 models. Similar to Mo- ViTv. we integrated pretrained MobileViTv3 network (SSD) (Liu et al. 2016) and the standard convolutions in the SSD head are replaced with separable convolutions create SSDLite network. This StDLite with pre-trained MobileViTv3
backbone network in Single Shot Detection

---

Under review as a conference paper at ICLR 2023
Table Comparing MobileViTv. segmentation task results on PASCAL VOC 2012 and ADE20K datasets. # params of MobileVit ViT models denotes the number of parameters of the encoder/backbone architecture only.
(a) Segmentation on PASCAL VOC 2012 dataset
Backbone
# Params (M))
mIOU(S) f
(b) Segmentation on ADE20K dataset
VI-XS
1.9
73.6 74.04 (+0.44%)
MobileiIV3-XXS
196
Backbone
Params (M))
mIOU(S) 7
MobileV//VV-0055
6.2
75.07
Mobile
6.31
31,93
MobileViIv3-00.5
6.3
76.48 (+1.419)
6.37
33.57 (+1.64%)
MobileViTVI-XS
2.9
77.1
MobileV/TVV--0.5
9.6
34.7
78.77 (+1.69)
36.43 (+1.733)
MobileViTvii-S
79.1
MobileViTv2-110
37,06
VIV
7.2
79.59 (+0.49%)
13.62
39.13 (+2.00%%)
MobileV//V2-1.0
13.32
78.94
MobileVi1vv--10
13,56
80.04 (+1.10%)
backbone is fine-tuned on MS-COCO dataset. No change to training hyperparameters. Detailed
list of hyperparameters in appendix C.2 Performance evaluation is done on validation using
mAP@IoU 0. 0.50:0.05:0.95 metric.
Results: Table 4a and 4b show the detection results on COCO dataset. # params of Vib models indicates number of parameters of the encoder/backbone architecture only. Mobile comparison with other light-weight CNN models is shown Table 4a MobileViTv3-XX outper- forms MobileViTv 1-XS by 0.8% and MNASNet by 2.6% mAP. Comparison with heavy-weight CNNs detailed in Table [4b] Mobile and MobileViTv3-1. surpasses MobileViTvI-XS and Mobile by 0.8% and 0.5% mAP respectively.
Table 4: Comparing VibileViTv3 detection task results on COCO dataset with light-weight and heivy- CNNs. # params of MobileViT models denotes the number of parameters of the encoder/backbone architecture only.
(a) Comparison welihht-eeight CNNs
Backbone
Params (M)
mAP(S))
(b) Comparison wheavy- CNNs
MobileViTVv-X0N
1.5
18.5
MobileViTva-XXS
(10.88%
Backbone
Params (M))
mAP((S) f
MohileViTv2-0.5
21.24
MobileViTv3-0.5
21.8 (10.56%)
VGG
35.6
25.1
ResNe150
22,9
(10.0%)
MobileViTVV2-0755
24.57
VobileViTv --S
2.7
24.8 (10.4%)
MobileViTva-0.75
3.7
25.0 (10.43%)
Mobile VIT
2.7
25.6 (10.4%)
MobileNetv.
4,9
22.0
MobileViTv2-1.0
5.6
26.47 (11.27%)
MobileViTv33..0
5.8
27.0 (118%)
MobileNetv2
4.3
22.1
MobileNetvl
MixNet
4.5
22.2
22.3
VibileViTv3-S
MobileViTvi-S
5.7
5.5
27.7 (12.5%)
(12.1%)
MNASNet
4.9
23.0 (10.0%)
VobIleVIIVI-XS
2.7
24.8 (71.89)
2.7
25.6 (12.6%
4.4 ABLATION STUDY OUR PROPOSED MORILEVITV3 BLOCK
Implementation details: We study the effect of the four proposed changes on MobileViTvl-S block by adding changes one by one. The final model with all the four changes our unscaled version and we name it: Mobile VobileViTvv-SSunneeee ViTss-S(usscalee). To match the number of parameters of Mobile we increase the width of S giving us MobileViTv3...
this ablation study we train models for 100 epochs, use batch size of 192 (32 images per GPU) and other hyper-parameters are default as given in section 4.1 In Table 5 'conv-3x3': 3x3 convolutional layer in fusion block, "conv-1x1': 1x1 convolutional layer in fusion block, Input-Concat' concatenating input features with global representation in the fusion block, 'Local-Concan concatenating recal-representation block output features with global representation in the fusion block, 'Input-Add': adding input
8

---

Under review as a conference paper at ICLR 2023
features to the output of fusion block, 'DWConv': using depthwise convolutional layer in the local representation block and 'Top-1': Top-1 accuracy on ImageNet- 1K dataset.
Table 5: Ablation study of MobileViTv3 block. 'unscaled' indicates that the number of channels in the architecture are kept same as the baseline MobileViTvi. v represents incorporating the change in the block.
Model
Conv 3x3
Conv 1xl
Input Concat
Local Concat
Input Add
DW Couv
Top-1 (18)
MobileViTvi-S
73.7 (10.0%)
Mobuleeiiv3-S(uuscaeee))
74,8 (11.1%)
Mobile-iiVv--Suuscalldd)
74.7 (11)
MothileViTv3-S(umcalee)
(66%)
MobileViTvV3 S(luscaled))))))
75.0 (1133%)
Results are shown in Table 5 The baseline MobileViTvI-S, in fusion block, concatenates input features with global representation block features and uses 3x3 convolu- tional layer. Also, it uses normal 3x3 convolutional layer in the local representation block. This baseline achieves an accuracy of 73.7%. 1. Replacing 3x3 convolution with /x/ convolutional layer in fusion block, Mobile VibilleVTvv-Siie V achieves 1.1% improvement. This result supports the assumption that simplifying the fusion block's task (allowing fusion layer to fuse local and global features independent of the other location's 's local and global features) should help optimization to attain better performance. 2. Along with 1xl convolutional layer in fusion block, concatenating local representation features instead of input features results in similar performance gains of 1% compared to concatenating input features. 3. This allows us to incorporate the next change i.e, to add input features to the output of fusion block to create a residual connection for helping opti- VoobileViVv--SSunseee
With 100 training epochs:
mization of deeper layers in the model. With this change, accuracy gain over the baseline MobileViTvl-S and 0.6% gain over the last change demonstrating the clear advantage of this residual connection. 4. To further reduce number of parameters and 3- depth-wise convolutional layer is used instead of normal convolu- vitvlleeiit--uuume
attains 1.6%
FLOPs in tional layer in the local representation block. gains by achieving 1.3% gain over the baseline. 0.3% accuracy drop can be observed when com- pared to the previous change. We adopt this change since it reduces parameters and FL.OPs without significantly impacting performance and helps in scaling of Mobile V-lokk
maintains high accuracy
3-S(usscalle,), MobileViTv1-S and Vobile Top-1 ImageNet-IK ac- curacy comparisons. With similar parameters and FLOPs after scaling, MobileViTv3-5 is able to exhibit better performance than baseline MobileViTvl-S.
Table 6:
Model
Training batch size
FLOPs (M)
# Params (M) L
Top-1 (%)t
MobileVivvi-S
192
2009
5.6
75.6
MobileWiIv3-Stuneaalld))
192
1636 (118.66)
4.3 (122.7%)
77.5 (119%%
MobileVitvi-S
1024
2009
5.6
78.4
MobileViTv3-S
384
1841 (18.3%)
5.8 (13.6%)
79.3 (10.9%)
With 300 epochs training: Results shown in Table When trained for 300 epochs with the batch size of 192, the baseline MobileViTvI-S achieves Top-1 accuracy of 75.6%, which is lower by 2.8% compared to reported accuracy on Mobile ViTv1-S trained on 1024 batch size. With all the four proposed changes implemented VobileViTvl-S architecture to form the model reaches Top-1 accuracy of 77.5%, which outperforms the baseline by 1.9% with 22.7% and 18.6% less parameters FLOPs respectively. MobileViirv3-Sunssaall
architecture though bet- ter than the baseline MobileViTv1-S with training batch size of 192, performs worse than the MobileViTv1-5 trained at batch size of 1024. Therefore, MobileViTv3-S, XS and XXS models scaled to have similar parameters and FLOPs with batch size of 384. debonstddes that after scaling. MobileViTv3.. is able to outperform Mobile MobileViTv1-S VIT by achieving 79.3% accuracy with similar parameters and FLOPs.
ViTv1-S. XS and XXS and trained
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, al. Xeit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014-20027, 2021.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05/50, 2020.
Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan. High- large-salle image recognition without normalization, 2021. URL https://aarxv..orr/as/////
06171
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. Proceedings of the IEEE/CVF international con- ference on computer vision, pp. 357-366, 2021.
Siang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint vxXiv:/766005887 2017.
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2270-5279, 2022.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882) 2021.
Zihang Dai, Hanxiao Liu, Quoe Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-3977, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98-133 136, January 2015.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 6824-6835, 2021.
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 12175 12185, 2022.
Bharath Hariharan, Pablo Arbeléez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. 2 international conference on computer vision, pp- 991-998. IEEE, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pallern recognition, pp. 770-778, 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang- -hing-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukum Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF international conference computer vision, pp. 1314-1324, 2019.
Jie Hu, Shen, and Gang Sun. Squeeze-and -ndeeetttinieiiitiiionn networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018
10

---

Under review as a conference paper at ICLR 2023
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp.
4814, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context, In European conference on computer vision, pp. 740-755. Springer, 2014.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10012-10022. 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11976-1 1986, 2022.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
Xiaofeng Mao, Gege Qi. Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 12042-12051, 2022.
Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, ceneral-purpose and mobile- friendly vision transformer. arXiv preprint 10.0::2110002178, 2021.
Sachin Mehta and Mohammad Rastegari. Separable self-attention for mobile vision transformers, 2022. URL http:ps://axi...oggg//// http///xwwvv..r//a////.0..00 02680
Sachin Mchta, Mohammad Rastegari, Linda Shapiro, and Hannanch Hajishirzi. Espnetv2: A light- weight, power efficient, and general purpose convolutional neural network. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 9190-9200, 2019.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 10428-10436, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chinh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114, PMLR, 2019.
Mingxing Tan and Quoe V. Le. Efficientnetv2: Smaller models and faster training. 2021. doi: Itttts::://aan
org. abs 2004 00298
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficien image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.

---

Under review as a conference paper at ICLR 2023
Wenhai Wang, Enze Xie, Xiang Li, Deng- Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 568-578, 2021.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415-424, 2022.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22-31, 2021.
Weijian Xu, Yifan Xu, Tyler Chang. and Zhuowen Tu. Co-scale conv-attentional image transform- ers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9981- 9990, 2021a.
Yufei Xu, Qiming Zhang. Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by ex- ploring intrinsic inductive bias. Advances in Neural Information Processing Systems, 34:28522- 28535, 2021b.
Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2 2021.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convo- lution designs into visual transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 579-588, 2021a.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-toker vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 558-567, 2021b.
Jinnian Zhang, Houwen Peng. Kan WU, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12145-12154, 2022.
Pengchuan Zhang, Xiyang Dai, Jianwei Yang. Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A vision transformer for high-resolution image encoding. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2998-3008, 2021.
Qinglong Zhang and Yu-Bin Yang. Rest: An efficient transformer for visual recognition. Advances in Neural Information Processing Systems, 34:15475-15485, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302-321, 2019.
A FINETUNING
Table 7 compares finetuning results of MobileViTv3-LLO, 0.75 and 0.5 with Mobile VITv2-100, 0.75 and 0.3. Implementation details are similar to section 4.1
12

---

Under review as a conference paper at ICLR 2023
Table 7: Fine-tuned Vi V2 and V3 comparison in Top-1 ImageNet- Ik accuracy, parameters and operations. <Model n3me>(384) indicate that the model was fine-tuned using an image size of 384x384. Models with similar parameters and operations are grouped together for clear comparison.
Model
Training
(M).
Params, (Mi)
Top-I (90)
MobhileViTv2 (384)
1048
1.4
72.14
MobileViTv3-0051 (384)
1083
1.4
74.01 (+1.87%)
(384)
VobileVITV.-0.75 (384)
64
2395
3.0
77.81 (+0.839))
MobileeiTvv.-L0
64
4083
4.9
79.68
MobileVTTvA.L10 (384)
4220
5.1
79.74 (+0.06%)
B IMPROVING LATENCY AND THROUGHPUT
Implementation details: We use GeForce RTX 2080 Ti GPU for obtaining latency timings. Results are averaged over 10000 iterations. The timing results may vary ++0.1 ms. Throughput for XXS, XS and S are calculated on 1000 iterations with batch size of 100. 'Blocks' in Table | represenns in 'layer4' of MibileViTv3 architectures (Table I To improve the latency, we reduce the number of Mobile ViT-blocks in 'layer4' from 4 to 2.
number of MobileViTv3-bloow
Results: Table 8 shows the latency and throughput results. MobileViTv3-XXB V with similar pa- rameters and FLOPs the baseline Mobile 1-Xx- along with 1.98% accuracy improvement achieves similar latency of ~7.1 mss ViTv33eV with two Mobile ViT-blocks instead of four, has 30% less FLOPss and achieves latency of 6.24 ms which is ~-1 ms faster than the baseline ViTvv 1-XXS.. With similar changes in Mobile ViTV3 ViTv3-XS and MobileViTv3s. architecture, FLOPs are reduced by 13.5% and 17.82% respectively and latency is reduced by ~ ms and ~0.7 ms respectively.
Blocks
FLOPs
#
Top-1 (?)
Throughput (f)
Time (ms) (U)
MobileViTv1-XX5
364
2124
7.24
MobileViI33-XSS
4
289
1,25
70.98 (71.98%%
2146
7.12
MobileViT+3-XXS
2
256
(130%)
1.14
70.23 ("1.23%)
2308
6.24
MobileVETV-XX
4
986
2.3
74.8
1097
7,32
VolsleViTv3-XS
76.7 (71.9%)
7.2
VobileVITv3-XS
2
853 (L113,5%)
2.3
76.1 (71.3%)
1129
6,35
MobileViTvi-S
4
2009
5.6
78.4
822
7.34
MobileViTv3-S
4
1841
5.8
79.3 (*0.9%
824
7.29
ViVv
(L17.8298)
79.06 (10.6%)
6.6
Table Latency and throughput comparison between MobileViTv3-XXX. XS, and S and ViTvieViTvl-Xe -XXS, XS, and S. While keeping the parameters and Top-1 accuracy similar to Mo- bile Mobile with 2 blocks reduces the number of FLOPs and improves the throughput and latency.
C HYPERPARAMETERS
C.I CLASSIHICATION
Mobile ViTY3-S, XS and XXS: Default hyperparameters used from MobileViTvl include us- ing AdamW optimizer, multi-scale sampler (S (160,160), (192,192), (256,256), (288,288), (320,320)), learning rate increased from 0,0002 to 0.002 for the first 3K iterations and then annealed to 0.0002 using cosine schedule, L2 weight decay of 0.01, basic data augmentation i.e, random resized cropping and horizontal flipping.
Mobile v--.0- 0.75 and 0.5: Default hyperparameters used from MobileViTv2 include using Adam W as optimizer, batch-sample (S = (256,256)), learning rate increased from le-6 to 0.002 for the first 20K iterations and then annealed to 0.0002 using cosine schedule, 1.2 weight decay of 0.05,
13

---

Under review as a conference paper at ICLR 2023
advanced data augmentation ie, random resized cropping, horizontal flipping, random augmenta- tion, random erase, mixup and cutmix.
C.2 COCO
Mobile XS and XXS: Default hyperparameters include using images of input resolution of 320 x 320, AdamW optimizer, weight decay of 0.01, cosine learning rate scheduler, total batch size of 128 (32 images per GPU), smooth LI and cross-entropy losses are used for object localization and classification respectively.
Mobile 0.75 and 0.5: Default hyperparameters include using images of input size 320 x 320, AdamW optimizer, weight decay of 0.05, cosine learning rate scheduler, total batch size of 128 (32 images per GPU), smooth and cross-entropy losses are used for object localization and classification respectively.
D DISCUSSION AND LIMITATIONS
This work is an effort towards improving performance of models for resource constrained environ- ments like mobile phones. We looked at reducing memory (parameters), computation (FLOPs), latency while boosting accuracy and throughput. With the proposed changes Mobile blocks we achieve higher accuracy. with same memory and computation as the baseline Mobile ViTv and v2 as seen in section 4.1 Table 7 shows fine-tuning results which also outperform the fine-tuned MobileViTv2 models. Section shows how we can achieve better latency and throughput with min- imal impact the accuracy of the model. While Mobile V3 has higher accuracy and lower or similar parameters as compared to other mobile-CNNs, it's higher FLOPs can be an issue for edge devices (Figure 3). This limitation of Mobile ViTv3 architecture is inherited from the self-attention module of ViTs. To solve this issue, we will further explore optimization of the self-attention block. Table shows results on Imagenet-Ik. The reported accuracies of ViTV3-XS XS and models on Imagenet-IK can potentially be further improved by increasing the training batch size 1024 similar the baseline model. The proposed fusion of input features, local features (CNN features) and global features (VIT features) shown in this paper can also be explored in other hybrid architectures.
E OBJECT DETECTION AND SEMANTIC SEGMENTATION RESULTS
E. OBJECT DETECTION ON COCO DATASET
Figure shows object detection results COCO validation images using SSD-Lite with Mobile ViTv3-S as its backbone. Figure 5 shows object detection results on COCO validation im- ages using SSD-lite with VobileViTv3- 1.0 its backbone. The images shown in figure 4 include challenging object detection examples (blurred human/person and complex background),
E.2 SEMANTIC SEGMENTATION ON PASCALVOC2012 DATASET
Figure shows segmentation results on PascalVOC2012 validation images using Deeplabv3 with Mobile V S as its backbone. Figure shows segmentation results PascalVOC2012 validation images using Deeplabv3 with VobileViTv3-L1 1.0 as its backbone. In figure 6 and moving from left to right provide the input image, the corresponding segmentation output, and the overlay of segmentation output on the input image.
14

---

Under review as a conference paper at ICLR 2023
parsoni 1.0
or 0.94
personi 0.36
tablet 0,666
CUP 0:92
cake:
PuU OGEE
Figure 4: Object detection results using SSD-Lite model with Vobile as its backbone.
15

---

Under review as a conference paper at ICLR 2023
Aizelanes
top sign: 1.00
Menlano: 0.33
STOP
Figure 5: Object detection results using SSD-Lite model with MobileViTv3-1.1 as its backbone.
16

---

Under review as a conference paper at ICLR 2023
Figure 6: Semantic segmentation results using Deeplaby3 with MobileViTv3.5 as its backbone.
17

---

Under review as a conference paper at ICLR 2023
Figure 7: Semantic segmentation results using Deeplabv3 with MobileViTv3-1.0 1.0 as its backbone.
18


Reviews:
[{"review_id": "_8AiJAY-RK", "reviewer": "Reviewer_kTXc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a light-weight model for mobile vision tasks, namely MobileViT-v3. The proposed model is based on the previous MobileVit v1 and v2. Specifically, several modifications have been conducted to improve the previous architectures, including replacing 3x3 conv with 1x1 conv, fusing local and global features, adding a residual connection between the input feature and the fusion feature, and replacing normal 3x3 conv with depthwise 3x3 conv. The above changes result in more efficient and effective mobile backbones. Experiments have been performed on the MobileViT families to validate the effectiveness.", "summary_of_the_review": "Overall, the paper provides extensive experiments to verify its effectiveness. The motivation is clear. I admire the authors’ efforts in engineering exploration. However, the technical novelty is limited since the previous works have investigated the core components (i.e., 1x1 conv, depthwise 3x3 conv, and residual connection). Besides, the paper lacks in-depth analysis. I will give my first rating as `reject`.", "strength_and_weaknesses": "### Pros:\n\n- The motivation is clear. It’s practical to design a model that is lightweight and efficient on mobile devices.\n- The technical comparisons between the MobileViT family are clear, as shown in Figure 2. The intuitive and straightforward illustration makes the methodology easy to follow.\n- Extensive experiments (including image/video classifications, object detection, and semantic segmentation) have been provided to validate the effectiveness. However, I still hold some concerns about the experimental comparisons, which are shown in the cons part.\n\n### Cons:\n\n- ***Lack of in-depth analysis and limited technical contribution***\n\nThe primary concern is the technical contribution. The components proposed in this paper are the common techniques to achieve the corresponding purpose. For example, adopting 1x1 conv to minimize the network parameters and employing depthwise 3x3 conv to reduce parameters and FLOPs without significantly impacting performance have already been investigated in previous works [1,2,3]. Besides, this paper needs to include an in-depth analysis of the usage of the above modules. From my perspective, I admire the authors’ efforts in the engineering exploration. However, the main ideas have been explored in many previous works, which makes the works seem to be incremental.\n\n- ***Over-claimed contribution and unclear writing***\n\nThe paper emphasizes that this work addresses the challenges of scaling and simplifies the learning tasks. However, I can not see it’s an essential problem in the previous MobileViT v1 and v2, since there are also a series of scaled models in MobielViT v1 and v2. The contribution seems to be over-claimed. Please present this part more clearly.\n\n- ***Insufficient comparisons***\n\nThe experimental comparisons are mainly conducted on the Top1 accuracy, parameters, and Flops. However, for mobile applications, the actual latency also matters. Please report the throughput for comprehensive comparisons.\n\n- ***Others***\n\n(a) *Extended comparisons:* Since depthwise 3x3 conv is utilized in the MobileViT v3, I would like to see the ablation studies of this module on object detection and segmentation since depthwise conv usually performs well in the dense prediction tasks.\n\n(b) Please add the conclusion part to make the paper more complete.\n\n---\n\n[1] Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions, 2021.\n\n[2] PVT v2: Improved Baselines with Pyramid Vision Transformer, 2021.\n\n[3] Co-Scale Conv-Attentional Image Transformers, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XjT8Yv8-MT4", "reviewer": "Reviewer_jAum", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attempts to resolve the scaling issues of MobileViTv1-blocks which hinder the learning. Specifically, the authors propose MobileViTv3-block which has a more simplified architecture. The proposed blocks have been added to both MobileViTv1 (with transformer blocks) and MobileViTv2 (with linear transformer blocks) models and experiments on various tasks such as classification (ImageNet), detection(COCO) and segmentation (ADE20K) demonstrate improved results. ", "summary_of_the_review": "This work presents new MobileViT blocks that make the architecture more efficient and purportedly more accurate in different tasks (e.g. classification, segmentation, detection, etc.). Despite the great engineering effort behind this work, the novelty and contribution are very limited and as a results don't merit acceptance. ", "strength_and_weaknesses": "Strengths: \n\n1. The paper is well-written and easy to follow. The visualizations allow for better understanding of the proposed technique. \n\n2. The proposed method has been rigorously tested on many different tasks and datasets. \n\n3. The authors have performed many optimizations to further perfect the proposed blocks. This is reflected in consistent improvements as well as the reduction in number of parameters and flops \n\nWeaknesses/Concerns\n\n1. My main criticism of this work is lack of novelty. As mentioned above, this work is an amazing engineering effort through which various building blocks (e.g. 3 x 3 conv layers, etc.) have been replaced with seemingly better alternatives. However, it is hard to pin-point a major contribution or novelty which is specific to this work (e.g. novel design of an attention layer)\n\n2. It is understandable that authors attempt to further optimize the already existing MobileViTv1 and MobileViTv2 blocks. However, one may wonder why the transformer layers should  proceed the conv-based layers ? why not processing features in parallel streams ? the current design is very limited in novelty, and the motivation behind it seems to be the reduction of number of flops, parameters, etc. However, there should also be better justification of the intuition. \n\n3. Experimental comparisons are presented in a somewhat contrived settings. It is preferred to know how the proposed method performs compared to other competing approaches.  \n\n4. Performance gains in downstream tasks of semantic segmentation on ADE20K dataset and object detection on COCO are higher than classification. What could be the reason behind this ?  \n\n5. Another concern is the throughput of the model with the proposed blocks on devices other than GPU. Is there any comparison on mobile devices ? in fact, the real latency on such devices could be drastically different than what is measured on the GPU. In addition, it is recommended to use more modern GPU hardware (e.g. A100) to report latency numbers to better illustrate the performance differences. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H9EbTvBpl_a", "reviewer": "Reviewer_8bQB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propsed mobilevit v3 which did modification based on mobilevit by reducing conv3x3 to conv1x1, adjusting the routing from input and pre-conv embedding [local feature as named in the paper] layers for feature fusion.\n\nIt produces a better efficiency especially at low-parameter cases. ", "summary_of_the_review": "good performance against SoTA baselines, while the modification and pruning of network follows prior arts with limited novelty. ", "strength_and_weaknesses": "Strength: \n\n1) From fig3, Performance of mobilevitv3 is significantly improved comparing with v2/v1, when # of  parameters is smaller than 3.5m. \n\n2) The model is well evaluated based on various tasks:  imagenet for classification, coco for detection and pascal ade20k for segmentation. In all tasks the model shows significant improvement especially on XS cases. \n\n3) Extensive related works are studied in the paper such that the results are compared against strong enough SoTA methods.\n\nCons:\n\nThe ablation study of modifications is lacked in the paper, i.e. step-by-step performance [flops, parameters, acc] comparison of doing each modification, so that we may got the idea of the contribution of each part. \n\nThe novelty of the paper is somehow limited since most modification is already proposed in prior works [depth-wise conv etc.], and the improvement is marginal when paraemters > 3.5m on imagenet. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-mdfsKdNTWU", "reviewer": "Reviewer_AJcM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a series of modifications to the MobileViT V1 and V2 architectures. It introduces a few tweaks to the original model architecture, including replacing regular conv by depthwise conv to reduce latency, using 1x1 conv instead of 3x3 conv to reduce number of parameters, and adding residual connection in the proposed MobileViT V3 block. Such modifications brings in additional quality gain compared to original MobileViT V1 and V2, while being more efficient in terms of FLOPs and number of parameters in some cases. Experiments on downstreaming tasks such as detection and semantic segmentation also show that the proposed modifications can benefit dense prediction tasks as well.", "summary_of_the_review": "In summary, this work tries to study a good problem and propose valid modifications to improve existing models. However, it falls short regarding technical details, latency benchmark and comparison with recent works. \n\nI would like authors to address my comments, especially regarding lack of latency benchmark and comparison with more recent models. ", "strength_and_weaknesses": "It is of great interest to study possible solutions for optimizing transformer models regarding the speed while maintain its superior quality. Although there are abundant works on pushing the boundary of transformer models in vision tasks in terms of quality, there is far less study on optimizing those heavy-lifting models for on-device use cases, where convents such as MobileNet and EfficientNet still dominate. \n\nThis work builds on top of two recent works, MobileViT V1 and V2, and proposes simple way to improve the model performance without altering the model architectures significantly. The figures in the paper provide a clear view of the landscape of recent on-device ViT models, and the place of the proposed MobileViT V3 models. This gives users a good sense of how the proposed model compares to existing works. This is highly appreciated.\n\nTechnically, this work is sound: using depth wise conv for speed-up and 1x1 conv to shrink model parameters are both widely used techniques to optimize a model. From experiments, the ablation study does show that these modifications indeed contribute to performance improvement. \n\nDespite of the strength, there are a lot of problems in the current manuscript that weakens this work.\n\n1. The paper claims it “addresses the scaling and simplifies the learning task” However, from the experimental results, it is not clear why the proposed modifications can achieve the claimed goal. Even the original MobileViT V1 and V2 are not difficult to scale up (especially V2 as the authors already have several variants of that by scaling it up and down): there are just a matter of changing number of filters or hidden size, and possibly adding or remove certain number of blocks. In addition, I do not see the original MobileViT models are not easy to train. In this sense, this paper’s claim does not hold: it is not supported from the experiments. \n\n2. Another big issue is benchmarking. This paper presents the plots of accuracy v.s. model FLOPs and number of parameters, where the proposed model has fewer number of parameters or FLOPs compared to its rivals and existing convnets. It gives a false impression that the proposed models outperforms all existing convents and ViT models. This is not exactly true. For on-device scenarios, FLOPs and number of parameters are not the golden rule to decide a model’s real performance, but latency is. Especially, although MobileNet has larger number of parameters, it actually runs much faster than ViT model (MobileViT) due to CPU-friendly ops. As pointed in [1] and [2], number of parameters and FLOPs are not positively correlated to latency. In fact, when a model contains excessive transpose/reshape ops or many branches, it can lead to higher latency even though the number of parameters are less. \nUnfortunately, the only real latency benchmark is present in Appendix B Table 8, on MobileViT model variants only. It does not provide a holistic picture on the real latency comparison of existing commonly used mobile models. \nTherefore, a latency benchmark on real mobile devices (not only on desktop GPU) would be more convincing, just like what MobileViT V1 and V2 have done. Without such benchmark, this work is less convincing that it will run fast as expected. \n\n3. Technical details\nThere are many unclear details that need further clarification. For example:\n\n    - When replacing 3x3 conv by 1x1 conv, you will have smaller receptive field as 1x1 conv generally serves a channel reduction method. Due to such loss, how do you ensure the information from global features is proper preserved?\n\n    -  How do you decide the scaling factor in Table 1? They seem pretty ad-hoc. For example, 1.3, 1.6 and 1.7 are not quite standard numbers to scale a model.\n\n   -  In implementation details, it is not clear why the model can only be trained with a much smaller batch size. Given that the model has fewer FLOPs and number of parameters comparing to MobileViT V1, it should be trainable using the same batch size as MobileViT V1, which is 1024.\n\n4. Related works miss a few recent works, for example:\n\n    - LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference, 2021\n    - EfficientFormer: Vision Transformers at MobileNet Speed, 2022\n    - Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios, 2022\n\nPlease consider adding these papers and do a more comprehensive comparison. All released code so it should not be difficult to reproduce those works. This is also related to experimental validation. The current comparison is mostly on MobileViT variants and old convnets. It is far from enough to justify the proposed model superiority. More recent models should be included and compared.\n\n- [1] An Improved One millisecond Mobile Backbone, 2022\n- [2] EfficientFormer: Vision Transformers at MobileNet Speed, 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject