An Investigation on Hardware-Aware Vision Transformer Scaling
Authors: Chaojian Li
Kyungmin Kim
Bichen Wu
Peizhao Zhang
Hang Zhang
Xiaoliang Dai
Peter Vajda
Yingyan Lin
Venue: ICLR 2022 Submitted (2022)
Keywords: Model Scaling
Vision Transformer

Abstract: Vision Transformer (ViT) has demonstrated promising performance in various computer vision tasks, and recently attracted a lot of research attention. Many recent works have focused on proposing new architectures to improve ViT and deploying it into real-world applications. However, little effort has been made to analyze and understand ViT’s architecture design space and its implication of hardware-cost on different devices. In this work, by simply scaling ViT's depth, width, input size, and other basic configurations, we show that a scaled vanilla ViT model without bells and whistles can achieve comparable or superior accuracy-efficiency trade-off than most of the latest ViT variants. Specifically, compared to DeiT-Tiny, our scaled model achieves a $\uparrow1.9\%$ higher ImageNet top-1 accuracy under the same FLOPs and a $\uparrow3.7\%$ better ImageNet top-1 accuracy under the same latency on an NVIDIA Edge GPU TX2. Motivated by this, we further investigate the extracted scaling strategies from the following two aspects: (1) "can these scaling strategies be transferred across different real hardware devices?''; and (2) "can these scaling strategies be transferred to different ViT variants and tasks?''. For (1), our exploration, based on various devices with different resource budgets, indicates that the transferability effectiveness depends on the underlying device together with its corresponding deployment tool; for (2), we validate the effective transferability of the aforementioned scaling strategies obtained from a vanilla ViT model on top of an image classification task to the PiT model, a strong ViT variant targeting efficiency, as well as object detection and video classification tasks. In particular, when transferred to PiT, our scaling strategies lead to a boosted ImageNet top-1 accuracy of from $74.6\%$ to $76.7\%$ ($\uparrow2.1\%$) under the same 0.7G FLOPs; and when transferred to the COCO object detection task, the average precision is boosted by $\uparrow0.7\%$ under a similar throughput on a V100 GPU. 

Under review as a conference paper at ICLR 2022
AN INVESTIGATION ON HARDWARE-AWARE VISION TRANSFORMER SCALING
Anonymous authors Paper under double-blind review
ABSTRACT
Vision Transformer (ViT) has demonstrated promising performance in various computer vision tasks, and recently attracted lot of research attention. Many recent works have focused on proposing new architectures to improve ViT and deploying it into real-world applications. However, little effort has been made to analyze and understand ViT's architecture design space and its implication of hardware-cost on different devices. In this work, by simply scaling VIT's depth, width, input size, and other basic configurations, we show that scaled vanilla ViT superior accuracy-
model without bells and whistles can achieve comparable efficiency trade-off than most of the latest ViT variants. Specifically, compared to DeiT-Tiny, scaled model achieves a 1.9% higher ImageNet top-1 accuracy under the same FLOPS and a 1 3.7% better ImageNet top-1 accuracy under the same latency on an NVIDIA Edge GPU TX2. Motivated by this, we further investigate the extracted scaling strategies from the following two aspects: (1) "can these scaling strategies be transferred across different real hardware devices?"; and (2) "can these scaling strategies be transferred to different VIT variants and tasks?". For (1), our exploration, based on various devices with different resource budgets, indicates that the transferability effectiveness depends on the underlying device together with its corresponding deployment tool; for (2), we validate the effective transferability of the aforementioned scaling strategies obtained from a vanilla ViT model on top of an image classification task to the PiT model, a strong ViT variant targeting efficiency, as well as object detection and video classification tasks. In particular, when transferred to PiT, our scaling strategies lead to boosted ImageNet top-1 accuracy of from 74.6% 76.7% (f 2.1%) under the same 0. 7.7G FLOPs; and when transferred to the COCO object detection task, the average precision is boosted by t 0.7% under similar throughput on V100 GPU.
INTRODUCTION
Transformer (Vaswani et al. 2017). which was initially proposed for natural language processing (NLP) and is type of deep neural networks (DNNs) mainly based the self-attention mechanism, has achieved significant breakthroughs in NLP tasks. Thanks to its strong representation capabilities, many works have developed ways apply Transformer computer vision (CV) tasks, such as image classification (Dosovitskiy et al. 2021). object detection (Carion et al. 2020), semantic segmentation (Wu al. 2002j, and video classification (Bertasius al.] (021). Among them, Vision Transformer (ViT) (Dosovitskiy et al. 2021) stands out and demonstrates that a pure Transformer applied directly to sequences of image patches can perform very well on image classification tasks, e.g., achieving comparable ImageNet (Deng et al., 2009) top-1 accuracy as ResNet (He et al.) 2016). Motivated by ViT's promising performance, fast growing number of works follow to explore pure Transformer architectures in order to push forward its accuracy- -efuraccefffccencyy trade-off and deployment
into world applications (Touvron et al. 2020 Heo et al. 2202 Graham et al. 202 Liu et al.
2021 Wu et al. (021), , achieving even better performance than EfficientNerVI Tan & Le 2019
a widely used efficient convolutional neural network (CNN).
The success of recent ViT works suggests that the model architecture is critical to VIT's achievable performance. Therefore, in this work we explore ViT architectures from a new perspective, aiming to analyze and understand VIT's architecture design space and real hardware-cost across different

---

Under review as a conference paper at ICLR 2022
devices. Despite the recent excitement towards VIT models and the success of model scaling for CNNs, little effort has been made into exploring ViT's model scaling strategies or hardware-coss Note that directly applying the scaling
strategies for CNNs (Tan & Lej 2019
Sun et al. 2017) Transformer on NLP
ImageNet top-1 accuracy (%)
tasks Kaplan et al., 2020; Henighan et al. 2020) will lead sub-optimality, as dis- cussed in Section 3.2 Furthermore, scaling strategies targeting one device/task might not be transferable to another device/task. Interestingly, we find that simply scaled ViT models can achieve comparable or trade-off
82
80-
78
Deeplet-
76-
Twinst
Thanfformet
even better eccuracy-efficicnn than dedicatedly designed ViT variants, shown in Figure [] Motivated by this, we further explore the transferability of our scaling strategies (1) across different real hardware devices and (2) different ViT variants and tasks. In particular, we make the following contributions:
DelT
Deff-Scaled (Our)
74
PT-iccated (Dues)
denctes 'ho dieti lntion
72
10'
10'
FLOPs (G)
Figure I: Our scaled ViT models achieve comparable or better -urracy-eficienccy trade-off as compared to some recent dedicatedly designed VIT variants.
We are the first to show that simply scaled vanilla ViT models can achieve comparable or even better accuracy- accuracy-effccieng
trade-off as compared to dedicatedly designed VIT vari- ants (Touvron et al. 202 1 Yuan et al.) 2021] et al. [202]; Chen et al. [002] Wu et al.)
2021 Liu et al. 2021 Chu et al., [2021] Touvron et al., 2020) Heo et al. (021), as illustrated
scaled model achieves f 1.9% higher ImageNet top-1 accuracy under the same FLOPs and a 3.7% better ImageNet top-1
Figure I Specifically,
compared to DeiT-Tiny,
accuracy under the same latency on an NVIDIA Edge GPU TX2.
We study the transferability of the scaled ViT models across different devices and show that the transferability effectiveness depends on the underlying devices and deployment tools. For example, scaling strategies targeting FLOPs the throughput V100 GPU (NVIDIA) LLC.) can be transferred to the Pixel3 (Google LLC.] a) device with little or even no performance loss, but those targeting the latency TX2 (NVIDIA Inc., (c) may not be transferred to other devices due to the obvious performance loss. Additionally, we provide ViT models' cost breakdown and rank correlation between their hardware-cost on different devices for better understanding it.
We show that our scaling strategies can also be effectively transferred to different ViT variants and recognition tasks to further boost the achieved accuracy-efficieme tradc-off, e.g., achieving
2.1% higher accuracy under a similar FLOPs when being transferred to the PřT model and f 0.7% higher average precision under similar inference throughput when being transferred to an object detection task.
RELATED WORKS
Vision Transformers. Transformer first proposed for machine translation (Vaswani et al. 2017
Motivated by its state-of-the-art (SOTA) performance in NLP tasks, there has been a growing interest mechanism CV tasks, e.g., by proposing novel attention
in applying the Transformer/seff-atteenii
mechanisms for CNNs (Hu et al., 2018 Li et al. 2019 Zhang et al., 2020). fusing Transformer
and CNN designs within the same model Bello al. [2019] Carion al. 2020 Wu al., 2020). or designing pure Transformer models (Dosovitskiy et al. [021] Chen et al. 2020). Among them, ViT (Dosovitskiy et al. 2021) has achieved SOTA performance by directly applying the Transformer architecture for NLP tasks to the input raw image patches of vision tasks. Nevertheless, ViT's powerful performance largely depends on its pre-training on JFT-300M (Sun et al.) 2017) (a giant private labelled dataset). As such, DeiT (Touvron et al. 2020) further develops improved training recipe (i.e., the setting of optimization augmentation and regularization, to achieve comparable performance while removing the necessity of the costly pre-training. In order to build more efficient VIT models, (Chen et al. 2021) leverages multiple branches to extract and fuse features different scales; (Heo et al. 2021 Graham et al.
(apeer-prammters) including a distillation setup and stronger data
2

---

Under review as a conference paper at ICLR 2022
2021
Liu et al. 2021 Fan et al. [021 Wang et al. 2021) apply pyramid like architecture commonly
used in CNNs to ViT; and (Graham et al., 2021 Liu et al., 2021 Wu et al. 2021) propose more
efficient attention mechanisms or feature projection blocks.
-rrasffomme (i.c., Transformer in NLP tasks) to boost its accuracy lower its computational resource requirements, e.g., ResNet can be scaled along its depth dimension (He et al. 2016) and MobileNets can be scaled along its width (i.e., the number of channels) and input resolution dimensions (Sandler et al.) 2018: Howard et al. 2019). Notably, EfficientNet further points out that it is critical to scale CNNs in a compound manner (i.e., simultaneously scaling the model width, depth, and input resolution) and does so to achieve SOTA accuracy-efficiee trade-off (Tan & Le (2019). Nevertheless, as (Bello et al.) (2021) demonstrates, the scaling strategies obtained from a specific model (e.g., EfficientNet-BO) can result in sub-optimal accuracy-efficiens develop a more general scaling strategies extracted from grid search experiments based on the chosen training recipe rather than specific model, achieving improved trade-off. In addition to scaling the model architecture, (Kaplan et al.] 2020 Henighan et al. 2020) show that scaling up the dataset size and the number of computations used for training can also help to achieve a smaller cross-entropy loss for Transformer in NLP tasks. Recently, Zhou et al. (2021) demonstrates that the accuracy of ViT will decrease when it is scaled up along only the depth dimension (i.e., number of layers), and proposes Re-attention to resolve it.
Model scaling, Prior works have explored scaling
trade-off for another model; motivated by this observation, they
Nevertheless, none of the prior works has targeted scaling strategies for ViT with multiple scaling factors or study its real-hardware efficiency across different platforms featuring diverse computational and storage capabilities. Additionally, is not clear whether their insights on scaling CNNs can be directly applied to ViT because of their different scaling factor definitions, e.g., while the number of channels represents the width in CNNs, the number of heads and embedding dimensions can both represent the width in VIT. As such, scaling strategies dedicated to ViT are highly desirable and our scaling strategies can provide unique insights to inspire more innovations towards efficient ViT models. Although there is some model scaling strategy explorations in (Dosovitskiy et al. 2021 /Zhai cct al., 202I), our work distinguishes with them in providing more discoveries and insights. Specially, we focus more on the accuracy vs. efficiency trade-off when scaling ViTs instead of merely the accuracy; and (2) provide additional analysis on the transferability of the extracted scaling strategies across different devices, VIT variants, and tasks.
3 SCALING VIT: HOW AND WHY DO WE SCALE VIT?
In this section, first analyze the scaling factors of ViT, then study the effectiveness of prior scaling strategies, which are dedicated to CNNs or Transformers, on ViT, and finally present our iterative greedy search approach to scale ViT.
Ciessdicttio Header
3.1 SCALING FACTORS IN VIT
Linarr
+ Unear
Feed Forward
e xxx
As analyzed in Kaplan et al. 2020), the scaling fac- tors in Transformers include the number of layers (d), the number of heads (h), the embedding dimen- sion for each head (e), and the linear projection ratio
Layerhorm
BuF-4aation
Uitor
exh
Mutl-Head Altention
(z/p)²
LayerNorm
(r). ViT, which directly adopts the Transformer ar- chitecture for NLP tasks and splits the raw images into patches to serve as the Transformer input, adds additional scaling factors, including image resolu- illustrates summarizes our considered scaling factors for VIT.
Spin & enboor
Linear
(1/2)*
(I/p²²
Figure 2: Illustrating the effect of scaling fac- tors on a ViT architecture clats/distillattin token is omitted for better visual clarity).
tion (7) and patch size (p). Figure Filurrelil and
3.2 PREVIOUS SCALING STRATEGIES FAIL ON VIT
CNN and VIT scaling faactors do not match. Scaling strategies dedicated CNNs (Tan & Le 2019
Sun et al., 2017 Feichtenhofer 2020) mostly come with CNN-specific scaling factor definitions (e.g., the number of channels in convolution layers represents the model width), which cannot be directly transferred to VIT. For example, doubling (2x) the width in CNNs can be achieved via various combinations of the number of heads (h) and embedding dimension for each head (e) in VIT.
3

---

Under review as a conference paper at ICLR 2022
Furthermore, there extra scaling fac-
ImageNet top-1 accuracy (%)
85
tors for ViT, c.g., the linear projection ratio (r)) and the patch size (p), which do not exist in the scaling factors for CNNs but are important for ViT as shown in Ap-
75
pendix D thus directly transferring the
70-
scaling strategies from CNNs to ViTs can
lead to ambiguity and sub-optimal perfor-
18861%
mance.
6M Params (DeiT-Trry Level)
Transformer scaling strategies for NLP sub-optimal on ViT. (Kaplan et al. 2020) noted that for NLP, model performance (i.c., accuracy or training loss) depends "strongly scale (i.e., the number of parameters), but weakly the model shape". However, when scaling ViT along the factors sum- marized in Figure 2 our observations suggest that this is not for VIT. As shown in Figure 3 when performing an extensive search on top of DeiT-Small (Touvron et al. 2020) following (Sun al. 2017). we observe that a model's shape has a great impact on the performance. Specifically, we change the aspect ratio, i.e., the ratio between the embedding dimension (e x h) and the number of layers (d), while keeping the model parameters to the same, the aceuracy drifts as much as 18.61%. This set of experiments motivates exploring scaling strategies dedicated to VIT.
60
(DeiT-Small Lavel)
67M Params (DeiT-Base Level)
10
10
10'
x h e
Aspect ratio
the model
d
Figure 3: The accuracy of ViT is sensitive to the aspect ratio. Note that the vertically aligned points are models with the same scaling factors except image resolution (T).
3.3 OUR SCALING METHOD BASED ON AN ITERATIVE GREEDY SEARCH
Starting from a relatively small model defined in Table I we
Table 1: The starting point model for our scaling method.
adopt a simple iterative greedy search to perform the ViT scal-
ing step by step, similar to the previous algorithms for exploring
CNN design spaces and feature selections (Feichtenhofer) 2020
Num. of layers (d)
6
John et al.) Jain & Zongker 1997] Guyon & Elisseeff 2003).
Num. of heads (h)
2
Embedding dim. per head (e)
64
Specifically. ly, based on the starting point model or the optimal one from the previous scaling step, we scale up the model
Lincar projection ratio (r)
4
Image resolution (J)
160
along each standalone scaling factor introduced in Section 3.1 to match the target hardware-cost (e.g., FLOPs, or latency on a specific hardware device), and select the one with the best eecuracy-efficient trade-off to be the starting point model of the next step. As analyzed in (Bello et al., 2021). unlike scaling strategies extracted from a specific model, scaling based such an iterative greedy search can avoid the unscalability of the resulting scaling strategies on a specific model. Our experiments in Section 4.1 also verify that such a scaling method simple yet effective for scaling ViT models, and only requires training a few models during each search step.
Patch size (p)
16
FLOPs (G)
0.15
Throughpur V100 (FPS)
20086
Latency on Pixel3 (ms)
30.05
Lateney on TX2 (ms)
4.42
EXPERIMENT RESULTS
In this section, we first present experiments for evaluating the scaled vanilla VIT models resulting from the iterative greedy search described in Section 3.3 in terms of accuracy-FLOL on ImageNet (Deng et al.) (2009). From this set experiments, we then extract a set of scaling strategies dedicated VIT. After that, we further conduct experiments study the transferability of our extracted scaling strategies (1) across different devices and (2) to different ViT variants and tasks.
trade-offs
4.1 SCALING VIT TOWARDS BETTER ACCURACY-FLOPS TRADE-OFFS
we set 2x FLOPs of the initial or selected model from the previous step as the target hardware-com in each step when individually scaling each factor, summarized in Figure 2 All networks are trained for 300 epochs ImageNet (Deng et al.) 2009) using the same training recipe with the one DeiT (Touvron et al. 2020), more details are included in Appendix E We summarize our observations as follow:
Following the scaling approach described in Section 3.3
4

---

Under review as a conference paper at ICLR 2022
Table 2: Our scaled ViT models outperform DeiT on ImageNet under the same FLOPs constrains.
Model
FLOPs (G)
Top-1 accuracy (%)
d
Deiy-TTnny
1.26
74,5
12
64
4
224
16
Deil-Sealed-Tiny
1.22
76.4 (11.9
14
64
160
16
DeiT-Small
4.62
81.2
12
64
4
224
16
DeiT-Scaled-Small
4.79
81.6 (10.4)
20
64
256
16
DeiT-Base
17.66
83.4
12
12
64
4
224
16
DefT-Scaled-Base
16.82
83.8 (10.4)
20
64
4
320
16
Scaled ViT models outperform SOTA DeÎT models. shown in Table our sealed ViT models (e.g., DeiT-Scaled-TinySSSaaJlll Tiny/Small/Base) achieve 10.4% ~ 1..9 higher top-1 accuracy on ImageNet under the same FLOPs constraints. Specifically, our DeiT-Scaled-Ting model chooses to use smaller image resolution (i.e., x60 160 vs. 224x224) and more layers and higher number of heads as compared to the SOTA DenT-Tiny model, and thus achieves a f1.9% higher accuracy at the same cost in terms of FLOPs, while DeiT-Scales SOOT-SSaledSmaallle Small/Base models choose use larger image resolution (i.e., 320/2262 320/256 vs. 224x224) and more layers, together with a lower number of heads compared the SOTA DeiT-Smalll/Base model, helping them to achieve 10.4% higher accuracy under similar FLOPs. This set of experiments shows that our simple search method can (I) effectively locate ViT models with better accuracy-FLOPs trade-off' and (2) automatically adapt different scaling factors towards the optimal accuracy-FLOPs trade-offs, e.g., different model shapes and structures different scales of FLOPs.
Random permutation further boosts the performance. Inspired by the coarse-to-fine architecture selection scheme adopted (Yu et al. 2020), we further randomly permute the scaling factors (i.e.. d, h, e, r. J, and p) of each scaled model in Table After the permutation, we select 24 architectures under the same target hardware-cost with the scaled model by iterative greedy search for each scaled model. Figure 4 demonstrates that (1) such a random permutation can slightly push forward the frontier of accuracy-FLo the scaled models resulting from the adopted simple scaling method); and (2) our adopted iterative greedy search alone is sufficiently effective while requiring a lower exploration cost (e.g., 6 vs. 30 (6+24) models to be trained for each step compared such search method together with the aforementioned permutation).
trade-off (e.g., a 10.4% higher accuracy under similar FLOPs on top of
83
Table 3: Scaled VIT models after training for 1000 epochs.
10.4%
10.4%
ImageNet top-1 accuracy
eo
79.
Model
FLOPS (G)
Top-1 accuracy (%)
DeiT-Tiny DedT-Sculed-
1.26
74.5
77-
0.5%
Dall
1.22
76.4 (f1.9)
26
11.9%
permuzato Scaldd
DeiT DeiT.Tiny 1000 epochs
1.26
76.6
+
DelT-Sclles-ep
DeiT-Scaled-Tiny 1000 epochs
1.22
(f1.7)
71.0
2.0
FLOPs
5.0
DeiT-Small DelT-Sealed-Smal
4.62
81.2
Figure 4: Random permutation on top of the DeiT-Scaled, where those on the Pareto fron- tier are marked as DeiT-Sccled-RP.
4.79
81.6 (10.4)
DelT-Small/ 1000 opachs
4.62
82.6
DelT-Sealed-Small 1000 epochs
4.79
82.9 (10.3)
Scaled VIT also benefits from longer training time. As pointed out by (Touvron et al., 2020). training VIT models for more epochs (e.g., 1000 epochs) can further improve the achieved accuracy. To verify whether the scaled ViT' models can benefit from more training epochs, we train the models in Table 2 for 1000 epochs following the training recipe in (Touvron al. (020). As shown in Table longer training epochs also help our scaled models (c.g., DeiT-Scaled- BeeT--Saled-TTyySSmallll to achieve higher accuracy, and thus, the advantage of our scaled models
DeiT is consistent under both the 300-epochs training recipe and 1000-epochs training recipe, e.g., a 1 11.9% higher accuracy over DeiT-Tiny with 300 epochs vs. 11.7% higher accuracy over DeiT-Tiny with 1000 epochs.
Drawn insights from scaling VIT. Based on the observations from the above experiments, especially the scaling strategies illustrated in Figure 5 draw the following scaling insights dedicated to ViT:
5

---

Under review as a conference paper at ICLR 2022
(I) When targeting relatively small models (i.e., with smaller FLOPs than DeiT-Scaled-Smie the optimal models tend to select "scaling h (i.e,, the number of heads)" or "scaling d (i..., the number of layers)" and "smaller I (i.e., the input image resolution)" (e.g., 160 160 instead of the commonly used 224 x 224).
(2) When targeting relatively large models (i.e., with larger FLOPs than DeiT-Scaled- Small), the optimal models mainly select to "scaling (i.e., the input image resolu- tion)", while "slowing down scaling h (i.e..
ImageNet top-1 accuracy (%)
DelT-Scaled-Small
Scale
Boals
Soale
80-
Scallo d
DelT Sca--edeae-BBae
DaIT-Scalad-Tiny
75
Scale
70-
number of heads)" as compared to the case when targeting relatively small models.
DelT-Scaled
65
Soale
SSate
Scale
60-
Scale
Scale J
4.2 TRANSFERABILITY OF THE EXTRACTED SCALING STRATEGIES ACROSS DIFFERENT DEVICES
Scalle
55
10a
10'
FLOPs (G)
Figure 5: Resulting models from our iterative greedy search, where models achieving the best accuracy-FL.O5
To evaluate the transferability of the ex- tracted scaling strategies across different real hardware devices, we consider 3 hard- ware devices which target different appli- cations as summarized in Table 4 More details about the setup of these devices are provided in Appendix B
trade-offs are marked as DeiT-Scaled- Tiny/Small/Base. The architecture configurations (i.e., sets of d, h, e, r, L, and p) leading to these best models are extracted our scaling strategies dedicated to ViT.
4.2.1 TRANSFERABILITY AMONG DIFFERENT DEVICES
To obtain the -ediicttd scaling strategies leading the best accuracy-efficieng trade-off on each device, we follow the scaling search method described in Section 4.1 but replace the target hardware-cost with (1) 0.5x throughput measured on an NVIDIA V100 GPU (i.e., V100) (NVIDIA) LLC.) (2) 2x latency measured on an NVIDIA Edge GPU TX2 (i.e.. TX2) (NVIDIA Inc. ( and (3) latency measured on a Google Pixe13 device (i.c., Pixel3) (Google LLC.| a). to simulate the model scaling for (I) cloud services with strong GPUs, (2) edge computing with weak GPUs, and (3) mobile deployment without GPUs, respectively. We then compare the scaled models that achieve the best securacy-efficiene
trade-off on each device, shown in Figure 6 aiming to answer "can scaling strategies be transferred across different real hardware devices?". This set of comparisons provides some interesting observations:
(1) The simple scaling approach is effective on different hardware devices. From the comparison
between the scaled models with FLOPs, throughput on V100, latency on TX2, and latency on Pixe13
as the hardware-coss during scaling (i.e., FLOPs (+), V100 (4). TX2 ) and Pixel3 (+) Scaling in Figure 6 and the SOTA DeiT model (i.e., Baseline
in Figure GD. as shown in Figure G (a), (b), (c). and (d), respectively. we can see that all the device-dedicaa iterative greedy search method described in Section 3.3 achieve better accuracy-efficieng trade-off than the baseline DeiT, indicating the necessity of device-dedicated scaling. Specifically, the scaled models targeting the TX2 device (+) can achieve f 3.7% higher accuracy under similar latency on TX2, as compared to the DeiT-Tiny model. This set of experiments verifies that the adopted scaling approach is simple yet effective across different devices or targeting hardware metrics.
scaled models resulting from the
3.3
(2) The transferability of our scaling strategies across different devices depends on the under- lying device. From Figure 6. we can observe that (i) the scaled models directly targeting a device indeed always lead to the best accuracy-efficieng ing search method can adapt to different devices; and (ii) the performance of the device-dedicated
trade-off the device, indicating that our scal-
Table 4: Important details about the hardware devices in the transferability exploration experiments.
Device
Deployment tool
Hardware-cost measurement tool
Target application
NVIDIA V100
Py TensorRT Tilite
Porch profiler
Cloud services w/ strong GPUs
NVIDIA Edge GPU TX2
TensorRT command-ling line wrapper
Edge computing w/ weak GPUs
Google PixeI3
Tiiite benchmark tools
Mobile deployment w/o GPUs

---

Under review as a conference paper at ICLR 2022
ImageNet top-1 accuracy (%)
ImageNet top-f accuracy (%)
20-
20-
75
75
20-
20-
Baseline
V100 Scaling
FLOPs Scaling
Pice Scaling
Piee Scaling
Scaling
FLOPs Scaling
V100 Scaling
101
10
10
50
10a
10a
FLOPs (G)
1/FPS on V100 (b)
02
20-
ImageNet top-1 accuracy (%)
ImageNet top-1 accuracy (%)
13.7%
20
70
62
Saseline
20-
Baseline
FLOPs Scaling
FLOPs Scating
Scaling
Scaling
Scaling
TX2 Scaling
Hive 3 Sca ing
52
6
100 150 200
250
300
350
Latency on TX2 (ms)
Latency Pixe13 (ms)
(c)
(d)
Figure 6: Comparing the optimal models resulting from scaling for different hardware devices. FLOPP/V1000TXX/PPeee Scaling represents the scaling strategies obtained on
FLOPS/1100TXX//xxx11. and DeiT models are marked the comparison Baseline.
scaled models when executed on other devices varies among different devices together with their corresponding deployment tools. For example, when executed on the Pixel3 device (see Figure (d)), as expected, the scaled models targeting the Pixel3 device (denoted as +) are always on the Pareto frontier (i.e., the best accuracy-efficiene trade-off); interestingly. the scaled models targeting FLOPs (x) and the V100 device (a) are also close to even the Pareto frontier; however, the scaled models targeting the TX2 device are obviously far from the Pareto frontier when executed on Pixel3 device.
As shown Table the scaled model targeting the TX2 device (+) suffers from a 4 0.82% lower accuracy at even 51. 11990% higher latency when executed on the Pixel3 device, as compared to the scaled models directly targeting the Pixel3 device (+), and vice versa for the performance of the scaled models targeting the Pixel3 device (+) when executed on the TX2 device, i.e., 15.74% higher latency and a 4 0.72% lower accuracy.
This set of experiments indicates that the scaling strategies obtained when targeting FLOPs (x) and the V100 device (a) can be transferred to the Pixel3 device with little or even no performance loss, but those obtained for the TX2 device leads degraded performance when being transferred.
4.2.2 ANALYSIS ON THE TRANSFERABILIT EFFECTIVENESS
To better understand why the transferability effectiveness depends the underlying devices, analyze the performance of ViT models executed on different hardware devices from the following
Table 5: Scaled models targeting Pixel3 are sub-optimal when executed on TX2, and vice versa.
Top-1 accuracy (%)
Latency on TX2 (ms)
Latency Pixel3 (ms)
Model
h
e
p
Pixe13 Scaling
74.8
20.91
181.07
2
108
160
16
TX2 Scaling
74.0 (J 0.8)
14.44 30.94%)
275.06 51.90%)
4
64
16
160
16
TX2 Scaling
78.2
23.70
456.41
64
16
Pixe13 Scaling (+)
77.5 (L 0.7)
27.43 15.74%)
297.58 34.80%
142
160
7

---

Under review as a conference paper at ICLR 2022
FLOPs breakdown
1/FPS breakdown on V100
Latency breakdown on TX2
Latency breakdown on Pixel3
2.2% %
11.1 %
9.2 %
1.3% %
14.3 %
25.0 %
83.4 %
21.1 %
62.5 %
34.3 %
70.5 %
42.7
5.2 %
3.3
MSA Sww.ssARAehhaaaaa..ooo Orathr
NSA-SA- MSA-MN
Figure 7: Cost breakdown of DiiT-Tiny on different devices in terms of (1) the number of FLOPs, (2) the I/FPS on V100, (3) the latency on TX2, and (4) the latency PixeI3, where MLP rep- resents the cost of all the Linear layers of ViT, MSA-SA-MatMuu represents the cost of ma- trix multiplication among Q(uery), K(ey), and V(alue) in ViT's multi-head attention, MSA-SA- &ranersseesssseaaGaaaa Gathe represents the cost of merely the data movement in ViT's multi-head attention, and the cost of all other operators are denoted Others.
two perspectives: (1) cost (e.g., latency) breakdown of the same model on different devices and (2) the rank correlation between the hardware-cost on different devices for the same group of models.
Connection between the breakdown and the transferability effectiveness. As shown in Figure] 7 the cost breakdown of the DeiT-Tiny model suggests that the breakdown terms of the number of FLOPs, the latency V100, and the latency on Pixel3 relatively similar, e.g., the breakdown's s cosine distance between any pair among them is smaller than 0.02. while the breakdown for the latency TX2 is quite different from that of the number of FLOPs, the latency V100, and the latency on Pixe13, e.g., the breakdown's 's cosine distance between the latency on TX2 and any other metric is larger than 0.28). This breakdown analysis explains why the scaled models targeting FLOPs (denoted as *), V100 (a), and TX2 have a different transferability performance in terms of the accuracy-latency trade-off when executed Pixe13.
Rank correlation between the hardware-cost on different devices can also indicate the trans- ferability effectiieness. Besides the above analysis based on the cost breakdown on different devices using one specific model (i.e.. DeiT--Tiy), we also perform analysis based on a group of ViT models.
Following the extensive search adopted in (Bello et al. (202)) we
generate group of VIT models by varying d in [3, 6, 12. 18, 24],
FLOPs
Latency Latency on V100 on TX2 on Pixel3
1/FPS
in [2. 3, 6, 8, 12], e in [32, 64, 96), r in [2, 4, 8),
[128, 160,
on Pixeis on TX2 on V100 FLOPs
224, 320]. and in [8, 16, 32], resulting in total of 2,700 differ-
1 00
0.90
0.82
0.82
ent VIT models. As shown in Figure 8 the Kendall Rank Cor-
relation Coefficient Coefficient (Abdi 2007). which is commonly used to benchmark the effectiveness of accuracy/hardwaar-- predictors in recent neural architecture search works (Li et al.,
Latency Latency t/FPS
0 9.
1.00
0.86
0.72
2021 You et al. 2020 Dai et af.] 2020), between the latency on
0.82
0.86
1.00
0.59
Pixel3 and TX2 (highlighted in the red box) the lowest one
among all the coefficients. This set of experiments indicates the weaker performance of using the latency on TX2/Pixel3 to be the proxy metric when scaling ViT targeting Pixel3/TX2, compared to other device pairs, which consistent with our observations on the transferability performance among different devices in Section 4.2.T i.e., scaled models targeting FI.OPs (denoted as x) and V 100 (denoted as a) have better transferability performance when executed on Pixel3 than those targeting TX2 (denoted as
0
0.82
0.72
0.59
1.00
Figure 8: The rank correlation co- efficient between the hardware- cost on different devices.
Along with the above analysis based the (1) cost breakdown and (2) rank correlation between the hardware-com cost on different devices, we further perform a deeper analysis from the hardware device specification perspective in Appendix for better understanding why the transferability effectiveness depends on the underlying devices.
4.3 TRANSFER OUR SCALING STRATEGIES ACROSS DIFFERENT MODELS AND TASKS
To answer "can these scaling strategies transferred to different ViT variants and tasks?", we transfer the extracted scaling strategies in Section 4. for DeiT (Touvron et al. 2020) on ImageNet (Deng
8

---

Under review as a conference paper at ICLR 2022
et al. 2009), as illustrated in Figure (1) PiT (Heo et al. 2021), strong ViT variant targeting efficiency, on ImageNet (Deng et al. 2 (2) COCO (Lin et al. 2014), e a popular benchmark for
object detection tasks, to build the backbone of the Deformable DETR detector (Zhu al.) (2020);
(3) Kinetics-400 (Kay et al., 2017). a commonly used dataset for video classification tasks, with a FimeSrormer (Bertasius
2021 style model extension, which included in Appendix
4.3.1 TRANSFER TO THE PTT MODELS
As shown in Table 6 when being transferred to PiT on ImageNet (Deng et al. 2009), the scaling strategies obtained from targeting DeiT (Tou- vron et al. 2020) on ImageNet (Deng et al.) 2009) still lead to advantageous accuracy- efficiency trade-offs for both the PiT-Scaled- Tiny and PiT-Scaled-XS models, e.g.. a f 2.1 % and 0.4% higher accuracy under similar num- ber of FLOPs, respectively. Although the accu- racy improvement for PYT-Scaled-Small is not as obvious as that for PiT-Scaled- Tiny//SS (i.e.,
Table Transferring the scaling strategies target- ing DeiT (Touvron al. 2020) to PiT (Heo et al.) 202T), where the resulting models are denoted as PiT-Scaled Secaled-inyXS//Small.
Model
Top-1 accuracy (%)
FLOPs (G)
Pil-Tiny
74.6
0.71
PrT-Scaled-Tiny
76.7 (1 2.1)
0.70
PIT-XS PiT-Scaled-XS
79.1 79.5 (f 0.4)
1.40
1.38
PrT-Small
81.9
2.9
PIT Small (Reproduced)
0.1% under similar FLOPs), the transferred scaling strategies at least do not lead to an in-
PiT-Scaled-Small
81.8 (1 0.1)
3.0
ferior model architecture. More details about the architectures of PiT-Scaled-TinyXXSSSmee provided in Appendix A
4.3.2 TRANSFER TO AN OBJECT DETECTION TASK
When transferred to object detection, DeiT (Tou- vron et al. 2020) and our scaled DeiT-Scaled mod- els are inserted into Deformable DETR (Zhu et al.) 2020) as the backbones, and the corresponding throughput on V100 is measured using the widely used Detectron2 tool (Wu et al. 2019). As listed in Table 7 our DeiT-Scaled models achieve 10.7% higher average precision under a similar inference throughput, which consistent with our observa- tion on the advantages of our DeiTe-d mod- els over the original DeiT (Touvron al.) 2020) models in terms of classification tasks, which discussed in Section
Table 7: COCO (Lin al., 2014) detection performance (val2017) of DeiT (Tovrro et al. 2020) and our DeiT-Scaled models with De- formable DETR (Zhu et al. 2020) detector.
Average precision (%)
Throughput (FPS) on V100
Backbone
DeiT-Tiny DelT-Scaled-Timg
35.0 35.7 (f 0.7)
13.31 13.05
DeiT-Small
41.0
10.81
DefT-Scaled-Small
41.7 (f 0.7)
All the above attempts of transferring the scaling strategies, extracted from scaling vanilla ViT models on an image classification task, into different ViT variants and tasks share the following common observations: (1) for some cases, such transfer still achieves advantegeous accuracy -efuracceffcccnncc trade- offs, even without any further exploration of scaling strategies dedicated to the new models/tasks; and (2) for the remaining cases, the transferred scaling strategies lead to models with securacy-efficiens trade-offs that are on par with the corresponding vanilla models. Notably, there is no extra exploration cost (c.g., g., re-extracting dedicated scaling strategies) during transfer and thus it can provide at least good starting point for further dedicated exploration on the new models/tasks.
CONCLUSION
In this work, we present the study for exploring hardware-aware VIT scaling and show that simply scaled vanilla VIT model can achieve a comparable or even better (e.g.. up to 3.7% higher accuracy) accuracy-efficiency trade-off as compared to dedicatedly designed SOTA VIT variants. Furthermore, we extract scaling strategies dedicated to ViT and study their transferbility across different hardware devices, ViT variants, and computer vision tasks. We believe that this work has demonstrated promising perspective towards more afficient/accurate ViT models and will inspire more following innovations both new VIT models via scaling and hardware-efficient ViT models.

---

Under review as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
Regarding our efforts that have been made to ensure reproducibility, we provide the implementation details in Appendix E
REFERENCES
Hervé Abdi. The kendall rank correlation coefficient. Encyclopedia of Measurement and Statistics. Sage, Thousand Oaks, CA, pp. 508-510, 2007.
Junjie Bai, Fang Lu, Ke Zhang, et al. Onnx: Open neural network exchange. Com/on onnx / / 2019.
https://gi /hub
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3286-3295, 2019.
Irwan Bello, William Fedus, Xianzhi Du, Ekin D Cubuk, Aravind Srinivas, Tsung-Y Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. arXiv preprint arXiv:2103.07579 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-ting time attention all you need for video understanding? arXiv preprint arXiv:2102.05095, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103. 14899, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1691-1703. PMLR, 13-18 Jul 2020. URL http:://pproeeddgg
/ (tpreess/v11//chn22222 html
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting spatial attention design in vision transformers, 2021.
Xiaoliang Dai, Alvin Wan, Zhang, B. Wu, Zijian He, Zhen Wei, K. Chen, Yuandong Tian, Matthew E. Yu, Péter Vajda, and J. Gonzalez. Fbnetv3: Joint architecture-recipe search using neural acquisition function. ArXiv, abs/2006. (00/2, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scall seage-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 1091100R..20000 20w.VP...000.22222 5500948.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv preprint arXiv:2104. 2021.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 203-213, 2020.
Francisco Massa. Script to calculate the throughput for DeiT. httppp//////ttttgithibb..oo com/
hwt.iiitttittuu....om accessed 2021-05-01.
Google L.C.C Pixe13 Mobile Phone,
accessed 2020-09-01.
10

---

Under review as a conference paper at ICLR 2022
Google LLC. Performance measurement,
https:/ww..ennsccioww.cooo jeefformance/mmsssueeetttt accessed 2021-05-21.
Google LLC. TensorFlow Lite: Deploy machine learning models mobile and IoT devices, https://wwrt wttps//wwwwteensrflloowo wwtp:/oogggittee/eee//e accessed 2019-11-21.
Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2204.01/ 136, 2021.
Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. Mach. Learn. Res., 3(null): 157-1182, March 2003. ISSN 1532-4435.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020.
Byeongho Heo, Sanghyuk Chun, Seong Joon Oh, Dongyoon Han, Sangdoo Yun, Gyuwan Kim, Youngjung Uh, and Jung- Jung-Woo Ha. Adamp: Slowing down the slowdown for momentum optimizers on scale-invariant weights. arXiv preprint arXiv:2006. 08217, 2020.
Byeongho Heo. Sangdoo Yun. Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Scong Joon Oh. Rethinking spatial dimensions of vision transformers. arXiv preprint arXiv:2103.16320 2021.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 1314-1324, 2019.
Jie Hu, Shen, and Gang Sun. Squeeze-and-excitate networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
A. Jain and Zongker. Feature selection: evaluation, application, and small sample performance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(2):153-11) 158, 1997. doi: 10.1 1..119994..54797..
George H. John, Ron Kohavi, and Karl Pfleger. Irrelevant features and the subset selection problem. In Proceedings of the Eleventh International Conference on International Conference on Machine Learning, ICML' 994, pp. 121-129.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 003611, 2020.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.066
2017.
Chaojian Li, Tianlong Chen, Haoran You, Zhangyang Wang, and Yingyan Lin. Halo: Hardware-aware learning optimize. In Proceedings of the European Conference on Computer Vision (ECCV), September 2020.
Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, and Yingyan Lin. Hw-nas beach Hardware-awand neural architecture search benchmark. arXiv preprint arxiv:210 103. 10584, 2021.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, pp. 510-519, 2019.
11

---

Under review as a conference paper at ICLR 2022
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
Hya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprini arXiv: 2017.
Maxim Lukiyanov, Guoliang Hua, Geeta Chauhan, and Gisle Dankel. Introducing PyTorch Profiler the new and improved performance tool. httpss//ppytorch.oorr///b. wer-prorrreeetteeeeeeeeee and- www.:edddddddooeeeerrerroeeeeooo
accessed 2021-05-21.
NVIDIA Inc. Performance Tuning Maximizing Performance,
https://
http:///...ea...ooo.oaaaaooanaaa Waw.eeiieeaaiieeeeeeeeeeea 4.
L/Performance Srtertrtmannneetatiinaaaaaaaaaap Perparomiaiinooeeeereerreeeeeee accessed 2020-09-01.
NVIDIA Inc. Tensortt open source software, b.
NVIDIA Inc. NVIDIA Jetson TX2, c. https://w..aviiia..
autonooous-maanee sreclineee/emeeddeeesstttteees
accessed / 2020-09-11.
NVIDIA Inc. TensorRT Command-Line Wrapper: trtexec, d. https:ttt/////.. httpp///:tpp//gtuuuuuuoo http:////ggtttuu.omo////1
[enoo/////ceee//aatet//////////p 2ttp:///eeteeteeeeacoooeeee///// 200e:sed0021--05-111
accessed 2021-05-21.
NVIDIA LLC. NVIDIA V100 TENSOR CORE GPU, https:/ww..vvddaa.comm///
www
data eeta-cenntr/v1000 accessed 2020-09-01.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances neural information processing systems, pp. 2019.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10428-10444 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Mennatullah Siam, Mostafa Gamal, Moemen Abdel-Razek, Senthil Yogamani, Martin Jagersand, and Hong Zhang. A comparative study of real-time semantic segmentation for autonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp- 587-597, 2018.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017. URL htt.attta//aaxi..oo.////////////
1707.02968
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6114. PMLR, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles. and Hervé Jégou. Training data-efficient image transformers distillation through attention. arXiv preprint arXiv:2012. 12877 2020.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2 17239, 2021.
12

---

Under review as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv: 0ttp::/7000031000 03762, 2017.
Wenhai Wang, Enze Xie, Xiang Li, Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.12122 2021.
Diana Wofk, Fangchang Ma, Tien-Ju Yang, Sertac Karaman, and Vivienne Sze. Fastdepth: Fast monocular depth estimation on embedded systems. In 2019 International Conference on Roboties and Automation (ICRA), pp. 6101-6108, IEEE, 2019.
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision. arXiv preprint arXiv:2006.0367) 2020.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint 15888, 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
https:://itthbe httpa//cooc....ccooaraaraaaaaata
Yunyang Xiong, Hanxiao Liu, Suyog Gupta, Berkin Akin, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Vikas Singh, and Bo Chen. Mobiledets: Searching for object detection architectures for mobile accelerators. arXiv preprint arXiv:2004. /4525, 2020.
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang. Greedynas: Towards fast one-shot nas with greedy supernet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1999-2008 2020.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. In European Conference Computer Vision, pp. 702-717. Springer, 2020.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint 116::20066045600, 2021.
Hang Zhang, Chongruo Wu, Zhongyue Zhang. Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R Manmatha, et al. Resnest: Split-attention networks. arXiv preprint arXiv:2004.08955, 2020.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11888 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint 11X 2020.
13

---

Under review as a conference paper at ICLR 2022
A ARCHITECTURE CONFIGURATIONS OF PIT-SCALED-TIN//XSSSMMA
Table 8: Architecture configuration of PiT-Scaled TrT-Scaled-Tiny/XSSSA Tiny/XS/Small, including image resolution (7), spatial size (i.e., spatial tokens), # of layers (d), heads (h), and the embedding dimension for each head (e). Here, h in the PiT models has to be in h-2h-4h format (e.g.. 2-4-8 in PiT-Tiny).
Model
FLOPs (G)
Top-1 1 accuracy (%)
I
Spatial size
d
h
DeiT-Tiny
1.26
74.5
224
14 x 14
12
3
64
DeiT-Scaled-Tiny
1.22
76.4 (1 1.9)
X 10
4
x 14 x 14
2
PiT-Tiny
0.71
74.6
224
6
4
32
7 X 7
4
8
32
19 X 19
2
3
32
PiT-Scaled-Tiny
0.70
76.7 (1 2.1)
160
10 10
6
32
5 x 5
12
32
x 14 X 14
2
PiT-XS
1.41
79.1
224
4
48
7 7
4
8
48
19 X 19
2
3
48
PiT-Scaled-XS
1.38
79.5 (1 0.4)
160
x 10
6
5 x 5
12
48
DeiT-Small
4.62
81.2
224
14
6
64
Deil-Scaled-Small
4.79
81.6 (1 0.4)
256
16 X 16
20
4
64
27 x 27
2
3
48
PiT-Small
2.90
81.7
224
14
6
7 x 7
4
12
48
X 31
2
48
PiT-Scaled-Small
3.04
81.8 (1 0.1)
256
16 X 16
10
4
48
8
8
48
Here we provide more details regarding how we transfer our extracted strategies to other ViT models, e.g., the PiT' models. Specifically, to obtain the corresponding TTT-ScaledTTin/XXSSSS
models based on the baseline PTT-Tiny/XSSSmall models and extracted scaling strategies, we 1) locate the most suitable architecture configuration our scaling strategies to be used in the new variant, i.c., DeiT-Scaled-Ting corresponds to PiT-Tiny/XS,, and DeiT-Scaled-Small corresponds to PiT-Small, considering that PiT- THTTTiny/XS/Smalll are designed to at a scale similar to DeíT- Tiny/Saanl (Heo) et al. 2021 2) adjust the scaling factors which the same in both DeiT and the new variant baseline models to match the located architecture configuration in the previous step, e.g., adjusting from 224 160 in PiT-Tiny build PiT-Scaled-Timm and 3) scale down/up the remaining scaling factors if the transferred models cost more/less FLOPS than the new variant baseline models, e.g., scaling up h from 2-4-8 to 3-6-12 12 in PiT-Tiny to build PiT-Scaled-Tiny The details of the finally obtained PiT-Scaled -inaleddiin//S/SSmaa models are summarized in Table S
B DEVICES SETUP
B.1 NVIDIA V100
Device specifications and target applications. NVIDIA V100 (V100) (NVIDIA LLC.) is of the most advanced data center GPUs that accelerate deep learning applications for cloud services and powered by 5120 NVIDIA CUDA cores and 640 NVIDIA Tensor cores. In all experiments, use the 16GB HBM2 GPU memory configuration type V100.
Pre-measurement setup. The V100 GPU system consists of an Intel Xeon Bronze 3204 Processor and 21GB RAM that are able to provide a high processing throughput (i.c., frames per second) of the given DNN models.
14

---

Under review as a conference paper at ICLR 2022
Measurement pipeline. Following (Tovrron et al. 2020), the maximum power-of-two batch size that can fit in the memory when measuring the throughput with the officially provided Py profiler (Maxim Lukiyanov, Guoliang Hua, Geeta Chauhan, and Gisle Dankel) based the PyTorch scripts provided in (Prancisco Massa).
Torch
B.2 NVIDIA EDGE GPU TX2
Device specifications and target applications. NVIDIA Edge GPU TX2 (TX2) (NVIDIA Inc. consists of quad-core Arm Cortex-A57, dual-core NVIDIA Denver2, 256-core Pascal GPU, and a 8GB 128-bit LPDDR4. commonly used in IoT and self-driving environments (Li et al. 2020 Siam et al. 2018 Wofk et al. 2019), working edge computing platform with relatively weak GPU.
Pre-measurement setup. In order to make full use of its resource following Wookk et al. 2019), we enable jetson.clock clock (NVIDIA Inc. a on TX2, pre-setting it into a max-N mode and adjusting the fan speed to 100%.
Measurement pipeline. When we measure the latency of a specific model on TX2, the model definition in PyTorch (Paszke et al. 2019) will be 1) exported into the onnx format (Bai al. 2019) and 2) passed to the TensorRT command-line wrapper (NVIDIA Inc. d). an officially provided binary file, be executed by TensorRT (NVIDIA Inc. b) that is a C++ library for high-performance
inference on NVIDIA GPUs. The corresponding latency is directly reported by the TensorRT command-line wrapper (NVIDIA Inc.
B.3 GOOGLE PIXEL3
Device specifications and target applications. Google Pixel3 (PixeI3) (Google LLC. consists of a quad-core 2.5 GHz Kryo 385 Gold CPU, quad-core 1.6 GHz Kryo 385 Silver CPU, and a 4GB RAM. It is one of the latest Pixel mobile phones, which is widely used the benchmark platform for deep learning targeting mobile devices (Xiong et al. 2020 Howard et al.) 2019 Google L.C..
Pre-measuremeet setup. In order to reduce the variance of the measured latency, the Pixel3 device is pre-configured to only use its big cores to perform the network inference, following the settings in (iiong al. 2020 Google LLC. b).
Mcasurement pipeline. To operate a given model in Pixe13, the model will be 1) converted into the tflite format (Google [ and 2) passed to the tflite benchmark tools (Google LLC. b) that are an officially provided binary file for fairly benchmarking different models in tilitt. The corresponding
latency then directly reported by the tflite benchmark tools (Google LLC. b).
C TRANSFER OUR SCALING STRATEGIES TO VIDEO CLASSIFICATION TASKS
When transferring our scaling strategies to video classification tasks, we follow (Berta- sius et al.) 2021) to (1) decompose an input video into sequence of frame- level patches and feed them into a Trans- former module and (2) include two atten- tion schemes, "Joint" (i.e., applying self- attention into space-time tokens jointly) and "Divided" (i.c., applying spatial and temporal attentions separately), to bench- mark the performance of different models. As shown T our DeiT-Scaled mod- els (e.g. DeiT-Scaled-Tins FLOPss by 33.2% under a similar accuracy (67.4% 67.7%) compared to DeiT- Tiny with the "Joint" attention scheme, and achieve accuracy-FLoC FLOPs trade-offs at least no worse than the original DeiT' (Tourron al. 2020 models in other settings.
Table Kinetics-400 Kay et al., 2017) video clas- sification performance (validation set) of extended DeiT (Tovvron et al. (020) and our Dei'--Scaled models
with a Tirmer (Bertasius al. 2021 style.
Attention Scheme
Top-1 Accuracy (%)
Model
FLOPs (G)
DeiT-Tiny DeiT-Scaled-Tim
67.7
19.9 13.3 (3.2%)
67.4 0.3)
Joint
Deil-Small
71.2
56.5
DeiT-Scaled-Small
71441 0.2)
61.9 (1 9.56%)
DeiT-Tmny
68.4
13.6 12.7 (4.6%2%%
Dell-Scaled-Ting
67.8 (4.00)
can reduce the
Divided
DeiT-Small
71.4
50.8
Del--Scaled-Smal
72.0 0.6)
54.2 (1 6.
15

---

Under review as a conference paper at ICLR 2022
Table 10: Specifications of the hardware devices in the transferability exploration experiments.
Specifications
NVIDIA V100 System (V100)
NVIDIA Edge GPU TX2 (TX2)
Google PixeI3 (Pixel3)
Archileclure
NVIDIA Volta
NVIDIA
Qualcoum
CUDA Cores
5120
256
EPYC
NVIDIA
Cortext
Kryo Gold/Kryo 385 Silver 2.8 GHb//17 Glfz
CPU Max Frequency
3.4 GHlz
2 GHzz2 Glfz
GPU/SeC Memorr
GB
GB 18 W
Power Consumption
300 W
15 W
Table 11: Detailed cost breakdown of Deny-Tiny on different devices for the operators of (1) multi- layer perceptron (MIP), (2) layer normalization (L.ayerNorm), (3) matrix multiplication in multi-head self-attention IMSA-MatMull (4) softmax in multi-head self-attention (MSA-Softmax) (5) reshape and transpose in multi-head self-attention (MSA-Reshee SeSS--eshape& Transpose), (6) gather in multi-head self-attention (MSA-Gather and (7) others.
Operators
1/FPS V100 (%)
Latency TX2 (%)
Latency on Pixel3 (%)
MLP LayerNorm MSA-MatMul
62.50
34.31
69,40
8.95
6.38
1.59
17.65
8.03
21.36
MSA-Softmax
3.48
5.75
3.20
MSA-Reshape&Traneee
5.17
6.32
3.30
MSA-Gather
<0.01
36.38
<0.01
Others
2.20
2.82
1.26
D ANALYSIS ON THE TRANSFERABILITY ACROSS DIFFERENT DEVICES FROM THE HARDWARE DEVICE SPECIFICATIONS PERSPECTIVE
By observing the specifications different hardware devices, which is summarized in Table I0 and the detailed cost breakdown different devices in Table I1 significant differences come from the MLP and e.g.. AISA-Gather costs much more (36.38% <0.01%) and MLP costs much less (34.31% vs. 62.50% (40%) in TX2 than in Pixel3/V 100 and (2) TX2 has the weakest CPU in terms of the maximum frequency among the three devices. Thus, conjecture the slow data movements in TX2 due to the weakest CPU cause the largest MSA-Gather cost percentage in TX2 among these devices. This can explain that the scaling strategies obtained when targeting FLOPs and V100 be transferred to Pixe13 with little or even performance loss,but those obtained for TX2 cannot do that, as mentioned in Section 4.2
conclude that (1) the most operators for all the three devices,
Interestingly, by comparing the extracted scaling strategies for V100 and TX2, we can observe that the scaled VIT in TX2 tends to enlarge more on linear projection ratio (r), which will not increase the cost of self-attention, (78. 17% vs. 78. (8%). as shown in Table 12 This matches the observation that the self-attention costs a large portion of the cost on TX2 (e.g., 56.48% for DeiT-Tiny), as demonstrated in Table I1
compared to the scaled VIT in V100 (16 vs. 4) under a similar accuracy
Table 12: Detailed architecture configurations of the scaled ViT models with throughput (i.e., FPS) on V100 (V100 Scaling) and latency on TX2 (TX2 Scaling) the hardware-cost during scaling, respectively.
Metries
V100 Scaling
TX2 Scaling
Accuracy (%)
78.10
78.17
FPS on V100
2488.81
1984.10
Latency TX2 (ms)
25.18
23.70
Num. of layers (d)
13
10
Num. of heads (h)
5
4
Embedding dim. per head (e)
64
64
Linear projection ratio (r)
16
Image resolution (I)
160
160
Patch size (p)
16
16
16

---

Under review as a conference paper at ICLR 2022
E IMPLEMENTATION DETAILS
In this section, we provide the implementation details of our experiments, including (1) our scaled
ViT (Dosovitskiy al. 2021] Touvron et al. 2020) models on ImageNet (Deng al. 2009) dataset
in Section 4.1 and 2 (2) our scaled PiT (Heo et al. [021] models on ImageNet (Deng al. 2009)
dataset in Section 4.3. (3) our scaled ViT (Dosovitskiy al. 202T] Touvron et al.) models on
COCO (Lin et al. 2014) dataset in Section 4.3.2, and (4) our scaled ViT (Dosovitskiy al. 2021
Touvron et al. 2020) models on Kinetics-400 Kay et al. 2017) dataset in Appendix
Scaled VIT models on ImageNet dataset. All the scaled ViT (Dosovitskiy et al.
2021 Touvron
et al. 2020) models on ImageNet (Deng et al. 2009) dataset reported in Section 4.1 and 4.2 follow the same training recipe (including the data pre- (re-processing) with the proposed in Touvrro et al.
2020), i.e., training on ImageNet for 300 epochs (1000 epochs for models in Table with batch size
1024, AdamW optimizer (Loshchilov & Hutter 2017), learning rate 0.001, cosine learning rate
decay, weight decay as 0.05, 5 warmup epochs, and distillation from RegNet (Radosavovic al. 2020).
Scaled PiT models on ImageNet dataset. To make a fair comparison with PiT (Heo et al. 2021 models, all our scaled PiT models (i.e., PTT-Scaled-TinyXXSSSaalll in Table follow the training recipe (including the data pre preppeessssing) in PiT (Heo et al. 2021), which uses the same learning rate, weight decay, warmup epochs, total epochs, and distillation settings with (Touvron et 2020 but using AdamP (Heo et al., 2020) the optimizer instead of AdamW (Loshchilov & Hutter 2017
Scaled ViT models on COCO dataset. Following the training recipe (including the data pre- processing) described in (Zhu et al. 2020), all the models are pre-trained on ImageNet (Deng et al., 2009) first, and then trained on COCO (Lin et al. 2014) dataset for 50 epochs with Adam optimizer, learning rate as 0.0002, weight decay 0.0001, and the learning rate is decayed the 40-th epoch by a factor of 0.1. Note that when adapting the models pre-trained on scale the positional embeddings of VIT via bilinear interpolation to
SoageNet(Dengg et al. 2009) to
COCO (Lin et al. 2014), match the differences of image resolutions and use the feature map before the final classifier and layernorm layer as the input feature map the Deformable DETR header.
Scaled ViT models on Kinetics-400 dataset. For Kinetics-400 (Kay et al. 2017 dataset, we follow the training recipes (including the data pre-processing) in (Bertasius et al.; [222]) start from the ImageNet (Deng et al. 2009) pre-trained models. Then clips of size 8 xx224 x 224 with frames sampled as rate of 1/32 are used for training. All models are trained for 15 epochs with learning rate as 0.005, batch size as 16, SGD optimizer with momentum 0.9, and the learning rate is decayed at the 10-th and 14-th epoch by factor of 0.1. We also include both the "Joint" (i.e., applying self-attention into space-time tokens jointly) and "Divided" (i.e., applying spatial and temporal attentions separately) attention schemes describled in (Bertasius et al. 2021) to make more fair comparison with the baseline models which use DeiT' (Touvron et al. 2020) models as backhones.
F ARCHITECTURES COMPARISON BETWEEN THE SCALED MODELS AND THE RANDOMLY PERMUTATED MODELS
To further explore why the architectures with the best accuracy efficiency trade-off after the random permutation on top the scaled models can achieve better performance than the scaled models, as shown in Figure 4 of the main content, here summarize their performance and architectures details in Table 13 As compared to the scaled models (i.e., DeiT-Scale-PV ('iny- Small)), those architectures with the best accuraccy vs. efficiency trade-off after random permutation (i.e., DeiT-Scale-(TTe (Tiny, RPP) adopt different scaling factors except the number of heads.
Table 13: Random permutation further boosts the performance of the scaled models.
Model
FLOPs (G)
Top-1 accuracy (%)
d
h
e
I
DerI-Tiny
1.26
12
64
Del--Seale--Tin
1.22
76.4 (?1.9)
14
64
160
16
MelT-Scaled-Tiny-RR
1.22
76.9 (12.4)
17
60
171
19
DefT-Small
4.62
81.2
12
64
224
16
DefT-Scaled-Smaa
4.79
81.6 (70.4)
20
64
256
16
DeeT-Sealed-Small
4.79
82.0 (10.8)
21
68
210
15
17

---

Under review as a conference paper at ICLR 2022
G MORE DETAILS ON THE ITERATIVE GREEDY SEARCH
To better demonstrate the iterative greedy search method introduced in Section 3.3
we summarize
the important details Algorithm | In nutshell, we 1) start from small model defined in Table 2) in each scaling step, we increase each scaling factor alone to meet the pre-set hardware-cost target;
3) we then select the one with the best accuracy vs. efficiency trade-off out of those architectures resulting from increasing each scaling factor standalone in the previous step; and 4) the selected architecture from the previous step will used as the starting point in the next step.
Algorithm 1 Iterative Greedy Search
Step:
2: Architecture: A + Ao (Ag is shown in Table
3: Hardware-cost metric: C()
4: Total steps in the scaling: N
5: Target hardware-cost in each step: [C1, C22 ..., Cv]
6: for in [1, ..., N] do
7: for scaling factor SF h, e, I,p] do
8: AsF +
9: while C(AsF) < C. do ++
11:
end while
A'ss ASF end for
13:
A < max accuracy(A3, (43, Ag, Ag. AG, A4, A5)
15:
A A
end for 17: Searched architectures A,, A2,..,, AN
18


Reviews:
[{"review_id": "hnmBkrAeYLM", "reviewer": "Reviewer_WJMX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.This work provides a new perspective to explore ViT architectures, aiming to analyze and understander ViT’s architecture design space and real hardware-cost across different devices.\n2. This work summarizes how to determine if a model searched on one hardware device applies to another.\n\nWeakness:\n1.Some typos such as “TRAFE-OFFS” in the title of section 4.1.\n2.The 24 different structures generated by random premutation in section 4.1 should be explained in more detail.\n3.The penultimate sentence of Section 3.3 states that \"iterative greedy search can avoid the suboptimality of the resulting scaling strategy on a particular model\", which is not a serious statement because the results of the iterative greedy search are also suboptimal solutions. \n4.The conclusion of \"Cost breakdown can indicate the transferability effectiveness\" in Figure 7 is not sufficient. We cannot extend the conclusion obtained from a few specific experiments to any different hardware devices or different architectures.\n5.Why not use the same cost for different devices instead of flops, latency, and 1/FPS for different hardware?\n6.The result comparison of \"Iteratively greedy Search\" versus \"random search\" on the model structure should be supplemented.\n", "summary_of_the_paper": "This work presents a study for exploring hardware-aware ViT scaling and demonstrates that a simply scaled vanilla ViT model can achieve a comparable accuracy-efficiency trade-off as compared to dedicatedly designed ViT variants. Furthermore, they discuss the transferability of the scaling strategies on different hardware devices, ViT variants, and computer vision tasks. ", "summary_of_the_review": "Please provide a short summary justifying your recommendation of the paper\nAlthough some parts of the paper are not clearly explained and the method is not very innovative, the analysis of the transferability of different scaling strategies on different hardware devices in this work is very positive and provides great inspiration to relevant fields. Therefore, it is recommended to accept this paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "g1ZJUNz1IZC", "reviewer": "Reviewer_d8q8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- Approach is tested on multiple devices under different constrains (FLOPs, latency etc). It is clearly shown that optimization for a specific hardware results in a better corresponding model. \n- Scaling factors for models are clearly indicated in the tables for reproducibility.\n- Improvements are shown over the baseline Deit models and are up to 1.9% better.\n\nWeakness:\n- For Figure 1. Did authors include all recent transformer models? For example LeViT is not there, it demonstrates higher accuracy and lower number of FLOPs (LeViT is 1.2 and 2.4x faster than PiT). If it holds then authors should be careful when comparing to hand designed ViT and claiming comparable or better accuracy-efficiency trade-off. Will be interesting to look at the updated figure with recent models. \n- Scaling method based on an iterative greedy search is not presented well. Going over the section 3.3 multiple times, it is still not clear how exactly the search was performed. Additionally, it is not obvious how different target constraints are reached (FLOPs or latency).  \n\nMinor:\n- In the introduction when comparing to CNNs, author mention EfficientNetV1, however, a much stronger EfficientNetV2 has been published. \n- Include top performing CNN models into Figure 1.\n- Existing calling methods for ViT. in section 3.2 authors first talk about CNN scaling strategies and conclude that those applied to ViTs can lead to ambiguity and sub-optimal performance. Where can we find supporting evidence for the statement? \n", "summary_of_the_paper": "The paper is focused on scaling width, input image resolution and other configurations of ViT model (6 in total) to make it hardware friendly and reach better accuracy when trained from the scratch. Results are verified on the edge devices - Nvidia TX2, Pixel3. Models optimized for different hardware result in higher performance on the targeted device. \n", "summary_of_the_review": "Paper contains an interesting empirical study on how iterative greedy search can be applied to the design parameters of ViT. Resulting models show speed-latency improvements over hand designed ViT models. The method of the search is not well explained (it seems to be the previously proposed method). Resulting models seem to be less generic and more engineered, not clear how presented results will scale beyond Tiny/Small/Base models. Presented study will be interesting for people designing ViT models, however its limitation is in the conclusions/findings specific only to ViT models (even with transfer to PiT). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GsnH291vdEr", "reviewer": "Reviewer_zFw4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The best part of the paper is about its \"usefulness\" and practical friendliness. The paper just studies the different factors that contributes to the model size/speed and accuracy (usually larger models on higher resolutions lead to better accuracy but slow down inference speed). An impressive part of the paper is that it shows measurements of speed beyond FLOPs -- it actually runs on various devices and measure the latency there. I think this practical mind is great.\n+ It also shows good generalization of the finding. E.g. it can be transferred to more recent models (PiT), other tasks (object detection, video). This suggests solid improvement.\n+ Overall the paper is well-written. The idea is clear, the organization is clean, and the figures are very illustrative. Overall it is even a good learning experience for me (e.g. knowing the latency break down of different devices) who is not so familiar with the hardwares.\n\n- Biggest concern about this paper is novelty. It has some good findings, it shows good numbers, and it also teaches us something. However, the method (for search) is based on existing work, the application to other devices is quite straightforward (though surely requires a lot of hard work), and to me, it is more like a technical report about something is well executed, rather than an interesting \"finding\" that has a good deal of scientific merit.\n- One more concern is about the word \"scale\". I don't think the study presented in this paper is about \"scaling\". To me the scaling is more about methods that scales *up* to larger models -- when people say about doing things \"at scale\", it indicates at \"a large\" scale. I come with the expectation that this paper can show how to scale our existing model to the next generation of model size, but only to find this paper is more about \"speed-accuracy\" trade-offs. So I would recommend replacing the term \"scale\" with \"speed-accuracy trade-off\" to be precise.", "summary_of_the_paper": "The paper studies different hyper-parameters in the ViT design space (e.g., the number of layers, the dimensions per head, the number of head, the MLP ratio, the input image size). It starts with a very small, basic model, and then scales it up based on an iterative, greedy search. Each step it will select the best model (in terms of speed-accuracy trade-off) as the starting point of the next step. The \"speed\" aspect is firstly measured by FLOPs, and then later replaced by empirical measurements like latency on some devices. Overall, the resulting architectures show better speed-accuracy trade-offs than previous DEiT architectures, and shows consistent gains when transferred to other architectures (PiT) and tasks (e.g., object detection).", "summary_of_the_review": "I think this paper is more on the \"technical report\" side, with quite limited novelty. I also find it gives me the wrong impression that the investigation is more about \"scaling-up\", but instead it just does search in the \"small-to-base\" scale regime. It definitely teaches me something and presents some interesting data points, but to me the scientific merit is not enough to warrant a publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1Ks7Js7Qq5p", "reviewer": "Reviewer_Lm1r", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n-This paper analyzes different scaling strategies of vision transformers and proposes a hard-ware scaling strategy for vision transformers. The authors conduct extensive experiments and bring some insights.\n\n-The proposed model achieves a better trade-off between FLOPs and accuracies than the existing models. It is interesting to see that only adjusting the model size can achieve better performance than SOTA transformer models.\n\nWeakness:\n- [1] has discussed how to scale vision transformers. A more detailed comparison with [1] is required.\n\n- Hardware-Aware architecture searching has been well investigated. The proposed method is not novel.\n\n[1] Scaling Vision Transformers, arxiv 2106\n\n\n", "summary_of_the_paper": "This paper proposes a model scaling strategy to change the model size. With a greedy search strategy, the author proposes a hardware-aware scaling method. Compared with the existing ViT variants, the proposed method achieves better performance.", "summary_of_the_review": "The proposed method is not novel, but the insights brought by the authors are interesting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores strategies for scaling vision transformers that can be transferable across hardware devices and ViT variants. While it presents some interesting observations as well as a useful practical guide, multiple reviewers expressed major concerns over the novelty and significance of the methods and findings. Besides novelty and significance, there are also some concerns about comparison with existing work as well as clarity of the presentation.
Decision: Reject