Real-time Core-Periphery Guided ViT with Smart Data Layout Selection on Mobile Devices
Authors: Zhihao Shu
Xiaowei Yu
Zihao Wu
Wenqi Jia
Yinchen Shi
Miao Yin
Tianming Liu
Dajiang Zhu
Wei Niu
Venue: NeurIPS 2024 poster (2024)
Keywords: Mobile DNN Acceleration
Model Pruning
ViT
Primary area: deep_learning_architectures

Abstract: Mobile devices have become essential enablers for AI applications, particularly in scenarios that require real-time performance. Vision Transformer (ViT) has become a fundamental cornerstone in this regard due to its high accuracy. Recent efforts have been dedicated to developing various transformer architectures that offer im- proved accuracy while reducing the computational requirements. However, existing research primarily focuses on reducing the theoretical computational complexity through methods such as local attention and model pruning, rather than considering realistic performance on mobile hardware. Although these optimizations reduce computational demands, they either introduce additional overheads related to data transformation (e.g., Reshape and Transpose) or irregular computation/data-access patterns. These result in significant overhead on mobile devices due to their limited bandwidth, which even makes the latency worse than vanilla ViT on mobile. In this paper, we present ECP-ViT, a real-time framework that employs the core-periphery principle inspired by the brain functional networks to guide self-attention in ViTs and enable the deployment of ViT models on smartphones. We identify the main bottleneck in transformer structures caused by data transformation and propose a hardware-friendly core-periphery guided self-attention to decrease computation demands. Additionally, we design the system optimizations for intensive data transformation in pruned models. ECP-ViT, with the proposed algorithm-system co-optimizations, achieves a speedup of 4.6× to 26.9× on mobile GPUs across four datasets: STL-10, CIFAR100, TinyImageNet, and ImageNet.

Real-time Core-Periphery Guided ViT with Smart Data Layout Selection on Mobile Devices
Zhíhao Shu
Xiaowei Yu
Zihao Wu'
Wenqi Jia Yinchen Shi
Miao Yin-
Tianming Liu²
Dajiang Zhu
Wei Niuè
University of Georgia 2 University of Texas at Arlington
NYU
7Zhihao.Shu Zihao. Wu1, wniu)ouga.edu
tliußcs. edu ys46530nyu. edu
fxxy1302, wxj1489)0mave wut.
{daj langgzzu,, miao. yin)euta.eee
Abstract
Mobile devices have become essential enablers for Al applications, particularly scenarios that require real-time performance. Vision Transformer (ViT) has become fundamental cornerstone in this regard due its high accuracy. Recent efforts have been dedicated to developing various transformer architectures that offer im- proved accuracy while reducing the computational requirements. However, existing research primarily focuses on reducing the theoretical computational complexity through methods such as local attention and model pruning, rather than considering realistic performance on mobile hardware. Although these optimizations reduce computational demands, they either introduce additional overheads related data transformation (e.g., Reshape and Transpose) or irregular computation/data-cco patterns. These result in significant overhead on mobile devices due to their limited bandwidth, which even makes the latency worse than vanilla VIT on mobile. In this paper, we present ECP-ViT real-time framework that employs the core-periphery principle inspired by the brain functional networks guide self-attention in ViT's and enable the deployment of ViT models on smartphones. We identify the main bottleneck in transformer structures caused by data transformation and propose serdware-friendde core-periphery guided self-attention to decrease computation demands. Additionally, we design the system optimizations for intensive data transformation in pruned models. ECP-ViT ViT, with the proposed algorithm-system co-optimizations, achieves speedup of 4.6x 26.9x mobile GPUs across four datasets: STL-10, CIFARI00, TinylmageNet, and ImageNet.
Introduction
In recent decades. there has been significant increase in applying deep neural network (DNN)
architectures across various fields, including autonomous driving [23), natural language processing (9)
extended reality (XR) [13), image processing [22]. and View Synthesis 32 Along with significant
progress hardware performance and public datasets, more and more complex DNN architectures
have been proposed, including Convolution Neural Network [52] RNN (36. Transformer (51). These architectures have led to groundbreaking breakthroughs in various application domains by leveraging their powerful feature extraction abilities. Meanwhile, mobile devices have become essential for deploying these applications, especially in scenarios that demand real-time performance. The mobile GPU Oneplus 1I can achieve theoretical peak performance of over 3T FLOPS (floating point
Equual contributions.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

88
BX Faster
MetaFormer-B
DeiT
MolaFarmer
Top-1 Accuracy (%)
Swih-S
MeteFormenT
MTT-B
82
ECPVIT-S
CSWIN
Swin-B
GrosaFome ormer
ConVIT
80
MT-S
SwinTT
78
ECPMT-T
MACs
VT-T
74
ResNat50
5G
10G
15G
20G
VGG
72
70
250
500
750
1000
1250
1500
1750
2000
Latency (ms)
Figure I: Accuragy-Latency-MACC comparison among various models on ImageNet- IK. Latency is measured on Oneplus 11 (Snapdragon 8 Gen 2 SoC). The radius of each circle represents the model's MACs MMutiply-Accumulle and offers the fastest inference speed among transformer- based models, providing 6x faster than ViT-Base. We simplify the notation as "Base" B, "Small" S, and "Tiny" = T.
Operations). Our ECP- ViT-Base achieves the highest top- accuracy
operations per second) [
Their widespread availability and increasing computational capabilities make them ideal platforms for extending the impact of these algorithm innovations.
Compared to traditional convolutional neural networks (CNNs), transformers have recently become widely used across various areas and tasks due to their high accuracy. The attention mechanism in the transformer allows neurons exchange messages effectively and efficiently, leading promising results in natural language processing [48] [6] and computer vision domains [17] [53] However, transformer architectures also known for their deeper network layers and require frequent reshaping and transposing of the feature maps. This results in more intermediate results and leads to memory bound for the computation. Optimizing transformers for efficient exccution becomes particularly challenging in environments where memory bandwidth is limited.
Advancements in transformer architecture design, including Vision Transformer (ViT) [17], have focused on improving message exchange mechanisms among spatial tokens through different Token Mixers. Other efforts include the shifted window attention in Swin (34), the token-mixing MLP in Mixer [44, and the pooling in MetaFormer [57]. These designs aim to enhance self-attention accuracy and hardware efficiency compared the original vanilla ViT (17), enabling more effective and efficient execution on various hardware platforms. Despite the significant progress made in transformer architecture, particularly in reducing theoretical computation demands, achieving real- time performance for transformer-based models on mobile devices remains major challenge. Figure T illustrates the latency, accuracy, and floating-poin operations (FLOPs) comparison of different models on mobile GPU. Despite having fewer FLOPs, Swin-T (local attention) is even more than x slower than ViT-T (global attention). This is because local attention requires more frequent data transformation Swin. contrast, these numbers are 43% in ViT and only 0.8% VGG16. Similarly, DeiT has broadly similar structure to VIT, but it involves more complex data layout transformations and larger intermediate results, as illustrated in Table Unlike powerful server platforms, mobile devices have limited memory bandwidth (24), making it challenging to benefit from reduced theoretical FLOPs. There has been a fundamental lack general principles that involve co-designing model architecture and system optimizations.
reorganize tokens, which accounts for 69% of execution time in
In this work, we present an integrated framework that incorporates co-optimizations by revisiting the model design and system optimizations. First, we propose hardware-efficient and computational- friendly sparse scheme (guided by the Core-Periphery principle brain networks) 61
59 60 58
62 that can be applied to guide the message exchange in self-attention. This scheme also helps reduce bandwidth pressure for the subsequent Softmax operation. Secondly, we develop a set of comprehensive compiler-base based optimizations supporting the proposed pruning scheme and

---

Table Comparison between ViT and DeiT. All data is measured on Oneplus (Snapdragon Gen 2 SoC). Layout Transformation indicates the time spent on transforming the tensor's ' layout, such as Transpose and Reshape. Computation indicates the time spent on pure tensor computation.
Model
Layout Transformation (ms)
Computation (ms)
Intermediate Results (MB)
ViT-Base 17
324
286
106.47
DeiT-Base
46
1303
633
125.42
fully eliminate the overhead for Transpose and Reshape. Enabled by the advanced compiler optimizations, it is possible to achieve both high accuracy and high acceleration simultaneously. We demonstrate that, 1) our proposed fine-grained structured core-periphery guided self-attention offers advantages both accuracy and speed, and 2) our compiler framework exhibits superior end-to-end acceleration performance for both the original and proposed ECP-ViT models.
We summarize our contributions in three aspects:
We incorporate the organizational principle of brain functional -irrookss-seccicclllly the core- periphery principle-to guide self-attentions ViTs. Additionally, we introduce a compiler code generation framework that efficiently supports the core-periphery structures on mobile devices, We develop comprehensive compiler-based optimizations that can fully eliminate the overhead of data transformation, bridging the gap between accuracy and latency.
We build ECP-ViT VIT, an end-to-end framework that combines algorithm and system design to achieve real-tims performance on mobile devices. ECP- CCT achieves the significant speedup on off-the- shelf mobile devices, reaching up to 16.1 x on ImageNet inference while maintaining an accuracy of 84.51 %.
2 Background and Motivation
Mixed blessing of attention. ViTs are widely utilized as robust backbones across various tasks. The two primary types of attention are Global Attention [3 33) and Local Attention [39
6 42 71.
Global Attention, as seen in standard transformer models, allows each token to attend to every other token in the input sequence. This mechanism ensures comprehensive context capture, leading high accuracy. However, the downside is its computational intensity, as it scales quadratically with the input length, making it less efficient for longer sequences or resource-constrained platforms.
restricts the focus of each token to a subset of adjacent tokens (or with specific patterns), resulting in the reduction of computational complexity. However, local Attention does always translate into proportionate realistic speedups when compared to the theoretical reduction in floating-po praaing--pont operations (FLOPs). This approach significantly reduces the computational complexity but at the cost of more layout transformations. These factors offset the expected gains from reduced computational complexity, particularly environments like mobile GPUs where memory bandwidth and efficient data handling are critical, as reflected in Figure I The actual performance gains need to be evaluated in the context of specific hardware and application requirements. Compared the above two types of attention, ECP-ViT VIT is the first work to explore a speed-award end-to-end framework that achieves real-time performance on real-world mobile devices for ViTs.
Local Attention 64 49
Efficient network design and architecture search. In this category, a significant methodology is 40 18
Some of the work leverages network pruning and sparse training to further reduce the theoretical FLOPss At the token level, Tang et al. 43 introduces a patch slimming method to remove redundant tokens. Evo-Vit ViT (54) updates selected informative and uninformative tokens through distinct computation paths, while VTP (66] reduced embedding dimensionality by introducing control coefficients. At the model architecture level, UP-ViTs (56) adopts unified approach to prune channels in ViTs. SViTE [IO) dynamically extracts and trains sparse sub-networks instead of training the entire model. Despite the significant progress made by these methods, both token- sammling and data--rivve strategies may heavily depend on specific datasets and tasks, limiting the generalization capability of vision transformers. Additionally, these loken-level pruning or selection introduce additional operations for the Reshape and Transpose to the feature map, leading to fewer benefits from reduced computation complexity. contrast, ECP-ViT achieves significant speedup mobile platforms through two key components: 1) the more efficient
Neural Architecture Search (NAS) 67
3

---

(6) Core-Periphey Graph Generation
(b1) Self Ameriien Re scheddlle
(c) Periphery Transformet
CPAtteution so/tmax CP RONT
MLP
Neatn
Core Nodes
Mullsffoe Atlentiee
Periphery Nodes
(b2) Core Peee-Perehey Guided Matris Multipication
Message
Ji
Vauu
Embesdddd
Proches
Suftaan-level Design
Handware-loee Design
Marmul
Reshape
Emrbeskisd
Patches
Restape
Keshhaee
Implicit
Matrol
Slice
Reshape
Intispeee
Softnux
Fased Patch Emhheddngg
sopose
Reshape
Fased Atlention
Figure 2: Co-Design Core-Periphery Principle Guided self-attention mechanism ECP-ViT VYT. At the software-level design, the rescheduled interactions between patches in ECP-ViT are guided by the generated Core-Periphe Prri-eerrhhry (CP) graphs. Moreover,
multiplication query, key, and value matrices in the self-attention mechanism of ECP-VI ViT is also under the guidance of the core-periphery graph. At the hardware-level design, the slice and transpose reshape operations are eliminated.
self-attention guided by the Peripherr principle, and 2) our compiler-based optimizations that eliminate data transformation overhead (i.e., Reshape and Transpose).
DNN frameworks on mobile devices. Recently, there has been dedicated focus on developing inference acceleration frameworks for mobile devices from both academia and industry. Some efforts include MCDNN [21), DeepSense (55), MobiSR (30), and PatDNN (38. However, none these frameworks support the execution of transformer models on mobile devices. Other efforts have been made to optimize the execution of transformer models on mobile devices, include TFLite (2], TVM [ΣI. MNN (26). Pytorch-Mobile (25), and DNNFusion (37]. They support optimizations including operator fusion, constant folding, and quantization on mobile devices. However, they are not able to eliminate the intensive Reshape and Transpose operations in transformer models. In this work, our goal is to find the most appropriate CP pruning scheme for mobile ViT acceleration and the corresponding full-stack acceleration framework.
Methodology
The entire framework of co-design for VIP-VIT is illustrated in Figure 2
comprises software- level design (algorithm) and hardware-level design, both of which we will discuss in detail in the subsequent sections.
3.1 Problem Definition and Research Issues
As mentioned earlier, the attention mechanism in transformer models involves large amount of data transformation, which poses significant challenges hardware efficiency and deployment on mobile devices. This is because (i) data transformation is memory- bound, requiring high memory bandwidth. mobile devices, which typically have restricted memory bandwidth, this leads increased time consumption for processing these transformations; and (ii) data transformation in transformers, especially within the multi-head attention mechanism, often results in irregular data access patterns. This is typically observed during operations like tensor reorganization. Such irregular access patterns

---

CP Grugh (Adjacency Manis) sDarhhate
Cumplete Graph
Aestape
Reti:
Nakmrd
NCCWM1
Wler Reshage
buttMox
Tranpoor
1956 (KGHMM)
INCAWA
Inplicat Roshape
Slice & Reslage
Reslege
Marral
Figure 4: Examples of Core-Periphery Nodes in the graph, where the core ratio is calculated by di- viding the number of core nodes by the total num- ber of nodes. The white area represents pruned nodes. In adjacency matrices, black color indi- cates connections between nodes, while white represents no edge.
Ourput
Figure 3: Illustration of VIT on attention module. In this example, we use the NCHW format for presenting our data. Words with red color are explicit data layout and transformative operators.
can significantly degrade performance, particularly in the absence of sufficient concurrent processing threads.
Explicit and Implicit. Explicit transformations (red color operators in round boxes) are designed by the algorithm developer to ensure the model follows its intended computational logic. On the other hand, Implicit transformations adopt data layouts that best suit layout-sensitive operators for increased effectiveness. implicit focuses more on performance optimization tailored to specific hardware platforms and depends on the inference framework. For instance, TFLite may prefer using NHWC layout for MatMul layer, while others may prefer NCHW ones. Based on the classification mentioned above, ECP- ViT specifically answers these three questions to achieve computation efficiency and data transformation elimination:
We classify the data transformation into two categories (as shown in Figure 3),
How design sondware-friendee pruning scheme for ViTs without compromising accuracy?
How to effectively minimize the data transformation overhead without affecting accuracy?
How flexibly reduce the memory pressure on mobile?
3.2 Pore-Periphery Guided Self-Attention in VIT
The design of VCT-, as shown in Figure is inspired by Brain Neural Networks. Specifically, Fig- ure iillustrates the selection of various CP graphs referenced in Figure 2 (a). The workflow involves CP graph generation, CP-guided self-attention, and CP-guided QKV multilicaaioon corresponding to Figure 2 (a), Figure 2 (b1), and Figure (b2). In these networks, the core nodes maintain connections to all other core nodes, while edge nodes only connect to a subset (or empty) of the core nodes. We employ Grad-CAM to identify important regions of the images and assign the core nodes to those regions during training. Accordingly, the QKV matrices of these patches are divided into core and peripheral components. For example, for images with resolution of 224x224 and patch size of 16x16, there are a total of 196 patch tokens as nodes. For a core ratio of 10%, around 20 patch tokens considered cores, and we choose the top 20 important regions as cores. This partitioning method is inspired by human brain networks [60], where different networks exhibit different core ratios. This architecture allows BNNs to effectively enhance information transmission and communication for integrative processing (5 19). To incorporate the Core- Periphhrr principle into the self-attention mechanism of ViT, we redefined the self-attention operations based on the generated Core-Periphery (CP) graphs, where the patches are regarded as nodes, and the new self-attentio relationships are represented by edges in the CP graph. Following this representation paradigm, a complete graph can depict the self-attention of the vanilla VIT. Similarly, the infusion of the Core-Peripher principle into the ViT architecture achieved by enhancing the complete graph with the generated CP graphs effectively and conveniently. The new self-attention rules can then be redefined: CP graph can be
represented by g = (V, E), with nodes set = (v) ..., ), edges set & S {(v,, 22) 14, Vy E V}. and adjacency matrix A. The CP graph guided self-attention for a specific node at r-th layer of ECP-ViT is defined as:
Li (K()))
(r)
2 (r+1) = -000(1(
WV N(O)).
(1)
Voh
5

---

where a(+) is the softmax function, of") is the query of patches in the i-th node in G, N(i)
faliv (i, E E are the neighborhood nodes of node i, dg is the dimension of queries and keys, and kj and vi()) are the key and value of patches in node j.
Vanilla Self-Anention
In ECP-ViT, each node can contain a single patch a set of multiple patches. We propose the following patch-assigning pipeline to map the original patches to the nodes of the CP graph. In vanilla ViT with patch size 16 x 16, one input image with resolution 224 x 224 is divided into 196 patches. When we use CP graph with n nodes to design the self- attention mechanism, 196 mod nodes will be assigned [166n]] + patches and the re- maining n - (196 mod n.) nodes will be as- signed [16//n] patches. For example, we use 5 node CP graph, the 5 nodes will have 40, 39, 39, 39, and 39 patches, respectively; and if we use 196 nodes CP graph in another case, each node will contain a single patch. Based on the above discussion, the CP graph- guided self-attention that is conducted at the node level can be formulated as:
KT
V
Core-Poriphery Guided Self-Atteation
C
(CP)
CP
h
h4
hy
Tiecsornni
b2
P
Figure 5: The Core-Periphery Principle guides Self-Attention where the Query, Key, and Value matrices are partitioned into core (C) and pe- riphery (P) components. The conventional self- attention mechanism is transformed into Core- Periphery (CP) attention through the guidance of Core- Core-Periphery graphs.
CPAttention(Q, Q,K,K,
Q.K
OnKe Vdk
concat(soflmax
) softmax ViL
Vo),
where queries, keys, and values of all patches are packed into matrices Q, K, and V. respectively, and subscript and p represent the core parts and periphery parts of the matrices. The graphical illustration of CPAttention shown in Figure(3] Similar to the multi-head attention in transformers 48 our proposed CP multi-head attention is formulated as:
MultiHead (Q, K, V) concat(headd,.... head,, W where head, CPAttention(QW, KW;K, VWY),
where the parameter matrices W.O. W.K, WY and W the projections. Multi-hean attention helps the model to jointly aggregate information from different representation subspaces at various positions. In this work, apply the CP principle each representation subspace. Therefore, the self-attention guided by CP graphs in ViT reduces the computational budgets, while still maintaining the hariware-friendly computation pattern, i.e., converting the traditional self-attention into two small parts of matrix multiplication.
3.3 Flexible Data Layout Selection
DNN execution on mobile devices is in manner of layer-wise computational graph (CG). The CG consists of nodes and edges, each node is an operator such as MatMul or LayerNorm, and the edge is an indicator to show the direction of the data flow. The main idea behind our optimization for eliminating data transformation is that, in a CG, each operator (such as MatMul or Convolution) has both producers and consumers. The producer generates layout based on the consumer's 's preferred data layout, resulting in relatively low additional overhead compared to explicitly reorganizing the data. For instance, if MatMul is followed by Transpose and another MatMul, we can make the first MatMul directly generate the desired data layout for the second MatMul, thus avoiding the need for explicit data reorganization within Transpose. Our compiler optimizations consist of three steps: (i) Identify nodes to fuse; (ii) Determine possible data layouts for the key nodes; (iii) Evaluate possible data layouts. We elaborate them in the following sections.
Identify nodes to eliminate starts by classifying the operator in CG into two types: 1) key nodes (nodes that perform the actual computation such as MatMul, Softmax, and Add are called key

---

nodes; and 2) nodes that only do layout transformation such as Slice and Transpose are called transformative nodes, which are the targets to eliminate. We first use the transf ormatiormative node as a breaking point to partition the graph into a set of sub-graphs. We also apply the fusion rules similar to [37] to identify all fusion opportunities within the sub-graphs. To fully eliminate the data transformation operations, our general strategy is to find a common data layout which works efficiently for both contiguous sub-grap -oobggrahss Note that, in this step, transf ormative nodes are still kept in the sub-graph because key nodes' data layouts are not the same.
Input Tensor V
Input Tensor >
Input Tensor
Determine and evaluate possible data layouts are to find out the best intermediate data layout to eliminate the necessity of introducing addi- tional operators solely for layout transformations. We conduct an exhaustive evaluation of all feasi- ble data format selections, intending to optimize hardware accelerations. It is worth noting that varying data layouts result in distinct access pat- terns within the mobile GPU, as shown in Figure Notably, noncontinuous data access patterns yield inferior data locality, thereby leading to in- creased lateney. Our dimension reduction heuris- tic is aiming to find the reduction dimension(s) from both ends and group the reducing dimen- sion continuously in the memory, in order to avoid the undesired data accessing pattern. For example, in a Matmul operation with A and BR... the dimension is our reducing dimension.
<
Manmull
Attemtion
Attemioon
Sliee
Slice
Slaee
Transpose
Transpose
Transpose
Matinal
Sothmax
Marmal
[ddd Original Ascution
Add
Add ECP Attention
Fised
Attention
Figure 6: Data Access Pattern Comparisons. On the leftmost, it is the naive attention computa- tional flow graph. In the middle part, it is the strided access pattern without any optimizations, and the rightmost graph is our optimized data access pattern.
Furthermore, specific operators in the subsequent operators may suitable for specific data formats. Take the LayerNorm operation an example, it usually applies the calculation one dimension only, we generally arrange the target dimension in the width continuously, and group other dimensions into the height dimension. It gives better data locality and GPU utilization. After using our heuristic algorithm to find the best data layout for each key node, we are then able to calculate the index transformation from the current node to its child nodes and map the index at the kernel level, which essentially eliminates all the trans-formate results within the GPU texture memory to mitigate the time required for data transfer between the CPU and GPU during inter-operator operations. This adjustment is critical, given that data transfer speed between the GPU and CPU remains persistent bottleneck mobile device performance.
operations. Additionally, we store the whole intermediate
3.4
Comprestoon-ompilation co-design in VIT
We evaluate the speed of pruned models by utilizing compiler code generation and on-device latency measurement before performing time-consuming pruning process. The compiler does not need absolute weight values for code generation and latency measurement, making the process more streamlined. Code generation with the compiler is much faster than Deep Neural Network (DNN) training. Therefore, before actually pruning the models, we use the compiler to evaluate them with different pruning ratios, using synthesized weights instead of real ones. This approach helps us establish predictive curve that shows how pruning ratios correlate with expected latency. Such strategy is crucial in determining the optimal pruning ratio that balances model performance and computational efficiency. It allows for more effective decision pruning before undertaking the computationally intensive DNN training process.
Evaluation
In this section, we evaluate the performance of our compiler-asisted framework with our ECP- -VT model deployed on mobile devices. We use CP-Level and core ratio interchangeably the following section.
7

---

4.1 Setting
Models and datasets. The ECP-ViT's implemented based on the ViT architecture [17 and evaluated on 4 different datasets, STL-10 12. CIFAR-100 (27]. TinyImageN [29). and ImageNet- K [I5] TinyImageNet is a subset of ImageNet-Ik Ik containing 100,000 images distributed across 200 classes. The parameters of ECP-ViT were initialized and fine-tuned from ViT-B/16 trained on ImageNet-2IIK (28 for TinyImageNet and ImageNet, and used parameters from pre- pre-trained ViT-S/16 for STL-10 and CIFAR-100. We trained the ECP-VIT for 50 epochs with batch size 256 for STL-10, C00,R-100, and 128 for TinyImageNe and ImageNet- IK, K, and used Adam optimizer and cosine learning rate schedule 33 with initial learning rate of 5c-4 and minimum of 1c-7.
Evaluation environment. We compare the latency of ECP-ViT on off-the-shelf mobile devices against three state-of-the-art deep learning frameworks: Alibaba MNN (26), Tencent TNN, and Apache TVM
We do not include emerging DNN inference frameworks like TFLite 2] and PyTorch Mobile [25] because they do not support ViT on mobile GPUs yet due to unsupported operators insufficient resources (e.g., memory capacity). Our evaluation focuses on GPU instead of CPU or NPU for two reasons. Firstly, compared mobile CPUs, mobile GPUs offer higher computational capacity with better power efficiency. Secondly, compared to mobile NPUs, the NPUs backend is often invoked via a system call provided by the Android Runtime System or specific hardware vendors which does not provide an interface for independent developers to support or optimize certain operators yet. We leave this as a future research direction.
The evaluations are conducted on Oneplus 11 cell phone, which features high-end Qualcomm Kryo octa-core CPU and a Qualcomm Adreno 740 GPU with 16 GB of unified memory. To demonstrate the portability of our methods, we also present results from testing on a low-end cell phone Xiaomi 6 with limited memory and com- putation capacity, as shown in Figure 7 Xiaomi 6 is equipped with an ARM Octa-core CPU, an Adreno 540 GPU, and 6 GB of unified memory. We use 16-bit floating point precision across all frameworks and models on the mobile GPU. All latency data is collected from running the tests 50 times. However, since the variance is small, we only report averages.
MNN
Curs
Speedup
4-
VIT-B
VITS
VIT-I
ECP-B
ECP-S ECP-T
Figure 7: Latency Speedup Comparison Over Low-end Device (Xiao Mi 6). B. S. T short for Base, Small, and Tiny models, respectively.
4.2 Accuracy Comparison
The accuracy comparison with other works (with pruning and without pruning) is presented in Table Notably, the accuracy of ECP-ViT is under the best ratio, showcasing its effectiivness. The core-periphery principle guided self-attention of ECP- ViT proves to be competitive with self-attention in complete graph form. This demonstrates that the message exchange of core-periphery derived from brain function networks, leverages the message communication in VIT.
We also assess the performance of the proposed ECP-VIT on ImageNet-1K K with varying core ratios, and the results are detailed in Table The baseline, denoted as vanilla ViT, featuring self-attention in complete graph form, is considered to possess core ratio of 1.0. Notably, for the ViT-base model scale, our ECP- -iT--V,eb integrating the core-periphery principle, demonstrates superior performance compared to ViT-base across spectrum of core ratios (0.7, 0.8. and 0.9). ECP-VIT surpasses the baseline by 0.73% under a ratio of 0.9, suggesting that the sparsity of self-attention could enhance performance, with the core-periphery principle guiding self-attention proving to be effective means of achieving this sparsity.
It's worth noting that our baseline is fine-tuned by us and already stands out compared to that reported in other works 65 and ECP- ViT-tinn,,,, the drop in accuracy is minimal when compared to vanilla VIT. Additionally, we conducted comparative analysis of ECP-ViT with other competitive models on TinylmageNet, as depicted in Table 4 The results highlight the advantages of ECP- VIT in terms of prediction accuracy.
[4]. Even for smaller model scales, such as ECP-ViT-smal
8

---

Table Comparisons of results on the ImageNet dataset. All reported model results are based on pre- tre-trained weights from ImageNet-21K. The accuracy, parameters, and MACs of ECP-ViT are under the best core ratio.
Model
Top-1 1 (%)
# Params.
# MACs
LTMP
S
75.4
5.7M
1.5G
PVT-Tiny 50
75.1
13.2M
1.97G
SCP-ViT-Tiny
75.8
5.83M
1.1G
PVT-Smal 550
79.8
24.5M
3.89G
ViT-S 17
81.1
22M
4.62G
T2T-ViT,-
65
80.7
22M
4.8G
ECP-ViT-SmalI
80.9
21.7M
4.3G
ViT-Base/16 17
83.9
86.6M
17.6G
PVT-Large (50
83.8
82.0M
11.84G
TNT-B 20
84.1
66.0M
14.16G
DeiT-Base/16 16
45
84.2
86.6M
17.76G
ECP-ViT-Base
84.6
86.5M
16.96G
Table 3: Performance evaluation of ECP-ViT on ImageNet under different core ratios. We fine-tune the VCP-ViT using the pre-trained weights on ImageNet-21K. Top-1 accuracy reported and shown in percentage. T, S. and B represents ECP-ViT-T. ECP-ViT-S, ECP-ViT-B, respectively.
Ratio
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0 (base)
T/16
65.50
69.14
69.39
69.75
71.38
73.13
74.55
75.48
75.84
75.90
S/16
73.24
74.42
74.98
76.32
77.82
78.78
79.82
80.34
80.88
81.12
B/16
76.15
77.98
79.40
80.51
81.80
83.22
84.03
84.51
84.62
83.89
Table 4: Comparison between ECP-VIT with other ViT variants on TinylmageNet. Top-1 Accuracy shown in percentage. The best result of ECP- ViT under different core ratios is selected. Model
Topl Acc.
Params.
Image Res.
ViT-B/I6 17
89.16
86.55M
224 x 224
DeiT-B/16 46
87.29
87.34M
224 x 224
BeiT-B/16
4
88.64
86.53M
224 x 224
ConViT-
14
90.52
86.54M
224 x 224
ECP-ViT-B/IE
90.55
86.50M
224 224
Table 5: Comparisons between ECP- ViT and vanilla ViT on STL10 and CIFAR100. The performance
evaluation under the best core ratio. Top-1 accuracy is reported and shown in percentage.
Model
STL-I0
CIFAR-100
ViT-S/16 [17]
96.36
89.51
VIT-VIT
98.18 (+1.82)
89.56 (+0.05)
Furthermore, we extend our evaluation to STL10 and CIFAR-100, with detailed results provided in Table
4.3 End-to-end Latency and Memory Comparison
Table compares peak memory, latency, and cache miss rates between ViT-Base and ECP-ViT. Table presents a comparison of latency among VIP-VIT, MNN, TNN, and TVM for vanilla ViT models. As the models retain their dense structure after CP pruning, they can still be executed on other frameworks. '_' means the model implementation or limited memory/computation resources. As shown in Table 7 compared to other -ftte-of-the-art frameworks, ECP-VIT achieves an average speedup ranging from 4.8x to 5.3x for vanilla ViTs. This is because our compiler optimizations, such as data transformation elimination and operator fusion, help significantly reduce memory pressure and bandwidth demands. Moreover, our compiler optimizations for ECP-VITs leverage the decreased computational complexity and yield speedup ranging from 10.6x 16. compared to vanilla ViT within our framework. Additionally, ECP-ViT VIT outperforms other frameworks with speedup ranging from 4.6x 26.9x, TVM exhibits
supported on the framework. due lack operator

---

even higher latency for ECP-ViT due to the absence of systematic optimizations. The extra data transformation resulting from CP pruning leads heightened demands on memory bandwidth. Our framework achieves real-time performance for small and tiny variants of ECP-ViT, meeting the requirement for real-time execution 30 frames per second.
Table 6: Comparison of Peak Memory, Latency, and Miss Rates for ViT-Base and ECP-ViT.
Cache Miss Rate (%)
Model
Peak Memory (MB)
Latency (ms)
L1
L2
L3
ViT-Base 48
454
421.25
0.77
5.94
15.12
RCP-ViT
403
99.84
0.66
5.38
15.05
Table 7: Latency comparison of 4 end-to-end frameworks on vanilla ViTs and CP-enabled ViT's using
the GPU on Oneplus 11. We CP level of 80% for all 3 variants (base, small and tiny). '-' means the models is not supported on the framework.
Model
TNN (ms)
TVM (ms)
MNN (ms)
Ours (ms)
Speedup (avg)
VIT-tiny
300.0
54.7
115.3
17.6
4.8
VIT-small
176.2
191.9
37.7
4.9
VIT-base
780.3
610.3
131.9
5.3
ECP-VTT-tiny
380.1
110.1
15.2
16.1
CCP-VVT-smmal
837.6
157.9
31.1
16.0
ECP-VIT-base
2033
563.2
122.8
10.6
Conclusion
This paper introduces ECP-VIT, framework that enhances the deployment of Vision Transformer (ViT) models on mobile devices for real-time Al applications. Our approach includes a hardware- friendly pruning method inspired by the brain network and set of compiler optimizations eliminate data transformation bottlenecks. The results show that ECP-ViT not only reduces computational size but also improves prediction accuracy. It achieves up 26.9x speedup compared other state-of-the-art frameworks, enabling real-time performance on off-the-shelf mobile devices.
Impact Statement
This paper aims advance the real-time implementation of brain-inspired on mobile devices. The integration of the brain-inspired core-periphery principle contributes to reducing the computation budget and enhancing prediction accuracy. Additionally, the optimization of hard-level data layout could significantly improve inference speed on mobile devices. This work has the potential to boost the development of brain- insi-inspired AI on mobile devices.
Acknowledgment
We would like express gratitude all those who contributed this work, with special thanks to the constructive comments from the anonymous reviewers. This work was supported in part by the National Science Foundation (NSF) under the awards of CCF-2428108. OAC-2403090. Any emoos and opinions are not those of the NSF and are attributable solely to the author(s).
10

---

References
[1] Snapdragon
gen
https://www.toocpp.. /n/cpu/
qualcomm snapdragon 8- gen-2
Accessed: Jan-26-2024
[2 Tensorflow lite. http
Accessed: Jan-26-2024.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint (et1:::40904433. 2014.
[4] Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. arXiv preprint arXiv:2106. 08254,, 2021.
[5] Danielle S Bassett, Nicholas F Wymbs, M Puck Rombach, Mason A Porter, Peter J Mucha, and Scott T Grafton. Task-based core-periphery organization of human brain dynamics. PLoS computational biology, 99))ee003171 2013..
[6] Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The documettu transformer, 2020.
[7] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. ArXiv, abs/2306. 12929, 2023.
[8] Maxim Bonnaerens and Joni Dambre. Learned thresholds token merging and pruning for vision transformers. arXiv preprint arXiv:2307. 10780, 2023.
[9] Erik Cambria and Bebo White. Jumping nlp curves: A review of natural language processing research. IEEE Computational intelligence magazine, 9(2):48-33 57, 2014.
[10] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34: 34:19974-19988 2021.
[11] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. (TVM}: An automated (End-to-End) optimizing compiler for deep learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 578-594, 2018.
[12] Adam Coates, Honglak Lee, and Andrew Y. Ng. An analysis of single layer networks in unsupervised feature learning. AISTATS, 2011.
[13] Arzu Çöltekin, Ian Lochhead, Marguerite Madden, Sidonie Christophe, Alexandre Devaux, Christopher Pettit, Oliver Lock, Shashwat Shukla, Lukáš Herman, Zdeněk Stachoñ, et al. Extended reality spatial sciences: A review of research challenges and future directions. ISPRS International Journal of Geo-Information, 9(7):439, 2020.
[14] Stéphane d' Ascooli Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103. 10697, 2021.
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. Ieee, 2009.
[16] Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.00 04.05 2018.
[17) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[18] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 1997)1197,, 2019.

---

[19] Shi Gu, Cedric Huchuan Xia, Rastko Ciric, Tyler M Moore, Ruben C Gur, Raquel E Gur, Theodore Satterthwaite, and Danielle S Bassett. Unifying the notions of modularity and core- periphery structure in functional brain networks during youth. Cerebral Cortex, (083):1087- 1 1102, 2020.
[20] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 34: 14:15908-15: 15919, 2021.
[21] Seungycop Han, Haichen Shen, Matthai Philipose, Sharad Agarwal, Alec Wolman, and Arvind Krishnamurthy. Mcdnn: An approximation- cessing under resource constraints. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys), pages 123- 233-136.4 ACM, 2016.
execution framework for deep stream pro-
[22] Zhe Huang, Xiaowei Yu, Dajiang Zhu, and Michael C. Hughes. Interlude: Interactions between labeled and unlabeled data to enhance semi-superrised learning, International Conference on Machine Learning, 2024.
[23] Rasheed Hussain and Sherali Zeadally. Autonomous cars: Research results, issues, and future challenges. IEEE Communications Surveys & Tutorials, 21(2):1275-1313, 2018.
[24] Loc N Huynh, Youngki Lee, and Rajesh Krishna Balan. Deepmon: Mobile gpu-based deep learning framework for continuous vision applications. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services, pages 82-95, 2017.
[25] Sagar Imambi, Kolla Bhanu Prakash, and GR Kanagaccidaambaresan. Pytorch. Programming with TensorFlow: Solution for Edge Computing Applications, pages 87-104, 2021.
[26] Xiaotang Jiang. Huan Wang, Yiliu Chen, Ziqi Wu, Lichuan Wang, Bin Zou, Yafeng Yang, Zongyang Cui, Yu Cai, Tianhang Yu, Chengfei Lv, and Zhihua Wu. Mnn: A universal and efficient inference engine. In MLSys, 2020.
[27] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[28] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84 90, 2017.
[29] Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
[30] Royson Lee, Stylianos Venieris, Lukasz Dudziak, Sourav Bhattacharya, and Nicholas D Lane. Mobisr: Efficient on-device super-resolution through heterogeneous mobile processors. In The 25th annual international conference on mobile computing and networking, pages 1-16, 2019.
[31] Zhengang Li, Geng Yuan, Wei Niu, Pu Zhao, Yanyu Li, Yuxuan Cai, Xuan Shen, Zheng Zhan, Zhenglun Kong, Qing Jin, et al. Npas: A compiler-aware framework of unified network pruning and architecture search for beyond real-time mobile acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14255-14266. 2021.
[32] Miaomiao Liu, Xuming He, and Mathieu Salzmann. Geometry-aware deep network for single- image novel view synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4616-4622 2018
[33] Yichao Liu, Zongru Shao, and Nico Hoffmann. Global attention mechanism: Retain information enhance channel-spatial interactions. arXiv preprint arXiv:2/ /2.05561, 2021.
[34] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[35] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
[36] Larry R Medsker and ICC Jain. Recurrent neural networks. Design and Applications, 5(64-67):2, 2001.
12

---

[37] Wei Niu, Jiexiong Guan, Yanzhi Wang, Gagan Agrawal, and Bin Ren. Dnnfusion: Accelerating deep neural networks execution with advanced operator fusion. PLDI 2021, page 883-898, New York, NY, USA, 2021. Association for Computing Machinery.
[38] Wei Niu, Xiaolong Ma, Sheng Lin, Shihao Wang, Xuehai Qian, Xue Lin, Yanzhi Wang, and Bin Ren. Patdnn: Achieving real-time dnn execution on mobile devices with pattern-basse weight pruning. In ASPLOS 2020, pages 907-922, 2020.
[39] Jack Rae and Ali Razavi. Do transformers need deep long-range memory? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. Association for Computational Linguistics.
[40] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang. Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR), 54(4):1-34, 2021.
[41] Aurko Roy*, Mohammad Taghi Saffar*, David Grangier, and Ashish Vaswani. Efficient content-based sparse attention with routing transformers, 2020.
[42] Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishray Chaudhary, Xia Song, and Furu Wei. A length-extraapolatbbe transformer. 2022.
[43] Yehui Tang, Kai Han, Yunhe Wang. Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12165-12174, 2022.
[44] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp- mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems, 34:24261-24272, 2021.
[45] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[46] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357, 2021.
[47] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In In Computer Vision-ECCV 2022: 17th European Conference, pages 459-479, 2022.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[49] Huiyu Wang. Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In European conference on computer vision, pages 108-126. Springer, 2020.
[50] Wenhai Wang, Enze Xie, Xiang Li. Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEFE/CVF international conference on computer vision, pages 568-578, 2021.
[51] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowic7, al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, 2020.
[52] Jianxin Wu. Introduction to convolutional neural networks. National Key Lab for Novel Software Technology. Nanjing University. China, 5(25) 2017.
13

---

[53] Zhenxiang Xiao, Yuzhong Chen, Junjie Yao, Lu Zhang, Zhengliang Liu, Zihao Wu, Xiaowei Yu. Yi Pan, Lin Zhao, Chong Ma, Xinyu Liu, Wei Liu, Xiang Li, Yixuan Yuan, Dinggang Shen, Dajiang Zhu, Dezhong Yao, Tianming Liu, and Xi Jiang. Instruction-vie Multi-modal prompts for instruction learning in vision transforme. Information Fusion,
2024.
[54] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li. Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision trans- former. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2022.
[55] Shuochao Yao. Shaohan Hu, Yiran Zhao, Aston Zhang, and Tarek Abdelzaher. Deepsense: A unified deep learning framework for time-series mobile sensing data processing. In Proceedings of the 26th International Conference World Wide Web, WWW '17, page 351-360, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee.
[56] Hao Yu and Jianxin Wu. A unified pruning framework for vision transformers. arXiv preprint arXiv:21 15127, 2021.
[57] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10819~10829, 2022.
[58] Xiaowei Yu, Zhang Lu, Haixing Dai, Lyu Yanjun, Lin Zhao, Tianming Liu, and Dajiang Zhu Core-periphery principle guided redesign of self-attention in transformers. arXiv preprint arXiv:2303. 15569, 2023.
[59] Xiaowei Yu, Zhang Lu, Tianming Liu, and Dajiang Zhu. Robust core- pore-ppriphery constrained transformer for domain adaptation. arXiv preprint arXiv:2308. /3515, 2023.
[60] Xiaowei Yu, Lu Zhang, Chao Cao, Tong Chen, Yanjun Lyu, Jing Zhang, Tianming Liu, and Dajiang Zhu. Gyri vs. sulci: Disentangling brain core- periphery functional networks via twin- transformer. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, 2024.
[61] Xiaowei Yu, Lu Zhang, Zihao Wu, and Dajiang Zhu. Core-periphery multi-modality feature alignment for zero-shot medical image analysis. IEEE Transactions on Medical Imaging, 2024.
[62] Xiaowei Yu, Lu Zhang, Zihao Wu, and Dajiang Zhu. Cp-clip: Core-periphery feature alignment clip for zero-shot medical image analysis. In International Conference on Medical Image Computing and Computer-Assiste Intervention, 2024.
[63] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEFF/CVF international conference computer vision, pages 558 567, 2021.
[64] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:1 13:17283-17297, 2020.
[65] Chuanyang Zheng, Kai Zhang, Zhi Yang, Wenming Tan, Jun Xiao, Ye Ren, and Shiliang Pu. Savit: Structure- aware vision transformer pruming via collaborative optimization. In Advances in Neural Information Processing Systems, pages 9010-9025. 2022.
[66] Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning. arXiv preprint arXiv:2104.08500, 2021.
[67] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint 111.0:16., 2016
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Main claims and brief results are in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
Justification:
Due to space space limit, we only have a little discussion in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
15

---

Justification:
We do not have any theory to prove.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provided all the essential parts
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[No]
Justification:
Because of copyright issues, we cannot make the framework public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
coioooooooycyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuii wwwpc//ggue////eeeeceooc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and test details are in 2 and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specified all the information in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htttps://eenn http:://euriss...oo https//ccccpp..cceticccccooeeeee
Answer: [Yes]
Justification:
We have read and confirmed to conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: put this section in Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper has no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all the assets and works as required.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.rrewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
19

---

Answer:
[NA]
Justification: No new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "rdDhXMmZd3", "reviewer": "Reviewer_gvXM", "summary": "This work proposes a ViT accelerating framework, ECP-ViT, to deploy ViT models on smartphones. This framework consists of two parts: 1) Core-Periphery Guided Self-Attention (reducing the computational and bandwidth cost of ViT) and 2) Data Layout Selection based on compiler optimizations (removing the time-consuming data transformation). Specifically, Core-Periphery Guided Self-Attention partitions the tokens in K, Q, and V matrices into core and periphery components. Tokens in the core component exchanged messages with all tokens, and tokens in the periphery component only exchanged messages with tokens in the core component. To fully eliminate the data transformation operations, this work attempts to find a common data layout that works efficiently for both contiguous operators. But the search space is large. They propose the dimension reduction heuristic algorithm to shrink the search space of the data layouts. ECP-ViT obtains competitive results on STL-10, CIFAR100, TinyImageNet, and ImageNet, achieving lower smartphone latency.", "questions": "1. How do you partition the KQV matrices into core and peripheral components? Are there any partitioning criteria? And why do you use this partitioning method?\n2. MobileViT [1] is a lightweight, low-latency network for mobile vision tasks. What significant advantages does ECP-ViT offer over MobileViT that we should consider?\n\n[1] Mehta, Sachin, and Mohammad Rastegari. \"Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer.\" arXiv preprint arXiv:2110.02178 (2021).", "limitations": "Not applicable.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The Core-Periphery Guided Self-Attention, which is inspired by Brain Neural Networks, is interesting. \n- The Data Layout Selection based on compiler optimizations can effectively speed up ViTs on Smartphones, as demonstrated by comprehensive experimental validation.", "weaknesses": "- Some details need to be elaborated on. The description of the dimension reduction heuristic needs to be more detailed, and it is encouraged to provide its pseudocode. \n- There seems to be a discrepancy in Equation 1. The conventional form of self-attention is softmax(qV)K, while in ECP-ViT, it appears to be softmax(qVK). This discrepancy should be carefully reviewed and corrected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 Explain dimension reduction heuristic\n\n**Response:**\n\nThanks for pointing this out. Below is the detailed pseudo code for the heuristic. The score is collected by running mini benchmarks, and the different layouts are defined by the frameworks. The process begins by identifying key nodes in the computational graph that perform actual computation (Step 1). For each key node, possible data layouts are determined (Step 2), taking into account the various layout options provided by the frameworks. Each possible layout is then evaluated by calculating a score based on data locality and GPU utilization using mini benchmarks (Step 2.1). The layout with the best score is selected and assigned to the key node (Step 3). Finally, the computational graph is updated with the new layouts for all key nodes to ensure that the entire graph benefits from the optimized layouts, improving overall performance (Step 4).\n\n\n### Algorithm 1: Dimension Reduction Heuristic Algorithm\n\n**Require:** Computational graph G\n\n**Ensure:** Optimized computational graph G'\n\n**Step 1: Find all the key nodes**\n- Identify key nodes K ← identify_key_nodes(G)\n- For each node n ∈ K do:\n    - **Step 2: Determine layouts for key nodes by Algo 3**\n    - P ← determine_possible_layouts(n)\n    - best_layout ← None\n    - best_score ← ∞\n    - For each layout l ∈ P do:\n        - **Step 2.1: Calculate scores by mini-benchmarks**\n        - score ← calculate_data_locality_score(l, n) + calculate_gpu_utilization_score(l, n)\n        - If score < best_score then:\n            - best_score ← score\n            - best_layout ← l\n    - End for\n    - **Step 3: Assign best layout for the key node**\n    - n.data_layout ← best_layout\n- End for\n- For each node n ∈ K do:\n    - **Step 4: Infer layout for all nodes**\n    - G.update_node_layout(n)\n- End for\n- Return G\n\n\n### Q2  A discrepancy in Equation 1. \n\n**Response:**\n\nThank you for pointing this out. We will correct it in Eq.1 and the corresponding figures. As stated in Equation 2, our ECP-ViT follows the conventional form of self-attention, which is softmax(QK)V.\n\n\n### Q3  How to determine the core and peripheral components.\n\n**Response:**\n\nIn our ECP-ViT, we consider the image patches as nodes and predefine a series of core ratios to divide nodes into cores and peripheries. During training, we employ Grad-CAM to identify important regions of the images and assign the core nodes to those regions. Accordingly, the QKV matrices of these patches are divided into core and peripheral components.\nFor example, for images with a resolution of 224x224 and a patch size of 16x16, there are a total of 196 patch tokens as nodes. For a core ratio of 10%, around 20 patch tokens are considered as cores, and we choose the top 20 important regions as cores.\nThis partitioning method is inspired by human brain networks [1], where different networks exhibit different core ratios. \n[1] \"Gyri vs. sulci: Disentangling brain core-periphery functional networks via twin-transformer,\" MICCAI 2024.\n\n\n\n\n### Q4 Comparison with MobileViT\n\n**Response:**\n\nThank you for your question regarding the advantages of ECP-ViT over MobileViT. While MobileViT is a lightweight, low-latency network designed for mobile vision tasks, it achieves a top-1 accuracy of 78.4% on ImageNet. In contrast, our ECP-ViT achieves a significantly higher top-1 accuracy of 84.6% on the same dataset, demonstrating superior performance.\nAdditionally, ECP-ViT employs a pruning method applied to the self-attention layer, which is an orthogonal approach to MobileViT’s design. Importantly, after pruning, each operation in ECP-ViT remains dense, making them friendly for execution on mobile GPUs. Our framework further eliminates all implicit and explicit data transformations caused by pruning and model design, which significantly reduces latency. This makes ECP-ViT highly efficient and suitable for deployment on mobile devices, offering substantial improvements in both performance and efficiency for self-attention-based transformer models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5jTzemMQBl", "reviewer": "Reviewer_r1Lw", "summary": "This paper introduces ECP-ViT, a framework designed to improve the performance of vision transformers (ViTs) on mobile devices. The authors observed the intensive and irregular memory access involved in the data transformation for self-attention layers, which significantly slows down transformers compared to traditional CNNs. To address this, they propose a hardware-friendly self-attention pruning technique motivated by the core-periphery structure in brain networks, thereby reducing the computational and memory access burdens. ECP-ViT also incorporates compiler optimizations to fuse and eliminate transformation operators to further boost the performance. These combined optimizations enable the work to speedup in ViT inference on mobile GPUs by 4.6x to 26.9x without sacrificing the inference accuracy.", "questions": "1. Can you provide a detailed comparison of memory behavior before and after applying the ECP technique?\n2. Can you show the power/energy improvement achieved by the proposed optimizations?", "limitations": "The authors have mentioned the plan to evaluate other model architectures in future work, which addresses some limitations. However, it would be beneficial to include additional evaluations such as memory studies, power/energy consumption analysis, and resource utilization metrics.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The problem is well-motivated with clear benchmarks in terms of accuracy, MACs, and latency comparisons across various ViTs.\n+ The core-periphery concept borrowed from brain networks for self-attention is interesting.\n+ The proposed compiler optimization effectively eliminates unnecessary data transformation operators, leading to significant performance improvements.\n+ The evaluation is conducted on real-world mobile phones, and includes both high-end and low-end devices.", "weaknesses": "- The evaluation section could be improved:\n    - It would be beneficial to discuss the memory access pattern differences before and after applying ECP to showcase its effectiveness in improving memory efficiency.\n    - The power and energy consumption evaluation should also be included, given the limited battery capacity of mobile devices.\n- There are several presentation issues; for example, the space between Table 3 and Table 4 is too dense, and the font size seems to change abruptly starting from the “Evaluation environment” in Section 4. Please ensure consistent font sizes throughout the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1  Need to provide a detailed comparison of memory behavior before and after applying the ECP technique.\n\n**Response:**\n\nThank you for your question regarding the memory access pattern differences before and after applying ECP-ViT. Our pruning technique ensures that computations remain dense, maintaining a memory access pattern similar to the original model. This consistency in memory access patterns, combined with fewer computations, results in faster performance.\n(We will add a fig to illustrate the memory access in the revision) If we just use naive fusion, the access pattern of the tensors between operators will be strided. This strided access pattern is determined by the data transforming operators like slice and transpose. By using our smart layout design, we can map the indices of tensors as desired, and the access pattern will be continuous. Compared to the strided access pattern, our access pattern can utilize the data locality and reduce the cache miss, which shows tremendous reduction in latency. \n\n| Model     | Peak Memory w/ ECP (MB) | Peak Memory w/o ECP (MB) | Latency w/ ECP (ms) | Latency w/o ECP (ms) |\n|-----------|-------------------------|--------------------------|---------------------|----------------------|\n| ViT-Base  | 403                     | 454                      | 99.84               | 421.25               |\n\n*Device – Oneplus 12 Snapdragon 8 Gen 3 SoC\n\n*Results are averaged by running 10 rounds and 5 warm-ups \n\n| Model     | L1 Cache Miss Rate | L2 Cache Miss Rate | L3 Cache Miss Rate |\n|-----------|--------------------|--------------------|--------------------|\n| ViT       | 0.77%              | 5.94%              | 15.12%             |\n| ECP-ViT   | 0.66%              | 5.38%              | 15.05%             |\n\nAfter ECP pruning, each part of our computation is still dense. Combining with our layout selections, we are able to increase data locality and reduce cache misses in all levels.\n\n\n\n\n\n\n### Q2 Need to show the power/energy improvement achieved by the proposed optimizations\n\n**Response:**\n\nTesting device: OnePlus 12, Snapdragon 8 Gen 3 SoC\n\nAll results are collected by running 10 rounds and pick the peak power usage\n\n| Model            | Power w/o ECP | Power w/ ECP |\n|------------------|---------------|--------------|\n| ViT-Base         | 2.46 W        | 1.05 W       |\n| Swin-Tiny        | 0.87 W        | 0.78 W       |\n| MetaFormer-Base  | 2.74 W        | 1.31 W       |\n\n\n### Q3 Table 3, 4 and section 4 font issues\n\n**Response:**\n\nThanks for the suggestions. We will revise it in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mNhhv6heHB", "reviewer": "Reviewer_MoNf", "summary": "This paper presents ECP-ViT, a real-time framework for deploying Vision Transformers (ViTs) on mobile devices. Inspired by the brain's core-periphery principle, this method guides self-attention in ViTs to reduce computational demands and eliminate data transformation operations. ECP-ViT integrates algorithm-system co-optimizations, achieving a speedup of 4.6× to 26.9× on mobile GPUs across various datasets while maintaining high accuracy.", "questions": "1. In line 67 of the introduction, it mentions support for a pruning scheme, which is confusing because there is no prior content about pruning before this line.\n\n2. In the background section, it states that the token pruning method introduces additional Reshape and Transpose operations to the feature map, leading to reduced benefits from reduced computation costs. However, in many token pruning methods [1] [2], pruning is applied to certain specific layers, so the overhead of Transpose and Reshape is not that large. Moreover, token pruning quadratically reduces computation cost, which is highly effective.\n\n3. In Section 3.1, it says it classifies data transformation into two categories, Explicit and Implicit, and Explicit transformations are denoted as red color operators in round boxes. But there are several colors in Figure 2, and some colors are quite similar, making it hard to distinguish between Explicit and other transformations. Also, it seems that there is no indication of the implicit transformations in the figure.\n\n4. It says the CP-guided self-attention design is shown in Figure 4. But Figure 4 seems to depict the ratio of core nodes and the number of edges. This is confusing as it does not illustrate any design component or workflow.\n\n5. The rationale behind the CP-guided self-attention design is not detailed enough. CP-guided self-attention is an efficient self-attention mechanism that reduces the number of self-attention operations, but why is it better than other efficient self-attention mechanisms such as Swin-Transformer and Criss-Cross Attention? Why do we adopt this CP-guided self-attention? How do you ensure that the information exchange through only core nodes is sufficient?  \n&nbsp;\n\n[1] Liang, Youwei, et al. \"Not all patches are what you need: Expediting vision transformers via token reorganizations.\" arXiv preprint arXiv:2202.07800 (2022).\n\n[2] Kong, Zhenglun, et al. \"Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 7. 2023.", "limitations": "Please see my comments.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The motivation data in Table 1 is interesting. \n\n2. This paper targets an interesting problem: eliminating the expensive transform operation in ViTs.", "weaknesses": "1. The paper writing can be improved.\n\n2. Some figures and their elaborations are not clear enough.\n\n3. The rationale behind the design is not well-explained.\n\nPlease see my comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 Confusions in line 67 of the introduction.\n\n**Response:**\n\nThank you for the suggestion. We will revise the introduction to provide a brief context about pruning before mentioning our support for it. Specifically, we will add a paragraph to explain the concept and relevance of pruning in the context of our work.\n\n\n### Q2  Token pruning methods add some overhead with Reshape and Transpose operations but are still highly effective in reducing computation costs by being applied to specific layers.\n\n[1] Liang, Youwei, et al. \"Not all patches are what you need: Expediting vision transformers via token reorganizations.\" arXiv preprint arXiv:2202.07800 (2022).\n\n[2] Kong, Zhenglun, et al. \"Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 7. 2023.\n\n**Response:**\n\nThank you for your feedback regarding the token pruning methods. Token pruning methods [1][2] can result in sparse data patterns that implicitly require more data transformations. Implicit data transformations occur when the underlying data layout needs to be reorganized to optimize performance for specific operations, leading to irregular data access patterns and increased memory bandwidth demands. This can be particularly unfriendly for mobile devices, which have limited memory bandwidth.\nIn ECP-ViT, we address this issue by grouping important patches together and pruning at the node level. This approach ensures that even though the model is pruned, each sub-matrix remains dense, allowing for more efficient computation. By maintaining dense computation patterns, ECP-ViT minimizes the need for implicit data transformations, thereby enhancing performance and efficiency on mobile devices. This makes ECP-ViT more suitable for mobile deployment compared to methods that result in sparse data patterns and increased implicit data transformations.\n\n\n\n\n\n### Q3  In Section 3.1, it states that data transformations are classified into Explicit (red, round boxes) and Implicit categories, but Figure 2 uses several similar colors making it difficult to distinguish Explicit transformations, and lacks indication of Implicit transformations.\n\n**Response:**\n\nThank you for your observation regarding the classification of data transformations in Section 3.1 and their representation in Figure 2. We will update the hardware design in the revision. In the updated figure, every computational operator will have its preferred data layout, and between each computational operator, there will potentially be an implicit reshape or transpose. These implicit reshapes occur because different computational operators often require their input data in specific formats or layouts for optimal performance. When the output layout of one operator does not match the required input layout of the next, an automatic reshape or transpose is needed to ensure compatibility and efficiency. In our framework, by using smart layouts, we will eliminate the implicit reshapes. Additionally, we will ensure that the colors used to denote different types of operations are distinct and easily distinguishable.\n\n\n### Q4 Confusions in Figure 4 (CP-Guided self-attention)\n\n**Response**\n\nThanks for pointing this out. We will revise the sentence to clarify that Figure 4 shows examples of core-periphery graphs with different core ratios. Specifically, Figure 4 illustrates the selection of various CP graphs referenced in Figure 2(a). The workflow involves CP graph generation, CP-guided self-attention, and CP-guided QKV multiplication, corresponding to Figure 2(a), Figure 2(b1), and Figure 2(b2).\n\n\n### Q5 More details are needed for CP-guided self-attention design. Why is it better than Swin and Criss-Cross Attention? Only core nodes are sufficient?\n\n\n**Response:**\n\nThank you for your question regarding the rationale behind the CP-guided self-attention design and its advantages over other efficient self-attention mechanisms such as Swin-Transformer and Criss-Cross Attention.\nFrom a hardware perspective, CP-guided self-attention can group core patches together and periphery patches together, which increases data locality. This grouping improves the efficiency of data access and reduces the need for frequent data movement across memory, thereby enhancing computational efficiency. While this design involves more implicit and explicit reshape and transpose operations, our framework successfully eliminates both, making it highly suitable for CP-guided self-attention.\nIn contrast, mechanisms like Swin-Transformer and Criss-Cross Attention do not inherently focus on optimizing data locality and may still involve significant overhead from data transformations. By reducing these overheads, CP-guided self-attention becomes more efficient and effective, particularly on hardware with limited memory bandwidth like mobile devices. Additionally, the core-periphery principle ensures that information exchange through core nodes is sufficient by maintaining connections among core nodes while selectively connecting periphery nodes, balancing comprehensive context capture with reduced computational complexity. This makes CP-guided self-attention a better fit for our optimization framework compared to other mechanisms.\nFurthermore, our method is actually a pruning mechanism that can also be applied to Swim-Transformer or Criss-Cross Attention. This flexibility allows for the benefits of our approach to be realized in various self-attention-based transformer models, enhancing their efficiency and performance across different applications and hardware configurations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aw4lgw5Qrr", "reviewer": "Reviewer_PnBL", "summary": "This paper introduces ECP-ViT, a framework that accelerates Vision Transformers on mobile devices using a core-peripheral guided self-attention mechanism. This approach reduces computational demands and achieves up to 16.1x speedup on a OnePlus 11 GPU, enabling efficient real-time deployment of ViT models while maintaining high accuracy.", "questions": "How does the core rate be selected for different devices and tasks? Would making it a sample-aware parameter further enhance the results?", "limitations": "Please refer to the weakness.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The compiler and model architecture were co-optimized, significantly improving computational efficiency on actual devices.\n- Based on the Brain Neural Networks, this paper cleverly distinguishes between important and less important parts of the attention mechanism. \n- The slice and transpose reshape operations are eliminated through the hardware-level design.", "weaknesses": "- Overall, I think this paper is above the acceptance bar. However, the comparison (or discussion) between ECP-ViT and other lightweight ViT methods is not comprehensive enough, for example, NAS for lightweight ViT [1][2], and advanced token optimization method [3]. \n- Showing the results on ImageNet seems like a reasonable choice, but insufficient to show that the method can be generalized to other domains, such as detection/segmentation. One possible way to do this is to directly transfer the weights for another downstream task and provide some comparison. By the way, the performance improvements in Tab.11 seem a bit marginal compared to the main results. \n\n[1] Elasticvit: Conflict-aware supernet training for deploying fast vision transformer on diverse mobile devices, ICCV 2023\n\n[2] Nasvit: Neural architecture search for efficient vision transformers with gradient conflict-aware supernet training, ICLR 2022 \n\n[3] Diffrate: Differentiable compression rate for efficient vision transformers, ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 More comparison between other lightweight ViT methods.\n\n[1] Elasticvit: Conflict-aware supernet training for deploying fast vision transformer on diverse mobile devices, ICCV 2023\n[2] Nasvit: Neural architecture search for efficient vision transformers with gradient conflict-aware supernet training, ICLR 2022\n[3] Diffrate: Differentiable compression rate for efficient vision transformers, ICCV 2023\n\n**Response:**\n| Model         | Top1 Acc on ImageNet-1k | Computation size (GFLOPs) |\n|---------------|-------------------------|--------|\n| ECP-ViT       | 84.6                    | 10.1   |\n| ElasticViT-L3 | 80.0                    | 0.86   |\n| NasViT-A5     | 81.8                    | 0.76   |\n| DiffRate-Base | 81.5                    | 11.5   |\n\nAs shown in the table, ECP-ViT achieves the highest top-1 accuracy on ImageNet-1k at 84.6%, demonstrating superior performance compared to other lightweight ViT methods. We will add the comparison and the speed comparison in the revision\n\n### Q2 Show results on other domains.\n\n**Response:**\n\nThank you for the suggestion. We have further applied the core-periphery guided ViT to detection and segmentation tasks, specifically using DETR on the COCO dataset and SegFormer on the Cityscapes dataset.\n\n|                 | DETR [1] | SegFormer [2] |\n|-----------------|----------|---------------|\n| Vanilla         | 42.0     | 83.8          |\n| w/ Our Method   |    42.6      |       84.2        |\n\n[1] End-to-end object detection with transformers. In ECCV 2020.\n\n[2] SegFormer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS 2021.\nA major contribution of our work emphasizes real-time performance on mobile devices. Our method reduces parameters and latency while maintaining or even improving model performance on devices. Although the performance improvements in Table 11 may seem marginal, they demonstrate that our method achieves a balance between efficiency and effectiveness.\n\n\n### Q3 How does the core rate be selected for different devices and tasks? Would making it a sample-aware parameter further enhance the results?\n\n**Response:**\n\nThank you for your question regarding the selection of the core ratio for different devices and tasks.\nSince our core-periphery principle guided ViT is inspired by brain functional networks, where different networks responsible for various tasks (such as vision and movement) have different core ratios, we have adopted a similar approach that we experiment with different core ratios to find the optimal ratio for specific datasets or tasks. Your suggestion to make it a sample-aware parameter is indeed interesting and a great idea for further exploration.\nAccording to Figure 8 and Figure 6, we employ different core ratios to run benchmarks and obtain a performance line in terms of accuracy. We then test the latency for these different core ratios. By evaluating both accuracy and latency metrics, we can identify the optimal core ratio that balances performance and efficiency. This systematic approach ensures that the selected core ratio is well-suited for the specific device and task, providing optimal results. Additionally, making the core ratio a sample-aware parameter could potentially further enhance results by allowing dynamic adjustment based on the specific characteristics of the data, thereby improving both accuracy and efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yqPqU8sQrD", "reviewer": "Reviewer_wumZ", "summary": "• This paper proposes ECP-ViT, which optimizes the ViT model by introducing the core-periphery (CP) principle.\n• The Core-Periphery Principle Guided self-attention mechanisms successfully reduce memory bandwidth by eliminating data transformation.\n• By applying pruning and removing data transformation, the optimization, which considers both software-level design (algorithm) and hardware-level design, achieves real-time performance on a mobile GPU (Snapdragon 8 Gen 2 SoC) with an average speedup of 4.8 times.", "questions": "• (Table 1, Table 9) Are there experimental results on latency for layout transformation and computation in other frameworks such as TFLite and Pytorch-Mobile?\n    o The experimental setup in Table 9 indicates a batch size of 1 to 18. Is this an average value? Do the experimental results for latency ratios align with different batch sizes?\n• (Table 3) Are there any comparative experimental results for Pytorch-Mobile?\n• (Figure 6) Are there experimental results on the computation complexity and memory usage of the model according to the core ratio? Are there results consistent with those presented in Table 1?\n• (Table 6) Were the experiments for TNN, TVM, and MNN conducted on NPUs while those for Ours were conducted on a mobile GPU? Are there experimental results for Ours on NPUs?\n• Are there experimental results for tasks such as Object Detection or Instance Segmentation, where real-time inference is crucial?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "• The performance generality of the proposed ECP-ViT is demonstrated across various datasets (STL-10, CIFAR-100, TinyImageNet, and ImageNet) and hardware environments (OnePlus 11, Xiaomi 6).\n• The justification for the proposed Core-Periphery Guided Self-Attention is well-explained, including background information, and effectively illustrated with figures and equations.\n• ECP-ViT achieves speed improvements by completely eliminating data transformation, which impacts memory bandwidth and causes network overhead.\n• (Table 8) The compiler speed is significantly improved compared to the MNN and TVM frameworks.\n• (Table 9) The overhead of data layout transformation and computation for the mobile GPU is analyzed in detail.", "weaknesses": "• The comparative experiments lack consistency.\n• Overall, the performance improvements are marginal compared to previous research.\n• The implementation uses Fixed Point 16-bit, but most current mobile environments typically utilize 8-bit or higher quantization, and there are no experiments addressing this. Data compression through quantization is essential for mobile environments.\n• More fair comparisons could be made if experiments were conducted in specific NPU environments.\n• Although a compiler for mobile GPUs is proposed, detailed information about the compiler itself is lacking.\n    o (Figure 4) There is insufficient explanation on how the specific allocation of the core/periphery nodes is determined, and there is no criterion for specifying core nodes.\n    o The detailed operation of the proposed dimension reduction heuristic algorithm is not explained. More detailed descriptions and algorithms are needed to optimize data layout.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 (Table 1, Table 9) Are there experimental results on latency for layout transformation and computation in other frameworks such as TFLite and Pytorch-Mobile? \n\n**Response:**\n| Model                   | Implicit Transformation (ms) | Explicit Transformation (ms) | Computation (ms) | Latency (ms) |\n|-------------------------|------------------------------|------------------------------|------------------|--------------|\n| ECP-ViT (our framework) | 0                            | 0                            | 99.83            | 99.83        |\n| ViT-Base                | 198.93                       | 0.92                         | 134.52           | 334.37       |\n| DeiT-Base               | 216.26                       | 5.294                        | 136.09           | 357.64       |\n*Device – Oneplus 12 Snapdragon 8 Gen 3 SoC\n\n*Platform – TFLite\n\n### Q2 The experimental setup in Table 9 indicates a batch size of 1 to 18. Is this an average value? Do the experimental results for latency ratios align with different batch sizes?\n\n**Response:**\n\nThank you for your question regarding the experimental setup in Table 9. We actually tested all batch sizes from 1 to 18 to ensure comprehensive evaluation. However, we observed that there were no major differences in latency across these different batch sizes. As a result, for simplicity and clarity in our presentation, we reported the results for batch size 1. The latency ratios remained consistent regardless of the batch size, ensuring that our conclusions are robust across varying batch sizes.\n\n### Q3 (Table 3) Are there any comparative experimental results for Pytorch-Mobile? \n\n**Response:**\n\nThank you for your question regarding comparative experimental results for Pytorch-Mobile in Table 3. Currently, Pytorch-Mobile does not support ViT model on mobile GPU. Therefore, we use TensorFlow Lite (TFLite) for our experiments instead and conducted all tests using OpenCL on GPUs. This approach allows us to provide a fair and comprehensive comparison of performance across different frameworks that support Vision Transformer models on mobile GPUs. Please refer to Q7 for the reason of choosing GPU instead of NPU.\n\n| Model      | Framework | Latency (ms) |\n|------------|-----------|--------------|\n| ECP-ViT    | Ours      | 99.83        |\n| ViT-Base   | TFLite    | 334.37       |\n| DeiT-Base  | TFLite    | 357.64       |\n\n*Device – Oneplus 12 Snapdragon 8 Gen 3 SoC\n\n*Results are averaged by running 10 rounds and 5 warm_ups\n\n### Q4 (Figure 6) Are there experimental results on the computation complexity and memory usage of the model according to the core ratio? Are there results consistent with those presented in Table 1? \n\n**Response:**\n\n| Model      | Peak Memory In core ratio 0.6 | Peak Memory In core ratio 0.7 | Peak Memory In core ratio 0.8 | Peak Memory In core ratio 0.9 |\n|------------|-------------------------------|-------------------------------|-------------------------------|-------------------------------|\n| ECP-ViT    | 361 MB                        | 378 MB                        | 403 MB                        | 420 MB                        |\n\nThese results are consistent with those presented in Table 1. After pruning, we separate computation into core-to-core and core-to-periphery nodes. The small size of core nodes results in intermediate results being similar across different core ratios. Additionally, our layout elimination technique stores intermediate results on the GPU and reuses finished buffers, saving more memory compared to other frameworks. This efficient handling keeps memory usage stable across different core ratios, aligning with the results in Table 1 and demonstrating the effectiveness of our ECP-ViT model in managing computation complexity and memory usage.\n\n### Q5 Are there experimental results for tasks such as Object Detection or Instance Segmentation, where real-time inference is crucial?\n\n**Response:**\n|                 | DETR [1] | SegFormer [2] |\n|-----------------|----------|---------------|\n| Vanilla         | 42.0     | 83.8          |\n| w/ Our Method   |  42.6         |   84.2            |\n\n[1] End-to-end object detection with transformers. In ECCV 2020.\n\n[2] SegFormer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS 2021.\n\n### Q6 Why not use quantization?\n\n**Response:**\n\nOur paper focuses on layout elimination techniques to reduce memory burden without sacrificing accuracy. Our main contribution lies in optimizing data layout and computation patterns, eliminating data transformation overhead, and enhancing performance on mobile devices. It’s important to note that quantization and our optimization techniques are orthogonal and can be applied independently or together for cumulative benefits. We plan to extend our research to include experiments with 8-bit quantization, aligning with current mobile environment practices.\n\n### Q7  Why Not using NPU? \n\n**Response:**\n\nAll the experiments were conducted on mobile GPUs because most frameworks like MNN or TNN do not support NPUs. We did not conduct experiments using NPUs because mobile NPUs, while faster than GPUs, lack low-level programmable interfaces for individual developers [1]. TFLite uses an NPU backend via system calls provided by the Android Runtime System, which do not provide an interface for independent developers to support or optimize certain operators [1]. \n\nWe have to put the comparison table in the PDF due to space limit. While the NPU theoretically outperforms the GPU, our GPU implementation of ECP-ViT runs at almost the same speed as TFLite’s NPU, demonstrating our framework’s efficiency in leveraging GPU capabilities effectively.\n\nQualcomm AI hub devices do not support Oneplus 12 yet, for TFLite with NPU we are using remote device Samsung Galaxy S24+ provided by Qualcomm AI hubs [1]\n\n[1] The Qualcomm® AI Hub Models https://github.com/quic/ai-hub-models?tab=readme-ov-file", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### Q1 (Figure 4) How are core nodes determined?

**Response:**

In our ECP-ViT, we consider the image patches as nodes and predefine a series of core ratios to divide nodes into cores and peripheries. During training, we employ Grad-CAM to identify important regions of the images and assign the core nodes to those regions. Accordingly, the QKV matrices of these patches are divided into core and peripheral components.
For example, for images with a resolution of 224x224 and a patch size of 16x16, there are a total of 196 patch tokens as nodes. For a core ratio of 10%, around 20 patch tokens are considered as cores, and we choose the top 20 important regions as cores.
This partitioning method is inspired by human brain networks [1], where different networks exhibit different core ratios. 
[1] "Gyri vs. sulci: Disentangling brain core-periphery functional networks via twin-transformer," MICCAI 2024.

### Q2 Need to explain dimension reduction heuristic algorithm.

**Response:**

Thank you for your feedback regarding the description of the dimension reduction heuristic. The detailed pseudo code for the heuristic has been attached in the uploaded PDF. The score is collected by running mini benchmarks, and the different layouts are defined by the frameworks. The process begins by identifying key nodes in the computational graph that perform actual computation (Step 1). For each key node, possible data layouts are determined (Step 2), taking into account the various layout options provided by the frameworks. Each possible layout is then evaluated by calculating a score based on data locality and GPU utilization using mini benchmarks (Step 2.1). The layout with the best score is selected and assigned to the key node (Step 3). Finally, the computational graph is updated with the new layouts for all key nodes to ensure that the entire graph benefits from the optimized layouts, improving overall performance (Step 4).

Meta-review: This paper presents ECP-ViT, a framework aimed at accelerating Vision Transformers (ViTs) for smartphone deployment. ECP-ViT enhances performance through Core-Periphery Guided Self-Attention and Data Layout Selection. The framework shows competitive results on STL-10, CIFAR-100, TinyImageNet, and ImageNet. The submission has received constructive feedback, particularly regarding the details and justification of the architecture design. We encourage the authors to carefully address the reviewers' feedback and incorporate the rebuttal into the final version.
Decision: Accept (poster)