Real-time Core-Periphery Guided ViT with Smart Data Layout Selection on Mobile Devices

Authors: zhihao  shu
xiaowei  yu
zihao  wu
wenqi  jia
yinchen  shi
miao  yin
tianming  liu
dajiang  zhu
wei  niu
convit  crossformer
  deit
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 16:27:23

Abstract: Mobile devices have become essential enablers for AI applications, particularly in scenarios that require real-time performance. Vision Transformer (ViT) has become a fundamental cornerstone in this regard due to its high accuracy. Recent efforts have been dedicated to developing various transformer architectures that offer im- proved accuracy while reducing the computational requirements. However, existing research primarily focuses on reducing the theoretical computational complexity through methods such as local attention and model pruning, rather than considering realistic performance on mobile hardware. Although these optimizations reduce computational demands, they either introduce additional overheads related to data transformation (e.g., Reshape and Transpose) or irregular computation/data-access patterns. These result in significant overhead on mobile devices due to their limited bandwidth, which even makes the latency worse than vanilla ViT on mobile. In this paper, we present ECP-ViT, a real-time framework that employs the core-periphery principle inspired by the brain functional networks to guide self-attention in ViTs and enable the deployment of ViT models on smartphones. We identify the main bottleneck in transformer structures caused by data transformation and propose a hardware-friendly core-periphery guided self-attention to decrease computation demands. Additionally, we design the system optimizations for intensive data transformation in pruned models. ECP-ViT, with the proposed algorithm-system co-optimizations, achieves a speedup of 4.6× to 26.9× on mobile GPUs across four datasets: STL-10, CIFAR100, TinyImageNet, and ImageNet.

Main paper: 
 introduction: 
 In recent decades, there has been a significant increase in applying deep neural network (DNN) architectures across various fields, including autonomous driving [23] , natural language processing [9] , extended reality (XR) [13] , image processing [22] , and View Synthesis [32] . Along with significant progress in hardware performance and public datasets, more and more complex DNN architectures have been proposed, including Convolution Neural Network [52] , RNN [36] , Transformer [51] . These architectures have led to groundbreaking breakthroughs in various application domains by leveraging their powerful feature extraction abilities. Meanwhile, mobile devices have become essential for deploying these applications, especially in scenarios that demand real-time performance. The mobile GPU on Oneplus 11 can achieve a theoretical peak performance of over 3T FLOPS (floating point Figure 1 : Accuracy-Latency-MACs comparison among various models on ImageNet-1K. Latency is measured on Oneplus 11 (Snapdragon 8 Gen 2 SoC). The radius of each circle represents the model's MACs (Multiply-Accumulate Operations). Our ECP-ViT-Base achieves the highest top-1 accuracy and offers the fastest inference speed among transformer-based models, providing 6× faster than ViT-Base. We simplify the notation as "Base" = B, "Small" = S, and "Tiny" = T. operations per second) [1] . Their widespread availability and increasing computational capabilities make them ideal platforms for extending the impact of these algorithm innovations.Compared to traditional convolutional neural networks (CNNs), transformers have recently become widely used across various areas and tasks due to their high accuracy. The attention mechanism in the transformer allows neurons to exchange messages effectively and efficiently, leading to promising results in natural language processing [48, 16] and computer vision domains [17, 53] . However, transformer architectures also known for their deeper network layers and require frequent reshaping and transposing of the feature maps. This results in more intermediate results and leads to a memory bound for the computation. Optimizing transformers for efficient execution becomes particularly challenging in environments where memory bandwidth is limited.Advancements in transformer architecture design, including Vision Transformer (ViT) [17] , have focused on improving message exchange mechanisms among spatial tokens through different Token Mixers. Other efforts include the shifted window attention in Swin [34] , the token-mixing MLP in Mixer [44] , and the pooling in MetaFormer [57] . These designs aim to enhance self-attention accuracy and hardware efficiency compared to the original vanilla ViT [17] , enabling more effective and efficient execution on various hardware platforms. Despite the significant progress made in transformer architecture, particularly in reducing theoretical computation demands, achieving realtime performance for transformer-based models on mobile devices remains a major challenge. Figure 1 illustrates the latency, accuracy, and floating-point operations (FLOPs) comparison of different models on a mobile GPU. Despite having fewer FLOPs, Swin-T (local attention) is even more than 3× slower than ViT-T (global attention). This is because local attention requires more frequent data transformation to reorganize tokens, which accounts for 69% of execution time in Swin. In contrast, these numbers are 43% in ViT and only 0.8% in VGG16. Similarly, DeiT has a broadly similar structure to ViT, but it involves more complex data layout transformations and larger intermediate results, as illustrated in Table 1 . Unlike powerful server platforms, mobile devices have limited memory bandwidth [24] , making it challenging to benefit from reduced theoretical FLOPs. There has been a fundamental lack of general principles that involve co-designing model architecture and system optimizations.In this work, we present an integrated framework that incorporates co-optimizations by revisiting the model design and system optimizations. First, we propose a hardware-efficient and computationalfriendly sparse scheme (guided by the Core-Periphery principle in brain networks) [61, 59, 60, 58, 62] that can be applied to guide the message exchange in self-attention. This scheme also helps reduce bandwidth pressure for the subsequent Softmax operation. Secondly, we develop a set of comprehensive compiler-based optimizations supporting the proposed pruning scheme and fully eliminate the overhead for Transpose and Reshape. Enabled by the advanced compiler optimizations, it is possible to achieve both high accuracy and high acceleration simultaneously. We demonstrate that, 1) our proposed fine-grained structured core-periphery guided self-attention offers advantages in both accuracy and speed, and 2) our compiler framework exhibits superior end-to-end acceleration performance for both the original and proposed ECP-ViT models.We summarize our contributions in three aspects:• We incorporate the organizational principle of brain functional networks-specifically, the coreperiphery principle-to guide self-attentions in ViTs. Additionally, we introduce a compiler code generation framework that efficiently supports the core-periphery structures on mobile devices. • We develop comprehensive compiler-based optimizations that can fully eliminate the overhead of data transformation, bridging the gap between accuracy and latency. • We build ECP-ViT, an end-to-end framework that combines algorithm and system design to achieve real-time performance on mobile devices. ECP-ViT achieves the significant speedup on off-theshelf mobile devices, reaching up to 16.1× on ImageNet inference while maintaining an accuracy of 84.51%.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: Main claims and brief results are in the abstract.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
background and motivation: 
 Mixed blessing of attention. ViTs are widely utilized as robust backbones across various tasks. The two primary types of attention are Global Attention [3, 33] and Local Attention [39, 41, 6, 42, 7] . Global Attention, as seen in standard transformer models, allows each token to attend to every other token in the input sequence. This mechanism ensures comprehensive context capture, leading to high accuracy. However, the downside is its computational intensity, as it scales quadratically with the input length, making it less efficient for longer sequences or resource-constrained platforms.Local Attention [64, 49] restricts the focus of each token to a subset of adjacent tokens (or with specific patterns), resulting in the reduction of computational complexity. However, local Attention does not always translate into proportionate realistic speedups when compared to the theoretical reduction in floating-point operations (FLOPs). This approach significantly reduces the computational complexity but at the cost of more layout transformations. These factors offset the expected gains from reduced computational complexity, particularly in environments like mobile GPUs where memory bandwidth and efficient data handling are critical, as reflected in Figure 1 . The actual performance gains need to be evaluated in the context of specific hardware and application requirements. Compared to the above two types of attention, ECP-ViT is the first work to explore a speed-aware end-to-end framework that achieves real-time performance on real-world mobile devices for ViTs.Efficient network design and architecture search. In this category, a significant methodology is Neural Architecture Search (NAS) [67, 40, 18, 31] . Some of the work leverages network pruning and sparse training to further reduce the theoretical FLOPs. At the token level, Tang et al. [43] introduces a patch slimming method to remove redundant tokens. Evo-ViT [54] updates selected informative and uninformative tokens through distinct computation paths, while VTP [66] reduced embedding dimensionality by introducing control coefficients. At the model architecture level, UP-ViTs [56] adopts a unified approach to prune channels in ViTs. SViTE [10] dynamically extracts and trains sparse sub-networks instead of training the entire model. Despite the significant progress made by these methods, both token-sampling and data-driven strategies may heavily depend on specific datasets and tasks, limiting the generalization capability of vision transformers. Additionally, these token-level pruning or selection introduce additional operations for the Reshape and Transpose to the feature map, leading to fewer benefits from reduced computation complexity. In contrast, ECP-ViT achieves a significant speedup on mobile platforms through two key components: 1) the more efficient At the software-level design, the rescheduled interactions between patches in ECP-ViT are guided by the generated Core-Periphery (CP) graphs. Moreover, the multiplication of query, key, and value matrices in the self-attention mechanism of ECP-ViT is also under the guidance of the core-periphery graph. At the hardware-level design, the slice and transpose reshape operations are eliminated.self-attention guided by the Core-Periphery principle, and 2) our compiler-based optimizations that eliminate data transformation overhead (i.e., Reshape and Transpose).DNN frameworks on mobile devices. Recently, there has been a dedicated focus on developing inference acceleration frameworks for mobile devices from both academia and industry. Some efforts include MCDNN [21] , DeepSense [55] , MobiSR [30] , and PatDNN [38] . However, none of these frameworks support the execution of transformer models on mobile devices. Other efforts have been made to optimize the execution of transformer models on mobile devices, include TFLite [2], TVM [11] , MNN [26] , Pytorch-Mobile [25] , and DNNFusion [37] . They support optimizations including operator fusion, constant folding, and quantization on mobile devices. However, they are not able to eliminate the intensive Reshape and Transpose operations in transformer models. In this work, our goal is to find the most appropriate CP pruning scheme for mobile ViT acceleration and the corresponding full-stack acceleration framework.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [No] Justification: Due to space space limit, we only have a little discussion in the appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 The entire framework of co-design for ECP-ViT is illustrated in Figure 2 . It comprises softwarelevel design (algorithm) and hardware-level design, both of which we will discuss in detail in the subsequent sections.As mentioned earlier, the attention mechanism in transformer models involves a large amount of data transformation, which poses significant challenges to hardware efficiency and deployment on mobile devices. This is because (i) data transformation is memory-bound, requiring high memory bandwidth. On mobile devices, which typically have restricted memory bandwidth, this leads to increased time consumption for processing these transformations; and (ii) data transformation in transformers, especially within the multi-head attention mechanism, often results in irregular data access patterns. This is typically observed during operations like tensor reorganization. Such irregular access patterns can significantly degrade performance, particularly in the absence of sufficient concurrent processing threads.We classify the data transformation into two categories (as shown in Figure 3 ), Explicit and Implicit. Explicit transformations (red color operators in round boxes) are designed by the algorithm developer to ensure the model follows its intended computational logic. On the other hand, Implicit transformations adopt data layouts that best suit layout-sensitive operators for increased effectiveness. Implicit focuses more on performance optimization tailored to specific hardware platforms and depends on the inference framework. For instance, TFLite may prefer using NHWC layout for MatMul layer, while others may prefer NCHW ones. Based on the classification mentioned above, ECP-ViT specifically answers these three questions to achieve computation efficiency and data transformation elimination:• How to design a hardware-friendly pruning scheme for ViTs without compromising accuracy?• How to effectively minimize the data transformation overhead without affecting accuracy?• How to flexibly reduce the memory pressure on mobile?The design of ECP-ViT, as shown in Figure 4 In these networks, the core nodes maintain connections to all other core nodes, while edge nodes only connect to a subset (or empty) of the core nodes. We employ Grad-CAM to identify important regions of the images and assign the core nodes to those regions during training. Accordingly, the QKV matrices of these patches are divided into core and peripheral components. For example, for images with a resolution of 224x224 and a patch size of 16x16, there are a total of 196 patch tokens as nodes. For a core ratio of 10%, around 20 patch tokens are considered as cores, and we choose the top 20 important regions as cores. This partitioning method is inspired by human brain networks [60] , where different networks exhibit different core ratios. This architecture allows BNNs to effectively enhance information transmission and communication for integrative processing [5, 19] . To incorporate the Core-Periphery principle into the self-attention mechanism of ViT, we redefined the self-attention operations based on the generated Core-Periphery (CP) graphs, where the patches are regarded as nodes, and the new self-attention relationships are represented by edges in the CP graph. Following this representation paradigm, a complete graph can depict the self-attention of the vanilla ViT. Similarly, the infusion of the Core-Periphery principle into the ViT architecture is achieved by enhancing the complete graph with the generated CP graphs effectively and conveniently. The new self-attention rules can then be redefined: CP graph can be represented by G = (V, E), with nodes set V = {ν 1 , ..., ν n }, edges set E ⊆ {(ν i , ν j )|ν i , ν j ∈ V}, and adjacency matrix A. The CP graph guided self-attention for a specific node i at r-th layer of ECP-ViT is defined as:EQUATIONwhere σ(•) is the softmax function, q (r) iis the query of patches in the i-th node in G, N (i) = {i|i ∨ (i, j) ∈ E} are the neighborhood nodes of node i, d k is the dimension of queries and keys, and K (r) j and V (r) j are the key and value of patches in node j.In ECP-ViT, each node can contain a single patch or a set of multiple patches. We propose the following patch-assigning pipeline to map the original patches to the nodes of the CP graph. In vanilla ViT with patch size 16 × 16, one input image with resolution 224 × 224 is divided into 196 patches. When we use a CP graph with n nodes to design the selfattention mechanism, 196 mod n nodes will be assigned ⌊196/n⌋ + 1 patches and the remaining n -(196 mod n) nodes will be assigned ⌊196/n⌋ patches. For example, if we use a 5 node CP graph, the 5 nodes will have 40, 39, 39, 39, and 39 patches, respectively; and if we use a 196 nodes CP graph in another case, each node will contain a single patch. Based on the above discussion, the CP graphguided self-attention that is conducted at the node level can be formulated as: Vanilla Self-Attention Q V K T ℎ 𝑤 ℎ 𝑤 ℎ 𝑤 • • Core-Periphery Guided Self-Attention C ℎ 1 P 𝑤 (CP) T ℎ 𝑤 CP ℎ 𝑤 • • ℎ 2 C T ℎ 1 C ℎ 1 = C ℎ 1 𝑤 P ℎ 2 Torch.concat = • • ℎ 𝑤EQUATIONwhere queries, keys, and values of all patches are packed into matrices Q, K, and V , respectively, and subscript c and p represent the core parts and periphery parts of the matrices. The graphical illustration of CPAttention is shown in Figure 5 . Similar to the multi-head attention in transformers [48] , our proposed CP multi-head attention is formulated as:MultiHead(Q, K, V ) = concat(head 1 , ..., head h )W o ,whereEQUATIONwhere the parameter matrices W Q i , W K i , W V i and W O are the projections. Multi-head attention helps the model to jointly aggregate information from different representation subspaces at various positions. In this work, we apply the CP principle to each representation subspace. Therefore, the self-attention guided by CP graphs in ECP-ViT reduces the computational budgets, while still maintaining the hardware-friendly computation pattern, i.e., converting the traditional self-attention into two small parts of matrix multiplication.DNN execution on mobile devices is in a manner of layer-wise computational graph (CG). The CG consists of nodes and edges, each node is an operator such as MatMul or LayerNorm, and the edge is an indicator to show the direction of the data flow. The main idea behind our optimization for eliminating data transformation is that, in a CG, each operator (such as MatMul or Convolution) has both producers and consumers. The producer generates a layout based on the consumer's preferred data layout, resulting in relatively low additional overhead compared to explicitly reorganizing the data. For instance, if a MatMul is followed by a Transpose and another MatMul, we can make the first MatMul directly generate the desired data layout for the second MatMul, thus avoiding the need for explicit data reorganization within Transpose. Our compiler optimizations consist of three steps: (i) Identify nodes to fuse; (ii) Determine possible data layouts for the key nodes; (iii) Evaluate possible data layouts. We elaborate on them in the following sections.Identify nodes to eliminate starts by classifying the operator in CG into two types: 1) key nodes (nodes that perform the actual computation such as MatMul, Softmax, and Add are called key nodes; and 2) nodes that only do layout transformation such as Slice and Transpose are called transformative nodes, which are the targets to eliminate. We first use the transformative node as a breaking point to partition the graph into a set of sub-graphs. We also apply the fusion rules similar to [37] to identify all fusion opportunities within the sub-graphs. To fully eliminate the data transformation operations, our general strategy is to find a common data layout which works efficiently for both contiguous sub-graphs. Note that, in this step, transformative nodes are still kept in the sub-graph because key nodes' data layouts are not the same.Determine and evaluate possible data layouts are to find out the best intermediate data layout to eliminate the necessity of introducing additional operators solely for layout transformations. We conduct an exhaustive evaluation of all feasible data format selections, intending to optimize hardware accelerations. It is worth noting that varying data layouts result in distinct access patterns within the mobile GPU, as shown in Figure 6 . Notably, noncontinuous data access patterns yield inferior data locality, thereby leading to increased latency. Our dimension reduction heuristic is aiming to find the reduction dimension(s) from both ends and group the reducing dimension continuously in the memory, in order to avoid the undesired data accessing pattern. For example, in a Matmul operation with A m,k and B k,n , the k dimension is our reducing dimension. Furthermore, specific operators in the subsequent operators may suitable for specific data formats. Take the LayerNorm operation as an example, it usually applies the calculation in one dimension only, we generally arrange the target dimension in the width continuously, and group other dimensions into the height dimension. It gives better data locality and GPU utilization. After using our heuristic algorithm to find the best data layout for each key node, we are then able to calculate the index transformation from the current node to its child nodes and map the index at the kernel level, which essentially eliminates all the trans-formative operations. Additionally, we store the whole intermediate results within the GPU texture memory to mitigate the time required for data transfer between the CPU and GPU during inter-operator operations. This adjustment is critical, given that data transfer speed between the GPU and CPU remains a persistent bottleneck in mobile device performance.We evaluate the speed of pruned models by utilizing compiler code generation and on-device latency measurement before performing a time-consuming pruning process. The compiler does not need absolute weight values for code generation and latency measurement, making the process more streamlined. Code generation with the compiler is much faster than Deep Neural Network (DNN) training. Therefore, before actually pruning the models, we use the compiler to evaluate them with different pruning ratios, using synthesized weights instead of real ones. This approach helps us establish a predictive curve that shows how pruning ratios correlate with expected latency. Such a strategy is crucial in determining the optimal pruning ratio that balances model performance and computational efficiency. It allows for a more effective decision on pruning before undertaking the computationally intensive DNN training process.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: we put this section in 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
evaluation: 
 In this section, we evaluate the performance of our compiler-assisted framework with our ECP-ViT model deployed on mobile devices. We use CP-Level and core ratio interchangeably in the following section.Models and datasets. The ECP-ViT is implemented based on the ViT architecture [17] and evaluated on 4 different datasets, STL-10 [12] , CIFAR-100 [27] , TinyImageNet [29] , and ImageNet-1K [15] . TinyImageNet is a subset of ImageNet-1k containing 100,000 images distributed across 200 classes. The parameters of ECP-ViT were initialized and fine-tuned from ViT-B/16 trained on ImageNet-21K [28] for TinyImageNet and ImageNet, and used parameters from pre-trained ViT-S/16 for STL-10 and CIFAR-100. We trained the ECP-ViT for 50 epochs with batch size 256 for STL-10, CIFAR-100, and 128 for TinyImageNet and ImageNet-1K, and used AdamW optimizer and cosine learning rate schedule [35] with an initial learning rate of 5e-4 and minimum of 1e-7.Evaluation environment. We compare the latency of ECP-ViT on off-the-shelf mobile devices against three state-of-the-art deep learning frameworks: Alibaba MNN [26] , Tencent TNN, and Apache TVM [11] .We do not include emerging DNN inference frameworks like TFLite[2] and PyTorch Mobile [25] because they do not support ViT on mobile GPUs yet due to unsupported operators or insufficient resources (e.g., memory capacity). Our evaluation focuses on GPU instead of CPU or NPU for two reasons. Firstly, compared to mobile CPUs, mobile GPUs offer higher computational capacity with better power efficiency. Secondly, compared to mobile NPUs, the NPUs backend is often invoked via a system call provided by the Android Runtime System or specific hardware vendors which does not provide an interface for independent developers to support or optimize certain operators yet. We leave this as a future research direction.The evaluations are conducted on a Oneplus 11 cell phone, which features a high-end Qualcomm Kryo octa-core CPU and a Qualcomm Adreno 740 GPU with 16 GB of unified memory. To demonstrate the portability of our methods, we also present results from testing on a low-end cell phone -Xiaomi 6 with limited memory and computation capacity, as shown in Figure 7 . Xiaomi 6 is equipped with an ARM Octa-core CPU, an Adreno 540 GPU, and 6 GB of unified memory. We use 16-bit floating point precision across all frameworks and models on the mobile GPU. All latency data is collected from running the tests 50 times. However, since the variance is small, we only report averages. The accuracy comparison with other works (with pruning and without pruning) is presented in Table 2 . Notably, the accuracy of ECP-ViT is under the best ratio, showcasing its effectiveness. The core-periphery principle guided self-attention of ECP-ViT proves to be competitive with self-attention in a complete graph form. This demonstrates that the message exchange of core-periphery, derived from brain function networks, leverages the message communication in ViT.We also assess the performance of the proposed ECP-ViT on ImageNet-1K with varying core ratios, and the results are detailed in Table 3 . The baseline, denoted as vanilla ViT, featuring self-attention in a complete graph form, is considered to possess a core ratio of 1.0. Notably, for the ViT-base model scale, our ECP-ViT-base, integrating the core-periphery principle, demonstrates superior performance compared to ViT-base across a spectrum of core ratios (0.7, 0.8, and 0.9). ECP-ViT surpasses the baseline by 0.73% under a ratio of 0.9, suggesting that the sparsity of self-attention could enhance performance, with the core-periphery principle guiding self-attention proving to be an effective means of achieving this sparsity.It's worth noting that our baseline is fine-tuned by us and already stands out compared to that reported in other works [65, 17, 47, 14] . Even for smaller model scales, such as ECP-ViT-small and ECP-ViT-tiny, the drop in accuracy is minimal when compared to vanilla ViT. Additionally, we conducted a comparative analysis of ECP-ViT with other competitive models on TinyImageNet, as depicted in Table 4 . The results highlight the advantages of ECP-ViT in terms of prediction accuracy. Furthermore, we extend our evaluation to STL10 and CIFAR-100, with detailed results provided in Table 5 .Table 6 compares peak memory, latency, and cache miss rates between ViT-Base and ECP-ViT. Table 7 presents a comparison of latency among ECP-ViT, MNN, TNN, and TVM for vanilla ViT models. As the models retain their dense structure after CP pruning, they can still be executed on other frameworks. '-' means the model is not supported on the framework, due to lack of operator implementation or limited memory/computation resources. As shown in Table 7 , compared to other state-of-the-art frameworks, ECP-ViT achieves an average speedup ranging from 4.8× to 5.3× for vanilla ViTs. This is because our compiler optimizations, such as data transformation elimination and operator fusion, help significantly reduce memory pressure and bandwidth demands. Moreover, our compiler optimizations for ECP-ViTs leverage the decreased computational complexity and yield a speedup ranging from 10.6× to 16.1× compared to vanilla ViT within our framework. Additionally, ECP-ViT outperforms other frameworks with a speedup ranging from 4.6× to 26.9×. TVM exhibits even higher latency for ECP-ViT due to the absence of systematic optimizations. The extra data transformation resulting from CP pruning leads to heightened demands on memory bandwidth. Our framework achieves real-time performance for small and tiny variants of ECP-ViT, meeting the requirement for real-time execution at 30 frames per second.  
conclusion: 
 This paper introduces ECP-ViT, a framework that enhances the deployment of Vision Transformer (ViT) models on mobile devices for real-time AI applications. Our approach includes a hardwarefriendly pruning method inspired by the brain network and a set of compiler optimizations to eliminate data transformation bottlenecks. The results show that ECP-ViT not only reduces computational size but also improves prediction accuracy. It achieves up to 26.9x speedup compared to other state-of-the-art frameworks, enabling real-time performance on off-the-shelf mobile devices. 
impact statement: 
 This paper aims to advance the real-time implementation of brain-inspired AI on mobile devices. The integration of the brain-inspired core-periphery principle contributes to reducing the computation budget and enhancing prediction accuracy. Additionally, the optimization of hard-level data layout could significantly improve inference speed on mobile devices. This work has the potential to boost the development of brain-inspired AI on mobile devices.NeurIPS Paper Checklist 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper has no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have cited all the assets and works as required. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This paper presents ECP-ViT, a framework aimed at accelerating Vision Transformers (ViTs) for smartphone deployment. ECP-ViT enhances performance through Core-Periphery Guided Self-Attention and Data Layout Selection. The framework shows competitive results on STL-10, CIFAR-100, TinyImageNet, and ImageNet. The submission has received constructive feedback, particularly regarding the details and justification of the architecture design. We encourage the authors to carefully address the reviewers' feedback and incorporate the rebuttal into the final version.

Reviews: review: paper_summary: paper_summary: This work proposes a ViT accelerating framework, ECP-ViT, to deploy ViT models on smartphones. This framework consists of two parts: 1) Core-Periphery Guided Self-Attention (reducing the computational and bandwidth cost of ViT) and 2) Data Layout Selection based on compiler optimizations (removing the time-consuming data transformation). Specifically, Core-Periphery Guided Self-Attention partitions the tokens in K, Q, and V matrices into core and periphery components. Tokens in the core component exchanged messages with all tokens, and tokens in the periphery component only exchanged messages with tokens in the core component. To fully eliminate the data transformation operations, this work attempts to find a common data layout that works efficiently for both contiguous operators. But the search space is large. They propose the dimension reduction heuristic algorithm to shrink the search space of the data layouts. ECP-ViT obtains competitive results on STL-10, CIFAR100, TinyImageNet, and ImageNet, achieving lower smartphone latency.
strength_weakness: strength_weakness: - Some details need to be elaborated on. The description of the dimension reduction heuristic needs to be more detailed, and it is encouraged to provide its pseudocode. 
- There seems to be a discrepancy in Equation 1. The conventional form of self-attention is softmax(qV)K, while in ECP-ViT, it appears to be softmax(qVK). This discrepancy should be carefully reviewed and corrected.
questions: questions: 1. How do you partition the KQV matrices into core and peripheral components? Are there any partitioning criteria? And why do you use this partitioning method?
2. MobileViT [1] is a lightweight, low-latency network for mobile vision tasks. What significant advantages does ECP-ViT offer over MobileViT that we should consider?

[1] Mehta, Sachin, and Mohammad Rastegari. "Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer." arXiv preprint arXiv:2110.02178 (2021).
limitations: limitations: Not applicable.
paper_summary: paper_summary: This paper introduces ECP-ViT, a framework designed to improve the performance of vision transformers (ViTs) on mobile devices. The authors observed the intensive and irregular memory access involved in the data transformation for self-attention layers, which significantly slows down transformers compared to traditional CNNs. To address this, they propose a hardware-friendly self-attention pruning technique motivated by the core-periphery structure in brain networks, thereby reducing the computational and memory access burdens. ECP-ViT also incorporates compiler optimizations to fuse and eliminate transformation operators to further boost the performance. These combined optimizations enable the work to speedup in ViT inference on mobile GPUs by 4.6x to 26.9x without sacrificing the inference accuracy.
strength_weakness: strength_weakness: - The evaluation section could be improved:
    - It would be beneficial to discuss the memory access pattern differences before and after applying ECP to showcase its effectiveness in improving memory efficiency.
    - The power and energy consumption evaluation should also be included, given the limited battery capacity of mobile devices.
- There are several presentation issues; for example, the space between Table 3 and Table 4 is too dense, and the font size seems to change abruptly starting from the “Evaluation environment” in Section 4. Please ensure consistent font sizes throughout the text.
questions: questions: 1. Can you provide a detailed comparison of memory behavior before and after applying the ECP technique?
2. Can you show the power/energy improvement achieved by the proposed optimizations?
limitations: limitations: The authors have mentioned the plan to evaluate other model architectures in future work, which addresses some limitations. However, it would be beneficial to include additional evaluations such as memory studies, power/energy consumption analysis, and resource utilization metrics.
paper_summary: paper_summary: This paper presents ECP-ViT, a real-time framework for deploying Vision Transformers (ViTs) on mobile devices. Inspired by the brain's core-periphery principle, this method guides self-attention in ViTs to reduce computational demands and eliminate data transformation operations. ECP-ViT integrates algorithm-system co-optimizations, achieving a speedup of 4.6× to 26.9× on mobile GPUs across various datasets while maintaining high accuracy.
strength_weakness: strength_weakness: 1. The paper writing can be improved.

2. Some figures and their elaborations are not clear enough.

3. The rationale behind the design is not well-explained.

Please see my comments.
questions: questions: 1. In line 67 of the introduction, it mentions support for a pruning scheme, which is confusing because there is no prior content about pruning before this line.

2. In the background section, it states that the token pruning method introduces additional Reshape and Transpose operations to the feature map, leading to reduced benefits from reduced computation costs. However, in many token pruning methods [1] [2], pruning is applied to certain specific layers, so the overhead of Transpose and Reshape is not that large. Moreover, token pruning quadratically reduces computation cost, which is highly effective.

3. In Section 3.1, it says it classifies data transformation into two categories, Explicit and Implicit, and Explicit transformations are denoted as red color operators in round boxes. But there are several colors in Figure 2, and some colors are quite similar, making it hard to distinguish between Explicit and other transformations. Also, it seems that there is no indication of the implicit transformations in the figure.

4. It says the CP-guided self-attention design is shown in Figure 4. But Figure 4 seems to depict the ratio of core nodes and the number of edges. This is confusing as it does not illustrate any design component or workflow.

5. The rationale behind the CP-guided self-attention design is not detailed enough. CP-guided self-attention is an efficient self-attention mechanism that reduces the number of self-attention operations, but why is it better than other efficient self-attention mechanisms such as Swin-Transformer and Criss-Cross Attention? Why do we adopt this CP-guided self-attention? How do you ensure that the information exchange through only core nodes is sufficient?  
&nbsp;

[1] Liang, Youwei, et al. "Not all patches are what you need: Expediting vision transformers via token reorganizations." arXiv preprint arXiv:2202.07800 (2022).

[2] Kong, Zhenglun, et al. "Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 7. 2023.
limitations: limitations: Please see my comments.
paper_summary: paper_summary: This paper introduces ECP-ViT, a framework that accelerates Vision Transformers on mobile devices using a core-peripheral guided self-attention mechanism. This approach reduces computational demands and achieves up to 16.1x speedup on a OnePlus 11 GPU, enabling efficient real-time deployment of ViT models while maintaining high accuracy.
strength_weakness: strength_weakness: - Overall, I think this paper is above the acceptance bar. However, the comparison (or discussion) between ECP-ViT and other lightweight ViT methods is not comprehensive enough, for example, NAS for lightweight ViT [1][2], and advanced token optimization method [3]. 
- Showing the results on ImageNet seems like a reasonable choice, but insufficient to show that the method can be generalized to other domains, such as detection/segmentation. One possible way to do this is to directly transfer the weights for another downstream task and provide some comparison. By the way, the performance improvements in Tab.11 seem a bit marginal compared to the main results. 

[1] Elasticvit: Conflict-aware supernet training for deploying fast vision transformer on diverse mobile devices, ICCV 2023

[2] Nasvit: Neural architecture search for efficient vision transformers with gradient conflict-aware supernet training, ICLR 2022 

[3] Diffrate: Differentiable compression rate for efficient vision transformers, ICCV 2023
questions: questions: How does the core rate be selected for different devices and tasks? Would making it a sample-aware parameter further enhance the results?
limitations: limitations: Please refer to the weakness.
paper_summary: paper_summary: • This paper proposes ECP-ViT, which optimizes the ViT model by introducing the core-periphery (CP) principle.
• The Core-Periphery Principle Guided self-attention mechanisms successfully reduce memory bandwidth by eliminating data transformation.
• By applying pruning and removing data transformation, the optimization, which considers both software-level design (algorithm) and hardware-level design, achieves real-time performance on a mobile GPU (Snapdragon 8 Gen 2 SoC) with an average speedup of 4.8 times.
strength_weakness: strength_weakness: • The comparative experiments lack consistency.
• Overall, the performance improvements are marginal compared to previous research.
• The implementation uses Fixed Point 16-bit, but most current mobile environments typically utilize 8-bit or higher quantization, and there are no experiments addressing this. Data compression through quantization is essential for mobile environments.
• More fair comparisons could be made if experiments were conducted in specific NPU environments.
• Although a compiler for mobile GPUs is proposed, detailed information about the compiler itself is lacking.
    o (Figure 4) There is insufficient explanation on how the specific allocation of the core/periphery nodes is determined, and there is no criterion for specifying core nodes.
    o The detailed operation of the proposed dimension reduction heuristic algorithm is not explained. More detailed descriptions and algorithms are needed to optimize data layout.
questions: questions: • (Table 1, Table 9) Are there experimental results on latency for layout transformation and computation in other frameworks such as TFLite and Pytorch-Mobile?
    o The experimental setup in Table 9 indicates a batch size of 1 to 18. Is this an average value? Do the experimental results for latency ratios align with different batch sizes?
• (Table 3) Are there any comparative experimental results for Pytorch-Mobile?
• (Figure 6) Are there experimental results on the computation complexity and memory usage of the model according to the core ratio? Are there results consistent with those presented in Table 1?
• (Table 6) Were the experiments for TNN, TVM, and MNN conducted on NPUs while those for Ours were conducted on a mobile GPU? Are there experimental results for Ours on NPUs?
• Are there experimental results for tasks such as Object Detection or Instance Segmentation, where real-time inference is crucial?
limitations: limitations: N/A

Comments: title: Assistance Needed: None of the reviewers has responded
comment: Thank the authors for their effort in the rebuttal. The feedback addressed most of my concerns. I raised my initial score from 4 to 5.
Thanks for the additional experiments and explanations. I will keep my score.
Dear AC and SAC

We hope this message finds you well! We deeply appreciate the hard work done by the AC and SAC in the review process.

We understand that you are very busy and we aim not to add to your burden. However, all 5 of our 5 reviewers have not yet indicated that they have read our rebuttal. We would greatly appreciate your assistance in soliciting responses from the reviewers, as we are eager for feedback.

We believe our rebuttal has adequately addressed the concerns raised by all reviewers, particularly since most of the questions are simple fact-based clarification questions or requests for additional experiments to support our claims. We also want to bring to your attention that one reviewer has requested numerous additional experiments. We have devoted considerable effort to conducting these experiments at the reviewer's request, which further support our claims in the paper. However, the reviewer has not responded.

Thank you for your assistance!

Best regards,

Authors