# Enhancing Split ViT Inference Through Sparsity-Driven Compression

Amira Dhaouadi, Nadjib Achir, Cédric Adjih

<figure><img src="image_1.png" /></figure>

To cite this version:

Amira Dhaouadi, Nadjib Achir, Cédric Adjih. Enhancing Split ViT Inference Through Sparsity-Driven Compression. VTC 2025 - Vehicular Technology Conference, IEEE, Jun 2025, Oslo, Norway. hal-05273377

HAL Id: hal-05273377

https://hal.science/hal-05273377v1

Submitted on 22 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Enhancing Split ViT Inference Through
Sparsity-Driven Compression

Amira Dhaouadi*, Nadjib Achir*, Cédric Adjih*

*Inria, Saclay, France

Email: firstname.lastname@inria.fr

***Abstract***—Vision Transformer (ViT) is a deep learning model that plays a significant role in advanced computer vision and pattern recognition tasks but faces challenges in inference due to high computational costs and energy consumption. Split computing, which distributes the load between an edge server and a mobile device, is proposed to address these issues. Unfortunately, most literature focuses on split computing for CNNs, with limited attention to ViT split computing. In this paper, we explore the use of split computing to optimize ViT's inference process by limiting usage of bandwidth. We propose compressing the latent space data by introducing more sparsity in the intermediate features. This sparsity is then exploited through a compression algorithm before transmitting the data through a communication channel. To understand and explain the performance of our approach, we analyze the latent space data using several metrics. Our approach obtains a significant compression ratio without causing a substantial decrease in accuracy. It is computationally efficient and do not require retraining the model.

***Index Terms***—Vision Transformer, Split Computing, Latent Space Compression, Global average pooling.

## I. INTRODUCTION

The development of AI models has become a central focus for addressing a wide range of tasks. Due to their efficiency and ability to facilitate workload, these models have become indispensable and are widely integrated into real-world applications. For example, advanced smartphones and autonomous vehicles increasingly rely on deep learning models. Apple and Samsung use computer vision-based AI for facial recognition to unlock screens, while Google's visual search generates results based on image similarity. In addition, AI models are critical in modern vehicles for detecting pedestrians, vehicles, cyclists, and other road obstacles, ensuring safe navigation. Among current models, the Vision Transformer (ViT) [1] variants have the best accuracies in many of these applications[10].

ViT (Vision Transformer) is a deep learning model applying the Transformer architecture [17] to images. One of its key strengths is related to the self-attention mechanism, which effectively captures long-range relationships across an image. This allows ViT to outperform convolutional neural networks (CNNs) models like ResNet in large-scale image classification. However, CNNs have the advantage of lower memory requirements, making them a better fit for real-time

tasks on devices with limited resources. Indeed, while CNNs are more efficient regarding memory usage, making them a better choice for real-time tasks on devices with limited resources, ViTs suffer from high computational and memory demands, particularly due to the self-attention mechanism. They require more processing power and memory to handle the large amounts of data involved in image classification and some specific tasks, which makes them less suitable for resource-constrained devices. To counter that, one standard method for running models on devices with limited resources, is to use Edge or Cloud Computing. This approach, known as AI edge/cloud offloading, reduces the computational load on resource-constrained devices and on and in-vehicle devices by shifting the workload to more powerful servers [16] [4].

However, offloading data to the remote servers introduces new challenges. For instance, there is an increased risk related to privacy and security [18]. A second approach is model compression, which seeks to reduce the complexity and size of learning models while preserving performance as closely as possible to the original model. Techniques such as pruning [7], quantization [3], and knowledge distillation [8] result in more resource-efficient models, making them more suitable for deployment on devices with limited computational capabilities. A final approach consists of splitting the model into two distinct parts: a first part (referred to as head) executed on the end-device and a second part (referred to as tail) executed on a remote server (Edge or Cloud). This model division can be exploited for training (learning) and inference (computing) tasks. In training, this model splitting allows the client to process the initial layers locally while the server handles the deeper layers, with communication between them to facilitate gradient backpropagation. In inference, the same split structure allows the end device to compute the early stages of the model locally to reduce the computation burden and offload the intermediate data (referred as latent space) to be handled by the tail on the server. This flexible approach leverages the benefits of split learning for collaborative training and split computing for efficient inference.

To the best of our knowledge, the literature lacks studies considering Split Computing on ViT models. Most existing studies on Split Computing consider CNN models and focus on reducing the inference delay and bandwidth consumption [11]. One common approach found in these studies is

<footnote>This work was funded by the French National Research Agency (ANR-22-PEFT-0007) as part of France 2030 and the NF-FITNESS project.</footnote>
compressing intermediate features before transmission. This technique reduces the amount of data sent between the device and server, which leads to improved real-time performance and enhanced resource efficiency. On the other hand, a few literature studies have focused on applying Split Learning to ViT models [13, 14]. These papers adopt a strategy where only the embedding layer and the classification layer are placed on the end device, which limits the capabilities of the model split between the client and server. Indeed, these approaches do not address the full potential of Split Learning, where several split point configurations are possible and where the main challenge is to find the optimal point that reduces the learning time. Both works are, in fact, closer to data offloading than true Split Learning. The primary goal is to enhance privacy, and they neglected the optimization of the bandwidth usage. However, given the large amount of data exchanged between the client and the server, bandwidth consumption becomes a significant concern since it poses practical challenges for real-world deployment.

In this paper, we address the problem of performing split computing for Vision Transformer (ViT) models, with a particular emphasis on reducing bandwidth usage and improving inference speed for real-time applications. One primary goal is to compress latent space data before transmitting it between devices and servers to reduce the volume of data to be sent, resulting in faster transmission speeds and shorter inference times. We analyzed the intermediate data for several potential split points to identify and understand its key properties, which informs the development of an effective compression strategy. Additionally, we compress the intermediate features without retraining the model by leveraging the Global Average Pooling layer concept. We aim to optimize the balance between compression efficiency and model accuracy, ultimately reducing inference time while maintaining performance.

The rest of this paper is structured as follows. In section II, we present the background and the state of the art. The section IV presents our proposed approach starting with our strategy of split computing vision transformer, then the intermediate features analysis, and arriving at our compression-based method the feature of the intermediate result tensor between possible split points. The results are presented in section V. Finally, Section VI concludes the paper.

## II. BACKGROUND AND RELATED WORK

### A. Vision Transformer (ViT)

The Vision Transformer (ViT) architecture [1] was introduced as an application and adaptation of the transformer architecture [17] to computer vision and pattern recognition tasks and specifically to image classification. The different parts of the ViT architecture are represented in Fig. 1. The ViT comprises three main parts: *Embeddings*, *Transformer Blocks*, and a final *Multi-Layer Perceptron* (MLP) for classification [1]. In the first stage (embedding), images with resolution $H \times W \times C$ (where C is the number of channels) are initially divided into a series of patches. Each patch has a resolution $P \times P$, and the resulting number of patches is $N' = \frac{H \times W}{P^2}$.

Each patch represented by $(P \times P \times C)$ pixel color values is then projected to a vector of dimension $D$. Various terms are used in the literature to refer to these vectors: *latent vector*, but also *token*, *intermediate features*, *feature vector* and *embedding*. Each latent vector is a vector of *values*, alternately called *coefficients*, *dimensions* or *features*. In order to represent an entire image and facilitate classification, an $N' + 1$<sup>th</sup> additional vector (class token [CLS]) is prefixed to that list of latent vectors. It occupies a unique place in a sequence and is itself a learnable parameter vector. For simplicity, denote $N = N' + 1$. The set of the $N$ latent vectors forms the *latent space* (sometimes called *smashed data*).

To maintain information of the positions of the patches (which has been lost at this step), trainable *positional embeddings* are added to each latent space vector. In the second stage, where the bulk of the ViT processing occurs, the sequence of patches are passed through multiple transformer encoder blocks. Each of them applies Multi-Head Self-Attention (MSA) and a Feed-Forward Network (FFN). The result of the Transformer encoder block is a sequence of $N$ embeddings with dimensions $D$, each embedding reflecting both the local and global context of the tokens. The MSA captures relationships between patches by computing attention. It uses three weight matrices (Query, Key, and Value) per head. Each layer also includes Layer Normalization and residual connections, which have negligible computational overhead. ViT can learn how the various components of an image contribute to its overall label thanks to this, which is crucial for image classification. A final MLP layer projects the output of the Transformer encoder blocks to class confidence scores.

### B. Split Computing

As introduced previously, in distributed machine learning, Split Computing is a recent method designed to distribute the computational load of deep learning models between mobile devices and edge servers during inference. A critical challenge is determining where to split the model to enhance inference efficiency optimally. To tackle this, the authors in [2] propose a method for dynamically identifying the best-split point in deep neural network (DNN), optimizing end-to-end inference time by factoring in the communication channel's state and the batch size. In [6], the authors proposed the Interpretable Split (I-SPLIT) algorithm, which identifies optimal split points by considering the importance of neurons. Their approach evaluates and ranks potential splitting points by determining how much each layer contributes to classification accuracy when the model is split and retrained at those points. Both papers focus solely on identifying optimal split points within the model architecture without addressing the potential impact of such splits on network performance. This omission is significant, as splitting the model could introduce high delays due to the large volume of data exchanged between the end device and server, potentially discrediting the benefits of split computing in real deployment.

To address this problem, the literature has proposed several solutions, which can be grouped into two main categories.
The first category contains approaches that minimize the data exchanged between the two entities by modifying the architecture of the model. For instance, in [5], the authors propose quantization, compression, and compression-augmented training with split computing. The first technique includes converting model weights and activations to lower-bit representations, then, the compression seeks to reduce the data size through lossless or lossy methods. Finally, compression-augmented training aims to mitigate accuracy losses by retraining the model with compressed data. However, this can increase training time and complexity, and may not fully compensate for degradation.

In [12], the authors combine both knowledge distillation in order to train a lightweight student model and neural data compression via learned entropy coding to reduce intermediate data. A bottleneck is then injected into the student model to enhance the efficient data transmission between mobile device and edge server. This bottleneck compresses the smashed data by allowing the model to focus on the most relevant information. However, during inference, an amount of the information can be restricted which can lead to loss of crucial features necessary for accurate predictions. Indeed, retraining the model after the injection of compression is necessary. Thus, the injection of bottleneck implies additional computational overhead, requiring time and resources, which can be a limitation for certain applications.

To address this issue, [3] proposes compressing the intermediate tensor without the need for model retraining. They performed a deep analysis of the latent space to assess the occurrence of zero values. They justified their approach by demonstrating how frequently redundant information appears in the data, which could be optimized through compression. Additionally, they explored spatial correlations across the three dimensions of the tensor (row, column, and channel) by differentially encoding consecutive coefficients—essentially subtracting the previous coefficient from the current one within the same row, column, or channel. They also employed Grad-CAM, a saliency-based technique that calculates the gradient of the predicted class score with respect to the input features. This approach identifies the regions of an image that have the greatest impact on the classification decision, allowing the model to focus on the pertinent areas while disregarding less relevant sections, thereby highlighting the object of interest.

The most significant tensor coefficients (the top 10% found by Grad-CAM) typically have larger values when applying Grad-CAM at the split point of a DNN, however, some significant coefficients can still have very small or zero values. Notably, Grad-CAM determined that many null coefficients for ResNet and related architectures were relevant, showing that even small or zero-valued coefficients can be important for the classification outcome. However, they didn't explore the result to compress intermediate data.

As we can see, most of the literature presented above has been done to optimize split computing for classical neural networks. They focus on improving tensor compression, identifying optimal split points, and minimizing communication

overhead. However, the literature lacks studies explicitly exploring Vision Transformers, which are becoming increasingly popular in many domains. Indeed, the architecture and self-attention mechanisms of ViTs present unique challenges and opportunities for split computing, which are largely unexplored in current literature. In this paper, we address this gap by focusing on split computing specifically designed for Vision Transformers as well as compression method to reduce the dimension of latent vectors.

### III. SPLIT COMPUTING VISION TRANSFORMER

#### A. Splitting Strategy

In our work, we focus on split computing; in other words, we split the vision transformer in the inference task. Figure 1 shows that we divided the model into two parts, one executed in the device and the other in the server. Thus, the device may have few Transformer blocks. The split point is, in general, between the encoder blocks. Then, we propose some analyses of intermediate features for different split points and explore the result to build the compression method. We propose to convert the maximum of amount of values to zero to increase redundancy. After that, the intermediate data will be passed through a compression algorithm (ZIP) to remove the redundancy. Finally, in the decoder part, before passing the data to the tail, we uncompressed that compressed intermediate data (which is still with zeros at the missing indices).

<figure><img src="image_3.png" /><figcaption>Fig. 1. The architecture of the Vision Transformer</figcaption></figure>

#### B. Intermediate Data Analysis

As introduced in the last section, our goal is to compress the latent space data by replacing the least important values by zero. But before that, it is essential to understand the characteristics of latent space data.

We examined the intermediate data values based on various potential split points in the initial step. Figure 2 illustrates these values' cumulative distribution function (CDF) from
<figure><img src="image_4.png" /><figcaption>Fig. 2. Cumulative Distribution Function of Latent Space Values</figcaption></figure>

thousands of images. The graph demonstrates that the distribution of latent space data across different split points follows a distribution with a mean value very close to 0 (and resembling a normal distribution). Indeed, the CDF reaches approximately 0.5 at 0, indicating a ≈ 50% probability that a value from the latent distribution is less than or equal 0. While the overall shape of the distribution remains consistent across split points, the spread of the cumulative distribution function varies with the split point; i.e. even though the form is the same, the probability mass is more or less dispersed.

In addition as we can see, when we increase the split point from 1 to 12, the distribution of the latent space becomes progressively less dispersed. At lower split points, the CDF exhibits a gentler slope, indicating a wider spread of data values. This suggests greater variability in the latent representation. On the other hand, as the split point increases, the CDF becomes steeper, reflecting a more concentrated distribution with reduced variability. This trend highlights how the choice of split point influences the distribution of latent features. With higher split points leading to a more compact and uniform representation of the data. However these small values can have significant implications in latent space data.

<figure><img src="image_5.png" /><figcaption>Fig. 3. Localization of Small Values in Latent Space Representation</figcaption></figure>

In Figure 3, we illustrate the locations of low magnitude values (≤ 0.1). As we can see, these small values are distributed throughout the tensor without any specific pattern governing their localization. We can choose between a column-wise or a row-wise approach when considering size reduction. In Vision Transformers (ViTs), each row corresponds to a token representing a specific patch of the input image. However, identifying the most relevant tokens is challenging due to the architectural structure of ViTs, where the self-attention mechanism distributes importance across multiple tokens. Unfortunately, techniques such as Grad-CAM, commonly utilized in convolutional neural networks (CNNs), are not very effective in identifying the most relevant tokens for ViT architecture [3].

In the following, we have opted for a column-wise value analysis. This method enables us to concentrate on the shared feature dimensions of all tokens, creating a more consistent optimization framework. By concentrating on columns, we can more effectively reduce the size of the token representation rather than trying to remove small values individually, which might compromise the integrity of the latent space. As a result, in the following we analyse the column-wise distribution of small values to enhance our understanding of their role and potential in optimizing the tensor representation.

Figure 4 presents the CDF of the average absolute values of the elements in each column. Notably, nearly 60% of the column means fall within the range of 0 to 1, regardless of the chosen split point. This observation indicates that small mean values are common and uniformly distributed across the dataset. However, while the CDF highlights the frequency of these small means, it does not offer direct insight into their relative importance or contribution to the overall data significance. To evaluate the contribution of each column, we analysed its energy distribution of each dimension. Basically, given latent space represented by a tensor L of size (N, D), where N represents the number of rows and D the number of columns (rows are latent vectors, columns are features/dimensions), we compute the contribution of the j-th dimension to the total energy as following:

$$C_j = \frac{E_j}{E} \qquad (1)$$

where $E_j$ is the energy if the j-th dimension, and computed as $\sum_{i=1}^{N} L_{i,j}^2$, and E denotes the total energy of the entire latent space and computed as $\sum_{j=1}^{D} E_j$. Figure 5 shows that some vectors have higher energy than others, indicating they contribute significantly more information to the overall tensor. High energy typically indicates that the values in those vectors have larger magnitudes. Larger values convey more significant information and exert a greater influence on the overall prediction. These features capture the most relevant variations in the data, making them essential for model performance.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Cumulative Distribution Function of mean of Dimension</figcaption></figure>

**Conclusion:** The latent space contains a significant number of small values. Each dimension exhibits specific low-magnitude values, which become evident when analyzing the
<figure><img src="image_7.png" /><figcaption>Fig. 5. Contribution (Importance) of each Dimension in Latent Space Tensor</figcaption></figure>

mean of each dimension. If a dimension's mean is insignificant, its contribution to the latent space data and subsequent predictions is also likely to be minimal. In this paper, we choose a strategy to rather than removing an entire token, we aim to retain it while reducing its size. This could be done by selecting some of the token's columns to be set to zero, therefore increasing redundancy and improving compression efficiency.

## IV. PROPOSED APPROACH: GLOBAL AVERAGE POOLING-BASED COMPRESSOR (GAPC)

Based on the analysis conducted in the previous section, we observed that the majority of values are relatively small in magnitude when considering their absolute values. However, some vectors exhibit significantly higher magnitudes compared to the others. In the following, we will show that leveraging this insight in conjunction with the concept of the Global Average Pooling (GAP) layer [9], we can effectively compress our intermediate data.

Global Average Pooling was designed to replace fully connected layers in classical neural networks model, specifically targeting the first layer, to significantly reduce memory consumption. By averaging every element in a feature map, GAP eliminates the need for large weight matrices and the associated storage requirements.

GAP operates on the spatial dimensions (height and width) of each feature map by calculating the average value for each channel. This process results in a vector where the length corresponds to the number of feature maps or channels. Each entry in the vector represents a single average value for its respective feature map.

In CNNs, GAP is intended to directly replace the last FC layer. In our case, we can not replace the MLP in the transformer block with GAP. The representational capacity of the model and the ability to capture complex patterns in the data are enhanced by the non-linear transformations performed by the MLPs. Global Average Pooling averages over spatial dimensions and reduces the feature map to a single value per channel. The pooling operation doesn't enhance the expressiveness of individual token features and cannot apply non-linear transformations. GAP is not suitable for ViTs, where each token represents an image patch and requires individual transformation rather than spatial aggregation, because it is not effective in architectures with spatial structure, such as CNNs. On the other hand, we mentioned before that we will

do compression without retraining models. We can explore the idea of GAP to compress our intermediate data. Our goal is to reduce the size of tokens by adding the redundancy in the dimensions and remove it using the ZIP algorithm.

To achieve this, we propose the GAP-based Compressor (GAPC): we use global average pooling on the columns to identify which ones can be removed. Precisely, this method calculates the average of absolute values of each dimensions. Compare the mean with a threshold. Finally, for the dimensions with an average below the threshold, we replace them with a null vector. The algorithm formalizes the suggested approach.

### Algorithm 1 Global Average Pooling-based Compressor

1: **Input:** Tensor $L$, Float threshold

2: **Output:** Compressed Tensor $L_{compressed}$

3: Compute the average feature per column:

$$ \text{avg\_feature}[j] = \frac{1}{N} \sum_{i=1}^{N} |L_{i,j}|, \quad \text{for all } j \in \{1, \dots, D\} $$

4: Determine selected columns:

$$ \text{selected\_col}[j] = \begin{cases} \text{True}, & \text{if } \text{avg\_feature}[j] \ge \text{threshold} \\ \text{False}, & \text{otherwise} \end{cases} $$

5: Extract selected indices:

$$ \text{selected\_indices} = \{j \mid \text{selected\_col}[j] = \text{True}\} $$

6: Reduce the tensor using selected indices:

$$ L_{reduced} = L \cdot \mathbb{1}_{[\cdot, j \in \text{selected\_indices}]} $$

7: Apply compression:

$$ L_{compressed} = \text{ZIP}(L_{reduced}) $$

8: **Return:** $L_{compressed}$

One can select the threshold to match available bandwidth by adjusting the compression ratio, ensuring an optimal trade-off between bandwidth efficiency and model performance.

## V. RESULTS

To evaluate our proposal, we utilized a pre-trained Vision Transformer (ViT) from Torchvision and fine-tuned it using the CIFAR-10 dataset. Thereafter, we modified the model's implementation to allow it to be split at a configurable point, defined by a parameter that specifies where the split occurs. This design enables a flexible head and tail architecture rather than a rigid separation. Before applying compression, our model achieved an accuracy of approximately 94%.

Figure 6 shows the accuracy compared to the compression ratio achieved by applying GAPC with a splitting point set to 6, evaluated across different threshold values. Note that the compression ratio is defined as the ratio of the original zipped latent space size to the compressed latent space size after applying GAPC.

As we can see, the compression ratio improves when we increase the threshold. This is because increasing the threshold
<figure><img src="image_8.png" /><figcaption>Fig. 6. Accuracy and Compression Ratio after Splitting ViT and Using GAPC (for a Split Point at Encoder Block 6, Averaged over 5000 Images)</figcaption></figure>

leads to an increase in the number of columns set to zeros, which implies higher redundancy in the latent space. However, the drawback is that the obtained accuracy gradually decreases as more information is discarded from the latent space. This observation reveals a trade-off between compression efficiency and accuracy. In other words, the compression ratio could be maximized without leading to a significant loss in accuracy. Indeed, we can clearly see that a threshold slightly greater than 1 achieves a compression ratio of almost 4 while still maintaining an accuracy close to 90%. However, a threshold greater than 1.2 decreases drastically the accuracy.

<figure><img src="image_9.png" /><figcaption>Fig. 7. Comparison Between GAPC, Random ZEO and Top-k</figcaption></figure>

To evaluate our approach, we aim to compare it with classical methods from the literature that also focus on compression without requiring model retraining. For this purpose, we consider two baseline methods alongside our Global Average Pooling-based Compressor approach. The first method, Random Zero, involves randomly selecting columns that are filled with zeros. This approach introduces redundancy but lacks any selection criteria for preserving important features. The second method, Top-k [19] [15], retains the *k* most significant column vectors based on their infinity norm (largest coefficient in absolute value): only the *k* column vectors with the largest norms are preserved, while the others are set to zero.

Figure 7 shows that GAPC outperforms both methods in terms of accuracy and compression ratio. By leveraging the average magnitude of dimensions, GAPC prioritizes meaningful features to retain, resulting in less accuracy loss while achieving a high compression ratio.

Determining the best split point is essential for achieving a balance between global performance, which includes accuracy, inference delay and computational demands on the end device. Figure 8 illustrates the relationship between accuracy and compression ratio for all possible split points, varying the applied threshold. We can observe that a constraint on the compression ratio requires a specific split point in order to ensure maximum accuracy. For instance a compression ratio below 2.5, the best split point is the encoder block 5. When dealing with important compression ratios greater than 3, the optimal split point is at encoder block 9. However, a key trade-off to consider is the calculations performed on the device. Indeed, for the larger compression ratios (> 3), the general trend is that later split points allow a greater accuracy for the same compression ratio: hence later split points tend to yield better accuracy (consider the curves for split points 2, 4, 6, 8, and 9 for instance). But of course, at the same time, they imply more processing done on the device, thus an higher energy processing cost. This leads to the question of how to minimize energy consumption when selecting these split points.

<figure><img src="image_10.png" /><figcaption>Fig. 8. Accuracy and Compression Ratio for different split points in ViT</figcaption></figure>

In Figure 9, we analyze the total inference time obtained when applying GAPC. This time is an estimation. It encompasses the processing time, on the device the communication time, and the processing time on the server. We evaluate several potential bandwidth scenarios between the end device and the server, including 10 Mb/s, 30 Mb/s, 60 Mb/s, and 100 Mb/s. The observations indicate that as bandwidth decreases, the benefits of latent space compression increases. Specifically, at low bandwidths of 10 Mb/s and 30 Mb/s, the inference time with compression is consistently shorter than the inference time without compression. At a bandwidth of 60 Mb/s, compression is advantageous only when the compression ratio exceeds 3. At a bandwidth of 100 Mb/s, compression is unnecessary. The exact timing figures depends on the specific inference processing times, which in turn will vary with the hardware (here estimated by measurements on machines with intermediate computational capabilities), but in all cases, the key conclusion is the amount of available bandwidth highly influence the decision to apply compression.
<figure><img src="image_11.png" /><figcaption>Fig. 9. Inference Time for Different Bandwidth and Compression Ratio</figcaption></figure>

## VI. CONCLUSION AND FUTURE WORK

In this work, we proposed a method to optimize Vision Transformer inference at the edge by splitting the model (and the processing) between the device and server during the inference stage. Based on a study of the statistical properties of the latent space data, we propose compressing it, with a new method called GAPC. GAPC maintains only the dimensions of the latent space that are expected to contain the most meaningful information, zeroing out the others. The method effectively reduces the size of latent space data with minimal impact on model performance. GAPC decreases communication delays, which helps reduce inference time. Furthermore, one of the important aspects in our work is the optimisation of the full system: choosing the optimal split point, and also selecting a good level of compression driven by a selectable internal parameter (the threshold). We have demonstrated the excellent performance of the method (compared to the baselines) and have quantified key trade-offs between accuracy and compression ratio. We also evidenced that the best split points tend to be towards last blocks, leading to heavier computation and increased energy consumption on the edge device: this introduces other trade-offs involving energy consumption. Finally, we have also shown that depending on network bandwidth, the impact of compression varies, introducing another parameter in the trade-offs when optimizing the full system. Future research directions include developing a full framework for automatic optimization of the split computing system with ViT, by selecting automatically parameters depending on constraints, network conditions, and predefined objective functions.

## REFERENCES

[1] Dosovitskiy Alexey. "An image is worth 16x16 words: Transformers for image recognition at scale". In: *arXiv preprint arXiv: 2010.11929* (2020).

[2] Arian Bakhtiarnia et al. "Dynamic split computing for efficient deep edge intelligence". In: *ICASSP 2023. IEEE. 2023*, pp. 1–5.

[3] Damiano Carra and Giovanni Neglia. "DNN Split Computing: Quantization and Run-Length Coding are Enough". In: *IEEE GLOBECOM. 2023*.

[4] Hewon Cho, Ying Cui, and Jemin Lee. "Energy-efficient computation task splitting for edge computing-enabled vehicular networks". In: *2020 IEEE International Conference on Communications Workshops (ICC Workshops)*. IEEE. 2020, pp. 1–6.

[5] Hyomin Choi and Ivan V Bajić. "Deep feature compression for collaborative object detection". In: *2018 25th IEEE International Conference on Image Processing (ICIP)*. IEEE. 2018, pp. 3743–3747.

[6] Federico Cunico et al. "I-split: Deep network interpretability for split computing". In: *2022 26th International Conference on Pattern Recognition (ICPR)*. IEEE. 2022, pp. 2575–2581.

[7] Yihui He, Xiangyu Zhang, and Jian Sun. "Channel pruning for accelerating very deep neural networks". In: *Proceedings of the IEEE international conference on computer vision*. 2017, pp. 1389–1397.

[8] Geoffrey Hinton. "Distilling the Knowledge in a Neural Network". In: *arXiv preprint arXiv:1503.02531* (2015).

[9] Ting-Yun Hsiao et al. "Filter-based deep-compression with global average pooling for convolutional networks". In: *Journal of Systems Architecture* 95 (2019).

[10] Quoc-Vinh Lai-Dang. "A survey of vision transformers in autonomous driving: Current trends and future directions". In: *arXiv preprint arXiv:2403.07542* (2024).

[11] Yoshitomo Matsubara, Marco Levorato, and Francesco Restuccia. "Split computing and early exiting for deep learning applications: Survey and research challenges". In: *ACM Computing Surveys* 55.5 (2022), pp. 1–30.

[12] Yoshitomo Matsubara et al. "Supervised compression for resource-constrained edge computing systems". In: *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*. 2022, pp. 2685–2695.

[13] Sangjoon Park and Jong Chul Ye. "Multi-task distributed learning using vision transformer with random patch permutation". In: *IEEE Transactions on Medical Imaging* 42.7 (2022), pp. 2091–2105.

[14] Sangjoon Park et al. "Federated split vision transformer for COVID-19 CXR diagnosis using task-agnostic training". In: *arXiv preprint arXiv:2111.01338* (2021).

[15] Felix Petersen et al. "Differentiable top-k classification learning". In: *International Conference on Machine Learning*. PMLR. 2022, pp. 17656–17668.

[16] Wassim Seifeddine, Cedric Adjih, and Nadjib Achir. "Dynamic hierarchical neural network offloading in IoT edge networks". In: *IFIP/IEEE PEMWN*. 2021, pp. 1–6.

[17] A Vaswani. "Attention is all you need". In: *Advances in Neural Information Processing Systems* (2017).

[18] Cheng Wang et al. "The security and privacy of mobile edge computing: An artificial intelligence perspective". In: *IEEE Internet of Things Journal* (2023).

[19] Xiaoqin Zhang et al. "Top-k feature selection framework using robust 0-1 integer programming". In: *IEEE Transactions on Neural Networks and Learning Systems* 32.7 (2020), pp. 3005–3019.