Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devices

## Abstract

Abstract The scaling laws have become the de facto guidelines for designing large language models (LLMs), but they were studied under the assumption of unlimited computing resources for both training and inference. As LLMs are increasingly used as personalized intelligent assistants, their customization (i.e., learning through fine-tuning) and deployment onto resource-constrained edge devices will become more and more prevalent. An urgent but open question is how a resource-constrained computing environment would affect the design choices for a personalized LLM. We study this problem empirically in this work. In particular, we consider the tradeoffs among a number of key design factors and their intertwined impacts on learning efficiency and accuracy. The factors include the learning methods for LLM customization, the amount of personalized data used for learning customization, the types and sizes of LLMs, the compression methods of LLMs, the amount of time afforded to learn, and the difficulty levels of the target use cases. Through extensive experimentation and benchmarking, we draw a number of surprisingly insightful guidelines for deploying LLMs onto resource-constrained devices. For example, an optimal choice between parameter learning and RAG may vary depending on the difficulty of the downstream task, the longer fine-tuning time does not necessarily help the model, and a compressed LLM may be a better choice than an uncompressed LLM to learn from limited personalized data.

## 1 Introduction

The world has witnessed the growing interest in applying Large Language Models (LLMs) as a potential solution to personalized AI assistants []. This is particularly true when LLMs are deployed onto edge devices (edge LLMs) to meet
the increasing needs of people’s daily life assistance [, ], personalized companionship [, ], and real-time work assistance [], where data privacy is of high priority [] and constant internet connections may not be possible [].
Some exemplar edge LLMs include LLM on Nvidia IGX [], Chat with RTX [], and TinyChat []. Edge LLMs can keep locally generated data from users and locally learn from those data. Such a high assurance of privacy coupled with LLMs’ strong reasoning capabilities can induce even more user interaction with the personal assistant at a deeper personal level than otherwise [].

The scaling law [] has emerged as the standard for creating large language models (LLMs), but a key assumption behind it is the reliance on unlimited computing power for both training and inference. However, the design of edge LLMs is no easy feat [] because of the limited resources available on edge devices.
It requires allocating limited resources to accommodate multiple needs as shown in Figure [1](#S1.F1).
An ideal resource allocation strategy needs to balance many key, yet sometimes conflicting, factors, such as the
the types and sizes of pre-trained LLMs,
the learning methods and hyper-parameters for LLM customization, the amount of historical data used for learning personalization, the compression methods of LLMs, the amount of time afforded to learn, and the difficulty levels of the target use cases.
For example, models can be selected from a wide range of candidates with different model structures and sizes; while learning methods include parameter-efficient fine-tuning (PEFT) and retrieval-augmented generation (RAG). A poorly designed edge LLM
may render a poor user experience as it cannot learn well from user locally-generated content. As such, a pressing yet unresolved issue is how those constraints of limited computing resources influence the design decisions for personalized LLMs on edge devices, and what principles should guide the deployment of these edge LLMs. In this work, we empirically investigate this problem, focusing on the trade-offs between key design factors and their combined effects on learning efficiency and accuracy.

Figure: Figure 1: overview of the investigations on edge LLM
Refer to caption: /html/2406.03777/assets/x1.png

We formalize our empirical design guidelines for edge LLMs
through extensive experimentation and benchmarking, covering
a wide range of possibilities for the key factors and their combinations.
We draw a number of surprisingly insightful guidelines for deploying LLMs onto resource-constrained devices. These guidelines will be elaborated in six parts in Section [3](#S3). As a heads-up, a high-level summary of some interesting guidelines is as follows:

- •
The difficulty of the downstream task is a main factor for choosing the optimal edge LLM types and the learning method. Tasks that are either too easy or too hard tend to favor parameter learning, while medicore tasks tend to favor RAG.
- •
More historical user data for training is not always better. In most cases, the fine-tuning process does not necessarily need to use all the data and should stop early, especially when the model converges to some stable position (even at a very shallow local optimum).
- •
Among the three most popular model compression methods, distillation provides the most stable performance, quantization is less stable but has the highest peak performance, while pruning is not suitable for the edge LLMs.

To the best of our knowledge, this work is the first exploratory study that tries to address the full range of constraints on deploying LLMs on edge devices. By systematically examining the trade-offs among various factors, we offer guidelines and insights for the community and future research in this field. We hope that this work will both increase awareness of the limitations that LLMs will encounter in future edge deployments and shed light on the opportunities for future LLM designs.

## 2 Preparation for Evaluation

### 2.1 Background: Running LLMs on Edge Devices

In this paper, we use a range of common edge devices to profile the performance of LLMs running on them, as summarized in Appendix [A.1](#A1.SS1). Running an LLM on an edge device requires loading the model weights into random-access memory (RAM) of edge CPUs or edge GPUs, where model fine-tuning (or equivalently referred to as training in this paper) and inference can be performed. The RAM size on these devices commonly ranges from 4G to 16G, thereby limiting the size of model weights. Note that the
RAM capacity constraint applies to both training and inference, and the former is usually the bottleneck
as training consumes more memory than inference
does. In addition, LLM parameter learning on edge devices is time-sensitive, and the training time is affected by both the edge device’s hardware computation density (which depends on the hardware architecture as well as clock frequency), and the model complexity.
To rule out the impact of pure hardware factor, the training times reported in this paper are all scaled as if each device were running at 1.5 TFLOPS (corresponding to Apple A14 GPU), and they range from one to eight hours. Note that in the same training time, different models are able to process different amounts of data, which is shown in detail in Table  [5](#A1.T5) in Appendix [A.3](#A1.SS3).

### 2.2 Data

To investigate the customization of edge LLMs for each specific user, we concentrate on two task types: content classification and summarization. The edge LLMs should learn from the user history data and generate content to better adapt to the user’s expectations. Hence, we use the datasets specifically designed to evaluate the LLM’s personalization capabilities. Detailed reasons to use these datasets can be found in Appendix  [E.2](#A5.SS2). The detailed dataset structures and their prompt format can be found in Appendix [A.2](#A1.SS2). We prepare the data for our edge LLM evaluation from two perspectives: user-level and task-level.

User-level:
Each dataset contains many users, and each user has many samples of user history data and user query data. One sample of user history data and user query data are shown in Table [3](#A1.T3) and in Table [4](#A1.T4) in Appendix  [A.2](#A1.SS2).
In terms of generalization, we randomly select 100 users and calculate the average performance of edge LLMs over the 100 users in our evaluation.

Task-level:
When evaluating various edge LLMs under multiple datasets (tasks), the difficulty of each task can be a latent factor impacting our evaluation []. Within the same dataset, we assume that utterances have similar levels of difficulty []. However, difficulties may vary across datasets, which is a non-negligible factor that requires attention.
Zero-shot learning can be used to measure the difficulty of a provided dataset on a well-performing pre-trained model [].
Therefore, we select GPT-4 [], Claude 3 Opus [], Gemini 1.0 Pro [], and Llama 3 70B [] to measure their performance on the testing data we choose from each dataset, and summarize the results in Table [6](#A1.T6). For classification tasks, since each task has a different number of options, we consider a fairer assessment method similar to [], where they normalize the accuracy with respect to human expert performance. However, since we could not get objective human performance on these datasets, we take an alternative approach by dividing the actual accuracy obtained from LLMs with the accuracy from random guessing, which we refer to as “normalized accuracy” in the table. Mathematically, the normalized accuracy is computed as $\frac{\text{accuracy}}{1/{\text{No. choices}}}$. For the summarization task, the normalized accuracy is the same as the accuracy. For the same type of task, the normalized accuracy can represent the difficulty of the task, where the higher the value, the easier the task is. Through our examination, the normalized accuracy for classification datasets are: LaMP-1 with 0.995, LaMP-2 with 5.157, and LaMP-3 with 3.236; the normalized accuracy for summarization datasets are: LaMP-4 with 0.1460, LaMP-5 with 0.3123, LaMP-6 with 0.3475, and LaMP-7 with 0.2385. The detailed results of each dataset in each cloud-based LLM can be found in Appendix  [A.6](#A1.SS6).

### 2.3 Learning Methods

There are two learning methods for edge LLM customization: parameter-efficient fine-tuning (PEFT) [] and retrieval-augmented generation (RAG) []. While both in general can improve the cloud-based LLM performance [], the performance can fluctuate given different settings in different datasets. It is important to investigate whether similar situations exist on edge LLMs.

PEFT: Compared to various PEFT implementations, low-rank adaption (LoRA) has demonstrated promising capabilities [] in fine-tuning a wide range of LLMs and significantly improving their performance via updating a small portion of trainable parameters. The two hyperparameters rank and alpha can mostly impact the LoRA-tuning model performance []. Hence, we set all other hyperparameters to default, as described in Appendix [A.4](#A1.SS4), and explore a wide range of rank and alpha value combinations and their corresponding trainable parameters in the performance of the edge LLM.

RAG: RAG consists of a retriever that is commonly based on max inner product search (MIPS) and a generator which usually is an LLM. The retriever gets the appropriate query-relevant information from user history data to formalize the final prompt []. While this method does not consume resources to fine-tune the model, it requires saving all user history data embeddings for information retrieval. As user history data accumulates, the data embeddings can become a significant burden on the edge device []. In our experiments, we focus on the size of user history data and set all other hyperparameters to default, as shown in Appendix [A.4](#A1.SS4).

Our experiments investigate how different settings in each method can impact model performance and study which method can outperform the other under certain circumstances.

### 2.4 Models

In addition to the data and learning method, the model itself can be undoubtedly important to the entire edge LLM framework. Under the constraints of edge devices, only the model with a certain size can be deployed. There are two types of models that can satisfy such constraints, either small-scale uncompressed models or models compressed from large-scale models. In our experiments, we examine both types of models.

Uncompressed models: Their performance can mostly related to their initial architectures and how they are trained. While they may not contain certain pre-trained knowledge due to the limitation of size, their performance will not risk being deteriorated by the compression methods. For such models, we mainly investigate Pythia [], OPT [], Llama [], Gemma [], and StableLM [] series of models. To fit them into the corresponding edge device, we select each model with different weight sizes. Additionally, we profile these models under the edge environment settings to ensure that their peak resource usage will not exceed the resource limitation of the edge devices they are deployed on. The detailed model information along with their corresponding edge device information can be found in Appendix  [A.4](#A1.SS4), Table  [5](#A1.T5).

Compressed models: Other than their initial architectures and pre-training, the compression methods can also largely impact the model performance. There are three mainstream compression methods, including quantization [], pruning [], and knowledge distillation []. Model compression normally squeezes a large and pre-trained model into a smaller model. While each method can have certain advantages, the past works [] mainly study such advantages for cloud-deployed LLMs, and it is not clear which one works best for compressing LLMs for edge devices. Among the existing compressed models, we select Llama, Mistral [], Synthia [], Gemma, and StableLM quantized by GPTQ [] as the quantization models, Sheared-Llama [] as the pruning models, and Phi [] as the knowledge distillation models. Each series of the model contains various sizes, which allows us to deploy them on different edge devices and investigate their performance under a wide range of edge devices. The detailed model compression methods can be found in Appendix  [A.5](#A1.SS5)

## 3 Results Analysis and Empirical Guidelines

In this section, we provide experimental results analyses and explain how these analyses can lead us to formalize the empirical guidelines for edge LLMs. Due to the limitation of space, some of the detailed experimental results and all the discussion on hypotheses behind these observations are placed in Appendix  [B](#A2) to  [D](#A4). For the datasets we used, their difficulties can be ranked as LaMP-2 < LaMP-3 < LaMP-1 in classification task type, and LaMP-6 < LaMP-5 < LaMP-7 < LaMP-4 in generation task type, as shown in Table  [6](#A1.T6). While there is no existing literature directly comparing the difficulty across different types of tasks, we find that models often exceed human performance on classification tasks [], whereas summarization tasks remain an open problem []. Thus, we argue that classification tasks are generally easier than summarization tasks.
In the following analyses, we will use “difficulties” defined here loosely in multiple
guidelines in the context of better choices.

### 3.1 Optimal Model and Customization for LLMs on the Edge

The first step before anything to deploy an LLM on the edge device is to select the model and then the customization. Thus, we start our guidelines with the most important question: which LLM models and customizations are best suited for edge devices?

To answer the above question, we conduct experiments on over 30 different LLM models and seven datasets as described in Section [2](#S2). Due to the space limit, we provide the full set of experiments in Appendix [B.2.2](#A2.SS2.SSS2), and we select some representative results, shown in Figure [2](#S3.F2) and Table [1](#S3.T1). Figure [2](#S3.F2) includes 13 models of different sizes ranging from 160M to 13B and their performance on three tasks of increasing difficulty. Based on the ranking above, we choose LaMP-2 as the easy classification task, LaMP-6 as the easy summarization task, and LaMP-4 as the hard summarization task. In addition, we provide numerical results of 10 relatively big models on four tasks in Table [1](#S3.T1), also in increasing difficulty. The models are Pythia-2.8B, OPT-2.7B, Llama-2-3B, StableLM-3B, Gemma-2B, Phi-2, Mistral-7B quantized, OpenChat quantized, Gemma-7B quantized, and Sheared Llama-2.7B pruned, respectively. Through experiments, we conclude that the choice of model and customization depends on the difficulty of the downstream task. We provide the general guidelines below:

- •
For simple classification tasks, the optimal choice should be a small LLM with PEFT.
- •
As the task difficulty increases such as complex classification tasks and simple summarization tasks, the choice should be gradually shifted to RAG with the strongest model. Here, the strongest models are (quantized) LLMs that excel at general benchmarks and fit within the RAM constraint. Currently, some of the promising candidates of strong models include OpenChat-3.5 and Llama3-8b []. Their quantized versions fit in an 8G RAM.
- •
As the difficulty of downstream tasks further increases to difficult summarization tasks, RAG will not be sufficient, and PEFT with strong and quantized models will stand out as the optimal choice.
- •
When the difficulty of the task could not be assessed, the Phi family combined with PEFT is a safe option that provides decent performance.

Figure: (a) Easy classification task
Refer to caption: /html/2406.03777/assets/x2.png

**Table 1: Performance comparisons between parameter learning and RAG, across ten relatively big models on four datasets of increasing difficulty. The complete table can be found in Appendix [B.2.2](#A2.SS2.SSS2).**
| Models | Py-2.8b | OPT-2.7b | Lla-2-3b | Sta-3b | Gem-2b | Phi-2 | Mis-7b-G | OpenC-G | Gem-7b-G | S-Lla-2.7b-P |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LaMP-2 | PEFT | 0.475 | 0.480 | 0.430 | 0.480 | 0.430 | 0.450 | 0.485 | 0.460 | 0.420 | 0.455 |
|  | RAG | 0.320 | 0.110 | 0.295 | 0.245 | 0.205 | 0.320 | 0.345 | 0.290 | 0.300 | 0.035 |
| LaMP-3 | PEFT | 0.716 | 0.627 | 0.765 | 0.784 | 0.784 | 0.745 | 0.814 | 0.647 | 0.775 | 0.725 |
|  | RAG | 0.716 | 0.696 | 0.696 | 0.667 | 0.765 | 0.627 | 0.755 | 0.814 | 0.480 | 0.578 |
| LaMP-6 | PEFT | 0.285 | 0.155 | 0.264 | 0.280 | 0.229 | 0.241 | 0.296 | 0.304 | 0.244 | 0.319 |
|  | RAG | 0.186 | 0.134 | 0.209 | 0.256 | 0.299 | 0.208 | 0.327 | 0.355 | 0.223 | 0.240 |
| LaMP-7 | PEFT | 0.154 | 0.168 | 0.204 | 0.108 | 0.201 | 0.220 | 0.199 | 0.290 | 0.030 | 0.124 |
|  | RAG | 0.170 | 0.156 | 0.149 | 0.166 | 0.270 | 0.197 | 0.181 | 0.231 | 0.126 | 0.188 |
|  |  |  |  |  |  |  |  |  |  |  |  |

### 3.2 Choice of PEFT Strategies

As the usage of an edge LLM grows, it is expected to generate content that better satisfies user preferences. To achieve this, the model needs to learn from user history data. Typically, this is done through PEFT. When an LLM hosted on a cloud server is trained, the vast resources and large-scale training data available mean that different settings of PEFT can significantly impact the model’s performance. Achieving better performance for such models often requires substantial human labor and effort to test various settings. However, for an edge LLM, resources and training data are often highly restricted. In this context, we aim to study three questions. First, is it necessary to try different PEFT settings to achieve better model performance given different models and data? Second, is there an optimal range or fixed PEFT setting that works for all edge LLM PEFT? Knowing the answers to these two questions can benefit future edge LLM research and usage by saving the time and resources needed to optimize edge LLM performance. Furthermore, we want to understand the factors that lead to different PEFT behavior in edge LLMs compared to LLMs hosted on cloud servers.

For the two main hyper-parameters rank and alpha in LoRA, different settings of them can lead to different trainable parameters. Over the model with different sizes and different architectures, we study the impact of LoRA settings from two perspectives: the number of trainable parameters and the specific combinations of rank and alpha.

In Figure  [3](#S3.F3), we examine the performance of different-sized Pythia models given the value of rank equal 8, 16, 32, 40, 48, 56, 64, 72, and 80, under the training time from one to eight hours. In Figure  [3](#S3.F3), we examine the performance of StableLM-2-1.6b and StableLM-3b on eight commonly used rank and alpha combinations, under the training time 1 to 8 hours. Our findings are as follows:

- •
Compared to varying the alpha value with rank, fixing alpha can benefit edge LLM PEFT more.
- •
There are no benefits of increasing training time even when the rank increases.
- •
As shown in Figure [3](#S3.F3) and its supplemental results in Appendix [B.1.4](#A2.SS1.SSS4), even in larger StableLM models deployed on edge devices with 10G or 16G RAM, increasing rank or alpha does not necessarily improve the model performance. Setting alpha and rank to (16, 16) or (16, 32) can work in most cases.

Figure: (a) Py-410m ($\alpha$=32)
Refer to caption: /html/2406.03777/assets/x5.png

### 3.3 Identification of PEFT Trends for Edge LLMs

During our investigation of PEFT settings, we provide the edge LLM with the largest training time to investigate its learning boundary. However, for the usage of edge devices, such a large training time can be less practical since training on them can fully occupy the devices for hours. For the edge LLM, if its one-hour learning can bring it similar performance lifting as eight-hour learning, then the one-hour learning is definitely more efficient. Hence, we further need to investigate whether the performance of edge LLM can consistently increase along with the longer training time with more training data.

Figure: (a) Pythia’s on LaMP-3
Refer to caption: /html/2406.03777/assets/x17.png

To conduct our investigations, we selected four Pythia models of different sizes and four OPT models of different sizes. Each model can be deployed on an edge device with varying RAM requirements, including 4G, 6G, 8G, and 16G. We used the PEFT settings obtained from previous experiments, setting the rank to 8 and alpha to 32. Each model was given eight experiments to examine its performance across training time from 1 to 8 hours, allowing us to observe its performance when training from one hour up to eight hours. The experimental results are shown in Figure  [4](#S3.F4). Due to limited space, the performance on datasets LaMP-1, LaMP-2, and LaMP-7 is included in Appendix  [B.1.1](#A2.SS1.SSS1). Our findings are as follows:

- •
More training data does not necessarily mean better performance. Under most tasks, training with 3-4 hours is usually enough for customizing the LLMs towards downstream tasks on the A14 Bionic chip, and the time could be adjusted accordingly based on the edge device.
- •
While fine-tuning with PEFT is useful for almost all cases, increasing the training time only shows consistent improvement over time on the LaMP-2 dataset, the easiest task out of all seven, as shown in Figure  [8](#A2.F8) and Figure  [9](#A2.F9).

### 3.4 Impact of User History Data Volume on RAG Performance

Unlike PEFT, the resource-related settings for RAG can be simpler. RAG relies on max inner product search (MIPS) to find the appropriate information in user history data. To ensure the instant MIPS, all user history data will be converted into sentence embeddings and saved in RAM []. Compared to the computation cost of MIPS, the RAM usage of holding these embeddings might cost a higher resource burden, especially on the size of RAM []. Hence, we examine whether the higher volume of user history data necessarily leads to RAG performance improvement.

We select 100 users and each user has up to 1000 samples of user history data. For the history data, we randomly maintain 0% to 100% of them, where 0% corresponds to the case where no RAG is used. RAG takes an LLM as its content generator. We examine four Pythia models with different sizes and four miscellaneous models including Llama-3b-v2-GPTQ, TinyLlama-1.1b, TinyLlama-1.1b-GPTQ, and StableLM-2-1.6b. We calculate the performance improvement brought by RAG using different amounts of user history data, as shown in Figure  [B.2.1](#A2.SS2.SSS1).
In general, we observe that increasing the amount of user history data to 1000 samples does not significantly enhance RAG performance compared to using only 100 samples.
The RAG performance based on eight models is quite consistent across all different sizes of user history data. Furthermore, in part (c), more user history data can even lower the RAG performance based on Pythia-1b and Pythia-1.4b. The performance of RAG can be more related to the internal reasoning capabilities of the LLM rather than the amount of user history data.

Figure: (a) RAG(Pythia) Diff on LaMP-2
Refer to caption: /html/2406.03777/assets/x25.png

### 3.5 Comparison of Model Compression Techniques on Edge LLMs

In the previous sections, we have already explored the appropriate learning method and their settings for edge LLMs. Now, we investigate the other side of the problem: given an LLM, what is the best way to compress it?
As such large-scale LLMs demonstrated promising reasoning abilities when they are deployed on the cloud servers, their compressed versions are also expected to demonstrate similar reasoning abilities when they are deployed on edge devices.

Among various model compression techniques, there are three main approaches including quantization, pruning, and knowledge distillation. While each of them may have the potential to conduct a decent compressed LLM performance, it remains unknown which one of the methods can be more promising and appropriate under the constraints of edge devices. We take the representative quantization method GPTQ, pruning method structure pruning, and knowledge distillation model Phi to make investigations.

Our findings through the investigations are as follows (full experiments in Appendix [C.1](#A3.SS1) to [C.4](#A3.SS4)):

- •
Using distilled models such as Phi is a safe option when the type and difficulty of the downstream task cannot be determined. Phi-2 shows robustness towards both classification and summarization tasks, where it generally has a "brighter" performance heatmap shown in Figure  [6](#S3.F6), whereas Phi-3 excels in classification tasks.
- •
Different compression techniques are good at different types of tasks. Summarization tasks require larger (and better) LLMs to achieve emergent ability, thus GPTQ models such as OpenChat-3.5-GPTQ work better than the other two compression techniques.
- •
Shearing is a very promising technique that preserves better performance, but they are less efficient at saving RAM compared to quantization. Thus, shearing is generally not preferred for edge LLMs where RAM capacity is a critical constraint.

Figure: (a) Llama-2-3b-GPTQ
Refer to caption: /html/2406.03777/assets/x33.png

Figure: (a) Gemma-2b (L3)
Refer to caption: /html/2406.03777/assets/x39.png

### 3.6 Comparison between Compressed and Uncompressed Edge LLMs

After we compare the three compression techniques for their adaption on edge devices, we further need to investigate the impact of such compression: Consider an edge device with 16G RAM which is sufficient to host some uncompressed large-scale LLMs.
While the compressed models can learn and infer faster than their larger original versions, will such benefits compromise the model’s performance? Other than model performance, learning efficiency under compressed and uncompressed models is also important to consider. The results as shown in Figure  [7](#S3.F7).
Through the investigations, our findings are as follows:

- •
While it is true that quantization will lead to some accuracy drops in most cases [], it is noteworthy that with limited user history data for fine-tuning, the quantized model might perform better compared to unquantized counterparts on more challenging tasks.
- •
A larger model is not always better. Llama3-8B is not performing well on many of the benchmarks because it is too big and not able to adapt to downstream tasks with limited user history data. Especially during classification tasks, we see that 2B to 3B models (quantized) are good enough. On the other hand, referring to Table [7](#A2.T7), bigger models work better with summarization tasks that require more semantic understanding of the context.

## 4 Limitations

In this work we have only considered the RAM capacity limit on edge devices; however, there are other factors that may be important for certain applications, such as the reliability of hardware, potential thermal-induced changes, etc, which may vary across different hardware.
In addition, even though our results are the average from 100 users, due to the extensive amount of results that need to be presented, we are not able to include error bars to reflect the variances. Nonetheless, we categorize this work as an exploratory work in a field that is emergent, and future studies need to be done to provide more nuanced conclusions.

## 5 Conclusion and Future Directions

In conclusion, we present an empirical study for deploying large language models onto edge devices with multiple resource constraints, and we provide guidelines for choosing the optimal strategy for the deployment.
Through experiments, we show that the optimal choice of LLM and customization depends on the difficulty of the downstream task; during PEFT fine-tuning, the largest possible parameters and time are not the optimal settings; and compressed models are sometimes better than the original models on the edge due to their faster adaptation speed.

We hope that this work provides guidelines for future works and deployment of LLMs on the edge, and we also hope that this work will provide valuable directions that guide future research on LLMs under resource-constrained conditions. In addition to the findings, we offer some insights to the LLM community based on the experiment results:

- •
Small LLMs could sometimes solve domain-specific problems better than larger models. The community, especially the industry, should consider private small LLMs as alternatives to larger models on many edge services.
- •
When fine-tuning small LLMs on the edge, the tradeoff between time and performance shall be carefully considered. Larger trainable parameters and more training time are not always the best. Different from the cloud LLMs where there are no validation datasets, edge LLMs should employ a validation set to ensure the best fine-tuning performance.
- •
The RAG ability of edge LLMs is largely unexplored. Existing RAG frameworks heavily rely on the semantic understanding ability of LLMs, which might not exist on the edge.
- •
Model distillation is shown to be an effective solution that migrates the larger model’s semantic understanding ability to edge devices, but more research shall be directed into this area.

## Appendix

## Appendix A Supplements to Evaluations and Experiments

### A.1 Edge Device Constraints

Overview. Clusters and cloud servers that provide computing-as-a-service (CaaS) consist of millions of computation units like GPU with nearly unlimited RAM, storage, and energy. Common edge devices such as smartphones, on the other hand, are usually equipped with a multi-core CPU with limited resources. Emerging technologies are beginning to integrate GPUs into edge devices, exemplified by the Jetson Orin or phones powered by the Qualcomm Snapdragon 8 Gen, which offer computational powers of 3.5 TFlops — nearly half that of a cluster-level GPU like the Nvidia RTX 2070. As computational power in edge devices increases, the feasibility of running Edge LLMs is significantly enhanced.

Edge Devices for Experiments. In this paper we selected some common edge devices with different RAM capacities and computation densities for evaluation as shown in Table [2](#A1.T2). RAM is a critical resource for enabling Edge LLM learning. The model weights must be loaded into RAM so that they can be updated and optimized to adapt to user needs. In other words, when choosing to fine-tune an Edge LLM, we should consider not only the size of its weights but also the additional RAM usage due to parameter optimization and gradient calculation. Conversely, when opting to use RAG to facilitate Edge LLM learning, the consideration shifts to only the weights size and data embedding size, as RAG requires storing all data embeddings in RAM to perform MIPS. To eliminate the impact of pure hardware factors, we scale all the profiled training times as if a device were running at 1.5 TFLOPS (corresponding to Apple A14 GPU), i.e., the training time of a model is device-agnostic.

**Table 2: Profile of five common edge devices evaluated in this paper**
| Device | RAM (GB) | FLOPS (T) |
| --- | --- | --- |
| iPhone 12 (A14) | 4 | 1.50 |
| iPhone 15 (A16) | 6 | 3.58 |
| NVIDIA Jetson Orin Nano | 8 | 0.50 |
| Samsung Galaxy S24 Ultra | 10 | 3.5 |
| NVIDIA Jetson Orin NX | 16 | 3.76 |

### A.2 Datasets

Background. It is crucial to utilize appropriate datasets. The Edge LLM is primarily employed for handling prompts made by a single user. Consequently, user-specific datasets are necessary. Such datasets should feature data that is correlated and includes user-specific information.

**Table 3: One sample of user history data in each dataset**
| Dataset | User History Data |  |
| --- | --- | --- |
| LaMP-1 | [label] title | "DSP architectures: past, present and futures" |
| abstract | "As far as the future of communication is concerned, we have seen that there is great demand for audio and video data to complement text. Digital signal processing (DSP) is the science that enables traditionally analog audio and video signals to be processed digitally for transmission, storage, reproduction and manipulation. In this paper, we will explain the various DSP architectures and its silicon implementation. We will also discuss the state-of-the art and examine the issues pertaining to performance." |  |
| LaMP-2 | [label] tag | "classic" |
| description | "Young Dorothy finds herself in a magical world where she makes friends with a lion, a scarecrow and a tin man as they make their way along the yellow brick road to talk with the Wizard and ask for the things they miss most in their lives. The Wicked Witch of the West is the only thing that could stop them." |  |
| LaMP-3 | [label] score | "4" |
| text | "Amazing story of love that overcomes many obstacles. This book demonstrates that many of us are imprisoned by our views that have developed due to our upbringing, our culture, our environment and that it is possible to work through these problems of prejudice." |  |
| LaMP-4 | [label] title | "Five Things Women Can Do Today to Move Past Divorce" |
| text | "I know it sounds trite, but now you have the freedom to be you and to let it reflect in your home, surroundings and daily activities. Embrace it, run with it and most of all enjoy it." |  |
| LaMP-5 | [label] title | "Performability Studies of Hypercube Architectures" |
| abstract | "The authors propose a novel technique to study composite reliability and performance (performability) measures of hypercube systems using generalized stochastic Petri nets (GSPNs). This technique essentially consists of the following: (i) a GSPN reliability model; (ii) a GSPN performance model; and (iii) a way of combining the results from these two models. Models and performability results for an iPSC/2 hypercube system under the workload of concurrent matrix multiplication algorithm are presented." |  |
| LaMP-6 | [label] title | "Get paid real $$$$$ to drive your own car!" |
| abstract | "You are receiving this exclusive promotion because you agreed to receive special offers from an emailYOUlike marketing partner. If you have received this email in error or would like to no longer receive these special offers, please follow the instructions at the end of the message. This message is brought to you by emailYOUlike. To find out more about emailYOUlike, visit http:www.emailyoulike.com or write us at 212 Technology Dr., Suite P, Irvine, CA 92602. If you would prefer not to receive future marketing messages from us, click here or visit http://www.emailyoulike.com/remove.asp, enter your email address, and click on the unsubscribe button. Only unsubscribe requests submitted to this page can be fulfilled. emailYOUlike cannot fulfill unsubscribe requests submitted elsewhere or to the email boxes of individuals." |  |
| LaMP-7 | tweet | "Atleast Now, All fake political drama by parties (to get votes) in TN in the name of suffering Eelam Tamils will end, Thats it, Game over |
|  | HT Channel News: 25000 SL Tamils lie injured in NFZ w/o medical care and food. Thousands died in the final assault by SL army" |  |

Personalization Dataset. We have selected the LaMP datasets [] to evaluate our Edge LLM. In LaMP, each user has history data as shown in Table [3](#A1.T3). The history data for each user consists of numerous utterances that include a label and textual content. The Edge LLM will primarily learn from this historical data. The user prompts in the LaMP datasets are presented in Table [4](#A1.T4). A well-trained Edge LLM should be capable of accurately responding to the prompts using the history data.

The LaMP datasets consist of seven datasets. The first three datasets involve text classification, including LaMP-1 for citation identification, LaMP-2 for movie tagging, and LaMP-3 for product rating. The remaining four datasets pertain to text generation, including LaMP-4 for news headline generation, LaMP-5 for scholarly title generation, LaMP-6 for email subject generation, and LaMP-7 for tweet paraphrasing.

**Table 4: One sample of query data in each dataset**
| Dataset | Prompt |
| --- | --- |
| LaMP-1 | "For an author who has written the paper with the title "An application-specific protocol architecture for wireless microsensor networks", which reference is related? Just answer with [1] or [2] without explanation. [1]: "End-to-end Internet packet dynamics" [2]: "Energy-Neutral Source-Channel Coding with Battery and Memory Size Constraints" |
| LaMP-2 | "Which tag does this movie relate to among the following tags? Just answer with the tag name without further explanation. tags: [sci-fi, based on a book, comedy, action, twist ending, dystopia, dark comedy, classic, psychology, fantasy, romance, thought-provoking, social commentary, violence, true story] description: A snobbish phonetics professor agrees to a wager that he can take a flower girl and make her presentable in high society." |
| LaMP-3 | "What is the score of the following review on a scale of 1 to 5? just answer with 1, 2, 3, 4, or 5 without further explanation. review: If You Were Me and Lived In…Germany: A Child’s Introduction to Culture Around the World (If You Were Me and Lived) by Carole P. Roman, Kelsea Wienrenga (Illustrations) is a wonderful addition to the series. It’s a delight to read and look at the illustrations! Plus this book provides so many facts about the culture and customs in Germany but not in a dry, boring way. This series is a terrific way to spark interest in the world for your child and maybe for you! With thanks to the author for my copy." |
| LaMP-4 | "Generate a headline for the following article: Being an ex wife was very unexpected. I went into it kicking and screaming. And drunk texting. Oh-and a little bit of stalking. To those going through it now I can tell you, you will survive." |
| LaMP-5 | "Generate a title for the following abstract of a paper: Web caching is the process in which web objects are temporarily stored to reduce bandwidth consumption, server load and latency. Web prefetching is the process of fetching web objects from the server before they are actually requested by the client. Integration of caching and prefetching can be very beneficial as the two techniques can support each other. By implementing this integrated scheme in a client-side proxy, the perceived latency can be reduced for not one but many users. In this paper, we propose a new integrated caching and prefetching policy called the WCP-CMA which makes use of a profit-driven caching policy that takes into account the periodicity and cyclic behaviour of the web access sequences for deriving prefetching rules. Our experimental results have shown a 10%-15% increase in the hit ratios of the cached objects and 5%-10% decrease in delay compared to the existing scheme." |
| LaMP-6 | "Generate a subject for the following email: You are receiving this exclusive promotion because you agreed to receive special offers from an emailYOUlike marketing partner. If you have received this email in error or would like to no longer receive these special offers, please follow the instructions at the end of the message. This message is brought to you by emailYOUlike. To find out more about emailYOUlike, visit http:www.emailyoulike.com or write us at 212 Technology Dr., Suite P, Irvine, CA 92618. If you would prefer not to receive future marketing messages from us, click here or visit http://www.emailyoulike.com/remove.asp, enter your email address, and click on the unsubscribe button. Only unsubscribe requests submitted to this page can be fulfilled. emailYOUlike cannot fulfill unsubscribe requests submitted elsewhere or to the email boxes of individuals." |
| LaMP-7 | "Paraphrase the following tweet without any explanation before or after it: I concur with @peyarili that there is animosity, and I believe that the Indian media is exacerbating the situation for the students. It seems as though the foolish media desires conflict with Australia." |

### A.3 Selected LLMs and Profiling

In the previous [section](#A1.SS1), we have explained the edge device constraints. In Table  [2](#A1.T2), we categorize five RAM sizes based on the common edge devices. To run pre-trained LLMs under such RAM constraints, we first need to ensure the size of these LLMs can fit into their corresponding RAM. To satisfy 4G RAM, we pick five models from the family of Pythia [] and Llama []. To satisfy 6G RAM, we pick four models from the family of StableLM [], Pythia [], and Gemma []. To satisfy 8G RAM, we pick six models from the family of Phi [], Pythia, Llama [], Mistral [], Synthia [], and OpenChat [].To satisfy 10G RAM, we pick five models from the family of Llama, StableLM, and Gemma. To satisfy 16G RAM, we pick seven models from the family of Phi, Pythia, Gemma, Llama, and StableLM. The detailed model descriptions can be found in Table [5](#A1.T5). For each model we make profiling to find its peak RAM usage, showing as "Training Peak RAM" on Table  [5](#A1.T5). (^1^11Phi-3 theoretically could fit in a 16G RAM during training, but typical optimizations such as HuggingFace’s trainer require some extra RAM, which will cause overflow issues. Thus we do not include Phi-3 in Table [5](#A1.T5) but we still use the model for some of the experiments.) The model with smaller training peak RAM can also run on larger RAMs.

### A.4 Experimental Settings

For each [LaMP dataset](#A1.SS2), we analyze 100 users, each containing up to 1000 documents in their user history data. Each document encapsulates a single piece of user information, as illustrated in Table [3](#A1.T3). Unless specifically stated otherwise, we set the temperature to 0.1, top_p to 0.9, max_new_tokens to 100, and top_k to 10 for content generation by the LLM.

Retrieval-Augmented Inference (RAG).
RAG operates with two main parts: a retriever that obtains the user-specific documents from his historical data using max inner product search (MIPS), and a generator backed by an LLM. This generator takes both the user query and the retrieved document as inputs to create an informative prompt and generate the corresponding content. For MIPS to function effectively, all history documents must be converted into embeddings via a sentence embedding model. In our experiments, we use all-MiniLM-L6-v2 []. We select the highest-ranked data as the output in MIPS, setting the top_k parameter to 3.

**Table 5: Selected models with their initial model weights size and the training peak RAM. For the data size per hour, it indicates the estimated amount of data each model can learn or train from given one hour.**
| Total RAM | Pretrained LLM (ID from Hugging Face) | Training Peak memory (GB) | Weights Size (GB) | Data Samples per Hour |
| --- | --- | --- | --- | --- |
| 4G | EleutherAI/pythia-70m | 1.15 | 0.17 | 825 |
|  | EleutherAI/pythia-160m | 1.61 | 0.38 | 415 |
|  | EleutherAI/pythia-410m | 2.69 | 0.91 | 215 |
|  | facebook/opt-125m | 1.78 | 0.25 | 550 |
|  | facebook/opt-350m | 2.25 | 0.66 | 320 |
|  | TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ | 1.72 | 0.77 | 110 |
|  | TheBloke/open-llama-3b-v2-GPTQ | 3.39 | 2.09 | 95 |
| 6G | facebook/opt-1.3b | 5.68 | 2.63 | 245 |
|  | TheBloke/stablelm-zephyr-3b-GPTQ | 3.17 | 1.84 | 85 |
|  | TechxGenus/gemma-2b-GPTQ | 5.31 | 2.08 | 145 |
|  | EleutherAI/pythia-1b | 4.98 | 2.09 | 250 |
|  | TinyLlama/TinyLlama-1.1B-step-50K-105b | 5.32 | 2.20 | 165 |
| 8G | microsoft/phi-1_5 | 6.76 | 2.84 | 153 |
|  | EleutherAI/pythia-1.4b | 6.68 | 2.93 | 182 |
|  | TheBloke/Llama-2-7B-Chat-GPTQ | 6.07 | 3.90 | 82 |
|  | TheBloke/Mistral-7B-v0.1-GPTQ | 6.61 | 4.16 | 75 |
|  | TheBloke/Synthia-7B-v1.3-GPTQ | 6.61 | 4.16 | 75 |
|  | TheBloke/openchat_3.5-GPTQ | 6.16 | 4.16 | 76 |
| 10G | princeton-nlp/Sheared-LLaMA-1.3B-Pruned | 6.89 | 2.69 | 222 |
|  | stabilityai/stablelm-2-1_6b | 8.12 | 3.29 | 156 |
|  | TechxGenus/Meta-Llama-3-8B-GPTQ | 8.59 | 5.74 | 85 |
|  | princeton-nlp/Sheared-LLaMA-1.3B | 6.89 | 5.38 | 222 |
|  | facebook/opt-2.7b | 6.89 | 5.30 | 120 |
|  | TechxGenus/gemma-7b-GPTQ | 9.45 | 7.18 | 71 |
|  | TheBloke/Llama-2-13B-GPTQ | 9.95 | 7.26 | 50 |
| 16G | microsoft/phi-2 | 11.97 | 5.00 | 94 |
|  | microsoft/Phi-3-mini-4k-instruct | 13.57 | 7.64 | 84 |
|  | EleutherAI/pythia-2.8b | 12.97 | 5.68 | 106 |
|  | google/gemma-2b | 13.45 | 4.95 | 132 |
|  | princeton-nlp/Sheared-LLaMA-2.7B-Pruned | 12.26 | 5.40 | 89 |
|  | stabilityai/stablelm-3b-4e1t | 12.61 | 5.59 | 100 |
|  | openlm-research/open_llama_3b_v2 | 15.11 | 6.85 | 89 |
|  | princeton-nlp/Sheared-LLaMA-2.7B | 12.26 | 9.95 | 89 |
|  | meta-llama/Meta-Llama-3-8B* | 23.00 | 16.07 | 50 |
| * for comparison with quantized Llama-3-8B |  |  |  |  |
|  |  |  |  |  |

Parameter-Efficient Fine-Tuning (PEFT).
We choose LoRA [] as the PEFT implementation in our work. For LoRA, we enable it to fine-tune parameters for query, key, and value layers. We set the dropout rate for LoRA as 0.1. The initialized learning rate is 5e-4, along with a linear learning rate scheduler to optimize the learning rate. We use adamW as the optimizer in LoRA. Additionally, we set the fine-tuning task type as CAUSAL_LM. For the rank and alpha, since they directly relate to the size of trainable parameters, we set wide ranges. For rank, it can increment from 8 to 256. For alpha, it can increment from 8 to 32. Such ranks allow us to explore different trainable parameter sizes.

### A.5 Model Compression

The LLMs were originally designed and trained for cluster computing []. Their size can easily go beyond the size of edge device RAM size. Furthermore, the larger the model, the longer time it takes to train and make inferences []. Model compression can help such LLMs deploy on edge devices (Edge LLMs) and improve their training and inference efficiency. There are three common implementation methods for model compression: quantization, pruning, and distillation.

Quantization The default precision for model weights in LLMs is FP32 or FP16. By reducing the weight precision from 32 bits down to 3 or 4 bits (quantization), we can significantly decrease the model size and inference time due to simpler gradient computations. However, reducing the number of bits in weights can compromise their precision and, consequently, model performance. The primary goal of quantization is to reduce weight bits without degrading performance. The main strategy for LLM quantization involves quantizing and then calibrating the pre-trained model, a technique known as post-training quantization (PTQ) []. Several implementations of PTQ for LLMs exist, including SmoothQuant [], AWQ [], LLM-QAT [], and GPTQ [].

For experimental consistency, in our experiments, we selected the widely-used GPTQ, which has been applied to almost all LLMs and has proven effective. Furthermore, GPTQ has been specifically adapted for Edge LLMs [, ].

Pruning Different from quantization, pruning lightens the model by trimming off certain weights without lowering the model performance. Compared to quantization, pruning receives less attention in the beginning. One reason can be the high weight complexity of LLMs which makes it difficult to prune them while maintaining their performance. Furthermore, since the core modules of LLMs, the transformer blocks, make the structures of LLMs complex, pruning the LLMs can be more challenging than pruning on pure neural networks with simple structures. Within the existing pruning works including Wanda [], LLM-Pruner [], and Sheared-LLaMA [], Sheared-LLaMA has demonstrated decent performance with published LLM weights.

To simplify the experiments and avoid potential issues with implementing Wanda or LLM-Pruner to prune LLMs, we use Sheared-LLaMA in our experiments.

Distillation Other than quantization and pruning, another method involves using a smaller model to emulate and learn from a larger model, a process known as distillation. Microsoft introduced a small language model (SLM) called Phi, which leverages synthetic datasets specifically created by GPT-3.5 to learn domains such as common sense reasoning, general knowledge, science, daily activities, and theory of mind []. This approach represents a novel departure from traditional large language models (LLMs), allowing SLMs to acquire dense knowledge from LLMs while maintaining a smaller size. Although other works such as Distilling Step-by-Step [], MetaIE [], and MLFS [] have proposed distillation methods, they have not yielded a well-trained model comparable to Phi, which benefited from Microsoft’s extensive resources and data. Therefore, we have chosen to use Phi in our experiments.

### A.6 Task Difficulty

**Table 6: Difficulty (weighted average) comparisons of different datasets. The mean of four cloud-based LLMs’ zero-shot performance can be used as an evaluator of task difficulty.**
| Dataset | GPT-4 | Claude 3 Opus | Gemini 1.0 Pro | Llama 3 70B | Average | Normalized Accuracy | Task Type |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LaMP-1 (Accuracy) | 0.539 | 0.539 | 0.490 | 0.422 | 0.4975 | 0.995 | Classification |
| LaMP-2 (Accuracy) | 0.355 | 0.320 | 0.300 | 0.400 | 0.3438 | 5.157 | Classification |
| LaMP-3 (Accuracy) | 0.667 | 0.657 | 0.510 | 0.755 | 0.6472 | 3.236 | Classification |
| LaMP-4 (ROUGE-1) | 0.143 | 0.171 | 0.139 | 0.131 | 0.1460 | 0.1460 | Generation |
| LaMP-5 (ROUGE-1) | 0.386 | 0.374 | 0.405 | 0.084 | 0.3123 | 0.3123 | Generation |
| LaMP-6 (ROUGE-1) | 0.351 | 0.356 | 0.405 | 0.278 | 0.3475 | 0.3475 | Generation |
| LaMP-7 (ROUGE-1) | 0.326 | 0.136 | 0.237 | 0.255 | 0.2385 | 0.2385 | Generation |
|  |  |  |  |  |  |  |  |

## Appendix B Results of Edge LLM Learning

### B.1 Experimental Results for PEFT

#### B.1.1 Experiments: model size and training time

The following results shown in Figure  [8](#A2.F8) and Figure  [9](#A2.F9) correspond to the question "Under the constraint of edge devices, does longer training time always result in better learning outcomes?" in experiment analysis section  [3.3](#S3.SS3).

Figure: (a) Performance on LaMP-1
Refer to caption: /html/2406.03777/assets/x57.png

Figure: (a) Performance on LaMP-1
Refer to caption: /html/2406.03777/assets/x64.png

#### B.1.2 Experiments: Trainable parameters based on fixed alpha and varying rank

The experiments for fixed value of alpha to 32 and changing value of rank from 8 to 80 have results shown in Figure  [10](#A2.F10), Figure  [11](#A2.F11), Figure  [12](#A2.F12), Figure  [13](#A2.F13) correspond to the question "Does larger portion of trainable parameters conduct better performance?" in experiment analysis section  [3.2](#S3.SS2).

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x71.png

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x78.png

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x85.png

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x92.png

#### B.1.3 Experiments: Trainable parameters based on varying aligned alpha and rank

The experiments for aligning alpha and rank and changing their values from 8 to 80 have results shown in Figure  [14](#A2.F14), Figure  [15](#A2.F15), Figure  [16](#A2.F16), Figure  [17](#A2.F17) correspond to the question "Does larger portion of trainable parameters conduct better performance?" in experiment analysis section  [3.2](#S3.SS2).

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x99.png

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x106.png

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x113.png

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x120.png

#### B.1.4 Experiments: Combinations of several commonly used alpha and rank

The experiments for different combinations of commonly used alpha and rank values as shown in Figure  [18](#A2.F18) and Figure  [19](#A2.F19). These experiments can correspond to the question "Does larger portion of trainable parameters conduct better performance?" in experiment analysis section  [3.2](#S3.SS2).

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x127.png

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x134.png

### B.2 Experimental Results for RAG

#### B.2.1 Experiments: RAG performance on different sizes of user history data

Below, we present the experiments for the impact of the amount of user history data on RAG performance. These experiments can correspond to the question "What is the threshold of the user history data volume to ensure a good RAG performance?" in the section  [3.4](#S3.SS4)

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x141.png

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x148.png

#### B.2.2 Experiments: Examine RAG and PEFT

**Table 7: Performance comparisons of PEFT and RAG for selected LLMs. For PEFT we use the default experimental settings and let rank = 8, alpha =16. Additionally, we keep the training hours to 8 so the model can fully learn from the user history data. The PEFT performance compares with RAG. Their zero-shot learning performance can be found in Table [8](#A2.T8).**
| LLM | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | FT | RAG | FT | RAG | FT | RAG | FT | RAG | FT | RAG | FT | RAG | FT | RAG |
| Pythia-70m | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Pythia-160m | 0.019 | 0.000 | 0.000 | 0.000 | 0.647 | 0.000 | 0.054 | 0.000 | 0.076 | 0.000 | 0.001 | 0.000 | 0.011 | 0.000 |
| Pythia-410m | 0.275 | 0.500 | 0.460 | 0.010 | 0.725 | 0.657 | 0.119 | 0.055 | 0.198 | 0.157 | 0.253 | 0.182 | 0.128 | 0.156 |
| Pythia-1b | 0.039 | 0.500 | 0.465 | 0.185 | 0.716 | 0.578 | 0.109 | 0.051 | 0.211 | 0.165 | 0.273 | 0.221 | 0.096 | 0.169 |
| Pythia-1.4b | 0.559 | 0.490 | 0.470 | 0.510 | 0.716 | 0.667 | 0.118 | 0.061 | 0.216 | 0.187 | 0.281 | 0.253 | 0.128 | 0.163 |
| Pythia-2.8b | 0.500 | 0.451 | 0.380 | 0.320 | 0.716 | 0.716 | 0.129 | 0.065 | 0.207 | 0.220 | 0.285 | 0.186 | 0.154 | 0.170 |
| opt-125m | 0.049 | 0.284 | 0.535 | 0.135 | 0.725 | 0.588 | 0.113 | 0.033 | 0.189 | 0.135 | 0.121 | 0.212 | 0.211 | 0.187 |
| opt-350m | 0.196 | 0.500 | 0.470 | 0.260 | 0.735 | 0.569 | 0.110 | 0.039 | 0.204 | 0.171 | 0.091 | 0.236 | 0.156 | 0.199 |
| opt-1.3b | 0.363 | 0.373 | 0.475 | 0.270 | 0.627 | 0.657 | 0.120 | 0.057 | 0.199 | 0.175 | 0.123 | 0.112 | 0.121 | 0.173 |
| opt-2.7b | 0.010 | 0.520 | 0.480 | 0.110 | 0.627 | 0.696 | 0.127 | 0.060 | 0.222 | 0.184 | 0.155 | 0.134 | 0.168 | 0.156 |
| TinyLlama-1.1b | 0.500 | 0.520 | 0.000 | 0.085 | 0.775 | 0.618 | 0.088 | 0.051 | 0.182 | 0.150 | 0.158 | 0.081 | 0.181 | 0.206 |
| Llama-v2-3b | 0.320 | 0.569 | 0.430 | 0.295 | 0.765 | 0.461 | 0.129 | 0.026 | 0.252 | 0.136 | 0.264 | 0.253 | 0.204 | 0.250 |
| Llama-v3-8b | 0.324 | 0.000 | 0.130 | 0.000 | 0.627 | 0.000 | 0.065 | 0.000 | 0.180 | 0.000 | 0.172 | 0.000 | 0.189 | 0.000 |
| StableLM-2-1.6b | 0.480 | 0.390 | 0.440 | 0.210 | 0.755 | 0.765 | 0.118 | 0.098 | 0.181 | 0.208 | 0.265 | 0.276 | 0.097 | 0.200 |
| StableLM-3b | 0.529 | 0.520 | 0.480 | 0.245 | 0.784 | 0.667 | 0.136 | 0.094 | 0.246 | 0.248 | 0.280 | 0.256 | 0.108 | 0.166 |
| Gemma-2b | 0.471 | 0.010 | 0.430 | 0.205 | 0.784 | 0.765 | 0.119 | 0.079 | 0.239 | 0.245 | 0.295 | 0.299 | 0.201 | 0.270 |
| Phi-1.5 | 0.255 | 0.451 | 0.405 | 0.270 | 0.696 | 0.422 | 0.119 | 0.068 | 0.252 | 0.203 | 0.265 | 0.197 | 0.181 | 0.270 |
| Phi-2 | 0.206 | 0.529 | 0.450 | 0.340 | 0.745 | 0.627 | 0.133 | 0.090 | 0.243 | 0.241 | 0.241 | 0.208 | 0.220 | 0.197 |
| Phi-3 | 0.471 | 0.333 | 0.470 | 0.345 | 0.784 | 0.647 | 0.103 | 0.048 | 0.211 | 0.242 | 0.198 | 0.285 | 0.199 | 0.158 |
| Llama-v2-3b-G(^1^11G represents GPTQ, the quantization technique) | 0.520 | 0.559 | 0.470 | 0.305 | 0.784 | 0.706 | 0.113 | 0.095 | 0.234 | 0.215 | 0.275 | 0.218 | 0.220 | 0.151 |
| StableLM-3b-G | 0.490 | 0.559 | 0.440 | 0.390 | 0.725 | 0.598 | 0.118 | 0.082 | 0.234 | 0.251 | 0.226 | 0.258 | 0.200 | 0.144 |
| TinyLlama-1.1B-G | 0.265 | 0.422 | 0.410 | 0.120 | 0.765 | 0.588 | 0.115 | 0.057 | 0.225 | 0.182 | 0.173 | 0.112 | 0.197 | 0.171 |
| Gemma-2b-G | 0.167 | 0.461 | 0.490 | 0.275 | 0.814 | 0.716 | 0.124 | 0.091 | 0.240 | 0.247 | 0.201 | 0.273 | 0.227 | 0.169 |
| Llama-v2-7b-G | 0.451 | 0.520 | 0.495 | 0.365 | 0.755 | 0.657 | 0.000 | 0.105 | 0.000 | 0.278 | 0.000 | 0.275 | 0.249 | 0.157 |
| Llama-v3-8b-G | 0.539 | 0.000 | 0.140 | 0.000 | 0.461 | 0.000 | 0.073 | 0.000 | 0.229 | 0.000 | 0.174 | 0.000 | 0.277 | 0.000 |
| Mistral-7b-G | 0.529 | 0.529 | 0.485 | 0.375 | 0.814 | 0.755 | 0.128 | 0.097 | 0.255 | 0.274 | 0.296 | 0.327 | 0.199 | 0.181 |
| OpenChat-3.5-G | 0.539 | 0.520 | 0.460 | 0.290 | 0.647 | 0.814 | 0.129 | 0.113 | 0.227 | 0.279 | 0.304 | 0.355 | 0.290 | 0.231 |
| Synthia-7b-G | 0.451 | 0.529 | 0.480 | 0.365 | 0.725 | 0.549 | 0.123 | 0.088 | 0.244 | 0.302 | 0.301 | 0.338 | 0.263 | 0.213 |
| Gemma-7b-G | 0.529 | 0.539 | 0.420 | 0.365 | 0.775 | 0.480 | 0.105 | 0.048 | 0.213 | 0.184 | 0.198 | 0.223 | 0.030 | 0.126 |
| Llama-13b-G | 0.529 | 0.569 | 0.465 | 0.350 | 0.804 | 0.657 | 0.143 | 0.106 | 0.259 | 0.264 | 0.253 | 0.303 | 0.121 | 0.138 |
| S(^3^33S represents Sheared)-Llama-1.3b-P(^2^22P represents Pruned) | 0.069 | 0.049 | 0.410 | 0.100 | 0.686 | 0.569 | 0.090 | 0.040 | 0.215 | 0.157 | 0.268 | 0.195 | 0.096 | 0.178 |
| S-Llama-1.3B | 0.461 | 0.471 | 0.510 | 0.255 | 0.784 | 0.559 | 0.113 | 0.064 | 0.205 | 0.197 | 0.247 | 0.234 | 0.126 | 0.183 |
| S-Llama-2.7B-P | 0.167 | 0.120 | 0.455 | 0.035 | 0.725 | 0.578 | 0.100 | 0.051 | 0.221 | 0.207 | 0.319 | 0.240 | 0.124 | 0.188 |
| S-Llama-2.7B | 0.294 | 0.529 | 0.415 | 0.250 | 0.775 | 0.627 | 0.108 | 0.069 | 0.216 | 0.242 | 0.218 | 0.260 | 0.234 | 0.181 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

**Table 8: Performance comparisons of zero-shot learning for all selected LLMs.**
| LLM | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Pythia-70m | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Pythia-160m | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Pythia-410m | 0.420 | 0.000 | 0.441 | 0.034 | 0.103 | 0.045 | 0.112 |
| Pythia-1b | 0.420 | 0.000 | 0.343 | 0.028 | 0.134 | 0.047 | 0.115 |
| Pythia-1.4b | 0.400 | 0.000 | 0.373 | 0.033 | 0.120 | 0.000 | 0.058 |
| Pythia-2.8b | 0.429 | 0.000 | 0.598 | 0.035 | 0.128 | 0.053 | 0.107 |
| opt-125m | 0.039 | 0.000 | 0.304 | 0.018 | 0.098 | 0.034 | 0.096 |
| opt-350m | 0.429 | 0.000 | 0.255 | 0.021 | 0.106 | 0.056 | 0.116 |
| opt-1.3b | 0.429 | 0.000 | 0.284 | 0.033 | 0.130 | 0.041 | 0.102 |
| opt-2.7b | 0.429 | 0.000 | 0.451 | 0.025 | 0.113 | 0.051 | 0.152 |
| TinyLlama-1.1b | 0.471 | 0.000 | 0.588 | 0.032 | 0.088 | 0.029 | 0.143 |
| Llama-v2-3b | 0.469 | 0.000 | 0.461 | 0.026 | 0.136 | 0.053 | 0.150 |
| Llama-v3-8b | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| StableLM-2-1.6b | 0.449 | 0.000 | 0.735 | 0.033 | 0.138 | 0.066 | 0.135 |
| StableLM-3b | 0.429 | 0.000 | 0.441 | 0.040 | 0.177 | 0.067 | 0.257 |
| Gemma-2b | 0.078 | 0.000 | 0.696 | 0.025 | 0.145 | 0.077 | 0.130 |
| Phi-1.5 | 0.500 | 0.000 | 0.382 | 0.023 | 0.094 | 0.056 | 0.062 |
| Phi-2 | 0.429 | 0.010 | 0.471 | 0.064 | 0.096 | 0.059 | 0.259 |
| Phi-3 | 0.441 | 0.000 | 0.549 | 0.049 | 0.108 | 0.104 | 0.098 |
| Llama-v2-3b-G(^1^11G represents GPTQ, the quantization technique) | 0.402 | 0.000 | 0.647 | 0.041 | 0.173 | 0.065 | 0.158 |
| StableLM-3b-G | 0.490 | 0.005 | 0.559 | 0.049 | 0.225 | 0.098 | 0.166 |
| TinyLlama-1.1B-G | 0.059 | 0.000 | 0.176 | 0.026 | 0.069 | 0.043 | 0.109 |
| Gemma-2b-G | 0.402 | 0.000 | 0.676 | 0.026 | 0.114 | 0.051 | 0.107 |
| Llama-v2-7b-G | 0.429 | 0.275 | 0.696 | 0.048 | 0.190 | 0.147 | 0.181 |
| Llama-v3-8b-G | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| Mistral-7b-G | 0.429 | 0.000 | 0.343 | 0.034 | 0.210 | 0.079 | 0.166 |
| OpenChat-3.5-G | 0.480 | 0.000 | 0.735 | 0.052 | 0.182 | 0.134 | 0.202 |
| Synthia-7b-G | 0.500 | 0.000 | 0.373 | 0.033 | 0.120 | 0.000 | 0.271 |
| Gemma-7b-G | 0.402 | 0.000 | 0.676 | 0.026 | 0.114 | 0.051 | 0.112 |
| Llama-13b-G | 0.429 | 0.000 | 0.696 | 0.027 | 0.252 | 0.068 | 0.174 |
| S(^3^33S represents Sheared)-Llama-1.3b-P(^2^22P represents Pruned) | 0.294 | 0.000 | 0.255 | 0.028 | 0.111 | 0.038 | 0.176 |
| S-Llama-1.3B | 0.429 | 0.000 | 0.206 | 0.030 | 0.107 | 0.029 | 0.122 |
| S-Llama-2.7B-P | 0.010 | 0.000 | 0.422 | 0.027 | 0.128 | 0.043 | 0.124 |
| S-Llama-2.7B | 0.429 | 0.000 | 0.549 | 0.029 | 0.174 | 0.074 | 0.118 |
|  |  |  |  |  |  |  |  |

## Appendix C Results of Edge LLM Design

### C.1 Experiments: quantization models

Experiments about quantization models only. These experiments can correspond to section  [3.5](#S3.SS5). We first examine quantized Gemma-2b and Gemma-7b models, showing results in Table  [9](#A3.T9) and Table  [10](#A3.T10). Then, we examine the quantized Llama family models, showing results in Figure  [22](#A3.F22). Additionally, we examine the quantized models with similar size but different architectures, showing results in Figure  [23](#A3.F23).

**Table 9: Performance comparisons between two quantized models with different sizes after training 8 hours.**
| Gemma-GPTQ | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 2b | 7b | 2b | 7b | 2b | 7b | 2b | 7b | 2b | 7b | 2b | 7b | 2b | 7b |
| R(8) A(8) | 0.520 | 0.529 | 0.490 | 0.420 | 0.814 | 0.775 | 0.124 | 0.105 | 0.240 | 0.213 | 0.201 | 0.244 | 0.227 | 0.030 |
| R(8) A(16) | 0.408 | 0.451 | 0.455 | 0.440 | 0.794 | 0.784 | 0.122 | 0.099 | 0.240 | 0.184 | 0.220 | 0.220 | 0.289 | 0.032 |
| R(8) A(32) | 0.475 | 0.304 | 0.440 | 0.425 | 0.725 | 0.784 | 0.119 | 0.095 | 0.209 | 0.187 | 0.241 | 0.228 | 0.231 | 0.093 |
| R(16) A(16) | 0.570 | 0.441 | 0.490 | 0.400 | 0.775 | 0.833 | 0.125 | 0.099 | 0.226 | 0.203 | 0.267 | 0.245 | 0.209 | 0.032 |
| R(16) A(32) | 0.545 | 0.510 | 0.470 | 0.430 | 0.765 | 0.725 | 0.129 | 0.092 | 0.221 | 0.228 | 0.224 | 0.263 | 0.300 | 0.052 |
| R(32) A(32) | 0.421 | 0.441 | 0.505 | 0.415 | 0.755 | 0.755 | 0.113 | 0.096 | 0.202 | 0.200 | 0.231 | 0.221 | 0.186 | 0.088 |
| R(64) A(32) | 0.520 | 0.480 | 0.455 | 0.435 | 0.745 | 0.745 | 0.108 | 0.094 | 0.214 | 0.190 | 0.229 | 0.208 | 0.190 | 0.090 |
| R(256) A(32) | 0.480 | 0.520 | 0.435 | 0.440 | 0.765 | 0.755 | 0.114 | 0.094 | 0.203 | 0.199 | 0.218 | 0.231 | 0.263 | 0.051 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

**Table 10: Performance comparisons between two quantized models with different normalized data sizes under the LoRA setting: rank = 8 and alpha = 32.**
| Gemma-GPTQ | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 2b | 7b | 2b | 7b | 2b | 7b | 2b | 7b | 2b | 7b | 2b | 7b | 2b | 7b |
| Training 1 Hour | 0.469 | 0.480 | 0.265 | 0.240 | 0.745 | 0.706 | 0.094 | 0.074 | 0.223 | 0.222 | 0.119 | 0.160 | 0.300 | 0.037 |
| Training 2 Hours | 0.445 | 0.480 | 0.310 | 0.285 | 0.725 | 0.765 | 0.104 | 0.076 | 0.201 | 0.204 | 0.155 | 0.171 | 0.283 | 0.036 |
| Training 3 Hours | 0.520 | 0.471 | 0.360 | 0.365 | 0.784 | 0.735 | 0.105 | 0.073 | 0.226 | 0.207 | 0.205 | 0.186 | 0.294 | 0.048 |
| Training 4 Hours | 0.410 | 0.186 | 0.415 | 0.350 | 0.784 | 0.775 | 0.116 | 0.088 | 0.205 | 0.200 | 0.169 | 0.195 | 0.263 | 0.036 |
| Training 5 Hours | 0.441 | 0.373 | 0.450 | 0.305 | 0.775 | 0.765 | 0.122 | 0.089 | 0.208 | 0.215 | 0.171 | 0.188 | 0.294 | 0.023 |
| Training 6 Hours | 0.429 | 0.480 | 0.470 | 0.360 | 0.765 | 0.716 | 0.110 | 0.079 | 0.200 | 0.195 | 0.226 | 0.205 | 0.244 | 0.090 |
| Training 7 Hours | 0.408 | 0.480 | 0.465 | 0.395 | 0.775 | 0.775 | 0.114 | 0.099 | 0.243 | 0.201 | 0.208 | 0.177 | 0.277 | 0.061 |
| Training 8 Hours | 0.421 | 0.304 | 0.440 | 0.425 | 0.725 | 0.784 | 0.119 | 0.095 | 0.209 | 0.187 | 0.241 | 0.228 | 0.231 | 0.093 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x155.png

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x162.png

### C.2 Evaluation LLMs Pruning

Experiments to investigate the pruned models: Sheared-Llama-1.3b-Prune and Sheared-Llama-2.7b-Prune. These experiments can correspond to the question "Quantization, pruning, and knowledge distillation: what are their pros and cons? can we identify the best way to compress the model?" in section  [3.5](#S3.SS5) and the question "Do we need to compress the model or can we simply use their original version? What factors can impact our decision?" in section  [3.6](#S3.SS6).

We first examine Sheared-Llama-1.3b-Prune and Sheared-Llama-1.3b across different data size and different lora setting. The results can be seen in Table  [11](#A3.T11) and Table  [12](#A3.T12). Then, we examine Sheared-Llama-1.3b-Prune vs Sheared-Llama-1.3b vs TinyLlama-1.1b. The results can be seen in Figure  [24](#A3.F24). Additionally, we examine Sheared-Llama-2.7b-Prune vs Sheared-Llama-2.7b across different data size and different lora setting. The results can be seen in Table  [13](#A3.T13) and Table  [14](#A3.T14). Finally, we examine Sheared-Llama-2.7b-Prune vs Sheared-Llama-2.7b vs Sheared-Llama-1.3b-Prune vs Sheared-Llama-1.3b vs Llama-3b vs Llama-3b-gptq. The results can be seen in Figure  [25](#A3.F25).

**Table 11: Performance comparisons between pruned (+P) and unpruned (-P) models with different LoRA settings under normalized data size of 8.**
| Llama-1.1b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | -P(^2^22P represents Pruning) | +P | -P | +P | -P | +P | -P | +P | -P | +P | -P | +P | -P | +P |
| R(8) A(8) | 0.461 | 0.069 | 0.510 | 0.410 | 0.784 | 0.686 | 0.113 | 0.090 | 0.205 | 0.215 | 0.247 | 0.268 | 0.126 | 0.096 |
| R(8) A(16) | 0.245 | 0.118 | 0.480 | 0.445 | 0.765 | 0.676 | 0.099 | 0.097 | 0.193 | 0.188 | 0.298 | 0.297 | 0.129 | 0.071 |
| R(8) A(32) | 0.010 | 0.186 | 0.500 | 0.450 | 0.725 | 0.696 | 0.093 | 0.091 | 0.221 | 0.184 | 0.291 | 0.281 | 0.125 | 0.084 |
| R(16) A(16) | 0.108 | 0.108 | 0.475 | 0.430 | 0.725 | 0.706 | 0.099 | 0.084 | 0.231 | 0.184 | 0.264 | 0.296 | 0.119 | 0.079 |
| R(16) A(32) | 0.029 | 0.098 | 0.520 | 0.445 | 0.784 | 0.657 | 0.098 | 0.079 | 0.194 | 0.204 | 0.258 | 0.305 | 0.154 | 0.065 |
| R(32) A(32) | 0.000 | 0.059 | 0.480 | 0.455 | 0.804 | 0.686 | 0.103 | 0.083 | 0.208 | 0.212 | 0.280 | 0.268 | 0.098 | 0.074 |
| R(64) A(32) | 0.108 | 0.157 | 0.535 | 0.435 | 0.735 | 0.706 | 0.105 | 0.093 | 0.237 | 0.212 | 0.263 | 0.297 | 0.099 | 0.084 |
| R(256) A(32) | 0.020 | 0.137 | 0.485 | 0.440 | 0.784 | 0.686 | 0.107 | 0.079 | 0.232 | 0.191 | 0.288 | 0.287 | 0.114 | 0.100 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

**Table 12: Performance comparisons between pruned (+P) and unpruned (-P) models given LoRA setting where rank = 8 and alpha = 32, under training hours from 1 to 8.**
| Llama-1.1b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | -P | +P | -P | +P | -P | +P | -P | +P | -P | +P | -P | +P | -P | +P |
| Training 1 Hour | 0.069 | 0.020 | 0.325 | 0.220 | 0.735 | 0.735 | 0.091 | 0.058 | 0.239 | 0.188 | 0.172 | 0.170 | 0.192 | 0.042 |
| Training 2 Hours | 0.000 | 0.039 | 0.380 | 0.335 | 0.755 | 0.667 | 0.080 | 0.074 | 0.259 | 0.188 | 0.169 | 0.190 | 0.139 | 0.060 |
| Training 3 Hours | 0.059 | 0.069 | 0.445 | 0.370 | 0.725 | 0.755 | 0.114 | 0.071 | 0.233 | 0.176 | 0.252 | 0.232 | 0.134 | 0.126 |
| Training 4 Hours | 0.049 | 0.069 | 0.490 | 0.400 | 0.843 | 0.755 | 0.103 | 0.083 | 0.191 | 0.186 | 0.264 | 0.260 | 0.126 | 0.080 |
| Training 5 Hours | 0.049 | 0.049 | 0.495 | 0.395 | 0.775 | 0.716 | 0.107 | 0.076 | 0.208 | 0.208 | 0.234 | 0.238 | 0.141 | 0.086 |
| Training 6 Hours | 0.000 | 0.059 | 0.480 | 0.415 | 0.784 | 0.647 | 0.098 | 0.089 | 0.204 | 0.213 | 0.313 | 0.288 | 0.146 | 0.089 |
| Training 7 Hours | 0.010 | 0.078 | 0.505 | 0.440 | 0.716 | 0.686 | 0.097 | 0.087 | 0.230 | 0.196 | 0.323 | 0.293 | 0.113 | 0.097 |
| Training 8 Hours | 0.010 | 0.186 | 0.500 | 0.450 | 0.725 | 0.696 | 0.093 | 0.091 | 0.221 | 0.184 | 0.291 | 0.281 | 0.125 | 0.084 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x169.png

**Table 13: Performance comparisons between pruned (+P) and unpruned (-P) models with different LoRA settings under training hours of 8.**
| Sheared-Llama-2.7b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | -P(^3^33P represents Pruning) | +P | -P | +P | -P | +P | -P | +P | -P | +P | -P | +P | -P | +P |
| R(8) A(8) | 0.020 | 0.167 | 0.480 | 0.455 | 0.735 | 0.725 | 0.112 | 0.100 | 0.213 | 0.221 | 0.278 | 0.319 | 0.001 | 0.124 |
| R(8) A(16) | 0.069 | 0.245 | 0.490 | 0.455 | 0.784 | 0.745 | 0.109 | 0.089 | 0.201 | 0.206 | 0.286 | 0.285 | 0.000 | 0.116 |
| R(8) A(32) | 0.078 | 0.059 | 0.495 | 0.450 | 0.755 | 0.765 | 0.112 | 0.089 | 0.208 | 0.217 | 0.332 | 0.301 | 0.147 | 0.133 |
| R(16) A(16) | 0.029 | 0.137 | 0.495 | 0.445 | 0.775 | 0.755 | 0.110 | 0.091 | 0.181 | 0.205 | 0.296 | 0.302 | 0.178 | 0.117 |
| R(16) A(32) | 0.039 | 0.314 | 0.485 | 0.460 | 0.775 | 0.745 | 0.103 | 0.085 | 0.223 | 0.209 | 0.316 | 0.302 | 0.151 | 0.100 |
| R(32) A(32) | 0.029 | 0.088 | 0.510 | 0.445 | 0.784 | 0.755 | 0.109 | 0.102 | 0.227 | 0.213 | 0.329 | 0.312 | 0.183 | 0.093 |
| R(64) A(32) | 0.039 | 0.049 | 0.495 | 0.435 | 0.725 | 0.716 | 0.103 | 0.086 | 0.208 | 0.222 | 0.307 | 0.314 | 0.144 | 0.071 |
| R(256) A(32) | 0.020 | 0.108 | 0.480 | 0.420 | 0.784 | 0.755 | 0.112 | 0.080 | 0.227 | 0.219 | 0.294 | 0.327 | 0.184 | 0.082 |

**Table 14: Performance comparisons between pruned (+P) and unpruned (-P) models given LoRA setting where rank = 8 and alpha = 32, under training hours from 1 to 8.**
| Sheared-Llama-1.3b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | -P | +P | -P | +P | -P | +P | -P | +P | -P | +P | -P | +P | -P | +P |
| Training 1 Hour | 0.010 | 0.039 | 0.320 | 0.200 | 0.706 | 0.676 | 0.097 | 0.074 | 0.212 | 0.241 | 0.180 | 0.093 | 0.000 | 0.128 |
| Training 2 Hours | 0.059 | 0.137 | 0.410 | 0.285 | 0.765 | 0.755 | 0.100 | 0.079 | 0.232 | 0.219 | 0.205 | 0.211 | 0.000 | 0.113 |
| Training 3 Hours | 0.069 | 0.010 | 0.400 | 0.345 | 0.804 | 0.765 | 0.101 | 0.072 | 0.220 | 0.234 | 0.219 | 0.193 | 0.000 | 0.087 |
| Training 4 Hours | 0.294 | 0.039 | 0.415 | 0.395 | 0.775 | 0.775 | 0.108 | 0.070 | 0.216 | 0.210 | 0.218 | 0.185 | 0.234 | 0.131 |
| Training 5 Hours | 0.029 | 0.137 | 0.440 | 0.460 | 0.755 | 0.784 | 0.101 | 0.087 | 0.204 | 0.216 | 0.257 | 0.227 | 0.128 | 0.069 |
| Training 6 Hours | 0.020 | 0.069 | 0.475 | 0.440 | 0.755 | 0.686 | 0.104 | 0.083 | 0.207 | 0.188 | 0.278 | 0.246 | 0.168 | 0.123 |
| Training 7 Hours | 0.000 | 0.078 | 0.505 | 0.470 | 0.755 | 0.794 | 0.112 | 0.093 | 0.202 | 0.209 | 0.311 | 0.304 | 0.172 | 0.108 |
| Training 8 Hours | 0.078 | 0.059 | 0.495 | 0.450 | 0.755 | 0.765 | 0.112 | 0.089 | 0.208 | 0.217 | 0.332 | 0.301 | 0.147 | 0.133 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

Figure: (a) LaMP-1
Refer to caption: /html/2406.03777/assets/x176.png

### C.3 Evaluation LLMs Distillation

The experiments examine distillation LLM including Phi-1.5, Phi-2, and Phi-3 across different data size and different lora setting. The results can be seen in Table  [15](#A3.T15) and Table  [16](#A3.T16). These experiments can correspond to section  [3.5](#S3.SS5).

**Table 15: Performance comparisons between pruned (+P) and unpruned (-P) models with different LoRA settings under training hours of 8.**
|  |  | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Phi-1.5 | R(8) A(8) | 0.255 | 0.405 | 0.696 | 0.119 | 0.252 | 0.265 | 0.181 |
|  | R(8) A(16) | 0.500 | 0.430 | 0.686 | 0.115 | 0.250 | 0.261 | 0.159 |
|  | R(8) A(32) | 0.441 | 0.435 | 0.716 | 0.119 | 0.224 | 0.280 | 0.112 |
|  | R(16) A(16) | 0.490 | 0.410 | 0.706 | 0.115 | 0.237 | 0.277 | 0.133 |
|  | R(16) A(32) | 0.137 | 0.440 | 0.706 | 0.114 | 0.247 | 0.248 | 0.171 |
|  | R(32) A(32) | 0.490 | 0.420 | 0.706 | 0.120 | 0.256 | 0.243 | 0.148 |
|  | R(64) A(32) | 0.441 | 0.435 | 0.706 | 0.116 | 0.220 | 0.255 | 0.104 |
|  | R(256) A(32) | 0.402 | 0.430 | 0.696 | 0.117 | 0.229 | 0.273 | 0.139 |
| Phi-2 | R(8) A(8) | 0.206 | 0.450 | 0.745 | 0.133 | 0.243 | 0.241 | 0.220 |
|  | R(8) A(16) | 0.529 | 0.445 | 0.725 | 0.132 | 0.237 | 0.281 | 0.221 |
|  | R(8) A(32) | 0.520 | 0.465 | 0.765 | 0.137 | 0.264 | 0.273 | 0.208 |
|  | R(16) A(16) | 0.373 | 0.435 | 0.745 | 0.130 | 0.253 | 0.254 | 0.219 |
|  | R(16) A(32) | 0.529 | 0.455 | 0.765 | 0.141 | 0.259 | 0.285 | 0.208 |
|  | R(32) A(32) | 0.529 | 0.455 | 0.775 | 0.137 | 0.247 | 0.297 | 0.209 |
|  | R(64) A(32) | 0.529 | 0.450 | 0.784 | 0.134 | 0.240 | 0.290 | 0.215 |
|  | R(256) A(32) | 0.520 | 0.440 | 0.745 | 0.135 | 0.232 | 0.268 | 0.191 |
| Phi-3 | R(8) A(8) | 0.471 | 0.470 | 0.784 | 0.103 | 0.211 | 0.198 | 0.199 |
|  | R(8) A(16) | 0.529 | 0.450 | 0.804 | 0.102 | 0.209 | 0.248 | 0.176 |
|  | R(8) A(32) | 0.520 | 0.470 | 0.794 | 0.109 | 0.224 | 0.221 | 0.159 |
|  | R(16) A(16) | 0.549 | 0.455 | 0.833 | 0.100 | 0.210 | 0.226 | 0.142 |
|  | R(16) A(32) | 0.529 | 0.465 | 0.745 | 0.098 | 0.210 | 0.222 | 0.197 |
|  | R(32) A(32) | 0.216 | 0.505 | 0.775 | 0.094 | 0.218 | nan | 0.125 |
|  | R(64) A(32) | 0.255 | 0.480 | 0.804 | 0.090 | 0.233 | 0.125 | 0.191 |
|  | R(256) A(32) | 0.402 | 0.490 | 0.833 | 0.094 | 0.252 | 0.129 | 0.127 |
|  |  |  |  |  |  |  |  |  |

**Table 16: Performance comparisons between pruned (+P) and unpruned (-P) models given LoRA setting where rank = 8 and alpha = 32, under training hours from 1 to 8.**
|  |  | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Phi-1.5 | Training 1 Hour | 0.108 | 0.255 | 0.696 | 0.084 | 0.219 | 0.163 | 0.166 |
|  | Training 2 Hours | 0.010 | 0.280 | 0.676 | 0.095 | 0.226 | 0.178 | 0.165 |
|  | Training 3 Hours | 0.108 | 0.390 | 0.716 | 0.100 | 0.249 | 0.180 | 0.135 |
|  | Training 4 Hours | 0.441 | 0.370 | 0.686 | 0.106 | 0.239 | 0.226 | 0.163 |
|  | Training 5 Hours | 0.382 | 0.415 | 0.676 | 0.119 | 0.218 | 0.234 | 0.098 |
|  | Training 6 Hours | 0.216 | 0.380 | 0.716 | 0.116 | 0.232 | 0.258 | 0.145 |
|  | Training 7 Hours | 0.480 | 0.435 | 0.716 | 0.114 | 0.247 | 0.285 | 0.095 |
|  | Training 8 Hours | 0.441 | 0.435 | 0.716 | 0.119 | 0.224 | 0.280 | 0.112 |
| Phi-2 | Training 1 Hour | 0.539 | 0.295 | 0.706 | 0.108 | 0.228 | 0.156 | 0.179 |
|  | Training 2 Hours | 0.529 | 0.315 | 0.735 | 0.116 | 0.249 | 0.157 | 0.235 |
|  | Training 3 Hours | 0.529 | 0.330 | 0.706 | 0.129 | 0.228 | 0.217 | 0.156 |
|  | Training 4 Hours | 0.529 | 0.350 | 0.765 | 0.126 | 0.204 | 0.190 | 0.211 |
|  | Training 5 Hours | 0.529 | 0.375 | 0.794 | 0.126 | 0.236 | 0.210 | 0.164 |
|  | Training 6 Hours | 0.529 | 0.420 | 0.765 | 0.136 | 0.251 | 0.249 | 0.164 |
|  | Training 7 Hours | 0.471 | 0.425 | 0.755 | 0.127 | 0.231 | 0.274 | 0.227 |
|  | Training 8 Hours | 0.520 | 0.465 | 0.765 | 0.137 | 0.264 | 0.273 | 0.208 |
| Phi-3 | Training 1 Hour | 0.529 | 0.325 | 0.765 | 0.094 | 0.213 | 0.158 | 0.091 |
|  | Training 2 Hours | 0.431 | 0.315 | 0.775 | 0.076 | 0.199 | 0.205 | 0.107 |
|  | Training 3 Hours | 0.500 | 0.395 | 0.725 | 0.100 | 0.208 | 0.200 | 0.130 |
|  | Training 4 Hours | 0.569 | 0.385 | 0.775 | 0.088 | 0.202 | 0.192 | 0.111 |
|  | Training 5 Hours | 0.549 | 0.385 | 0.775 | 0.095 | 0.215 | 0.186 | 0.145 |
|  | Training 6 Hours | 0.510 | 0.470 | 0.814 | 0.092 | 0.221 | 0.215 | 0.121 |
|  | Training 7 Hours | 0.441 | 0.425 | 0.755 | 0.102 | 0.245 | 0.260 | 0.114 |
|  | Training 8 Hours | 0.520 | 0.470 | 0.794 | 0.109 | 0.224 | 0.221 | 0.159 |
|  |  |  |  |  |  |  |  |  |

### C.4 Experiments: Compare the three compression techniques

The experiments examine how each compressed model can perform compared to each other. We select a wide range of models from each compression technique. For each task, we use the same heatmap scale to show the optimal model and condition. These experiments correspond to the section  [3.5](#S3.SS5).

Figure: Figure 26: Performances of Gemma-2b-GPTQ on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x183.png

Figure: Figure 27: Performances of TinyLlama-1.1B-GPTQ on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x184.png

Figure: Figure 28: Performances of Llama-2-3b-GPTQ on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x185.png

Figure: Figure 29: Performances of Llama-2-7B-GPTQ on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x186.png

Figure: Figure 30: Performances of Llama-3-8B-GPTQ on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x187.png

Figure: Figure 31: Performances of Mistral-7b-GPTQ on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x188.png

Figure: Figure 32: Performances of OpenChat-3.5-GPTQ on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x189.png

Figure: Figure 33: Phi-1.5
Refer to caption: /html/2406.03777/assets/x190.png

Figure: Figure 35: Performances of Phi-2 on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x191.png

Figure: Figure 36: Performances of Phi-3 on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x192.png

Figure: Figure 37: Performances of Sheared-LLaMA-1.3B-Pruned on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x193.png

Figure: Figure 38: Performances of Sheared-LLaMA-1.3B on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x194.png

Figure: Figure 39: Performances of Sheared-LLaMA-2.7B-Pruned on eight commonly used combinations of alpha and rank, over eight hours training time, across seven datasets.
Refer to caption: /html/2406.03777/assets/x195.png

Figure: Figure 40: Sheared-LLaMA-2.7B
Refer to caption: /html/2406.03777/assets/x196.png

### C.5 Experiments: compare the compressed model with uncompressed models

These experiments provide additional study concentrating on quantization due to the popular trend of this technique. Thse experiments can correspond to section  [3.6](#S3.SS6)

**Table 17: Performance comparisons between quantized (+G) and quantized (-G) models with different LoRA settings under training hours of 8.**
| Llama-v2-3b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | -G(^1^11G represents GPTQ, the quantization technique) | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G |
| R(8) A(8) | 0.020 | 0.520 | 0.430 | 0.470 | 0.765 | 0.784 | 0.129 | 0.113 | 0.252 | 0.234 | 0.264 | 0.275 | 0.204 | 0.220 |
| R(8) A(16) | 0.010 | 0.343 | 0.475 | 0.470 | 0.814 | 0.784 | 0.118 | 0.104 | 0.251 | 0.216 | 0.241 | 0.312 | 0.132 | 0.189 |
| R(8) A(32) | 0.147 | 0.490 | 0.460 | 0.475 | 0.775 | 0.784 | 0.117 | 0.117 | 0.217 | 0.233 | 0.248 | 0.307 | 0.111 | 0.215 |
| R(16) A(16) | 0.294 | 0.373 | 0.450 | 0.470 | 0.833 | 0.794 | 0.116 | 0.098 | 0.230 | 0.190 | 0.270 | 0.280 | 0.097 | 0.251 |
| R(16) A(32) | 0.010 | 0.510 | 0.470 | 0.475 | 0.804 | 0.784 | 0.114 | 0.108 | 0.232 | 0.211 | 0.294 | 0.291 | 0.061 | 0.229 |
| R(32) A(32) | 0.000 | 0.529 | 0.460 | 0.450 | 0.647 | 0.804 | 0.117 | 0.106 | 0.215 | 0.223 | 0.263 | 0.311 | 0.109 | 0.291 |
| R(64) A(32) | 0.196 | 0.480 | 0.455 | 0.475 | 0.794 | 0.647 | 0.115 | 0.109 | 0.235 | 0.191 | 0.263 | 0.296 | 0.125 | 0.290 |
| R(256) A(32) | 0.324 | 0.588 | 0.465 | 0.440 | 0.765 | 0.765 | 0.112 | 0.096 | 0.244 | 0.234 | nan | 0.281 | 0.095 | 0.233 |
| Llama-v3-8b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
|  | -G(^1^11G represents GPTQ, the quantization technique) | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G |
| R(8) A(8) | 0.324 | 0.039 | 0.130 | 0.140 | 0.627 | 0.461 | 0.065 | 0.073 | 0.180 | 0.229 | 0.172 | 0.174 | 0.189 | 0.277 |
| R(8) A(16) | 0.529 | 0.020 | 0.125 | 0.115 | 0.676 | 0.559 | 0.088 | 0.067 | 0.246 | 0.195 | 0.166 | 0.162 | 0.128 | 0.199 |
| R(8) A(32) | 0.088 | 0.216 | 0.150 | 0.190 | 0.363 | 0.363 | 0.091 | 0.088 | 0.147 | 0.227 | 0.080 | 0.134 | 0.089 | 0.264 |
| R(16) A(16) | 0.265 | 0.039 | 0.115 | 0.180 | 0.637 | 0.412 | 0.094 | 0.070 | 0.186 | 0.192 | 0.179 | 0.172 | 0.155 | 0.263 |
| R(16) A(32) | 0.049 | 0.029 | 0.110 | 0.095 | 0.608 | 0.627 | 0.074 | 0.081 | 0.190 | 0.155 | 0.172 | 0.134 | 0.079 | 0.253 |
| R(32) A(32) | 0.029 | 0.216 | 0.135 | 0.260 | 0.324 | 0.363 | 0.057 | 0.075 | 0.203 | 0.207 | 0.097 | 0.110 | 0.076 | 0.194 |
| R(64) A(32) | 0.098 | 0.402 | 0.115 | 0.100 | 0.627 | 0.627 | 0.069 | 0.092 | 0.211 | 0.185 | 0.158 | 0.200 | 0.135 | 0.306 |
| R(256) A(32) | 0.010 | 0.059 | 0.115 | 0.110 | 0.588 | 0.363 | 0.089 | 0.082 | 0.199 | 0.218 | nan | 0.095 | 0.089 | 0.233 |
| Gemma-2b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
|  | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G |
| R(8) A(8) | 0.010 | 0.000 | 0.430 | 0.490 | 0.784 | 0.814 | 0.119 | 0.124 | 0.239 | 0.240 | 0.229 | 0.201 | 0.301 | 0.227 |
| R(8) A(16) | 0.000 | 0.078 | 0.465 | 0.455 | 0.794 | 0.794 | 0.126 | 0.122 | 0.202 | 0.240 | 0.239 | 0.220 | 0.165 | 0.289 |
| R(8) A(32) | 0.000 | 0.000 | 0.445 | 0.440 | 0.794 | 0.725 | 0.125 | 0.119 | 0.242 | 0.209 | 0.239 | 0.241 | 0.135 | 0.231 |
| R(16) A(16) | 0.039 | 0.000 | 0.460 | 0.490 | 0.755 | 0.775 | 0.121 | 0.125 | 0.211 | 0.226 | 0.248 | 0.267 | 0.194 | 0.209 |
| R(16) A(32) | 0.020 | 0.000 | 0.445 | 0.470 | 0.755 | 0.765 | 0.121 | 0.129 | 0.227 | 0.221 | 0.246 | 0.224 | 0.173 | 0.300 |
| R(32) A(32) | 0.059 | 0.010 | 0.450 | 0.505 | 0.804 | 0.755 | 0.121 | 0.113 | 0.234 | 0.202 | 0.220 | 0.231 | 0.164 | 0.186 |
| R(64) A(32) | 0.029 | 0.010 | 0.485 | 0.455 | 0.794 | 0.745 | 0.125 | 0.108 | 0.208 | 0.214 | nan | 0.229 | 0.238 | 0.190 |
| R(256) A(32) | 0.000 | 0.108 | 0.440 | 0.435 | 0.794 | 0.765 | 0.121 | 0.114 | 0.225 | 0.203 | nan | 0.218 | 0.193 | 0.263 |
| StabLM-3b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
|  | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G |
| R(8) A(8) | 0.529 | 0.490 | 0.480 | 0.440 | 0.784 | 0.725 | 0.136 | 0.118 | 0.246 | 0.234 | 0.280 | 0.226 | 0.108 | 0.200 |
| R(8) A(16) | 0.020 | 0.529 | 0.485 | 0.485 | 0.627 | 0.725 | 0.123 | 0.101 | 0.234 | 0.205 | 0.285 | 0.245 | 0.112 | 0.176 |
| R(8) A(32) | 0.176 | 0.520 | 0.455 | 0.440 | 0.627 | 0.775 | 0.118 | 0.105 | 0.235 | 0.251 | 0.276 | 0.269 | 0.098 | 0.218 |
| R(16) A(16) | 0.255 | 0.471 | 0.480 | 0.425 | 0.618 | 0.755 | 0.139 | 0.115 | 0.241 | 0.207 | 0.297 | 0.266 | 0.133 | 0.231 |
| R(16) A(32) | 0.343 | 0.431 | 0.500 | 0.485 | 0.627 | 0.725 | 0.124 | 0.104 | 0.243 | 0.232 | 0.245 | 0.262 | 0.109 | 0.237 |
| R(32) A(32) | 0.304 | 0.549 | 0.455 | 0.420 | 0.627 | 0.627 | 0.122 | 0.105 | 0.230 | 0.184 | 0.251 | 0.269 | 0.082 | 0.219 |
| R(64) A(32) | 0.304 | 0.520 | 0.525 | 0.435 | 0.775 | 0.775 | 0.119 | 0.109 | 0.230 | 0.218 | 0.304 | 0.246 | 0.075 | 0.201 |
| R(256) A(32) | 0.020 | 0.392 | 0.450 | 0.450 | 0.775 | 0.627 | 0.120 | 0.111 | 0.252 | 0.226 | 0.266 | 0.268 | 0.066 | 0.204 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

**Table 18: Performance comparisons between quantized (+G) and quantized (-G) models given LoRA setting where rank = 8 and alpha = 32, under training hours from 1 to 8.**
| Llama-v2-3b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | -G(^1^11G represents GPTQ, the quantization technique) | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G |
| Training 1 Hour | 0.098 | 0.559 | 0.320 | 0.330 | 0.667 | 0.765 | 0.093 | 0.083 | 0.203 | 0.219 | 0.174 | 0.187 | 0.162 | 0.169 |
| Training 2 Hours | 0.020 | 0.412 | 0.310 | 0.370 | 0.569 | 0.676 | 0.105 | 0.102 | 0.230 | 0.227 | 0.191 | 0.186 | 0.118 | 0.107 |
| Training 3 Hours | 0.000 | 0.225 | 0.375 | 0.345 | 0.765 | 0.735 | 0.115 | 0.088 | 0.238 | 0.191 | 0.192 | 0.241 | 0.103 | 0.187 |
| Training 4 Hours | 0.000 | 0.196 | 0.390 | 0.395 | 0.794 | 0.755 | 0.112 | 0.097 | 0.223 | 0.246 | 0.202 | 0.203 | 0.115 | 0.306 |
| Training 5 Hours | 0.059 | 0.500 | 0.365 | 0.420 | 0.745 | 0.775 | 0.106 | 0.086 | 0.228 | 0.228 | 0.186 | 0.248 | 0.115 | 0.253 |
| Training 6 Hours | 0.088 | 0.539 | 0.420 | 0.435 | 0.775 | 0.804 | 0.116 | 0.104 | 0.221 | 0.228 | 0.250 | 0.261 | 0.081 | 0.157 |
| Training 7 Hours | 0.363 | 0.559 | 0.425 | 0.455 | 0.647 | 0.765 | 0.118 | 0.105 | 0.243 | 0.223 | 0.289 | 0.299 | 0.107 | 0.180 |
| Training 8 Hours | 0.147 | 0.490 | 0.460 | 0.475 | 0.775 | 0.784 | 0.117 | 0.117 | 0.217 | 0.233 | 0.248 | 0.307 | 0.111 | 0.215 |
| Llama-v3-8b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
|  | -G(^1^11G represents GPTQ, the quantization technique) | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G |
| Training 1 Hour | 0.353 | 0.500 | 0.065 | 0.075 | 0.569 | 0.363 | 0.077 | 0.055 | 0.263 | 0.208 | 0.052 | 0.114 | 0.087 | 0.306 |
| Training 2 Hours | 0.304 | 0.324 | 0.095 | 0.120 | 0.569 | 0.324 | 0.066 | 0.060 | 0.239 | 0.144 | 0.067 | 0.103 | 0.107 | 0.268 |
| Training 3 Hours | 0.098 | 0.098 | 0.110 | 0.150 | 0.696 | 0.627 | 0.060 | 0.075 | 0.174 | 0.168 | 0.106 | 0.069 | 0.115 | 0.272 |
| Training 4 Hours | 0.020 | 0.039 | 0.120 | 0.100 | 0.627 | 0.539 | 0.066 | 0.066 | 0.157 | 0.240 | 0.112 | 0.177 | 0.089 | 0.193 |
| Training 5 Hours | 0.020 | 0.127 | 0.125 | 0.115 | 0.627 | 0.627 | 0.083 | 0.059 | 0.253 | 0.202 | 0.088 | 0.135 | 0.103 | 0.200 |
| Training 6 Hours | 0.029 | 0.078 | 0.100 | 0.130 | 0.559 | 0.676 | 0.088 | 0.074 | 0.155 | 0.255 | 0.151 | 0.139 | 0.077 | 0.190 |
| Training 7 Hours | 0.039 | 0.020 | 0.115 | 0.120 | 0.627 | 0.657 | 0.064 | 0.072 | 0.132 | 0.216 | 0.130 | 0.150 | 0.082 | 0.188 |
| Training 8 Hours | 0.088 | 0.216 | 0.150 | 0.190 | 0.363 | 0.363 | 0.091 | 0.088 | 0.147 | 0.227 | 0.080 | 0.134 | 0.089 | 0.264 |
| Gemma-2b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
|  | -G(^1^11G represents GPTQ, the quantization technique) | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G |
| Training 1 Hour | 0.010 | 0.069 | 0.275 | 0.265 | 0.696 | 0.745 | 0.102 | 0.094 | 0.231 | 0.223 | 0.127 | 0.119 | 0.154 | 0.300 |
| Training 2 Hours | 0.098 | 0.000 | 0.320 | 0.310 | 0.755 | 0.725 | 0.113 | 0.104 | 0.226 | 0.201 | 0.159 | 0.155 | 0.140 | 0.283 |
| Training 3 Hours | 0.000 | 0.000 | 0.390 | 0.360 | 0.706 | 0.784 | 0.112 | 0.105 | 0.208 | 0.226 | 0.176 | 0.205 | 0.165 | 0.294 |
| Training 4 Hours | 0.186 | 0.010 | 0.390 | 0.415 | 0.794 | 0.784 | 0.114 | 0.116 | 0.233 | 0.205 | 0.207 | 0.169 | 0.139 | 0.263 |
| Training 5 Hours | 0.010 | 0.000 | 0.460 | 0.450 | 0.784 | 0.775 | 0.116 | 0.122 | 0.213 | 0.208 | 0.197 | 0.171 | 0.167 | 0.294 |
| Training 6 Hours | 0.010 | 0.029 | 0.435 | 0.470 | 0.784 | 0.765 | 0.115 | 0.110 | 0.223 | 0.200 | 0.213 | 0.226 | 0.141 | 0.244 |
| Training 7 Hours | 0.039 | 0.000 | 0.480 | 0.465 | 0.824 | 0.775 | 0.118 | 0.114 | 0.200 | 0.243 | 0.217 | 0.208 | 0.153 | 0.277 |
| Training 8 Hours | 0.000 | 0.000 | 0.445 | 0.440 | 0.794 | 0.725 | 0.125 | 0.119 | 0.242 | 0.209 | 0.239 | 0.241 | 0.135 | 0.231 |
| StableLM-3b | LaMP-1 | LaMP-2 | LaMP-3 | LaMP-4 | LaMP-5 | LaMP-6 | LaMP-7 |  |  |  |  |  |  |  |
|  | -G(^1^11G represents GPTQ, the quantization technique) | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G | -G | +G |
| Training 1 Hour | 0.020 | 0.480 | 0.280 | 0.325 | 0.706 | 0.676 | 0.111 | 0.088 | 0.248 | 0.200 | 0.206 | 0.191 | 0.152 | 0.137 |
| Training 2 Hours | 0.078 | 0.510 | 0.380 | 0.285 | 0.716 | 0.716 | 0.108 | 0.094 | 0.188 | 0.179 | 0.221 | 0.210 | 0.113 | 0.128 |
| Training 3 Hours | 0.245 | 0.539 | 0.370 | 0.370 | 0.725 | 0.716 | 0.103 | 0.103 | 0.209 | 0.223 | 0.216 | 0.211 | 0.098 | 0.152 |
| Training 4 Hours | 0.059 | 0.500 | 0.390 | 0.365 | 0.804 | 0.784 | 0.125 | 0.094 | 0.227 | 0.193 | 0.220 | 0.227 | 0.059 | 0.149 |
| Training 5 Hours | 0.324 | 0.510 | 0.430 | 0.380 | 0.833 | 0.735 | 0.123 | 0.101 | 0.223 | 0.208 | 0.186 | 0.234 | 0.094 | 0.235 |
| Training 6 Hours | 0.490 | 0.510 | 0.450 | 0.380 | 0.716 | 0.765 | 0.112 | 0.113 | 0.247 | 0.238 | 0.244 | 0.244 | 0.109 | 0.180 |
| Training 7 Hours | 0.441 | 0.471 | 0.455 | 0.390 | 0.755 | nan | 0.130 | 0.105 | 0.240 | 0.229 | 0.246 | 0.242 | 0.116 | 0.227 |
| Training 8 Hours | 0.176 | 0.520 | 0.455 | 0.440 | 0.627 | 0.775 | 0.118 | 0.105 | 0.235 | 0.251 | 0.276 | 0.269 | 0.098 | 0.218 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

## Appendix D Hypotheses behind findings

In this section, we provide (unproven) discussions on the intuitions and theoretical foundations behind the findings. Each subsection here corresponds to a subsection in Section [3](#S3). While these discussions shall not be treated as fully verified and correct, we feel the need to provide some thoughts on how future research could be facilitated to provide deeper investigations into those areas.

### D.1 Why does RAG work the best with mediocre tasks?

We hypothesize that such behavior is because RAG depends on the semantic understanding ability of the base LLM, and it has an upper bound on improving the
LLM’s performance. Considering the emergent ability of LLMs [] which states that some abilities (on solving more complex problems) only present in larger models, it makes sense for small LLMs to work with simple tasks but not difficult ones. On the other hand, the LLMs that could be deployed on the edge are not too big, which means that those 7B-sized models might not have the ability to solve too complex tasks (with RAG). Under those scenarios, fine-tuning at least teaches those models some semantic structure of the answer, which makes parameter learning become better than RAG.

### D.2 Why is fixing alpha needed but not rank for LoRA fine-tuning?

We hypothesize that fixing $\alpha$ for those tasks is an implicit measure to avoid overfitting. Since increasing $r$ will increase the model’s adaptation towards the training dataset, fixing LoRA’s effect on the original model with a constant $\frac{a}{r}$ will cause the model to adapt more towards details of the training dataset, which might cause overfitting issues. Nonetheless, we do acknowledge that the results are rather counterintuitive, and future research needs to be done on this topic, especially for resource-constrained fine-tuning.

### D.3 Why does test accuracy not improve after training for some time?

We hypothesize that the interesting behavior of LLMs not improving after some training is due to two possible reasons. First, since the evaluated tasks are personalized datasets, the training data might not be very diverse, which raises the possibility of overfitting. The LLMs might learn patterns associated with specific tokens, rather than the desired features of the target population. Second, due to the resource constraints on the edge, we simulate the practical scenarios and limit the training data size to be rather small. Recent work [] argues the phenomenon of “pre-scale law”, which states that loss during LLM’s fine-tuning is not linearly decreasing. At the beginning of fine-tuning, the test loss does not decrease very fast. What we observe in the experiments might correspond to the “pre-power law” and even the previous stage of “pre-power law”, where LLM learns an initial adaptation towards the downstream task. In the first stage, the LLM learns the task’s representation, which causes performance improvement. Then, it enters the pre-power-law stage, where performance starts to oscillate. Finally, it will reach the power-law stage, but that requires at least 10K-100K samples according to [], which is almost impossible under the constraint of edge devices.

### D.4 Why does using less history data have marginal effects on RAG performance?

Such results are not surprising, especially if we consider that personalized data might not be as diverse as general domain data. On the other hand, these results further validate our hypothesis on the learning curves of parameter learning. Firstly, the user history is not very diverse, which raises the possibility of overfitting. Secondly, by providing grounded context on the target’s format and basic information, RAG lifts the LLM to the stage before “pre-power law”. In this case, the intertwined performance of RAG and PEFT on some tasks, as shown in Figure [2](#S3.F2).b, aligns with the hypothesis that PEFT will reach the “pre-power law” stage after a few hours.

### D.5 What are the foundations behind each compression technique?

From the results, we see that model distillation is the most stable method to fit a large LLM on the edge, but it seldom (never) stands out as the best option. On the other hand, GPTQ is not very stable, but it can reach the best performance most of the time with some combination of LoRA parameters and training data. Shearing and pruning are just not very good compared to those two methods.

First of all, it should be evident why sheared and pruned models are not suitable for edge devices. Pruning is a technique that removes certain weights in the model, but because those models need to be fine-tuned for downstream tasks, those missing weights will inevitably be added back during fine-tuning. Thus, pruned models neither have advantages in RAM nor inherit semantic understanding ability from large models. Then, we hypothesize that the quality and memorization of pre-training datasets affect the fine-tuning of those models. It is well known that LLMs memorize part of their training datasets [, ], but two things remain unknown. The first one is that we do not know the quality of the memorized data (i.e. do they memorize only high-quality/significant data or anything?), and the second one is that where they memorize the data. It is possible that models with low-quantity pre-training data will have unstable behavior with downstream tasks, but further investigations are needed to prove this.

### D.6 Why is a compressed model sometimes better?

We hypothesize that such a phenomenon shares the same intuition with the previous subsection. An LLM memorizes its pre-training data, and they are in the model’s weights in a numerical form that we cannot interpret. Fine-tuning toward the downstream task is essentially the process of letting the LLM forget some of the pre-trained data and memorize data from the target domain. Thus, quantization removes a lot of the pre-training information that is stored in the less significant bits of the LLM and makes the fine-tuning faster.

## Appendix E Background and Related Works

### E.1 Large Language Models Customization for Downstream Tasks

Large Language Models (LLMs), such as GPT-4 [], BERT [], and T5 [], are pre-trained on vast amounts of text data []. This pretraining enables them to learn a wide range of language patterns, structures, and knowledge. However, to achieve optimal performance on specific downstream tasks such as classification, summarization, or translation, these models often require further customizations [, ]. Several approaches can be employed to tailor LLMs for specific downstream tasks, and we can generally group them into two categories: fine-tuning and prompt engineering. Fine-tuning uses targetted datasets to update the weights of LLMs so that the models can be familiar with the desired knowledge and structure of the outputs []. The dominant approach uses variations of low-rank approximation (LoRA) techniques [, , , ], which freeze the original LLM weights and add additional low-rank decomposition weights that simulate weight updates. On the other hand, prompt engineering achieves downstream optimization without changing the LLM’s weights. Typical approaches include few-shot in-context learning [] and retrieval-augmented generation (RAG) []. In particular, RAG retrieves relevant documents from a knowledge base and adds the knowledge pertinent to the prompt. It grounds the generated answer with the retrieved knowledge and significantly increases response quality on commercial Cloud LLMs.

### E.2 Large Language Model Evaluations

The field of Large Language Models (LLMs) has seen a significant amount of research focusing on various aspects of model evaluation [, , ]. Numerous studies have proposed benchmarks to systematically evaluate the capabilities of LLMs across different tasks. On the benchmark side, notable benchmarks include MMLU [], GPQA [], HumanEval [], and GSM-8K [], but they are not suitable for edge LLMs due to the reasons explained in the introduction section. In this paper, we focus on the LaMP dataset [], which closely aligns with the common use cases of edge LLMs. On the side of optimizations, research has explored the impact of quantization on LLM performance, aiming to make these models more efficient without significantly degrading their accuracy [].
Optimal fine-tuning techniques have been extensively studied by works like [], which provides a framework for choosing the optimal fine-tuning techniques given the task type and data availability, and [] which focuses on the selection of LLM. However, to the best of our knowledge, those existing evaluation papers either focus on cloud LLMs or do not fully address the constraints outlined in the introduction, thus being unable to provide a guidebook for edge LLMs.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.	The remark in Section 3.1 suggests that increasingly complex tasks require stronger models. This is a commonly understood point and lacks novelty.
2.	In Section 3.1, it is suggested that RAG is more suitable for tasks of moderate difficulty. However, according to Figure 2 and Table 1, there is no significant difference between LoRA and RAG in terms of performance.
3.	Five fine-tuning methods are compared in Section 3.1, but the first three are weaker than LoRA and RAG, offering limited practical insight.
4.	Section 3.2 suggests that smaller values for rank and alpha are more suitable for resource-constrained environments, but this finding also lacks innovation. Additionally, the models discussed are relatively small, making them inherently more compatible with LoRA in limited-resource scenarios, which somewhat disconnects the findings from real-world edge limitations.
5.	The discussion on training duration in Section 3.2 does not specify which type of device is being considered. In the Appendix, the devices listed range from 4GB to 16GB of RAM, which would result in significantly different feasible training times.
6.	Section 3.4 proposes using only a limited amount of historical data in RAG, yet given the privatized edge LLM scenario suggested by the authors, it is realistic that users would only have access to a finite amount of data rather than unlimited data.
7.	The performance loss due to model compression is a well-known trade-off rather than a novel finding specific to edge LLMs.
8.	Although the paper is framed as addressing the deployment of edge LLMs, it mainly focuses on the private fine-tuning of models. Important aspects of deployment, such as inference, are not covered.
soundness: 3
strengths: This study is very comprehensive, employing a wide range of models and constructing various scenarios and methods for comparison.
summary: The authors conducted extensive experiments and benchmarking to provide empirical guidelines for deploying large language models (LLMs) on edge devices, with a focus on fine-tuning models in private settings. However, the final conclusions of the paper align largely with common sense, and in some areas, the study lacks novelty.
weaknesses: Although this paper provides an empirical guideline through extensive experimentation, many of the conclusions are quite intuitive, lacking some innovative findings.

---

confidence: 5
contribution: 1
initial_rating: 5
presentation: 2
questions: The paper primarily evaluates popular pre-trained LLMs like Llama and OPT, with various modifications. There is little exploration of alternative architectures that could be inherently better suited for edge deployment.
soundness: 2
strengths: 1. Adequate experimental evaluation is carried out in this paper.
2. The topic of deploying LLM at the edge is interesting.
summary: This paper focuses on providing empirical guidance on LLM deployment on resource-constrained edge devices. The research focuses on how to optimize the design choices of LLMs in a resource-limited environment, balancing the computational demands with the capabilities of the devices.
weaknesses: 1. The paper primarily restates existing strategies for model deployment and optimization, lacking substantial innovation. The guidelines and strategies discussed, such as model compression, parameter-efficient fine-tuning (PEFT), and retrieval-augmented generation (RAG), are already well-documented methods in the machine learning field. The paper offers an empirical evaluation rather than a novel methodological contribution.

2. The experiments are largely confined to synthetic and benchmark datasets, which may not adequately represent the diversity of real-world scenarios where edge LLMs are deployed. This limits the applicability of the guidelines to practical use cases involving more dynamic environments.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Please see above.
soundness: 1
strengths: This is an important and timely topic. If done correctly, such study can provide practical guidelines for researchers and developer.
summary: The paper studies LLM personalization at resource-constrained edge devices, investigating the effect of design choices (e.g. what model to use, what personalization technique to apply) on the performance (e.g. accuracy). This is done by running a set of experiments to observe the effect of each choices. According to these observations, a set of guidelines are proposed.
weaknesses: The study does not have the necessary depth. In particular, it reports only single-run experiments. It is however essential for an experimental study of this nature to perform multiple experiments per setting, enabling statistical comparison of different design choices. For instance, by providing means/medias and confidence intervals, one can assess if design choice A achieves a statically significant improvement over design choice B. I recommend the authors to follow approaches such as experimental design [1] to enhance the robustness of the study.
 
Besides, the paper fails to provide clear guidelines on how select the design choices. For example, the text below is from Section 3.1.:

“As task difficulty increases, such as with complex classification tasks and simple summarization tasks, the choice should gradually shift to RAG with the strongest model. Here, the strongest models are (quantized) LLMs that excel at general benchmarks and fit within the RAM constraint.”

What are specific criteria on deciding what is a complex classification task? Does it depend on the number of classes? Or on the task? Is it possible to provide some quantitative measures on what is a complex classification task? Also, what does mean gradually? What I get from this guideline is some general roles, but it does not help me to make a clear decision.

Finally, as I understand, the selected datasets for the fine-tuning process are available online, at Github. So, there is a possibility that the models which are studied in this paper have been already exposed to these datasets during the pre-training. This could change how we interpret the results, as fine-tuning a model over a subset of its training data is usually an easier task than fine-tuning over new (unseen) dataset. Please elaborate more on this aspect.

[1]. R. A. Fisher et al. The Design of Experiments. Number 5th ed. Oliver and Boyd, London and Edinburgh, 1949.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Including some curiosity in weakness, I also raise the following questions.

- Appendix A.1 shows different edge devices. I suggest the authors include and compare the computing resources, not only RAM, of the devices to give readers a better understanding of the range of resources considered in the study
- In Table 3, the data samples per hour measured by the A10 GPU are illustrated. However, as stated earlier, processors like CPUs, GPUs, and mobile-targeted SoCs show significantly different computation times. Is the focus on computing resources solely on memory?
- RAG and LoRA are quite different techniques, with one altering the model structure and the other not. I also think they serve different purposes. Can they be considered as alternatives to one another?
- (minor) Rather than “performance,” please specify the exact metrics measured
- Figure 9 shows five models, not 6, as mentioned in the caption.
- The results suggest that insights vary significantly based on task type or complexity. Is there a way to categorize or quantify task characteristics or difficulty before running the tasks?
soundness: 4
strengths: - As LLM deployment differs significantly on edge devices due to limited computing resources and the use of private data, the comprehensive evaluation and criteria for optimization techniques are crucial topics
- The authors conduct comprehensive and well-defined experiments
- The authors’ findings are novel, as they can guide future research directions
summary: This paper addresses various design and deployment choices for running LLMs on edge devices. I enjoyed reading this paper, as the authors provide insights with clear and empirical experimental results.
weaknesses: - In the introduction, many statements and findings seem to pertain to training (fine-tuning). Clearly stating the focus and deployment scenario would make the paper’s sections clearer (fine-tuning, inference, or both?)
- Instead of focusing on individual techniques for LLM deployment, what about combining two or three methods? For example, RAG and LoRA could potentially be applied together
- I understand the page limit and the authors’ efforts to address this issue, but many supporting results are in the Appendix, making it somewhat difficult to follow thoroughly.
- The authors experiment with specified models on particular devices, considering memory constraints. What about testing similar models on different devices? Training a model for one hour on different devices might yield different insights
- I believe many aspects, including the combination of design choices, remain unexplored in this study. Clearly specifying which aspects or potential experiment cases are covered and which are not would help readers better understand the study’s scope