Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inference (2024)

Abstract: The customization of large language models (LLMs) for user-specified tasks gets important. However, maintaining all the customized LLMs on cloud servers incurs substantial memory and computational overheads, and uploading user data can also lead to privacy concerns. Ondevice LLMs can offer a promising solution by mitigating these issues. Yet, the performance of on-device LLMs is inherently constrained by the limitations of small-scaled models. To overcome these restrictions, we first propose Crayon, a novel approach for on-device LLM customization. Crayon begins by constructing a pool of diverse base adapters, and then we instantly blend them into a customized adapter without extra training. In addition, we develop a device-server hybrid inference strategy, which deftly allocates more demanding queries or non-customized tasks to a larger, more capable LLM on a server. This ensures optimal performance without sacrificing the benefits of on-device customization. We carefully craft a novel benchmark from multiple questionanswer datasets, and show the efficacy of our method in the LLM customization.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3720–3731 August 11-16, 2024 ©2024 Association for Computational Linguistics
The customization of large language models (LLMs) for user-specified tasks gets important. However, maintaining all the customized LLMs on cloud servers incurs substantial memory and computational overheads, and uploading user data can also lead to privacy concerns. Ondevice LLMs can offer a promising solution by mitigating these issues. Yet, the performance of on-device LLMs is inherently constrained by the limitations of small-scaled models. To overcome these restrictions, we first propose Crayon, a novel approach for on-device LLM customization. Crayon begins by constructing a pool of diverse base adapters, and then we instantly blend them into a customized adapter without extra training. In addition, we develop a device-server hybrid inference strategy, which deftly allocates more demanding queries or non-customized tasks to a larger, more capable LLM on a server. This ensures optimal performance without sacrificing the benefits of on-device customization. We carefully craft a novel benchmark from multiple questionanswer datasets, and show the efficacy of our method in the LLM customization.

heading: 1 Introduction
text: Large language model (LLM) has achieved unprecedented success on diverse natural language processing tasks such as machine translation, question and answering, text summarization and stylization, etc. Now then, it is expected for LLMpowered artificial intelligence (AI) to understand and satisfy each user’s unique needs such as recommendation systems, personalized assistance, and personalized search. To this end, a pivotal cornerstone is customized LLM where the LLM is highly advanced to a user-requested task. Indeed, several ∗ The authors contribute equally. † indicates corresponding author. ‡Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
web services related to LLM customization have been emerged such as GPTs (OpenAI, 2023) and PersonaAI (Character.AI, 2023; Meta, 2023).
However, due to the significant scale of LLM, keeping all the customized LLMs in the servers imposes a tremendous burden. Also, the privacy issues is inevitably raised by uploading the user’s data which entail user-requested task. Then, the focus is shifting towards on-device LLM. However, as the limited computing power of edge devices, it is impractical to address models as large as those on the servers. Therefore, for on-device LLM customization, it is crucial to maintain the performance on user-defined tasks, while constraining the model sizes. The practical method for the on-device LLM customization, however, has been less explored.
Recently, several works are developed to further lead out the ability of LLMs, and they may be exploited to cover the performance limit of LLMs on smaller size (device-level). Brown et al. (2020) introduced few-shot learning where a few example query-answer prompts are given together with users’ queries. Chain-of-thought (COT) (Wei et al., 2022) tried in-context learning by encouraging LLM to generate evidences as well as final answers. Also, for knowledge-intensive NLP tasks, retrieval augmented generation (RAG) (Lewis et al., 2020) made up query-relevant prompts by retrieving a given database. These prompt-based approaches have a intrinsic problem of increasing inference cost as the prompts get long and complex, and hence they are not suitable for edge devices.
Moreover, we can consider fine-tuning the ondevice LLM in order to internalize these promptbased knowledge for user-specific tasks to the model. Freezing the pre-trained LLM, adapterbased methods (Hu et al., 2021; Houlsby et al., 2019; Wang et al., 2022) have tried to facilitate LLM fine-tuning, and the low-rank-based adapter LoRA (Hu et al., 2021) have been most in the limelight. Despite on-device-scale LLM armed with the
3720
adapters, fine-tuning is time-consuming process and also needs a certain-level of training dataset. However, computing power of edge devices is limited and collecting enough user-specific data is also impractical. Thus, we raise inquiry: How about simply customizing LLM without on-device training?
For this purpose, given a target customization task, we propose Crayon customizing the ondevice LLM via a single customized adapter which is blended on-the-fly from a set of base adapters, called adapter pool. To cover a wide range of user requests, the base adapters are learned to contain different knowledge each other. As depicted in Fig. 1, our approach requires no additional training cost in both of server and edge device when blending the customized adapter. In addition, we develop a device-server hybrid inference strategy to effectively leverage the better-generalized larger model of the server for handling unexpected queries (out-of-scope for the customized model). Our contributions are summarized as follows:
• We propose the first practical approach for customization of on-device small-scale LLM.
• We develop Crayon where the base adapters are learned satisfying their diversification by instantly blending the base adapters, and device-server hybrid inference to cover the out-of-customized tasks.
• We present an on-device LLM customization benchmark by tailoring the public questionanswer datasets, and analyze our method.

heading: 2 Problem Set-up
text: Defining & processing customized task. In the context of few-shot learning (Brown et al., 2020), an LLM is prompted by several query-answer pairs to better understand testing queries. It has been proven that these few-shot prompts are helpful for increasing the generalization capability even in smaller LLMs. However, prompting increases the inference cost of LLMs as well, which is not preferred to on-device use case. Rather than prompting the few-shot examples Dc, we define it as a target task specified to the user. From them, we immediately generate an adapter customized for the target task on the server, and deploy it to the on-device LLM. Note that, considering the privacy issue and communication cost with the server, Dc itself is never transmitted to the sever in our method. Baseline LLM. For autoregressive, causal language model, most of popular LLMs such as GPT (Brown et al., 2020), LLaMA (Touvron et al., 2023a), Mistral (Jiang et al., 2023), and Falcon (Penedo et al., 2023) have adopted the decoderonly transformer (Vaswani et al., 2017) architecture. Hence, in our work, we select the smallest LLaMA (LLaMA-7B) as the baseline on-device LLM, which is reasonable size for edge devices1. Adapter for customized LLM. To reduce training cost, parameter efficient fine-tuning (PEFT) injects small trainable adapters, and only updates them for LLM fine-tuning. As a widely-used PEFT approach, LoRA (Hu et al., 2021) approximates the gradient of pre-trained weights into low-rank
1https://github.com/Bip-Rep/sherpa
matrices, and use them as the LLM adapters. As such, we also employ LoRA as our LLM adapter. Note that the learning LoRA does not take place on edge devices in our method. Briefly explaining our approach, we only train a set of N base LoRAs {lθn}Nn=1 (i.e., LoRA pool) given a training set Dtr on the server, and then they are combined and deployed for instant LLM customization to the target task (i.e., Dc) without additional training. Device-server hybrid inference. Although an ondevice LLM is well-customized to a user-specified task, there is inevitable performance gap between the device-level and server-level LLMs. Especially, the on-device LLM suffers from more performance drop when the inputs are out of the target task. Hence, we devise a device-server hybrid inference strategy. When output of an on-device LLM is unconfident, the output is replaced from the server’s larger model. To reduce frequent use of the server LLM, we develop a method to determine the reliability of on-device LLM’s output inside the device.

heading: 3 Methodology
text: In this section, we introduce Crayon which consists of LoRA pool construction (Sec. 3.2) and customized LoRA generation (Sec. 3.3). Also, we develop device-server consistent inference (Sec. 3.4).
3.1 Overall Framework of Crayon
As illustrated in Fig. 1, given Dtr that consists of various tasks and a baseline LLM MΦ0 where Φ0 is the initial weight before customization, we jointly train N base LoRAs {lθn}Nn=1 to have different characteristics and knowledge, respectively, in the server. Here, θn is the weight of lθn . We also simultaneously learn the base LoRA indicator cn which is allocated to lθn . After training, N base LoRAs (i.e. LoRA pool) and the indicators are located in the server and device, respectively.
Then, for a small-scaled customization dataset Dc, we first obtain the relationship between a LoRA pool and Dc by computing the similarities between the indicators and Dc on the device. This similarities are sent to the server, and then used to determine the weights of the base LoRAs in blending the customized LoRA. This customized LoRA is finally deployed to the user’s device to customize the on-device baseline LLM to the target task. Notice that we only upload the similarities of Dc to the indicators, but do not Dc itself. This is why our customization is privacy-friendly.
Algorithm 1: Learning LoRA pool Input: baseline LLM MΦ0 , base LoRAs {lθn}Nn=1,
training set Dtr 1 # Extract intermediate embeddings 2 Qtr = {qx|M1Φ0(x), x ∈ Dtr} 3 # Set the indicator of each base LoRA 4 {cn}Nn=1 = K-MEANS_Centroids(Qtr, N) 5 # Update the base LoRA weights 6 while not done do 7 Compute relationship {αn(qx)}Nn=1 (Eq. 2) 8 Compute the combined LoRA weight Θx (Eq. 3) 9 Update {θn}Nn=1 optimizing over Θx (Eq. 4)
10 end Output: Weights of the base LoRAs θ1, . . . , θN
3.2 Crayon: LoRA Pool Construction To address a variety of target customization tasks, it is important to diversify the base LoRAs’ knowledge and characteristics. To this end, we introduce an indicator for each base LoRA. In specific, for ∀ x ∈ Dtr, we first obtain the intermediate embeddings (empirically, the query embeddings of the first self-attention layer) as
qx = M1Φ0(x) (1)
Then, we apply unsupervised k-means clustering (k = N ) with qx since the text corpora in Dtr have no specific task label. To suppress noise and focus on significant features, we reduce the dimensional of the embeddings using PCA before the clustering (see Appendix C for more details). For brevity, applying PCA is not explicitly notated. N centroids cn’s resulting from k-means clustering are assigned to each base LoRA, dubbed base LoRA indicators. In the aftermentioned section, the base LoRAs are differently updated depending on the similarity between each corresponding indicator and embedding qx during training. Learning base LoRAs. For a training input x in Dtr, we extract its query feature qx as in the base LoRA indicators. Then, we compute its relationship with the base LoRA lθn by using the corresponding indicator cn:
αn(qx) = cos_sim(cn, qx) + 1
2 (2)
where cos_sim(·, ·) denotes the cosine similarity. To obtain a LoRA lΘ specified to x, the N base LoRAs are combined by
Θx = α1(qx)θ1+α2(qx)θ2+···+αN (qx)θN (3)
Following (Hu et al., 2021), we only train the base LoRAs while freezing the baseline LLM. As
Algorithm 2: Generate customized LoRA Input: Base LoRAs {lθn}Nn=1, indicators {cn}Nn=1,
MΦ0 , a few customized data Dc 1 —————– On-device Processing —————– 2 # Get query embeddings from Dc 3 Qc = {qx,c|M1Φ0(x), x ∈ Dc} 4 # Get user embedding 5 qc =
1 |Qc| ∑|Qc| k=1 qx,c, where qx,c ∈ Qc
6 # Compute relationship between qc and {lθn}Nn=1 7 αcn = cos_sim(cn, qc), n = 1, . . . , N 8 ——————- Server Processing ——————- 9 # Generate customized LoRA
10 θ̂ = ΣNi=1α c iAiBi (Eq. 6)
Output: customized LoRA lθ̂
such, we update the baseline LLM’s weights Φ0 into Φ0 +∆Φ(Θx) optimizing over Θx:
max Θ
∑
(x,y)∈Dtr
|y|∑
t=1
log(pΦ0+∆Φ(Θx)(yt|x, y<t))
(4) where y is the label for x. The entire process is presented in algorithm 1.
3.3 Crayon: Generation of Customized LoRA We explain step-by-step in algorithm 2. When a user provides a few examples Dc describing the customized task, we can generate the customized LoRA instantly from the LoRA pool. To do so, we first obtain an user embedding qc to represent the customized task by averaging the query embeddings in Dc on the edge device:
qc = 1 |Dc| ∑
x∈Dc M1Φ0(x) (5)
After then, following Eq. 2, we compute the combination ratios {αci}Ni=1 based on the cosine similarities of qc with the base LoRAs indicators {cn}Nn=1. We upload {αci}Ni=1 to the server instead of Dc to generate the customized LoRA, and thus Crayon can protect the users’ privacy. On the server side, since the base LoRA’s weight θi can be decomposed by two low-rank trainable weights Ai and Bi, we instantly generate the customized LoRA lθ̂:
θ̂ = ΣNi=1α c iAiBi (6)
At last, the customized LoRA generated from the LoRA pool is deployed to the edge device, and it customizes the on-device LLM. Accordingly, we can effectively and efficiently customize the ondevice LLM without additional training in both server and edge device.

heading: 3.4 Device-Server Hybrid Inference
text: Although the on-device LLM is customized, it cannot accommodate all kinds of input queries. For instance, the user can raise queries outside the scope of the customized task. Also, even for the queries raised from the customized task, the customized ondevice LLM can suffer from its inherent limitation. i.e. relatively small model size. To overcome these difficulties, we intermittently turn to a larger LLM (e.g., LLaMA-30B) which is placed on the server due to high computational cost. This server LLM is not customized, but can make better response owing to its superb versatility.
Notice that when deciding if an input query x is routed to the server, we cannot utilize the server LLM MΦs . Instead, we pre-compute a set of prototypes S = {s = MΦs(xc)|xc ∈ Dc} where the server LLM’s characteristics is represented. We consider that S is deployed from the server to the edge device, together with lθ̂.
Then, supposing that the desirable output of the on-device LLM may be close to S, we compare the on-device LLM’s output ox with S. In specific, we compute the routing score rx in the edge device:
rx = 1
|S|Σs∈Scos_sim(ox, s). (7)
Finally, we route x to the server LLM when rx < rth, or hold the on-device output otherwise. The routing threshold rth is empirically determined.

heading: 4 Experiment
text: As there is no established benchmark for on-device LLM customization, we present a novel benchmark for this field in Sec. 4.1. Then, we show comprehensive evaluation and analyses in Sec. 4.2& Sec. 4.3. Further details on experimental set-up and baselines in Appendix A.

heading: 4.1 On-device Customization Benchmark
text: Datasets. QA (question-answering) datasets are widely used in evaluation of LLM. Hence, to verify the effectiveness of Crayon, we select three public multiple choice QA datasets for training the pool of base LoRAs; Social Interaction QA (SIQA) (Sap et al., 2019) that focuses on the reasoning about people’s actions and their social implications, MedMCQA (MCQA) (Pal et al., 2022) that addresses real-world medical entrance exam questions, and Openbook QA (OBQA) (Mihaylov et al., 2018) that contains open book exams for assessing human understanding of a subject. For validating task generalization, we utilize MMLU (Press et al., 2022) that contains 57 subjects across STEM, the humanities, the social sciences, and more. Customized task configuration. In customization datasets, it is expected to contain conversations for different users, where each conversation comprising a series of user’s question and LLM’s response. However, there are no publicly available datasets. Despite several dialog datasets such as shareGPT (Tey, 2022) and ChatAlpaca (Bian et al., 2023), they lack user identities for the dialogue and then it is difficult to measure the customization results in these datasets. In contrast, the MMLU dataset including individual question-answer data provides annotated by the subject categories. We therefore consider each subject as the interest specified to a user, i.e. a distinct customization task. In specific, we assume 57 distinct users with their own customization tasks. From this, we can quantitatively evaluate customization results in terms of accuracy (Acc). In this experiment, we set the size of customized dataset |Dc| as 10 for every user.

heading: 4.2 Main Results
text: Crayon attains customization. We evaluate the effectiveness of our method in comparison with several baselines: LLaMA (Touvron et al., 2023a), LoRA (Hu et al., 2021), and LoraHub (Huang et al., 2023). For LLaMA, we compare the proposed Crayon with the larger LLaMA-13B as well as the LLaMA-7B. For fair comparison with our approach, we report both zero and few-shot results. In zero-shot, the input is prompted as in our unified template Fig. 2. In few-shot, task-specific few-shot prompt including Dc is used following (Touvron et al., 2023a). In single LoRA, we train LoRA on top of the LLaMA-7B with Dtr, and it is used universally all the customized tasks. Similar to our method, LoraHub combines pre-trained LoRAs to create a new one suitable for a new task. However, there are several limitations that assume specific upstream tasks with the corresponding datasets to individually pre-train all the LoRAs, and also requires a time-consuming searching process to determine the combination ratios of the pre-trained LoRAs (More details in Tab. 3).
As shown in Tab. 1, Crayon outperforms all the compared methods with only on-device LLM. Once the base LoRAs are established in the server, our method enables superior customization (40.7% vs 44.6%) compared to Single LoRA with r = 4 at an identical inference cost in the edge device. Interestingly, Crayon surpasses LoraHub (Huang et al., 2023) by 2.2% in average, although the base LoRAs are combined instantly. It is intuitive that the variety of base LoRAs is beneficial for the generalization ability of LoRA pool. Hence, the outperforming performance of the proposed Crayon indi-
cates that our joint training of all the base LoRAs at once produces their diversity, more effectively. This experiment will be addressed in Sec. 4.3. Device-server hybrid inference. To assess the efficacy of device-server hybrid inference, we employ LLaMA-30B as the server LLM, which yields 53.2% Acc on MMLU in average. Moreover, we set the routing threshold rth to satisfy 20% routing ratio, empirically. As shown in Tab. 1, with only 20% routing to sever model, the proposed hybrid inference ‘Crayon + hybrid 20%’ obtains 47.6% Acc, which is even better than fully using the large 13B model. Thus, our approach can boost the customized on-device LLM by efficiently intervening a more versatile server LLM.

heading: 4.3 Further Analyses
text: We extensively analyze the key components of the proposed method. Impact of customized data. To assess the efficacy of customized data Dc, we randomly select two different subjects (i.e., two customization datasets) from each category in MMLU, and summarize the results in Tab. 2. This demonstrates that customized LoRAs, when generated from their corresponding subjects, perform better on their matched subjects than when they are created from unrelated subjects. For instance, on the Management subject, the LoRA generated from Management data can obtain 7.8% higher Acc than the LoRA generated from Jurisprudence data. We can see a similar trend in other subjects. Hence, in spite of a very small-scale customization data Dc, it can contain significant generalization cue for the target customization task. Then, unlike the compared methods, our Crayon can effectively leverage it for LLM customization. Diversity in LoRA pool. Fig. 3 plots that density distribution 2 of α for each task contained in the
2It represents the proportion of the data in each range.
training dataset, which shows the focus of the base LoRAs on the training tasks. From 32 base LoRAs, we select the four ones. We can identify that the trained base LoRAs have different weights for each task. 0th base LoRA (Fig. 3a), 24th base LoRA (Fig. 3c), and 28th base LoRA (Fig. 3d) more specialized to SIQA, MCQA, OBQA, respectively. Unlike above three base LoRAs, 11th base LoRA is likely to evenly trained with all the tasks as shown in Fig. 3b. Note that we do not provide any information to specify or define the task (e.g. task name) during training. However, Crayon produces a diversity of base LoRAs, enabling the LoRA pool to accommodate a wide range of customization tasks. Device-server hybrid inference. In deep models, the confidence level is usually estimated by the maximum softmax score, and hence it can be a straightforward choice as decision rule for the hybrid inference. Hence, as in Fig. 4, we compare
the proposed routing strategy with the maximum softmax-based approach varying the routing ratio. We can see the proposed approach beats the maximum softmax-based approach for all the three routing ratios in Fig. 4a. Hence, our method can effectively detect the failure cases of the customized on-device LLM under a routing ratio. To consider a more general setting, we also assess the capability to address the input queries outside the customized task. To this end, in Fig. 4b, the testing queries are configured from a non-customized subject as well as the customized one. In this setting, we can see a similar trend. Hence, the our routing effectively complements the customized on-device LLM, and completes the practical use of on-device LLM. Hyperparameter sensitivity. Fig. 5 examines Crayon changing the size of the customization dataset (|Dc|) and LoRA rank r. In Fig. 5a, when the number of base LoRAs N is set as 32 (default setting), the proposed method is capable of customization, irrespective of |Dc|. This trend is mirrored when N is reduced to 16. Whereas, despite the same |Dc|, the performance gap between N = 16 and N = 32 is notable. Hence, the number of base LoRAs highly impact the quality of customization. As the number of customized data will differ from user to user and N is usually predetermined in practical use, our method can be effectively applied in a real world scenario.
Further, we investigate the performance difference as varying rank of LoRA in Fig. 5b. When r = 2, the performance gap is marginal. We infer that the LoRA pool with too small rank might not effectively represent different customization tasks. Nevertheless, it is still slightly better than the a single universal LoRA without any additional training cost. However, once r ≥ 4, the performance gap between them highly increases. Thus, for customiza-
Training base LoRAs Determining α Task-wise Joint Learning-based Cos. sim. Acc (%) Time (s)
tion, the proposed task-wise base LoRA blending is more beneficial than generalizing a single LoRA.
Component ablation study. In Crayon, customization consists of two steps; i) constructing multiple base LoRAs and ii) deploying customized LoRA via blending the base LoRAs. First, we jointly learn the base LoRAs with no task definitions from the training datasets, which effectively diversifies the base LoRAs. Second, we obtain the relationship α via simply mapping to the LoRA indicators. LoraHub combines multiple task-specific LoRAs trained on different upstream tasks, and exhaustively search their relationship with a few examples Dc. To validate the efficacy of those components, we change each component to the matched LoraHub’s one. As shown in Tab. 3, the first row where all the components are ablated corresponds to LoraHub. In the second row, we can infer that our task-agnostic joint learning is more beneficial to learn diverse base LoRAs, compared to the LoraHub’s individual LoRA learning with taskdefinition. Also, in the last row, rather than the time-consuming searching of the relationship α, the proposed simple LoRA indicator mapping is more proper to the jointly learned base LoRAs, since the base LoRAs are well-aligned with the LoRA indicators during the training.

heading: 5 Related Works
text: Task Generalization. In NLP, generalizing the language models to a wide range of unseen task has been important for its practical use. Addressing models with relatively small sizes (under 0.5B), several works have attempted to adapt them with few-shot examples into unseen tasks. However, CrossFit (Ye et al., 2021) has necessity of prompting the task names as hard prefixes. ReCross (Lin et al., 2022) alleviated this constraint via exploiting the retrieved training data, and yet requires an additional cost to retrieve task-friendly data.
Including a way more parameters, LLMs such as GPT (Brown et al., 2020), LLaMA (Touvron et al., 2023a,b), and Gemini (Team) has shown impressive results to a wide range of queries without specifically trained in the task of queries. However, their model sizes are too large for edge devices. LLaMA and Gemini also released their smaller versions under 10B training parameters which are feasible in edge devices, but their task generalization ability largely lags behind the larger models. To leverage these smaller LLMs, Mistral-7B (Jiang et al., 2023) applied sliding window attention and rolling buffer cache. Nevertheless, it cannot cover various tasks as much as the larger models. Recently, LoraHub (Huang et al., 2023) pre-trains LoRA adapters for multiple upstream tasks, and generate task-specific LoRA by mixing the pretrained ones. However, it requires manipulating upstream tasks, and time-consuming process to determine mixing ratio of the pre-trained LoRAs. Contrarily, our method constructs the LoRA adapters which has different knowledge and characteristic each other, with no task ques. Mixture of Experts (MOE). In MOE (Jacobs et al., 1991; Masoudnia and Ebrahimpour, 2014; Riquelme et al., 2021), each expert is controlled by a unique gating network, activated based on the distinct nature of the input data. Especially, in language domain, the MoE network identifies and engages the most suitable experts for every token. MoLoRA (Zadouri et al., 2023) and SiRA (Zhu et al., 2023) propose mixture of LoRA, and all the LoRAs and partial LoRAs (i.e., top-K) are participating in the gating for every token, respectively. Moreover, very recently, Mixtral 8x7B (Jiang et al., 2024) has been introduced and surpasses LLaMA2 70B across all evaluated benchmarks. It employs eight specialized experts that focus on dense matrices within fully connected layers. During the
processing of a token, a routing mechanism selects two of these experts, and their resulting outputs are then merged together. Since edge devices has limited storage to contain several LoRAs, token-wise MoE methods are hard to be applicable in our setup. Several MoE works such as Task-MoE (Kudugunta et al., 2021) and Skill Selection (Ponti et al., 2023) selects experts for every task, and can be adapted to on-device customization. However, they still assumed that the task id should be given with inputs while both training and inference phases. Speculative Decoding. To accelerate LLM decoding, speculative decoding (Chen et al., 2023; Leviathan et al., 2023; Yang et al., 2023; Gupta et al., 2024) exploits a small (draft) model to predict what the larger target model will produce using uncertainty metric (Fadeeva et al., 2023), and then use the target model just to check if the prediction is correct. Device-server hybrid inference has a problem that the draft (i.e., on-device) model cannot verify the prediction of its own for routing decision without target (i.e., server) model. Even though the draft model’s output is correct, it should be verified after communication with the server. To solve it, AutoMix (Madaan et al., 2023) utilized meta-verifier to double-check the self-verification results. To improve both accuracy and efficiency, we develop advanced device-server hybrid inference where the on-device LLM’s reliability inside the device. Further, owing to the robust customization via Crayon, our customized on-device LLM do not need frequent intervention of the server LLM.

heading: 6 Conclusions
text: We propose Crayon and device-server hybrid inference for customizing on-device LLM for the first time. In Crayon, we jointly train a pool of base LoRAs, which has distinct knowledge and characteristics each other. Using this base LoRA pool, we can instantly blend the base LoRAs into a customized LoRA for user-defined task, without additional training or transferring user data to the server. To encompass complex queries, we develop the device-server hybrid system that measures the reliability of the customized LLM to identify the necessity of server LLM. We also present a new benchmark for quantitative evaluation of on-device LLM customization, incorporating commonly-used QA datasets, and our method shows superior performance. We hope that this benchmark can be a valuable tool for future research in this field.

heading: 7 Limitations
text: This work has demonstrated that through an adapter pool elaborately learned from a variety of datasets, it is possible to create customized adapters suitable for unseen target QA tasks. Our methodology is not limited to QA but is also anticipated to be applicable across more NLP domains that requires sequential output tokens. Alongside this, we expect that increasing the number of adapters in the adapter pool together with the utilization of more large-scale datasets will lead to the creation of more diversified customized adapters for a wider scope of unseen tasks.

heading: 8 Potential Risks
text: As with any LLM, the customized on-device LLM’s outputs might inadvertently perpetuate biases present in the training data, requiring careful oversight and potential intervention to ensure fairness and ethical use. Moreover, it is crucial to consider the battery life of the edge device of deploying additional computational resources, as the use of edge devices for intricate LLM tasks could result in increased energy consumption.

heading: A Details on Experiments
text: A.1 Training setting
We implemented the proposed and baseline methods based on the Huggingface PEFT library (Mangrulkar et al., 2022). We set the rank r and scaling factor of a LoRA as 4, and the number of base LoRAs as 32. For training, we use the AdamW optimizer with a learning rate 0.0001 which is cosine annealed. We also set the batch size as 128 and the maximum iteration as 800. For all the methods, we unified the prompt template as shown in Fig. 2 where task cue is not prompted, which is proper to practical use. All the proposed and baseline methods are implemented with PyTorch 2.0.1 and executed on a single NVIDIA A5000 GPU.
A.2 Baselines
Since on-device LLM customization is understudied, we carefully selected three baselines to validate the efficacy of Crayon.
1. LLaMA (Touvron et al., 2023a,b) released publicly available models, and also reported the score on the MMLU dataset. However, the reported scores are obtained using the fewshot prompt as in the upper part of Fig. 2, where both the subject name (i.e., the task name) and examples of the subject are given. This few-shot prompt is not applicable to our on-device customization. For fairness, as well as the score from the literature, we also disclose scores using zero-shot prompt of Fig. 2.
2. Single LoRA follows the training recipe in the literature (Hu et al., 2021). Since the Single LoRA is trained using the entire training dataset Dtr, the same LoRA is used for all the customized tasks. Additionally, for a fair comparison, we fine-tune the Single LoRA (named as "Single LoRA (few-shot)"), which is originally trained on the training dataset, using a few number of examples from customized task Dc. We observed that this additional finetuning yields severe performance drop, and it can be attributed to the insufficient size of Dc for customizing the LoRA to the specific task.
3. LoraHub (Huang et al., 2023) did not focus on tailoring their work for on-device LLM customization, but it can offer a proper baseline for validating Crayon. It outlines how to generate LoRAs specified to a new task by using a given few examples and LoRAs trained for other upstream tasks. However, the downside is the lengthy process required to generate new LoRAs due to the reliance on fewshot learning with the new task’s examples. Moreover, due to necessities that all upstream tasks should be clearly defined (i.e., a meticulously refined dataset is needed), it cannot be seamlessly integrated into on-device LLM customization scenarios.

heading: B Details of Tab. 1
text: Tab. 4 extends the results from Tab. 1 to show the accuracy for each of the 57 subjects in the MMLU dataset. This allows us to see which subjects fall under each category (i.e., STEM, Humanities, Social Science, and Other). Additionally, methodologies with a higher average accuracy also tend to yield higher accuracy across individual subjects.

heading: C Effectiveness of PCA
text: Fig. 6 illustrates the deviation of the relationship α corresponding to the base LoRAs for one example in the training set, both with and without the use of PCA. The deviation of α when using PCA is larger 100 times than when not using PCA, implying that we can train the base LoRAs more diversely as experts. In line with the one of the objectives of our method, which is to train the base LoRAs with different types of knowledge, we employ PCA to our methodology when getting embeddings.

heading: Future Work
text: We hope that this benchmark can be a valuable tool for future research in this field