GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment (2024)

Abstract: The burgeoning size of Large Language Models (LLMs) has led to enhanced capabilities in generating responses, albeit at the expense of increased inference times and elevated resource demands. Existing methods of acceleration, predominantly hinged on knowledge distillation, generally necessitate fine-tuning of considerably large models, posing a challenge for average users. Furthermore, present techniques for expediting inference and reducing costs operate independently. To address these issues, we introduce a novel and intuitive Guidance-based Knowledge Transfer (GKT) framework. This approach leverages a larger LLM as a “teacher” to create guidance prompts, paired with a smaller “student” model to finalize responses. Remarkably, GKT requires no fine-tuning and doesn’t necessitate the teacher and student models to have the same vocabulary, allowing for extensive batch generation to accelerate the process while ensuring user customization. GKT can be seamlessly integrated into cloud-edge collaboration architectures, and is versatile enough for plug-and-play application across various models. When utilizing ChatGPT as teacher model and Llama2-70B as the student model, we can achieve 95.00% of ChatGPT’s performance at 52% of the cost. The results highlight substantial enhancements in accuracy and processing speed on the GSM8K and CSQA datasets, surpassing the performance of using either the student or teacher models in isolation. Our code is publicly available at https://github.com/Zoeyyao27/GKT * Corresponding author. This research was supported by the Joint Research Project of Yangtze River Delta Science and Technology Innovation Community (No. 2022CSJGG1400), the National Natural Science Foundation of China (No. 62306216), the Natural Science Foundation of Hubei Province of China (No. 2023AFB816), the Fundamental Research Funds for the Central Universities (No. 2042023kf0133). Figure 1: Cloud-edge collaboration: The GKT framework facilitates cloud-edge collaboration by deploying the larger teacher model on remote cloud servers and the smaller student model on lightweight mobile devices. GKT allows for brief guidance prompts to be easily transmitted to mobile devices, significantly reducing data transmission costs. In cloud-edge collaboration, users can also perform simple personalized generation settings on their mobile devices.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 3433–3446 August 11-16, 2024 ©2024 Association for Computational Linguistics
The burgeoning size of Large Language Models (LLMs) has led to enhanced capabilities in generating responses, albeit at the expense of increased inference times and elevated resource demands. Existing methods of acceleration, predominantly hinged on knowledge distillation, generally necessitate fine-tuning of considerably large models, posing a challenge for average users. Furthermore, present techniques for expediting inference and reducing costs operate independently. To address these issues, we introduce a novel and intuitive Guidance-based Knowledge Transfer (GKT) framework. This approach leverages a larger LLM as a “teacher” to create guidance prompts, paired with a smaller “student” model to finalize responses. Remarkably, GKT requires no fine-tuning and doesn’t necessitate the teacher and student models to have the same vocabulary, allowing for extensive batch generation to accelerate the process while ensuring user customization. GKT can be seamlessly integrated into cloud-edge collaboration architectures, and is versatile enough for plug-and-play application across various models. When utilizing ChatGPT as teacher model and Llama2-70B as the student model, we can achieve 95.00% of ChatGPT’s performance at 52% of the cost. The results highlight substantial enhancements in accuracy and processing speed on the GSM8K and CSQA datasets, surpassing the performance of using either the student or teacher models in isolation. Our code is publicly available at https://github.com/Zoeyyao27/GKT
* Corresponding author. This research was supported by the Joint Research Project of Yangtze River Delta Science and Technology Innovation Community (No. 2022CSJGG1400), the National Natural Science Foundation of China (No. 62306216), the Natural Science Foundation of Hubei Province of China (No. 2023AFB816), the Fundamental Research Funds for the Central Universities (No. 2042023kf0133).

heading: 1 Introduction
text: The swift advancement of large language models (LLMs) has dramatically pushed the frontiers of AI technology. LLMs, with their vast number of parameters, are exceptionally adept at comprehending human intentions, offering high-quality reasoning, and responses (Wei et al., 2022b; Zhang et al., 2023; Yao et al., 2023; Kojima et al., 2022). However, the immense size of these models is a double-edged sword. While it improves model performance, it also leads to slower inference times and higher computational costs. As the demand for LLM usage increases, relying solely on LLMs for auto-regressive inference actually demands an overwhelming amount of computational resources and time, posing a substantial deployment challenges for cloud services and resource-constrained devices.
Consequently, many recent studies (Leviathan
3433
et al., 2023a; Ning et al., 2023; Jiang et al., 2023) have taken steps to improve the inference efficiency of LLMs. One prevalent and widely adopted approach is knowledge distillation (Hinton et al., 2015; Yim et al., 2017; Tunstall et al., 2023; Jiang et al., 2023; Chiang et al., 2023; Li et al., 2023). The majority of knowledge distillation frameworks utilize large language models as “teacher” models to generate training samples. These samples are then used to train more compact “student” language models, effectively teaching them to mimic the performance and capabilities of teacher model. However, this process still demands a carefully crafted data generation mechanism and the subsequent training of the student model. Despite being smaller in size, to achieve satisfactory results in general tasks, many student models still maintain a considerable parameter size, often around 7B (Tunstall et al., 2023) and 13B (Jiang et al., 2023; Chiang et al., 2023). This is in line with the observations by Wei et al. (2022a), who noted that the emergent abilities of LLMs for language understanding typically become evident when the model size exceeds 10 billion parameters.
To solve this limitation, another research line focuses on speculative decoding (Leviathan et al., 2023a; He et al., 2023; Leviathan et al., 2023b), which involves using a more efficient, smaller model to generate token predictions, which the larger target model then evaluates. If a token prediction is accepted, it’s used; if not, it’s discarded, and the target model generates a new token. The method is shown to significantly speed up inference without needing changes to the model’s architecture or training procedures. However, users often have diverse generation requirements, such as the desire to adjust generation parameters like temperature and top_p. While speculative decoding can ensure consistency with the output of the final large model, employing batch generation to save time during high concurrent access can impede the abil-
ity to meet user-specific generation settings. Considering the limitations of these methods, we were inspired by the common human experience of “Getting started is the hardest part.” and sociological studies (Goldberg et al., 2014) showing that effective prompts provided by teachers in classrooms can significantly improve student performance in exams. Thus, we propose a novel knowledge transfer framework: Guidance-based Knowledge Transfer (GKT). Our framework involves two steps: firstly, using an LLM as a teacher model to generate guidance prompts from concurrent user inputs through batch generation. Secondly, a smaller LM acts as the student model, which simply completes the answers based on the guidance prompts, allowing for user-customized generation settings.
GKT reduces the burden of LLM inference, thereby speeding up response generation. Unlike knowledge distillation, our framework does not require generating dataset from teacher model and fine-tuning the student model. It also doesn’t necessitate the teacher and student models to have the same vocabulary, allowing for extensive batch generation to accelerate the process while ensuring user customization. GKT can also be seamlessly integrated into cloud-edge collaboration architectures as shown in Figure 1. GKT deploys the larger teacher model on remote cloud servers and the smaller student model on lightweight mobile devices, such as smartphones. This setup allows for brief guidance prompts to be easily transmitted to mobile devices, significantly reducing data transmission costs. In cloud-edge collaboration, users can also perform simple personalized generation settings on their mobile devices. The table 1 succinctly delineates the pros and cons between the Guidance-based Knowledge Transfer (GKT) framework, Knowledge Distillation (KD) and Speculative Decoding (SD), providing a clear comparative perspective.
Finally, our study follows the philosophy that “No such thing as bad student. Only bad teacher.” We conducted various experiments to explore the optimal guidance strategies for the teacher model, seeking answers to questions such as how much and what type of help a teacher should provide to maximize the student model’s accuracy while minimizing the teacher model’s inference time. The results demonstrate that GKT not only achieves a significant accuracy improvement of 14.18% on the GSM8K dataset, but also enhances speed by 10.72
×. Furthermore, on the CSQA dataset, it records a noteworthy accuracy increase of 14.00% along with a 7.73 × acceleration in inference speed.

heading: 2 Method
text: The schematic representation of our GuidanceBased Knowledge Transfer (GKT) framework is depicted in Figure 2. GKT encompasses a two-step process: guidance generation and response completion. During the guidance generation phase, a large language model serves as the “teacher” model. This model processes concurrent user inputs and employs batch generation to craft guidance prompts. Subsequently, in the response completion stage, a smaller language model functions as the “student” model. This model offers flexibility in generation settings, allowing for user customization. The guidance prompt created in the first stage is then fed into this smaller model, facilitating the completion of the response with enhanced efficiency. We will now elaborate on each stage:
Guidance Generation Given concurrent user inputs Q = {q1, q2, ..., qn}, where qi represents the input question from user i, the teacher model Mt batch generates the guidance prompts:
G = {g1, g2, ..., gn} = F(Mt(Q))
Here, G represents the batch-generated guidance, gi is the guidance for user i, and F(·) denotes the projection operation that generates the guidance prompt from the generated text. In this paper, we explore different projection operations including: (1) Cut-off guidance generation (2) Concise guidance generation (3) Hint guidance generation. We will elaborate on these methods in Section 4.2
Response generation In response generation, we use a smaller language model Ms as student model. For every user i, Ms generates the final response ri by:
ri = M i s(gi)
Where M is stands for Ms under the user i’s custom generation setting. Figure 6 in Appendix D demonstrates a concrete example for GKT framework.

heading: 3 Experimental Settings
text: Datasets In this paper we use two challenge but widely used dataset: GSM8K (Cobbe et al., 2021) for arithmetic reasoning and CSQA (Talmor et al., 2019) for commonsense reasoning. GSM8K focus on arithmetic reasoning which is a collection of grade school math word problems, each requiring 2 to 8 steps to solve. The solutions mainly involve a series of basic arithmetic calculations to arrive at the final answer. CSQA focuses on commonsense question answering which includes multiple-choice questions that require commonsense knowledge for answering. The detailed dataset statistics can be found in Appendix A. In all our experiments, we use the same prompt settings in Manual-CoT (Wei et al., 2022b) otherwise stated. The full prompt can be found in Appendix E
Hyperparameter All experiments were conducted on an NVIDIA A800 GPU. Detailed hyperparameter settings for the experiments are provided in Appendix B. In our experiments, we tested various teacher models, including Flan-t5-xl (Scao et al., 2022), Bloom-7B(Scao et al., 2022), Llama270B(Touvron et al., 2023) and Llama2-13B. Correspondingly, the student models used were Flan-t5large, Bloom-3B, Llama2-13B and Llama2-7B.

heading: 4 Results and Exploration
text: In this section, we delve into the empirical results of our comprehensive analysis using the GKT framework. The essence of this exploration lies in quantifying the effectiveness of GKT in enhancing the accuracy of student models while ensuring computational efficiency. We first report the overall results for the GKT on the GSM8K and CSQA datasets. Then, we explore various dimensions
of knowledge transfer including the optimal guidance generation methods, the intriguing dynamics between different types of teacher and student models, the influence that a teacher can exert on student and the optimal guidance length.

heading: 4.1 Overall Results
text: The overall results for the Guidance-based Knowledge Transfer (GKT) framework on the GSM8K and CSQA datasets are presented in Table 2. The
results indicate that on the GSM8K dataset, we achieved a maximum accuracy improvement of 14.18 % compared to simply using the student model, along with a 10.72× speed up relative to the teacher model. On the CSQA dataset, we observed a accuracy improvement of 14.00 % compared to the student model, and a maximum 7.73× speed-up in comparison to the teacher model.

heading: 4.2 How to facilitate student learning effectively?
text: To investigate how to generate better guidance to assist student models in answering questions, we experimented with three different guidance generation methods as described in Section 2: (1) Cut-off Guidance Generation, (2) Concise Guidance Generation, and (3) Hint Guidance Generation.
Cut-off Guidance Generation: We employed the simplest method of cutting off, where the teacher model generates only a fixed number of the first m tokens as guidance.
Concise Guidance Generation: In this approach, we added the prompt: “Provide the answer in a brief manner:” to guide the model to generate more concise guidance responses.
Hint Guidance Generation: Here, we introduced the prompt “Provide a brief hint for the question:” to encourage the teacher model not to give direct answers but to offer hints in a guiding manner, aiding the student model in generating responses.
The detailed results of these experiments on GSM8K can be seen in Table 3. To more intuitively understand the acceleration effect of each component in the GKT framework, we have created a trace diagram of GKT’s performance on the GSM8K dataset, as illustrated in Figure 3. Intriguingly, we found that providing hints leads to poorer outcomes compared to directly giving the answer. We speculate that this may be due to the limited inferential and reasoning capabilities of the smaller models. Instead of giving hints for them to infer, it might be more effective to provide direct answers. We also observed that Concise Guidance Generation was relatively effective, as the brevity of the guidance reduces the inferential workload for the student model. By prompting the teacher model to produce shorter answers, this method not only improved the accuracy of the model’s responses but also accelerated the inference speed.

heading: 4.3 GKT for Cloud-edge Collaboration LLM Deployment
text: Figure 3 demonstrates that utilizing only the Llama2-13B model for the GSM8K dataset, comprising a total of 1319 examples, results in an average response time of 10.78 seconds per example (14215.12 ÷ 1319 = 10.78s). This implies a single-user service capability within this timeframe. Conversely, the deployment of the GKT framework in a Cloud-Edge collaboration environment for LLMs markedly reduces the large model’s response time to 0.38 seconds on average (506.73 ÷ 1319 = 0.38s), and the small model processes each example in 7.86 seconds (10364.28÷ 1319 = 7.86s), culminating in a total response time of 8.24 seconds. Therefore, theoretically, by employing batch processing, the GKT framework can facilitate simultaneous service to 24 users within the 8.24-second window. This efficiency stems from the large model’s capacity for batch processing in the cloud, which can concurrently serve multiple users (with a batch size of 24). At the same time, the small models are deployed on distinct edge devices, facilitating parallel operations and enabling personalized user experiences. In stark contrast, reliance solely on the Llama2-13B model limits service to a single user within the 10.78-second timeframe. Hence, the GKT framework substantially augments the parallelism in user service provision in Cloud-Edge collaborative LLM deployments.
Additionally, compared to speculative decoding, GKT can control transmission losses in lowbandwidth environments by adjusting the length of the guidance prompt. We provide a comprehensive analysis of GKT and speculative decoding in Appendix C.

heading: 4.4 How To Identify the Right Teacher for the Right Student ?
text: To find the right student for the right teacher, we experimented with the decoder-only model Llama and the encoder-decoder model Flan-t5. Our experiments show that both types of models can achieve certain improvements through GKT. However, an intriguing phenomenon we observed was that replacing the teacher model of Flan-t5 with a larger Llama model led to an accuracy decrease, rather than an increase. We speculate that one possible reason could be that for encoder-decoder models, the encoder and decoder use different parameters.
Using a large model’s answers as input for the smaller model’s encoder may disrupt the coherence of thought, leading to reduced inferential ability. In contrast, for decoder-only models, directly concatenating inputs and targets seems to aid in better inference. The significant structural differences between T5 and Llama models could result in inconsistent thinking patterns between models, implying that the teacher and student are not perfectly in sync, thereby diminishing the overall effectiveness.

heading: 4.5 How Can a Teacher Influence His Student?
text: Based on the above findings, we conducted further experiments to explore the impact of a teacher’s guidance on student models, specifically focusing on decoder-only models. In these experiments, we used Llama2 and Bloom, both decoder-only models. The overall results can be seen in Table 5. Table 5 presents a detailed overview of the impact of teacher model choice on the performance and efficiency of student models, as tested on the GSM8K and CSQA datasets. The results highlight the differential effects on accuracy and processing speed,
depending on the combination of teacher and student models used. The results also revealed that the Llama model possesses stronger inferential capabilities and stores more common-sense knowledge, performing better on both datasets. When we provided the same Bloom student model with the more experienced teacher model Llama, under similarsized teacher model conditions, we observed that the Llama model, as a teacher, could enhance the student model’s accuracy by 20 %. In contrast, using Bloom-7B as the teacher resulted in a decrease in accuracy. This outcome underscores the critical influence of the teacher on the student.
Additionally, by comparing results on the GSM8K and CSQA datasets, we found that this approach of using larger models to guide smaller ones can better transfer the common-sense knowledge stored in the teacher model. However, it had a less pronounced effect on mathematical reasoning abilities in the GSM8K dataset. On the CSQA dataset, when employing a Llama2-7B teacher model, Bloom-3B model’s accuracy is improved by nearly 20 %, with a 1.52 times speed increase. These results convincingly demonstrate that our method can effectively transfer the knowl-
edge and reasoning abilities stored in larger models to smaller ones.
To delve deeper into the extent of influence a teacher model can have on a student model, we utilized the superior-performing ChatGPT (OpenAI, 2023) as the teacher model, with Llama2-7B, 13B, and 70B serving as the student model. The Table 6 clearly demonstrates that using a more powerful teacher model significantly enhances performance. When the number of guidance tokens is kept constant, employing ChatGPT as the teacher model results in a substantial improvement over using Llama2-13B as the teacher model. Specifically, on the GSM8K dataset, there’s an increase of 21.53 % compared to Llama2-13B teacher (from 19.26% to 40.79% ). We can also see from the table that stronger student model can preserve more teacher model abilities. From another perspective, this approach allows for more cost-effective outcomes. When utilizing ChatGPT’s API interface and employing the best performance Llama2-70B as the student model, we can achieve 95.00% of ChatGPT’s performance at 52% of the cost, effectively showcasing the GKT framework’s “cheap and cheerful” charm.

heading: 4.6 Further Exploration

heading: 4.6.1 Results on additional datasets
text: To better demonstrate the effectiveness of GKT across different datasets, we conducted supplementary experiments on the AQuA-RAT (Ling et al., 2017) dataset. The results for the AQuA-RAT dataset are presented in the Table 7 which shows that GKT also exhibits superior capabilities on the AQuA-RAT dataset, powerfully affirming the effectiveness of the GKT framework and its robustness across various datasets.
To validate the performance of the GKT frame-
work in more open-ended generative tasks, we also calculated and compared the Rouge scores of the rationales generated by the GKT model on the GSM8K dataset. The GSM8K dataset contains two types of labels. The first is the rationale for each question, which is a text string providing a solution to the question. The second label is the numeric format answer to the question. Examples of the GSM8K dataset can be found in Figure 7, where the blue part is the rationale, and the pink part is the answer. In our evaluation, we expanded beyond just assessing the final numeric format answers (Accuracy) by also evaluating the rationale provided for answering the question (Rouge). This approach helps in assessing how well the GKT framework adapts to tasks that require extensive content generation. The results in terms of Rouge scores are presented in the Table 8.

heading: 4.6.2 How much guidance should a teacher offer to students ?
text: To investigate the optimal amount of assistance a teacher should provide to a student, or in other words, to determine the most suitable guidance length for maximizing student benefits, we conducted experiments. Intuitively, one might assume that the more a teacher model outputs, the higher the accuracy of the model’s response. However, longer guidance tends to weaken the model’s acceleration effect. Therefore, we plotted a line graph showing the changes in accuracy as the guidance length varied from 10 to 40, in intervals of 10, as depicted in Figure 4. Based on our experiments, we established that for the GSM8K dataset, the teacher model should output the first 40 tokens of the answer. For the CSQA dataset, which generally requires shorter responses, we set the teacher model to output the first 30 tokens.

heading: 4.6.3 How few-shot exemplars affect students?
text: In this investigation, we focused on the role of fewshot exemplars in influencing the performance of student models. We used a range of few-shot exemplars, provided through Manual-CoT (Wei et al., 2022b), acros GSM8K and CSQA. The variation in
accuracy with the change in the number of few-shot exemplars is illustrated in Figure 5. As depicted in Figure 5, it can be observed that the performance of the student models improves with the increase in the number of exemplars. Thus, in this experiment, we select 8-shot for GSM8K and 7-shot for CSQA in line with the Manual-CoT.

heading: 5 Related Work
text: The aspect most relevant to our work is knowledge distillation(Tunstall et al., 2023; Jiang et al., 2023; Chiang et al., 2023; Li et al., 2023; Chen et al., 2023). Knowledge distillation involves condensing the expertise from larger models into more compact versions, thereby ensuring these smaller models retain high efficiency while still achieving impressive performance. Li et al. (2023) introduces Symbolic Chain-of-Thought Distillation (SCoTD), which trains a smaller "student" language model using the outputs (reasoning chains) of a larger "teacher" model. The teacher model first generates multiple reasoning chains for a given task, showcasing step-by-step problem-solving. The student model then are fine-tuned on these examples, essen-
tially mimicking the teacher’s reasoning process. This training enables the student model to perform complex reasoning tasks more effectively, despite its smaller size. Tunstall et al. (2023) presents a distillation framework named Distilled Direct Preference Optimization (dDPO) result in a 7B model named ZEPHYR. The method comprises three steps: (1) distilled supervised fine-Tuning (2) AI feedback through preferences and (3) distilled direct preference optimization. Different from knowledge distillation, our work focuses on leveraging the knowledge of larger models to enhance the overall efficiency and performance. The proposed GKT framework circumvents the usual requirements of producing a distillation dataset from a teacher model or fine-tuning the student model, streamlining the knowledge transfer process.
Another line of study that is related to our work is speculative decoding (Leviathan et al., 2023a). Speculative decoding uses two models: the original target model and a much smaller approximate model. The smaller model handles autoregressive sampling, while the larger assesses the output. Simple tokens are generated by the smaller model, with complex tokens handled by the larger. Based on speculative decoding, (He et al., 2023) proposed Retrieval-Based Speculative Decoding (REST) which combines speculative decoding with retrieval techniques. Instead of using a smaller LM for draft generation, REST bypasses the need for an additional small LM by retrieving draft tokens from a pre-built datastore containing contextcontinuation pairs. These drafts are then verified by a large LM. Recently, Ning et al. (2023) proposed skeleton-of-thought (SoT). SoT first guides LLMs to first create a concise “skeleton” of an
answer and then fill in each point of the skeleton in parallel, speeding up the response process. Xu et al. (2023) introduces Super In-Context Learning (SuperICL), a method enhancing the performance of LLMs by integrating them with smaller, locally fine-tuned models. These smaller models, acting as plug-ins, provide task-specific knowledge and predictions. The process involves fine-tuning a small model on task-specific data, using it to generate predictions and confidence scores for in-context examples, and combining these with the LLM’s general language understanding. While SuperICL also integrates LLMs and LMs, it primarily focuses on enhancing model performance without considering the efficiency of the framework. This focus on performance enhancement can even negatively impact overall efficiency. Whereas, GKT focuses on finding the optimal balance between efficiency and effectiveness

heading: 6 Conclusion
text: We introduce the innovative Guidance-based Knowledge Transfer (GKT) framework, designed to leverage the knowledge of larger models to enhance the efficiency and performance of smaller models, while maintaining the flexibility of personalized generation settings, allowing users to freely adjust the generation parameters. The unique collaborative framework of GKT seamlessly integrates into cloud-edge architectures, deploying smaller models on edge devices to minimize data transmission delays and expedite response generation. Our results demonstrate remarkable improvements: a maximum accuracy increase of 14.18% and a 10.72× speed-up on the GSM8K dataset, and a 14.00% accuracy enhancement with a 7.73× speed increase on the CSQA dataset. Moreover, when utilizing ChatGPT as teacher model and Llama2-70B as the student model, we can achieve 95.00% of ChatGPT’s performance at 52% of the cost. GKT signify major strides in performance metrics, combining accuracy with computational speed-ups - all wrapped up in a “cheap and cheerful” package.
Limitation
The Guidance Knowledge Transfer (GKT) framework exhibits certain limitations compared to traditional knowledge distillation methods. Knowledge distillation primarily relies on a pre-trained smaller model, often leading to faster inference speeds. However, the GKT framework still depends on a large model to generate guidance prompts during inference. This approach aims to reduce the limitations brought about by fine-tuning through performance loss during inference while supporting customized configurations of the framework.
Moreover, the length of guidance prompts generated by the GKT framework for different datasets is specific. Although the generation method is universally applicable, finding a universally appropriate length for guidance prompts that suits various scenarios remains a challenge. This means that additional adjustments and optimizations may be necessary for different datasets and application contexts to ensure optimal performance. Therefore, one of the future research directions is to explore how to more effectively determine the appropriate lengths of guidance prompts to enhance the universality and flexibility of the GKT framework.

heading: A Dataset Statistics
text: The detailed dataset statistics can be found in Table 9

heading: B Hyperparameters
text: The detailed hyperparameter settings can be found in Table 10

heading: C Speculative Decoding VS GKT
text: We provide a more comprehensive analysis of the deployment of speculative decoding and GKT within a cloud-edge framework in this section. For the Llama model, with a vocabulary size of 32K, the number of bits (b) required to store each token can be calculated using the formula:
b = ⌈log2(N)⌉
where ⌈x⌉ denotes the ceiling function. Therefore, the number of bits needed to store each token is 15 bits.
In the scenario of GSM8K (40->300), for the GKT framework, to respond to a user request, it only needs to transmit 40 tokens of guidance prompts from the cloud LLM. However, for speculative decoding, the model needs to transmit 300 * 2 = 600 tokens (the small model transmits the draft token to the cloud LLM, and the LLM sends back the validated tokens to the small model).
In a network environment with limited bandwidth, say 5K per second, transmitting a 15-bit token takes about 0.003 seconds. Therefore, for GKT, transmitting 40 tokens would only require 0.12 seconds. However, for speculative decoding, it would take 1.8 seconds. A significant advantage of the GKT framework over speculative decoding lies in its ability to control transmission losses by adjusting the length of the prompt. In contrast, the speculative decoding framework needs to transmit all tokens output by the LLM. Although speculative decoding can preserve the performance of LLM, it cannot be efficiently deployed in cloud-edge scenarios. On the other hand, GKT offers flexible applicability in these settings.

heading: D Example for GKT framework
text: An concrete example for GKT framework can be found in Figure 6, where we show the input and output of the cloud teacher LLM and the edge student LLM.

heading: E Prompt used for GSM8K and CSQA
text: The prompt used for GSM8K and CSQA can be found in Figure 7 and Figure 8 respectively.

heading: Future Work
text: Therefore, one of the future research directions is to explore how to more effectively determine the appropriate lengths of guidance prompts to enhance the universality and flexibility of the GKT framework