Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits
Authors: Yeshwanth Venkatesha
Souvik Kundu
Priyadarshini Panda
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) enable various applications on edge devices such as smartphones, wearables, and embodied robots. However, their deployment often depends on expensive cloud-based APIs, creating high operational costs, which limit access for smaller organizations and raise sustainability concerns. Certain LLMs can be deployed on-device, offering a cost-effective solution with reduced latency and improved privacy. Yet, limited computing resources constrain the size and accuracy of models that can be deployed, necessitating a collaborative design between edge and cloud. We propose a fast and cost-effective speculative edge-cloud decoding framework with a large target model on the server and a small draft model on the device. By introducing early exits in the target model, tokens are generated mid-verification, allowing the client to preemptively draft subsequent tokens before final verification, thus utilizing idle time and enhancing parallelism between edge and cloud. Using an NVIDIA Jetson Nano (client) and an A100 GPU (server) with Vicuna-68M (draft) and Llama2-7B (target) models, our method achieves up to a 35% reduction in latency compared to cloud-based autoregressive decoding, with an additional 11% improvement from preemptive drafting. To demonstrate real-world applicability, we deploy our method on the Unitree Go2 quadruped robot using Vision-Language Model (VLM) based control, achieving a 21% speedup over traditional cloud-based autoregressive decoding. These results demonstrate the potential of our framework for real-time LLM and VLM applications on resource-constrained edge devices.

Published in Transactions on Machine Learning Research (08/2025)
Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits
Yeshwanth Venkatesha
Deportment of Electrical Engineering Yale Universily
New Haven, CT, USA
Souvik Kundu
kondkk
Gintel. www.aannnndlannnnnnnmmnmnmm
Intel Labs
San Diego, CA, USA
Priyadarshini Panda
priya.panda@yole..c edu
Department of Electricol Engineering Yale University
New Haven, CT, USA
Reviewed on OpenReview:
Abstract
Large Language Models (LLMs) enable various applications edge devices such smart- phones, wearables, and embodied robots. However, their deployment often depends on expensive cloud-based APIs, creating high operational costs, which limit access for smaller organizations and raise sustainability concerns. Certain LLMs can be deployed on-device, offering a cost-effective solution with reduced latency and improved privacy. Yet, limited computing resources constrain the size and accuracy of models that can be deployed, necessi- tating collaborative design between edge and cloud. We propose fast and cost-effective speculative edge-cloud decoding framework with large target model the server and small draft model on the device. By introducing early exits in the larget model, lokens are generated
allowing the client to preemptively draft subsequent tokens before final verification, thus utilizing idle time and enhancing parallelism between edge and cloud. Using an NVIDIA Jetson Nano (client) and an A100 GPU (server) with Vicuna-6SM (draft) and TJama2-7B (target) models, our method achieves up a 35% reduction latency compared to cloud-based autoregressive decoding, with an additional 11% improvement from preemptive drafting. To demonstrate real-world applicability, on the Unitree Go2 quadruped robot using Vision-Languee Model (VLM) based control, achieving a 21% speedup over traditional cloud-based autoregressive decoding, These results demonstrate the potential of our framework for real-time LLM and VLM applications on resource-constrained edge devices.
deploy our method
1 Introduction
Large Language Models (LLMs) have become pivotal in advancing artificial intelligence, transforming natural language processing (NLP), and enabling wide range of applications such as chatbots, virtual assistants, robotics, translation, coding, and content generation Zeng et. al. (2023); Huang et al. (2024); Sun et al. (0024);
Zhang et al. (2023). Their importance lies in their ability to understand and generate human- like text, making interactions between humans and machines scamless and suggesting potential emergent capabilities
Wei et al. (2022). Recent advances include largo-scale models like OpenAl's GPT Radford et al. (2019):
Brown 2020): Achiam al. 2002),, Meta's L.aMAA Touvron et. al. 2023); Dubey et al. (2024), and Google's

---

Published in Transactions on Machine Learning Research (08/2025)
(8) Autoregressive Decoding;
Speculative Decoding
Vanilla Spec Edge-Cloud Decoding (4) Spec Edge-Clood Decoding
X
T/r
X
T/r
Ganerate
Verify
Draft
Varify
Drart
Venty
kole time
Server kile time
Steve craffing time and nerver icle time
High API oost
API
Low latency
Cost extecttive
Lowllstency
Coat offective
Lowi lanancy
Profic
Verifled tokkons
Draft tokonss
Draft Model
Target Model
Figure 1: Illustration of traditional cloud-based autoregressive decoding versus cloud-based speculative decoding, vanilla speculative edge-cloud decoding, and the proposed preemptive drafting mechanism.
Cost comparison between cloud autoregressive (AR) decoding and cloud speculative decoding (SD) and speculative edge-cloud decoding across different API providers on a set of candidate models. The measurement is based on million requests, each consisting of 100 input tokens and 500 output tokens, assuming draft length of 7 4 tokens and an average of 2.5 accepted tokens per draft.
Table 1:
Cost (In/Out per 1M tokens)
API Cost
API Provider Draft/Targe
Draft
Target
Cloud AR
Cloud SD
Edge-cloud SD
Together AI
Qwen1.5-0. 5-0.1..5-BBB.... 10t////e...5100
$0.1/50.1 $0.02/80.00
80.9/S0.9
$540
$300 (33% L
$270 (50% J)
OpenRouter
llama-3.188h 1a..1-1-8///laamma ab//llmm-33.1 1-405b
80.9/S0.9
$3-10
8312 (42% L
$270 (50% J)
Grog
Ilama 8b /1/2///mmm3 70b 8192
/8.05/80.08
$0.59 (..59/00.79
$454
8286 (37% L)
$217 (52% J)
OpenRouter
Qwen-2-VL- VL--BB WLe.VVLLLBLB/w-22LLL-oe-oo
Qwen-22VL--22B
80.2/80.2
80.7/80.7
$4120
$390 (7% J)
$210 (50% J)
Gemma Team et al. 2024). driving breakthroughs in applications ranging from personalized assistants to complex problem-sol ving across various domains.
However, running large models is costly, requiring extensive computational resources for training, often spanning thousands of GPU hours, while inference scale demands specialized hardware maintain responsiveness Samsi et al. (022).. This creates significant barriers for smaller organizations and researchers who rely on expensive cloud-based APIs; for example, GPT-4.1 text generation costs $2.00/1M /MM input tokens and $8.00/1M output tokens at the time of writing this paper
A potential solution is deploying LLMs on edge devices, which offers benefits like low latency, faster customization, and enhanced privacy in addition to cost-offoctiveness. This espocially critical for real-time robotics applications, where decisions must be made on the fly, and the server cost can add up. For example, robotic platforms such the Unitree Go2 quadruped are being equipped with language interfaces for real-world tasks like navigation, object interaction, and instruction following Cheng et al. (2024). However, such robots typically run on cooputte-onstrainet devices, making it infeasible to host large LLMs locally. For instance, the Unitree Go2 is powered by a Jetson Orin board with 16GB of unified memory, which is insufficient to run models over 10B parameters that require over 40GB of memory. Efficient decoding strategies like speculative decoding provide cost-effective solutions to bridge this gap. Speculative decoding Leviathan et al. (2023) uses a smaller model to generate tokens quickly, which are then verified by a larger model in parallel, significantly speeding up LLM inference. Despite its success on standalone machines, the application of speculative decoding edge devices remains underexplored.
In this work, we propose novel speculative edge-cloud decoding method to enable fast and cost-effective LLM inference at the odge. As shown in Fig. [ a), traditional cloud-based autorogressive decoding takes a prompt from the client and performs T forward passes on the target model generate T tokens, incurring
OOpenAT's API pricing as May 2025.
2

---

Published in Transactions on Machine Learning Research (08/2025)
API cost proportional to T. Speculative decoding on the cloud (Fig. [b)) reduces target model calls by a factor of the number of tokens generated per draft-verify round. However, it introduces additional draft model calls which comes with a non-negligible cost Yan et al. (2024). Shifting drafting to the edge can climinate this cost. Table 1 shows potential savings for example model pairs from various API providers Speculative edge-cloud decoding can reduce costs by up to 52% over cloud autoregressive decoding.
A straightforward edge speculation and cloud verification approach (Fig. [c]) suffers from inefficiencies: the client remains idle during server verification, and the server is unutilized while the client drafts tokens. To address this, propose preemptive drafting mechanism to maximize client-server utilization. As shown in Fig. dd), we introduce carly exits in the target model to produce verified tokens before full verification. These early tokens enable the client to draft the next set preemptively, & process we call pre-drafting. If the final verification confirms the early tokens, the next set draft tokens is readily available for verification, minimizing idle time and keeping both client and server continuously active. Our contributions are summarized as follows:
We propose novel framework that splits speculative decoding by hosting the draft model the edge and the target model the server, significantly reducing target model API costs.
We introduce early exits in the target model to generate verified tokens ahead of full verification, enabling the client to preemptively draft the next tokens, minimizing idle time for both client and server.
We conduct comprehensive evaluation across 6 generation tasks on 3 sets of models. With Vicuna- 68M as the draft model and Tlama2-7B as the target model, we show an average 35% latency reduction from autoregressive to vanilla edge cloud speculative decoding and a further speedup of 11% with our fast decoding method.
We demonstrate our approach on real-world robotics platform (Unitree Go2 equipped with an NVIDIA Jetson Orin), highlighting the applicability of our method for enabling edge-cloud collabora- tive inference in embodied intelligence applications.
2 Background
Speculative Decoding: Speculative decoding follows a Draft-and- Draft-and-Verify approach, where each step starts with generating multiple candidate tokens, which are then verified by the target LLM in parallel, speeding up inference Leviathan al.] (2023). Formally, given an input prefix In:t, and target model Mg, a smaller draft model Mp generates the next y tokens Teaty and their corresponding probability distribution Praty autoregressively:
Tetty, Pts+y DRAFT(Mpp To:))
(1)
The target model Mg verifies these tokens and decides how many to accept denoted by 5 (8 S 7), then produces the next token:
Tea+541 VERIFY(M Tust4y Praty)
(2)
The process repeats with the input prefix extended to t + 5 + and passed back to the draft model for the next round.
Early Exit in Large Language Models: Early exit strategies improve the efficiency of LLMs by terminating
the generation process early if a sufficiently confident output identified Panda et al. (2016): : Chen et al.
[2023). Given an LLM M with L layers and input sequence Fiat the hidden state at each layer 7 is computed as:
C = 10 x11))
where (0)) is the input embedding. At each layer 1, the model calculates logits by passing the hidden state through a language model (LM) head, denoted as (01) = LMHEAD(h()). It also computes a confidence score 5()) based on the softmax probability:
2APT cost of May 2025 based on https://wwwhhelccone comt..st.
3

---

Published in Transactions on Machine Learning Research (08/2025)
Prefix
Drafting
Draft Model
Server
Verification
Prefix
Prefix
Utiize Client Idie Time
Draft Tokens
Client EE Queue
Server Queue
EE,
Prefix
EE₂
Prefix
Prefix
Pre-drafting
EE Queuing
Nect set of Crah Tobans
Venitee Generated Tokenss
Target Model
Final Exit
Pre-Draft Cache
Miss
Hit
Server Time
Prefix
Prefix
Profix
Next Round of Drafting
Next Round of Verification
Prefix
Prefix
Figure 2: Illustration of our proposed approach. Given a prefix, the client generates two draft tokens and sends them to the server. The server verifies them using a target model with early exits, returning verified tokens and the next generated token. For each early exit, the client pre-drafts the next tokens and stores them in the pre-draft cache. If the final output matches cache entry, the draft tokens are sent immediately, roducing latency.
(4) The model exits carly at layer l' if the confidence score exceeds a predefined threshold, 50¹) 2 r, and the next loken Ft+1 is sampled from soft We leverage this mechanism generale early verified tokens in the target model, which are used preemptively produce the next set of draft tokens.
50)) = max (sx(
3 Methodology
In our distributed speculative decoding setup, the client runs lightweight draft model, denoted as Mp. while the server hosts large largel model Mg- As shown in Fig. 2 the algorithm takes the following sequence of steps:
Step 1: Given a prefix sequence To: = {Io, ( Ti...., 3t), the client uses the draft model MM to predict a sequence of y draft tokens (Eq. D These draft tokens Fatt, along with their probability distributions Pertr are transmitted the server for verification by the target model MM.
The target model Mg is designed with multiple early exits, denoted as M (1=x., Each early exit E (1,...,n) n} performs a verification step on the draft tokens (Eq. 2 and generates the next token. For example, if the early exit accepts g(6) tokens and generates the next token, the total generated tokens would bo:
Step 2):
6148041 = VERIFY(M
Seatn.Ptatt)
(5)
Here, 80) denotes the number of draft tokens accepted by early exit i.
Step 3 Given that the communication channel is typically the bottleneck, early exit outputs are queued in the server's 8 early exit queue as soon as they become available and are transmitted to the client sequentially.
Step The client, turn, stores the early exit outputs from the server its own queue and processes each one in a new thread, preemptively generating the subsequent of draft tokens for each early exit. This process is referred to as pre-drafting For early exit i, the newly verified /ggnneatedd tokens from the server
4

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Early Exit training details. # Params and Params denote the total number of trainable adapter parameters and their fraction compared total model paramcters respectively.
Model
# Exits
# Params
% Params
Context
GPU Hours
Imays/ Vicuna--B---- Zheng et al.
[2023
31
101M
1.48
1600
117
Imays/ Vicuma-13--vl. Zheng (2023
39
158M
1.20
800
122
meta-llama, metta-lamaa/Llanaa2271 -llama----aaa------ hi Touvron et al.
2023
31
101M
1.48
1600
119
Qwen Spenn/Qwen2VV--77--nnse Wang et
2024
88M
1.02
1600
136
41245041 are concatenated with the original prefix Ti.t; resulting in new prefix:
(1)
Vo.r
(4)
Concat x 62486042)
(6)
The pre-draft tokens represented as Ur and their corresponding probabilities Pvw--y: are then computed as:
(
ye'e Peiety = PREDRAFT(Mpp. Voxe)
These pro-drafted tokens are subsequently stored in cache referred to as the pre-draft cache.
Steps 5 & 6 Once the final output Fe++5+1 from the target model is received, the client checks whether these tokens were already processed in any of the early exits by looking at the pre-draft cache. there is hit, the corresponding pro-draft tokens are retrieved from the pre-draft cache and immediately sent the server for the next round of verification, avoiding any delay. If it is miss, new set of draft tokens is generated following the usual drafting process. The server proceeds with the next round of verification over the new set of draft tokens.
This design enhances efficiency by leveraging the client's idle time for pre-drafting and reducing the server's idle time between verification rounds whenever there is a pre-draft cache hit. Importantly, the output is identical that of standard speculative decoding since all tokens verified at the final exit of the target model, guarantecing loss in accuracy. For delailed system design and pseudocode please refer to Appendix A
Input
Trainable
Frozen
Backprop to Adapters
Layer 1
LM Head
EE, Loss
Layer
LM Head
EE, Loss
Layer 3
Early Exit Training: We add adapter layers after each layer of the target model train the early exils, as shown in Fig. Each adapter connects the language model (LM) head, and its loss is backpropagated to update only that adapter. This approach minimizes the number of trainable parameters and preserves the original model weights. Importantly, we do not define fixed confidence thresholds to determine when to exit; instead, all exit outputs are computed during inference, and their associated confidence scores are used to prioritize which outputs to communicate first.
LM Head
EE, Loss
Trainable Adapters
Layer L
LM Head
Final Exit
Figure 3: Ilustrationn of training early exit adapters.
For language generation models, we train early exit adapters the publicly available ShareGPT conversation dataset Ibf.RyyokoAA/SShrGPPPTSS three models-Vicuna-77
using single NVIDIA A100 GPU with 80GB of VRAM. We fine-tune Vicuna-13B, and Llama2-7B- 10 epochs cach, using batch size of 1 and a learning rate of 1e-4. Additionally, we train early exit adapters for a vision-language model based on Qwen2VL-7B using the Spacellava dataset aafremyyaiivqaayynt.. spacellava), which is generated using an open-source implementation of SpatialVIMM Chen al. (2024). Table 2 summarizes the number of carly exits, total training time, and the number of trainable parameters for each model. Note that the context length reduced during training to ensure compatibility with the memory limitations of single A A100 GPU.
5

---

Published in Transactions on Machine Learning Research (08/2025)
Table 3: Notations used our analysis.
Notation Description
Tp Time for single forward pass of the draft model Mp
Tg Time for forward pass of the target model M,
Te Communication latency between the client and server
C:
Latency ratio between the draft and target models (Tp/Tg)
7 : Number of draft tokens
T Effective number of tokens generated per draft-verify round (# accepted tokens +
gencrated).
n Total number of tokens generated
r: Cache miss rate
T, Latency of thread synchronization cache hit
4 Experiments
Following the standard speculative decoding literature Li et al. (2024) evaluate method three model sets:
Models and Benchmarks:
Vicuna-160MI Motina--60M/Vicnaaa--a Viiuna- 13B, and Vicuna-
68M/Llama2-7B. We show the experiments on 6 standard generalive task benchmarks spanning conversation
Zheng et al. (002)), code generation Chen et al. (2021). mathematical reasoning Cobbe et al. (2021), instruction
following Taori et al. (2023 summarization Nallapati et al. (2016), and question-answering tasks Kwiatkowski
et al. 2019).
Server Side Hardware: We utilize a high performance computing cluster node equipped with a single A100 GPU with 80GB VRAM, 16 CPU cores, and 8GB of CPU memory per
server.
Client Side Hardware: We remonst.rae system on two types of client devices:
NVIDIA Jetson Nano: A compact AI development board tailored for edge computing. It includes a quad-core ARM Cortex-A57 CPU, a 128-core Maxwell GPU, and 4GB of LPDDR4 RAM shared between the CPU and GPU. With a performance of up to 472 GFLOPs, the Jetson Nano is ideal for edge applications.
Cluster Node with RTX 2080 Ti: This setup features single RTX 2080 Ti GPU with 12GB VRAM, an 8-core CPU, and 4GB of RAM per core, providing more powerful alternative for our experiments.
Communication between cluster nodes is facilitated by InfiniBand high-speed interconnect. Ethernet is used for communication between the cluster node and the Jetson Nano.
Communication:
Latency Calculation: Table 3 summarizes the notations used in our analysis. Key vari- ables include TG, Tp, and Te, representing la- tencies of the target model, draft model, and communication, respectively. Speculative de- coding metrics include r, the effective tokens per draft-verification round. Hyperparameters are n, the total tokens, and y, tokens per draft- verification round. Additional factors for our fast speculative dccoding method include the cache miss rate r and synchronization latency T, for cache hits. The latency calculation for autoregressive (AR) decoding, vanilla speculative edge-cloud decoding (SD), and our fast spoculative odgo-cloud decoding (FSD) are presented in Table 4 The latency of FSD method depends on the cache miss rate, r. In case of cache hit, threads must synchronize, incurring a latency of T,. Unless otheewise specified, use 7 4 and 200 in experiments.
Table 4: Latency calculation. AR denotes cloud-based au- toregressive decoding. SD and FSD refer to vanilla and fast speculative edge-cloud decoding, respectively.
Method
Latency
Cloud AR
2T. + nT, "(T.. TT, + TT) (2T. rTT, + - rT,, +T,)
Edge-Cloud SD
Edge-Cloud FSD

---

Published in Transactions on Machine Learning Research (08/2025)
4.1 System Metrics
We report the syslem metrics in Table 5 including drafting latency (TTp) at 7- verification latency (Tg): the latency ratio c, and communication latency (To). On the Jetson Nano, drafting is about three times slower and communication twice as slow as on a cluster node with an RTX GPU. We report the maximum GPU VRAM and the number of early-exit threads supported by the system. On the RTX-equipped node, the system handles up to 30 threads for the Vicuna-68M model, but GPU VRAM (12 GB) limits the Vicuna-160M, model to 15 threads before encountering an out-of-memory (OOM) error. On the Jetson Nano, both CPU threading and RAM are bottlenecks. The maximum number of threads is capped at 15 for the Vicuna-68M model, while the 4 GB memory limit allows only 7 threads for the Vicuna-160X model.
Table 5: Average system metrics that are dataset agnostic.
Vicuna-68m Vicuma-BB
Vicuna-160m Vicuna- 13B
Vicuna-68m/ /Llmma2-78
Metric
Jetson
RTX
Jetson
RTX
Jetson
RTX
Drafting Latency (TTp,, 4)
334ms
102ms
1596ms
555ms
301ms
99ms
Verification Latency (T,)
497ms
442ms
616ms
618ms
522ms
467ms
Latency Ratio (c Tp)
0.17
0.06
0.65
0.22
0.14
0.05
Commmicatiio Latency (Te)
95ms
42ms
91ms
46ms
96ms
47ms
Max GPU memory
1.7G
3.2G
3.5G
8.9G
1.7G
3.2G
Num EE Threads
15
30
15
15
30
4.2 Speedup Results
Evaluation Metrics: Our fast decoding method with early exit is exact, with outputs identical to standard speculative decoding, ensuring no loss in accuracy. We define the following metrics to evaluate our method.
Speedup AR SD: Latency savings of vanilla speculative edge-cloud decoding (SD) compared to cloud based autoregressive (AR) baseline.
Speedup SD FSD: Latency savings of our fast speculative edge-cloud decoding (FSD) compared to the vanilla speculative edge-cloud decoding (SD).
Cache miss rate (lower the better): Frequency of cache misscs, that indicates how often we fail to find the final output in one of the early exits.
Average Early Exit (lower the better): The average early exit that produces the same output as the final exit.
Table 6 presents the evaluation metrics on the beuchmark datasets. In addition the aforementioned evaluation metrics, the effective number of generated tokens per verification (7) is also reported. Note, 7 remains identical to that of vanilla SD FSD method produces identical outputs but highlights the reduction in API call costs.
On average, using RTX, vanilla SD achieves 1.2x and 1.94x speedup over autoregressive decoding with the Vicuna-68M/Vieun /Vecna--B and MalaaaaM///LLaaa222 models, respectively. However, it results in a marginal slowdown with Moaum--60MM/iiunaaa with higher communication cost, reduces the speedup relative to autoregressive decoding, making slower for Vicuna-68M Wwe.//M////iiii /Vicuna-BB and Vieuma-160M/Vicnnaa Vicuna-68M, (aatna--6MLLLaaa2-7777 2-7m The speedup from autoregressive SD primarily depends on e, T, and Te, with ideally requiring low values of and T. and high т. Since Jetson has high and Te, it underperforms compared to autorogressive on (icuna-68M/iicnnaa
AR - SD:
Jetson, being slower at drafting coupled
models, though achieves a 1.34x speedup with
and Vicuna-160M 1ae./100M//nmmaa Stt//Vicnaaa113 13B models. In contrast, for Vicuna-68M/ L/lamm2a/laa22.... lower combined with a higher yields 1.35x speedup.
SD + FSD: Our FSD provides consistent speedup over vanilla SD across all datasets on both RTX and Jetson. On the RTX client, it achieves average spocdups of 1.06x, 1.10x, and 1.06x for Vicuma-68M CaeM///eeunaTT7 7B, Vicuna- Vicuna-160M/ 13B, and Vicuna-68M 68//Llma22-11111 7B, respectively. Similarly, on Jetson Nano, the speedups 1.04x, 1.06x, and 1.11x x for the same model pairs. The primary benefit of FSD lies in its pre-drafting
7

---

Published in Transactions on Machine Learning Research (08/2025)
Table 6: Speedup evaluation on standard language benchmark datasets.
Vicuna-G8m/ Vieuna-7B
Vicuna-160m/ Maana--60m//Vennnee Vicuna-13B
Vieuna-68m/ Llama2-7B
Benchmark
Metric
Jetson
RTX
Jetson
RTX
Jetson
RTX
Speedup SD
0.70x
0.42x
0.97x
1.34x
2.01x
Speedup SD FSD Avg Tokens
L.0x
1.0x
1.05x
1.0x
1.07x
1.02x
MT-bench
2.30
2.01
2.28
4.12
3.48
Cache miss
60.92%
20.07%
62.49%
16.40%
36.73%
13.38%
Arg EE
11
Speedup AR + SD
0.79x
1.4xx
0.47x
0.85x
1.53x
2.09x
Speedup SD - FSD Arg Tokens
1.0xx
1.03x
1.00x
1.15x
1.22x
1.07x
HvmlnBeal
2.04
2.14
3.83
Cache miss rate
64.73%
15.79%
57.64%
25.16%
17.6%
1.75%
Avg EE
Speedup ARt > SD
0.63x
1.11x
0.40x
0.77x
1.15x
1.69x
Speedup SD -> FSD
1.0xx
1.08x
1.06x
1.13x
1.07x
1.06x
GSM8K
Avg Thkens
2.08
1.96
2.23
2.29
3.48
3.29
Cache mies rate
61.21%
12.40%
58.04%
18.37%
41.87%
9.55%
Avg EE
8
13
9
14
10
Speedup AR - SD
0.63x
1.06x
0.42x
0.74x
1.42x
1.99h
Speedup SD - FSD
1.04x
1.07x
1.05x
1.12x
1.10x
1.18x
Alpacu
Avg Tokens
2.09
1.96
2.32
2.36
4.29
3.62
Cache miss rate
64.60%
19.29%
60.94%
25.45%
36.60%
4.28%
Avg EE
8
14
14
2
Speedup AR SD
0.72x
1.20x
0.38x
0.73x
1.41x
1.91x
Speedup SD FSD
1.03x
1.07x
1.03x
1.07%
1.07x
1.04x
CNN/DM
Avg Tokens Cache miss rate
2.32
1.95
2.08
2.10
4.32
3.43
70.70%
29.40%
69.92% 8
38.09%
47.13%
13.97%
Avg EE
10
15
16
14
13
Speedup AR SD
0.65x
1.10x
0.46x
0.82x
1.26x
1,93x
Speedup SD FSD
1.02x
1.06x
1.0xx
1.12%
1.05x
1,01x
NQ
Arg Tokens
2.08
2.05
2.14
2.62
3.83
3.62
Cache miss
71.50%
22.59%
63.25%
30.25%
57.11%%
14.88%
Avg EE
14
Speedup AR SD (1)
0.69x
1.20x
0.42x
0.94x
1.35x
1.94x
Speedup SD + FSD (t)
1.04x
1.06x
1.05x
1.10x
1.11x
1.06x
Average
Avg Tokens (1)
2.15
2.10
2.25
2.40
4.03
3.54
Cache miss rate (1)
65.61%
19.59%
62.03%
27.95%
39.94%
9.63%
Avg EE (L)
8
14
8
14
9
mechanism, which enables these improvements over vanilla SD. This mechanism's impact is reflected in the cache miss rale, indicating how often the final output available through early exits, allowing pre-drafting of the next of tokens. The average early exit metric further highlights how quickly verified tokens are obtained, enabling efficient generation of subsequent draft tokens.
4.3 Ablation Studies
Effect of Number of Threads: In Figure 4a we show the speedup of our FSD relative to vanilla SD and the cache miss rate as the number of early exit. threads increases up 30. Using the GSM8K dataset with the Vicuna-68M//icnnal
modcls on an RTX client, we find that the cache miss rate docreases as more threads are added, improving speedup. However, after around 15 threads, the speedup begins to plateau, and further increases in thread count yield minimal additional speedup. This is because the priority queues process the most promising carly exits first, making it more likely to match the final output with the initial threads rather than the later ones.
Effect of 7: The number tokens, 7, significantly influences the efficiency of speculative decoding. Fig. we plot speedup between SD and FSD, and between autoregressive (AR) and SD, as y increases up to 10. models on an RTX-based client. Our FSD method shows greater latency improvements compared to vanilla. SD as y increases, enhancing the benefits of
We use the GSM8K dataset with the Vicuma-68M /Ve/unina- Miiima--M///icaa-77
8

---

Published in Transactions on Machine Learning Research (08/2025)
Speedup (SD a FSD)
1.08
100
Cache Miss Rate (%)
Speedup (SD FSD)
1.1
12
Speedup (AR SD)
11
1.10
1.06
10
1.09
1.04
0.9
Miss Rate
40
1.08
0.8
1.07
1.00
20
1.06
0.7
10
15
20
25
30
Number of Threads
Number Draft Tokens Y
(a) Effect of number of early exit threads.
(b) Effect of number of draft tokens (7).
Figure 4: Ablation studies: (a) Effect of varying the number of early exit threads, (b) Effecc of varying the
number of draft tokens (7).
pre-drafting, However, an excessively large y can hinder the speculative decoding process, causing the overall speedup to decrease. As shown, the speedup of SD relative to AR falls below 1x when yy exceeds 5.
Since our sys- tem asynchronous, we need queues for graceful operation. Further, organize the queucs in pri- orily, determined by the confidence score of the generated token (Eq. 4. This prioritization is es- pecially beneficial when the number of threads limited. Table 7 presents an ablation study compar- ing different queue configurations. On the server side, we use either a priority queue or a FIFO queue, while on the client side, also include a random queue option. We report the cache miss rate (r) for systems with 3, 5, and 10 threads, denoted as 3T, 5T, and 10T, respectively.
Importance of Priority Queue:
Table 7: Ablation study of client-server queue strategies. indicates the average response time (lower is better) for different queue lengths.
Client
Server
(3T)
(5T)
(10T)
Priority
79.85
62.93
27.57
Priority
FIFO
80.40
63.48
27.73
Priority FIFO
82.32
63.25
26.92
Random
84.56
65.36
27.68
Priority 82.61
64.43
27.69
FIFO
FIFO
85.43
65.67
27.84
Our findings indicate that when the server uses a priority queue, significantly improves performance for any given queue lype on the client, although this benefit decreases with higher thread count. On the client side, priority queue consistently outperforms both the random and FIFO queues.
4.4 Robotics Case Study: Vision-Language Navigation on Unitree Go2
To evaluate the real-world applicability of approach, deploy the edge-cloud speculative decoding system the Unitree Go2 EDU quadruped robot. This platform features an onboard NVIDIA Jetson Orin board, which includes an 8-core ARM Cortex-A78AL v8.2 64-bit CPU and 16GB of 128-bit LPDDR5 unified memory, offering up to 157 TOPS of compute. Communication wetween the robot and the server is established over Wi-Fi 6.
The robot receives natural language instructions such as "go to the red chair" or "turn left at the hallway" and uses its front-facing RGB camera to perceive the environment. A vision-language model (VLM) processes the visual observations and language commands to generate mid-level navigation actions (e.g., move forward small/ haall//ediumm/lan /large), following the approach of Chenn et al. (2024). These actions are then executed by the robot's onboard controller. To enhance decision quality and interpretability, we additionally prompt the VLM to provide reasoning alongside its action outputs.
We deploy a quantized version of Qwen-2-VIL-BB as the on-device draft model and offload token verification VL-7B model hosted on an A100 GPU the cloud. Figure illustrates an example scenario in which the robot. is instructed to locate specific object- in this case, silver bottle. To test the
the full-size Qwen-2-Vil --v---2-V--BB
9

---

Published in Transactions on Machine Learning Research (08/2025)
Target
Distractor
Figure 5: Example run of the Unitree Go2 robot performing an object-finding task using vision-languagg-baseel control. The robot receives the instruction "find the silver bottle" and navigates the environment while distinguishing the correct object from similar distractor.
Table 8: System-level evaluation metries for our edge-cloud speculative decoding sctup on the Unitrec Go2 robot (Jetson Orin) and A100 server. (a) Reports core system and latency metrics. (b) Summarizes the performance gains with our FSD method.
Metric
Value
Metric
Value
Drafting Latency (TTpp, 7 = 4)
288ms
Avg Tokens (T) (t)
2.92
Verification Latency (T,)
620ms
Cache Miss Rate (4)
55.63
Latency Ratio (c TTy
0.11
Avg EE (L)
8
Communication Latency (Te)
120ms
Speedup AR + SD (t)
0.90x
Max GPU Memory
12.4G
Speedup SD + FSD (t)
1.34x
Num EE Threads
(b) FSD speedup and performance metrics
(a) System and latency metrics
model's reasoning and grounding capabilities, we introduce a distractor object of similar appearance. The robot succossfully navigates the environment and identifics the correct object, demonstrating the effectiveness of our method on a vision-language-bun based control task.
Table aa) reports key system-level metrics from our deployment, including drafting and verification latencies, communication overhead, and peak GPU memory usage. Table ({b) highlights the performance improvements enabled by method, showing speedups from standard autoregressive decoding (AR) to edge-cloud speculative decoding (SD) and further to Fast Speculative Decoding (FSD). It also includes average accepted tokens per round, cache miss rate, and average early exits. Overall, our system achieves 21% speedup over conventional cloud-based autoregressive decoding, validating the practicality of our approach for real-time, language-conditione robot control on resource-constraine edge platforms.
5 Related Work
There is a significant interest in enabling edge devices to run LLMs. The deployment of collaborative Al inference systems across the edge and the cloud introduces unique challenges such as latency constraints, bandwidth limitations, and inconsistent netw conditions. One straigghtforward approach smaller models Lu et al. (2024). While all the popular class of models such as OPT Zhang et al. (2022), Llama Touvron al. (2023); Dubey al. (2024), and Gemma Team al. (2024) have smaller scale models, edge device or not accurate enough to reliably deploy in practical applications. Quantization is one of the heavily focused methods to enable on-device LLMs Lin
design
they are either not small enough to
(2024) aim to compress the models with layer-wise pruning and quantization to enable edge LLMs. Qu et. al.
et al. 2024). Yu et al.
2024) discuss an approach enabling LLMSs to run on 6g edge devices. On system side, Xu et al. (2024a) focus on leveraging on-device Neural Processing Unit (NPU).
Early exit strategics, which allow intermediate layers of deep networks to make predictions without waiting for the full forward pass, have been extensively explored for resource-constrained devices. Pioneering works such
10

---

Published in Transactions on Machine Learning Research (08/2025)
Conditional Deep Learning Panda et al. (2016) and BranchyNet Teerapittayanon et al. (2016) introduced the idea of adding multiple exit points to deep neural networks to reduce computation time. Recent research has also explored layer skipping in LLMs for enhanced efficiency Fan et al. (2024), with dynamic compute allocation based on tokens Raposo et al. (2024). In terms of multi-device spoculative decoding, McDanel (2024)
has recently shown that asynchronous speculative decoding over multiple GPUs can be beneficial. However, it uses shared memory to communicate between devices, so it is not directly applicable to edge-cloud scenarios. To the best of our knowledge, this is the first work to show end-to-end speculative decoding with models split. between edge and cloud. Further, comprehensively analyze and demonstrate the system-level trade-offs during the implementation of collaborative edge-cloud decoding, which no prior work has investigated.
6 Conclusion and Discussion
We introduced a novel speculative edge-cloud decoding framework, offering a cost-effective alternative to traditional cloud-based deployment. By distributing the draft and target models between edge and server environments, our solution significantly reduces high APT costs. Early exits and pre-drafting allow us to enhance parallclism by leveraging idle client time and reducing server idle time. Our comprehensive end-to-end evaluation on the NVIDIA Jetson Nano highlights the feasibility of efficient edge-cloud collaborative LLM inference on resource- resotrce-lmmited edge devices. On Jetson Nano, speculative edge-cloud decoding achieves up to 35% speedup over cloud-based autoregressive decoding, with up to an additional 11% performance gain enabled by pre-drafting and early exits. Further, we validate our approach with execution of vision language models on the Unitree Go2 quadruped robot. We achieve an overall 21% speedup over standard cloud-based autoregressive decoding, demonstrating the effectiveness and real-world applicability of our framework for robotics use cases.
Our method operates effectively without making assumptions about communication delays, and we show that it remains practical under real-world conditions. However, in extreme scenarios -uch round-trip latencies exceoding 300-400 ms or highly variable throughput on congested mobile networks-on performance gains may diminish. Our use of priority queues is specifically designed to address these challenges by optimizing bandwidth utilization and maintaining responsiveness under constrained network conditions. Future work could explore adaptive communication strategies and dynamic scheduling policies to enhance robustness. Further, we believe that continued advancements in edge-cloud crtrastructure-nnchue lower-latency connectivity (e.g., 5G/6G), transport protocol optimizations, and hardware-accoelrrttel interconnects will roduce this bottleneck.
Additionally, while pre-drafting leverages idle client time for parallelism, it introduces compute overhead on the client. This is manageable for most edge platforms, and the ability to scale the number of threads provides a ficxible trade-off between latency and energy efficiency, especially for battery-powered devices. That said, we acknowledge that our approach introduces additional system complexity. Coordinating early exits, managing pre-draft caches, and maintaining token-level priority queues require careful orchestration. Although the performance gains justify this complexity in many scttings, future work could explore lightweight. abstractions runtime systems to streamline deployment. Another practical consideration is the training overhead introduced by early exit adapters. Fine-tuning these exit heads on target models incurs a one-time cost (approximately 100 GPU hours per model). Although the training cost is a one- time expense that can be amortized over repeated use, it can limit the frequency with which the model can be updated in practice. Our results are also sensitive to the quality of the early exit calibration. High exit misprediction or cache miss rates can degrade throughput and reduce pipclining efficiency. Adaptive confidence thresholds or dynamic calibration methods may mitigate this issue and offer further robustness.
Cache miss rates significantly affect the efficiency of speculative decoding and are primarily influenced by two factors: (i) the quality of the carly exit adapters and (ii) the computational capacity of the client device. Well-trained and calibrated early exits are more likely to produce outputs that match the final model, thereby increasing cache hit rates and improving decoding throughput. However, on resource-constraingg constrained devices like the Jetson Nano, limited memory and compute capabilitics restrict the number of early exit outputs that can be computed and stored in the pre-draft cache. Although we train early exit adapters on general-purpose conversational data, task-specific fine-tuning can improve their alignment with final outputs, thereby reducing
11

---

Published in Transactions on Machine Learning Research (08/2025)
cache misses. On the client side, prioritize speculative drafts from higher--onfidence exits. However, these typically correspond to later layers, which are delayed in arriving to the client. Future extensions could explore more sophisticated scheduling strategies beyond static confidence- based prioritization, potentially introducing controlled delays to improve hit rates while balancing latency and resource usage.
Our proof-of-concept is implemented in Python, which offers ease of experimentation but leaves room for performance optimization. A low-level C++ implementation with multithreading and shared memory could significantly enhance efficiency, particularly for latency-sensitive applications. While the current system supports only single-client interaction with the server, extending it to handle multi-client concurrency is a natural next step. This introduces new challenges, such as synchronizing carly exit queues, managing server resource contention, and ensuring fair allocation of communication bandwidth across clients. Future iterations may incorporate techniques like batched verification and client prioritization policies to improve scalability and support robust, concurrent edge-cloud inference with early exits. Future work could explore broader range of edge-cloud configurations beyond the Jetson Nano and A100 GPU used in this evaluation. This includes ultra-low-power microcontrollers, ARM-based edge devices, and emerging robotic platforms. On the cloud side, potential directions include evaluating newer accelerators such as Google TPUs, Groq LPUs, and Cerebras WSEs. It would also be valuable to assess performance under diverse network conditions, including 4G and 5G connectivity.
Broader Impact: FSD reduces reliance on always-on cloud infrastructure by offloading a portion of the inference workload to the edge, thereby lowering energy consumption per query. As a reference point, the NVIDIA A100 GPU has thermal design power (TDP) of 250-400W, whereas the Jetson Nano consumes only 5-10W-nearly two orders of magnitude less. This makes the additional power required by the edge device negligible relative to the server. Consequently, even partial offlodingg can lead to significant energy savings; for instance, our approach reduces server utilization by approximately 50%, potentially halving the associated power consumption. While we do not directly measure carbon emissions, this reduction in centralized compute contributes to a smaller environmental footprint. Although our cvaluation is based on NVIDIA hardware, the FSD Tramework both model- and hardware-aguostii, and can be adapted to open-source models and alternative edge platforms that support multi-threading or multi- poouti-processing. We do not anticipate any hardwaro-specifie dependencies exclusionary practices resulting from our framework.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ar preprin! wwXiv:230300877, 2023.
Mohammed Banafaa, Tbraheem Shayea, Jafri Din, Marwan Hadri Azmi, Abdulaziz Alashbi, Yousef Ibrahim Daradkeh, and Abdulraqeb Alhammadi. 6g mobile communication technology: Requirements, targets, applications, challenges, advantages, and opportumities. Alezandria Engineering Journal, 64:245 274, 2023.
Tom Brown. Language models are few-shot learners. arXio preprint orXiv:2005.14166, 2020.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the JEFE/CV Conference on Computer Vision and Pottern Recognition, pp. 14455 14465, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiu preprint ar arXio:2107. 03374, 2021.
Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism. arXiv preprint arXiv:2312.04916, 2023.
An-Chich Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xucyan Zou, Jan Kautz, Erdem Biyik, Hongxu Yin, Sifei Liu, and Xiaolong Wang. Navila: Legged robot vision-language-action model for navigation. Xix preprint or 12.:2412.0044444 04455, 2024.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arxiv preprint arXiv:2110. 10. 14168, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv.4077.2173 21783, 2024.
Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers of ilms are necessary during inference. arXiv preprint orXiv:2003.0211 021811 2024.
Othmane Friha, Mohamed Amine Ferrag, Burak Kantarci, Burak Cakmak, Arda Ozgun, and Nassira Ghoualmi-Zinne Zine. Llm-based edge intelligence: A comprehensive survey architectures, applications, security and trustworthiness. IEEE Open Journal of the Communications Society, 2024.
Kaiyu Huang, Fongran Mo, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, et al. A survey on large language models with multilingualism: Recent advances and new frontiers. arxiv preprint arx::22405. 10936, 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Iilia. Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-192880 PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406. 16858, 2024.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awar weight quantization for on-device Ilm compression and accoleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Zheng Lin, Guanqiao Qu, Qiyuan Chen, Xianhao Chen, Zhe Chen, and Kaibin Huang. Pushing large language models to the 6g edge: Vision, challenges, and opportunities. arXiv preprint or arXix:2309.1677 2023.
Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D Lane, and Mengwei Xu. Small language models: Survey, measurements, and insights. arXiv preprint arXiv: 15790, 2024.
Bradley McDanel. Amuud: Asynchronous multi-device speculative decoding for Ilm acceleration. arXiv preprint arXie:2410. 17375, 2024.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to-sequeen rmns and beyond. arXiv preprint arXt::1602.06023 2016.
Priyadarshini Panda, Abhronil Sengupta, and Kaushik Roy. Conditional deep learning for energy-efficient and enhanced pattern recognition. In 2016 design, automation @ test in europe conference & exhibition (DATE), pp. 475-480. IEEE, 2016.
Guanqiao Qu, Qiyuan Chen, Wei Wei, Zheng Lin, Xianhao Chen, and Kaibin Huang, Mobile edge intelligence for large language models: A contemporary survey. arXiv preprint axXiv:2407.18921, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 1lya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute transformer-based language models. arXiv preprint ar 2rXiv:2404.02258 2024.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From words to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1-9. IEEE, 2023.
Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyuc Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint ar arx::240 1479,, 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0820 08295, 2024.
Surat Teerapittayanon, Bradley McDanel, and Hsiang- exiting from deep neural networks. In 2016 23nd international conference on pattern recognition (ICPR), pp. 2464 2469. IEEE, 2016.
Hsiang-Tsung Tsung Kung. Branchynet: Fast inference via early
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ar preprint 200::2307.09288, 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. aixie preprint 0rXiv:2409.1214 12191, 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgcaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint Xiv: 07682, 2022.
Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. Empowering 1000 tokens/second on-device I1m prefilling with milm-npu. arXiv preprint ar xxXi:2447705888 2024a.
Minrui Xu, Dusit Niyato, Jiawen Kang, Zehui Xiong, Shiwen Mao, Zhu Han, Dong In Kim, and Khaled B Lotaief. When large language model agents meet 6g networks: Perception, grounding, and alignment. IEEE Wireless Communications,, 2024b.
Minghao Yan, Saurabh Agarwal, and Shivaram Venkataraman. Decoding speculative decoding, ar preprinl 2402v 0152, 2024.
Zhongzhi Yu, Zheng Wang, Yuhan Li, Haoran You, Ruijie Gao, Xiaoya Zhou, Sreenidhi Reedy Bommu, Yang Katie Zhao, and Yingyan Celine Lin. Edge-Iim: Enabling efficient large language model adaptation on edge devices via layerwise unified compression and adaptive layer tuning and voting. arXiv preprint XrXiv:2406.1 100.6.15755 15758, 2024.
Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S Yu. Large language models for robotics: A survey. arXiv preprint arXiv: 11.31110722666 2023.
Mingjin Zhang, Jiannong Cao, Xiaoming Shen, and Zeyang Cui. Edgeshard: Efficient. Ilm inference via collaborative edge computing. arXiv preprint arXiv:2405. 00.112405,,4,11,,, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0 010688, 2022.
Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. A survey on language models for code. Xrxiv preprint arXiv22311.07989, 2023.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Fric Xing, et al. Judging llm-as-a-judge with mi-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 46623, 2023.
A System Design
On the client side, the primary goal is to maximize idle time usage and increase cache hit rates. As shown in Algorithm| [ the client maintains priority queue Qp: pre-draft cache C, and RECEIVER thread. After sending draft tokens to the server for verification, the RECEIVER thread listens for server callbacks, which provide outputs from early exits.
Client:
Since the elient's 's bottleneck lies in the processing power required for generating pre-draft tokens, it is essential to prioritize the handling of carly exit outputs. The priority qucue Op organizes these outputs according populated to their confidence levels (Eq. 4) prioritizing the most promising ones for pre-drafting. asynchronously as early exit outputs are received by the client. If the client's device has multiple available threads, it can process several early exits in parallel to generate more pre-draft tokens. All the pre-draft tokens are stored in the pre-draft cache. Once the client receives the final exit output, it checks the pre-draft cache for the corresponding tokens. If there is cache hit, the pre-drafted lokens are sent to the server immediately for the next verification round.
Server: As detailed in Algorithm 2 the server consists of two asynchronous threads: LISTENER and SENDER. The LISTENER processes the verification requests from the client.. Tt. takes in the prefix 2ids draft tokens Ti:t+-y: and their corresponding probability distribution Pirt
As shown in Fig. communication typically becomes the bottleneck in the server early exit outputs are produced faster than the network can transmit. Farly exit outputs are placed in queue on the server side and transmitted sequentially to handle this. Let Qe represent the server-side queue storing the early exit outputs:
(L) Qe t- (0 (1488441
Asynchronously, the SENDER thread sends the early exit outputs from the queue based on priority determined by the confidence score (Eq.
Speedup Projection Analysis
Speedup SD
FSD (y=
Speedup SD
FSD (
7)
Speedup SD
FSD (y= 10)
100
100
100
Llama-458
Llama-
Liama-858
Cache miss rate ir)
80
80
80
1
60
V-88MN-78
V160M/9133
60
V68MA-78
V1668
60-
V-68MN-78 V-160N/V1138
5D
40
40
40
3 (uuu
V-00M12-78
V-60M12-78
V-60MKLL-70
Speeddy
20
20
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0,2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Latency Ratio (c)
Figure 6: Estimated speedup per round of our FSD relative vanilla SD method assuming no communication latency. We emphasize the contours for speedups of 1.1x, 1.2x, 1.5x, 2x, and 5x, and we indicate the position of our models within this landscape. Additionally, we highlight the operating range for the larger models OPT-66B and Llama-65B.
One of the bottlenecks in our system is communication between the devices. addition to directly adding to the latency, it also bottlenecks the number of early exit. verifications communicated back to the client.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm Client-Side Algorithm
1: Initialize: Draft model Mp: Queue Qp, Cache C, RECEIVER
2: Input: Prefix Fi:t: # Total Tokens T, # Draft tokens y
3: Output: Final tokens I:-1:T
4: for 1 7 do
P: DRAFT(Mp. F1:44-1)
end for
(END(
8: while 1 <T do
9:
while Qp not empty do
10:
11-1148415 s' t Qp-pop()
11:
if 441448841 not in then
Cla4114041 t PHaDiaA((onsss 13: end if
12:
14: end while
15: V1. < Concat (144444444
16: if I+41:45+1 E C' then
17:
Wea'rrPPy t C[12+104441]
18: else
19: for
1= to do 4e'+i: + DRAFT(RRF..... V1-
20:
21: end for
22: end if
23:
SEND(JU PI:7)
24:
1 I', x +- y, C.resel)) (), Qp.reset
25:
end while
26: function PREDRAFT
27: Input: Prefix 3ıd, Tokens 4441418441
28: Var +- Concat(rna, 11448+44))
29: for = 1 to y do
30:
ye+..P/ DRAFT( x1144111)
31: end for
32: return 3 P1n
33:
end function
34:
function RECEIVER
35:
Input: Tokens 11+144841 Priority 8', isfinaleril
36: if isfinalezit then
37:
I1-1116611 + 244144841
38: else
39:
40:
end if
41:
end function
This increases the cache miss rate further increasing the latency. While this a challenge at present due to limited communication network capabilities, several works have shown the vision having edge LLMs on 6g networks with a projected network speed up to 10 Tbps Banafaa al. (0023 : Lin et al. (2023) ; Xu et al.
(2024b): Friha al. (2024): Qu et al. (2024); Zhang al. (2024). In scenarios where communication latencies T. is negligible relative to drafting and verification time, and ignoring thread synchronization latency (T,) for simplicity, we can approximate the speedup SD FSD as:
Speedup (SD) + FSD) = 7C + 1 + r(rc) +1
(9)
16

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm Server-Side Algorithm
1: Initialize: Target Model Mag Verification criterion VERIFY, Queue Qe, LISTENER, SENDER
2: function LISTENER
3: Input: Prefix and draft tokens F1sty, Probs. Play
(1:2) f1.+11 VERIPY(M (144y)11
(1:2)
#1+1:4841
for all 1= I..... L do
8(0) < max (1::++1)
Qu.push(z (+14++111 (
end for
SEND( (L) SEND(#1414644) isfinalexit True)
10:
(e-res
11: end function
12: function SENDER
13: while Qe not empty do
14:
(z(_1144++1:5)) Q..pop()
15: SEND(E4+ s isfinalexit False)
16: end while
17:
end function
This formula reduces the final speedup to be affected by two factors Cache miss rate r and Latency Ratio c. Cache miss rate r depends on the redundancy in the larget model and how well the early exit adapters are trained. On the other hand, e is highly dependent on the compute capability of the edge device. Since edge devices are often slower, this pushes the e to be higher.
We visualize Eq. as a heatmap in Fig. 6 for values 4, 7 and 10. For reference, we plot the measured and values based on the Jetson implementation of our current set of models within this landscape. Naturally, having a lower r will improve speedup, but the usefulness of our FSD method becomes more pronounced as we get to higher e and y. For model sets with a latency ratio greater than 0.5 and well-trained carly exit adapters that achieve a cache miss rate of less than 10,, we can anticipate speedup over 5x
Extending our analysis to larger models, speeifically OPT-66B and Llama-65B with draft models OPT-125M and NoFT-Wide-796M.
reported latencies from Yan et al. (2024) (6.6 ms for draft, 67 ms for target) and factor in 3x slowdown on Jetson, arriving at c z 0.3. This value is illustrated by the yellow line in Fig. G For instance, to achieve a 2x speedup with values of 4, 7, and 10, the cache miss rate must remain below 10%, 25,, and 35%, respectively.
C Batch Processing
Table shows latency analysis for Vicuna-7B (A100) and Vicuna-68M (Jetson Nano). Batch processing improves throughput but not always latency; e.g., batch size 32 increases A100 latency by over 5x. However, API providers often offer discounts for batch processing (c.g., OpenAI provides 50% discount OpenAI Pricing), making it cost-saving approach. On the client, batch processing shows smaller latency increase batch sizes of and 8 differ by 15%. A batched pre-drafting approach could reduce latency but requires waiting to accumulate multiple carly exits, introducing a trade-off.
D Early Exit Analysis
In Fig. 7 we present detailed comparison of token match percentages and average confidence scores across early exit layers in the Vieuna-7B model. The token match percentage quantifies the proportion of tokens generated by each early exit that match the corresponding tokens in the final model output. As expected, this match percentage increases stcadily with deeper exits, indicating that later layers are more aligned with the final output distribution. Interestingly, the match percentage grows sharply beyond the halfway point of the network, suggesting that substantial refinement of predictions occurs in the latter half of the model's
17

---

Published in Transactions on Machine Learning Research (08/2025)
Table 9: Server and client latency for different batch sizes for Vicuna-68h and Vicuna-7B models on client and server respectively.
Batch Size
Server Latency (A100)
Client Latency (Jetson)
17.94
7.76
21.75
8.02
24.35
7.57
4
24.63
7.87
8
30.73
8.97
12
44.52
9.71
16
45.79
10.63
32
96.94
14.49
258.79
OOM
Token Match vs. Confidence Score Across Exits
0.7
80
Token Match (%)
Avg. Confidence
0.6
Token Match (%)
Avg. Confidence
Original Model Confidence
60
0.5
50
0.4
0.3
30
20
5
15
30
Early Exit Layer
Figure 7:
Token match percentage and Confidence Score across early exits measured with Early Exit adapters trained on the Vicuna-7B model.
depth. Notably, early exits in the initial layers achieve only 20 30% token-level agreement with the final model, underscoring the limited utility of very shallow exits for high-fidelity generation.
Alongside match rates, we also plot the average confidence score at each carly exit, defined as the softmax probability of the most likely token each timestep. While confidence generally correlates with layer depth, observe that confidence saturates earlier than match accuracy. This suggests that some early exits, particularly in the mid-range layers, can achieve high token match percentages despite lower confidence scores-indicating that accurate predictions may be made even before the model's certainty poaks. Such behavior can be leveraged for adaptive inference: exils that combine high match rales with sufficient confidence offer promising points for early stopping, enabling latency reduction without substantial loss in output quality. These findings highlight the value of jointly analyzing confidence and output fidclity when designing efficient. decoding strategies based early exits.
18


Reviews:
[{"review_id": "7pLqklXgA2", "reviewer": "Reviewer_zGA5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Novel Edge-Cloud Framework:\nThe paper introduces a well-motivated and original speculative decoding framework that splits draft and target models across edge and cloud, addressing a key challenge in deploying LLMs on resource-constrained devices.\n\n2. Effective Use of Early Exits and Pre-Drafting:\nThe incorporation of early exits in the target model to enable client-side pre-drafting is a clever design that improves parallelism and reduces latency without compromising correctness.\n\n3. Comprehensive Evaluation:\nThe authors provide thorough experiments across multiple model sizes, tasks, and hardware setups (Jetson Nano and RTX), with detailed breakdowns of latency components, cache behavior, and ablation studies. In addition, they implement the method on the Unitree Go2 quadruped robot for vision-language navigation tasks significantly strengthens the paper’s practical relevance.\n\n4. Clear and Structured Presentation:\nThe paper is well-organized, with clear diagrams and algorithms that enhance understanding of the proposed methodology and system implementation.\n\nWeaknesses:\n1. High Cache Miss Rates on Weaker Devices:\nThe Jetson Nano, a common low-end edge device, exhibits relatively high cache miss rates (up to 70%+), which limits the benefits of pre-drafting in some settings. Further tuning or adaptive mechanisms could be discussed to mitigate this.\n\n2. Limited Multi-Client Support:\nThe current system is designed for single-client use. Scalability to multi-client scenarios, which is important for real-world applications, is mentioned as future work but not explored experimentally.\n\n3. Early Exit Training Details Could Be Expanded:\nWhile early exit adapters are introduced effectively, the paper could benefit from more discussion on how training stability and exit thresholds impact performance across different models and datasets.\n\n4. Assumption of Stable Connectivity:\nThe method relies on frequent client-server communication. Although the authors acknowledge bandwidth constraints, the system may struggle in environments with unreliable or low-bandwidth networks, which could be better quantified.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel and practical framework for speculative edge-cloud decoding that enables fast and cost-effective deployment of large language models (LLMs) on resource-constrained edge devices. The key contributions include several aspects: Edge-Cloud Collaboration for Speculative Decoding, Early Exits with Preemptive Drafting, Real-World Deployment on Robotics Platform. Overall, the work introduces a significant advancement toward scalable, real-time LLM inference in edge-cloud scenarios, addressing both latency and cost-efficiency without sacrificing output fidelity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarify Cache Miss Implications on Low-End Devices: Provide more analysis or discussion on how this affects the overall system performance and whether adaptive exit thresholds or model-specific tuning can help mitigate it.\n\n2. Detail Early Exit Training Procedure and Threshold Selection: Expand on how thresholds are set (fixed, learned, heuristic?) and provide metrics or plots showing the confidence distribution at different layers to justify early exit effectiveness.", "broader_impact_concerns": "This work does not involve any ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LGtO70r62T", "reviewer": "Reviewer_n1bg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) The paper directly tackles the significant issues of high operational costs and latency associated with deploying large language models (LLMs) on edge devices, which is a major barrier for smaller organizations and real-time applications like robotics.\n(2) The proposed \"speculative edge-cloud decoding framework with early exits\" presents a novel solution that effectively leverages both edge and cloud resources. This split between a small draft model on the device and a large target model on the server is a key innovation.\n(3) The method demonstrates impressive quantitative improvements, including up to a 35% reduction in latency compared to cloud-based autoregressive decoding, and an additional 11% improvement from preemptive drafting. It also achieves cost reductions of up to 52% over cloud autoregressive decoding.\n(4) Real-World Validation: The deployment and testing on a Unitree Go2 quadruped robot for vision-language model (VLM) based control provides strong evidence of the framework's practical applicability and effectiveness in resource-constrained, real-time scenarios.\n(5) The authors claim this is the first work to demonstrate end-to-end speculative decoding with models split between edge and cloud, highlighting its pioneering nature.\n\nWeaknesses\n(1) While the paper states the method \"operates effectively without making assumptions about communication delays\"  and that priority queues help optimize bandwidth , it also acknowledges that \"communication latency can be a limiting factor in extreme cases\". Further elaboration on what constitutes \"extreme cases\" and how the system's performance degrades under such conditions would be beneficial.\n(2) While tested on Jetson Nano and A100 GPU, the performance in other edge-cloud configurations (e.g., different network latencies, lower-power edge devices, or various server infrastructures beyond a single GPU) could be further explored or discussed as a future work direction.\n(3) The authors state that the Python implementation \"leaves room for further optimization\" and suggest a low-level C++ implementation for latency-sensitive applications. While understood for a proof-of-concept, a brief discussion of the expected performance gains from such optimizations could provide more complete context.\n(4)  The current system supports \"single-client interaction with the server,\" with multi-client concurrency noted as future work. Given that many real-world LLM deployments involve concurrent requests, elaborating on the challenges and proposed solutions for scaling to multi-client scenarios would be important for broader impact.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A novel edge-cloud LLM decoding framework with early exits and preemptive drafting significantly cuts latency (up to 35%) and costs (up to 52%) while preserving accuracy, demonstrating real-time applicability on a robot.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Adjustments to Strengthen the Work:\n\n(1) The evaluation uses specific hardware configurations like NVIDIA Jetson Nano and A100 GPU. While this is a good start, discussing the potential performance across a wider range of edge-cloud configurations (e.g., lower-power edge devices, varying network latencies, or different server infrastructures beyond a single high-end GPU) would provide a more comprehensive understanding of the framework's scalability and robustness. This could be framed as future work but with more specific considerations.\n(2) The paper notes that the current system supports \"single-client interaction with the server\" and identifies multi-client concurrency as a natural next step. Given that many real-world LLM deployments involve concurrent requests, outlining the key challenges and conceptual approaches for extending the framework to handle multiple concurrent clients would significantly enhance its practical relevance.", "broader_impact_concerns": "The paper, \"Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits,\" primarily focuses on optimizing LLM deployment. However, it presents two areas of ethical implications that warrant a Broader Impact Statement would be (1) Environmental impact ---  While the paper emphasizes cost-effectiveness and efficiency, which can implicitly reduce energy consumption by optimizing API calls and leveraging edge devices, it does not explicitly quantify the potential environmental benefits or drawbacks of this framework. A broader impact statement could address the energy footprint of running LLMs, especially with the distributed nature of the proposed system, and whether this approach genuinely leads to a net reduction in carbon emissions compared to solely cloud-based solutions. (2) Dependence on Specific Hardware and Models: The evaluation highlights performance on NVIDIA Jetson Nano and A100 GPU with specific Vicuna and Llama models. This reliance on particular hardware and models could create dependencies or exclusionary practices if the framework is not easily adaptable to diverse open-source or more accessible alternatives.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UNTk9miiVg", "reviewer": "Reviewer_ni9w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Cost Reduction. The proposed method intelligently utilizes client idle time and server resources, leading to significant demonstrated reductions in both inference latency and operational API costs.\n\n2. The method's effectiveness is robustly evaluated across multiple LLM pairs, standard benchmarks, and different client hardware, all while maintaining output identity with standard speculative decoding. \n\nWeaknesses:\n\n1. Increased System Complexity. The proposed pre-drafting mechanism, with early exits and client-side caching, adds considerable implementation complexity compared to simpler decoding strategies. The system's performance benefits are also highly dependent on the quality of early exit predictions, edge device latency, and communication latency, which might be hard to predict.\n\n2. Early Exit Training Overhead. The necessity of training early exit adapters introduces an additional, non-trivial step and computational cost (100+ GPU hours per model). \n\n3. Communication overhead. Additional communication rounds are required to facilitate the verification at finer granularity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel framework for fast and cost-effective LLM inference. The core idea is to run a small draft model on the edge device to generate candidate tokens, while a large target model on the server verifies these tokens via early exit. A key contribution is the introduction of early exits in the target model, allowing partially verified tokens to be sent back to the client sooner. This enables the client to pre-draft subsequent tokens preemptively, utilizing its idle time and enhancing parallelism. The authors demonstrate significant latency reductions (up to 35% + 11%) compared to traditional cloud-based autoregressive decoding and vanilla edge-cloud speculative decoding.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Discuss System Complexity. Briefly acknowledge the increased system complexity (e.g., managing pre-draft caches, priority queues) in the Discussion section. Also, comment on the sensitivity of FSD's gains to the early exit quality and cache miss rate, perhaps suggesting scenarios or future work to improve early exit calibration or mitigate high miss rates.\n\n2. Address Training Overhead and Communication: In the Discussion or Future Work, briefly mention the early exit training overhead as a practical consideration. Also, touch upon how further advancements in edge-cloud communication infrastructure could unlock even greater potential for the proposed FSD method.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)