Cost-efficient Collaboration between On-device and Cloud Language Models
Authors: Avanika Narayan
Dan Biderman
Sabri Eyuboglu
Avner May
Scott Linderman
James Zou
Christopher Re
Venue: ICML 2025 poster (2025)
Keywords: Local-remote collaboration
reasoning
Primary area: deep_learning->large_language_models
TL;DR: We demonstrate a simple communication protocol between an on-device and cloud-hosted LLM reduces cost 5.7x while retaining 97.9% of cloud-only accuracy.

Abstract: We investigate an emerging setup in which a small, on-device language model (LM) with access to local data collaborates with a frontier, cloud-hosted LM to solve real-world tasks involving financial, medical, and scientific reasoning over long documents. 
*Can a local-remote collaboration reduce cloud inference costs while preserving quality?*
First, we consider a naïve collaboration protocol, coined MINION, where the local and remote models simply chat back and forth. Because only the local model ingests the full context, this protocol reduces cloud costs by 30.4x, but recovers only 87% of the performance of the frontier model.
We identify two key limitations of this protocol: the local model struggles to (1) follow the remote model's multi-step instructions and (2) reason over long contexts. Motivated by these observations, we propose MINIONS, a protocol in which the remote model decomposes the task into easier subtasks over shorter chunks of the document, that are executed locally in parallel. MINIONS reduces costs by 5.7× on average while recovering 97.9% of the remote-only performance. Our analysis reveals several key design choices that influence the trade-off between cost and performance in local-remote systems.

Cost-efficient Collaboration Between On-device and Cloud Language Models
Avanika Narayan 11 Dan Biderman 12 Sabri Eyuboglu" "1 Avner May?
Scott Linderman? 2 James Zou Chris Ré!
Abstract
standard million-token code repository with OpenAl's ol API costs > $15 per query. At the same time, smaller LMs (I-8B parameters) are rapidly improving and can now run on personal computers (Ollama, Ilama.cpp) and smartphones (Mehta et al., 2024; Yi et al., 2024; Xu et al., 2024). Yet, to- day, these small, on-device LMs are used mostly for simple tasks such as tone adjustment and text completion (Gunter et al., 2024). They do not play a role in data-intensive reasoning tasks.
We investigate an emerging setup in which a small, on-device language model (LM) with access local data communicates with frontier, cloud- hosted LM to solve real-world tasks involving financial, medical, and scientific reasoning over long documents. Can a local-remote collabora- tion reduce cloud inference costs while preserving quality? First, we consider a naive collaboration protocol where the local and remote models sim- ply chat back and forth. Because only the local model ingests the full context, this protocol re- duces cloud costs by 30.4x, but recovers only 87% of the performance of the frontier model. We identify two key limitations of this protocol: the local model struggles to (I) follow multiple instructions at once and (2) reason over long con- texts. Motivated by these observations, we pro- pose MINIONS, a protocol in which the remote model decomposes the task into easier subtasks over shorter chunks of the document, that are exe- cuted locally in parallel. MINIONS reduces costs by 5.7x on average while recovering 97.9% of the remote-only performance. Our analysis reveals several key design choices that influence the trade- off between cost and performance in local-remote systems.
Inspired by the growing literature on multi-agent sys- tems (Wang et al., 2024a; Guo et al., 2024), in this work we ask: how can a small, on-device LM collaborate with frontier LM in the cloud to reduce inference costs on data-intensive reasoning tasks? In particular, we study the communication protocols that govern how the two LMs talk to each other, focusing on the tradeoff between cost and accuracy. To mimic realistic use cases, we study tasks that involve varying levels of reasoning over large volumes of medical, financial, and academic data (Islam et al., 2023; Adams et al., 2024; Dasigi et al., 2021).
As our first attempt, we study a simple communication protocol we call MINION: an unconstrained chat between the local and remote models. MINION reduces cloud costs by only "reading" the data locally, and communicating a compressed version of the context to the remote model. We show that while MINION achieves 30.4x reduction in remote model costs, it trails behind the remote-only baseline by 9.4 accuracy points on average (with an 8B local model; see Section 4 for details). In isolated ablations, we identify two key limitations of small LMs that hinder MINION'S performance (Section 4):
1. Introduction
Today's 's cloud-hosted frontier Language Models (LMs) can perform data-intensive reasoning: they can generate and refactor code across entire repositories and make decisions based on financial, legal, and medical documents. How- ever, accessing these models is expensive: processing a
Small LMs struggle to follow multiple instructions at once. Large LMs often dispatch complex instructions. We find that splitting complex instructions into separate requests improves performance by 56%.
'Equal contribution 'Department of Computer Science, Stan- ford University 'Department of Statistics, Stanford Univer- sity 'Together AI 'Department of Biomedical Data Science, Stanford University. Correspondence to: Sabri Eyuboglu <eyuboglu @ std.f.
Small LMs get confused by long contexts. Inereasing context length from < 1K to > 65. decreases perfor- mance by 13% on a simple extraction task.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Motivated by these limitations, we propose MINIONS, an extension of MINION where the remote LM decomposes

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Data-intensive Reasoning
Minion Protocol
MinionS Protocol
Job Preparation
Query
Gased my files, can you
compute FY15 depreciation
Job
emortization margin for AMD?
Local-Remote Conversation
Here is function that genarates jobss...
Extract the D&A from documant. Output "none" pressent.
Context
What was the total revenue FY20157
free the seport teaut-fulSorp
The total reverue 2015 51.3bn.
User's working context
Instruction
Chunk
Find the D&A in the 10-K documant.
Job Execulion and filtering
Local-Remote Systems
can't find any mention DB/
Find the D&A the document.
We need more info.. let's jobs..
Job Aggregation
Fronber LM
Local
Until answer...
Until answer...
Frontier
Figure 1. Local-Remote Systems. MINION and MINIONS protocols. (Left) Problem set-up: local and remote LM collaborate on a data-intensive reasoning task. (Center) MINION: A simple communication protocol in which the local and remote models have an "unconstrained" back and forth chat. (Right) MINIONS: an extension of MINION where the remote LM decomposes a query into many jobs that are processed parallel by the local model. Each job is single instruction over a chunk of the context.
the problem into a set of single instructions to be performed on smaller chunks of the document. Crucially, the remote model has to do this without reading the full document, which it achieves by generating code that is later executed locally where the document is. More precisely, MINIONS involves loop over three steps:
structure parallel workloads at the edge to maximize perfor- mance? Section 6.3, we study three different strategies for increasing the parallel workload on-device: (a) repeated sampling, (b) decomposition, and (c) context chunking. We show that all three can independently improve quality at the expense of increased remote cost.
(c) Sequential communication protocols. Can multiple rounds of communication improve quality? At what cost? We show that by increasing the number of sequential rounds of communication, we can pay more to improve quality.
1. Decompose: Given task, the remote model writes code that decomposes it into "bite-sized"" subtasks.
2. Execute: The local LM then executes the subtasks in parallel and sends filtered selection of the responses back to the remote model.
To summarize, our main contributions are as follows:
Propose MINION, a naïve local-remote LM commu- nication protocol, that achieves 30.4 efficiency over remote-only workloads while recovering 87% of per- formance.
3. Aggregate: The remote model aggregates the local outputs and either finalizes the solution or loops back to the Decompose step.
Propose MINIONS, an extension that overcomes the limitations we identify in MINION, achieving 5.7x cost-reduction over remote-only workloads and recov- ering 97.9% of performance.
Averaged across tasks, MINIONS with an 8B parameter local LM can recover 97. .9% of the performance of remote- only systems 18.0% of the cloud cost (see Figure 2). With 3B parameter local LM, MINIONS achieves 93.4% of the performance of remote-only systems 16.6% of the cloud cost (see Figure 2).
Conduct an in-depth analysis of MINIONS, exploring design choices to traverse the cost-accuracy trade-off.
We perform a detailed analysis of the design and hyperpa- rameter space of MINIONS, Our analysis highlights several "knobs" that allow us to trade off cost for quality.
2. Related Work
See Appendix A for an extended discussion of related work.
(a) Model choice How does the size and family of the lan- guage models affect the cost and quality? We show that MINTONS became feasible only with mid-2024 releases like GPT4-TURBO and LLAMA-3. and is now performant with the latest 3B- parameter models running locally.
This study is inspired by large body of work that explores how to combine multiple LMs and tools to improve quality and reduce cloud inference costs. These include:
Retrieval-Augmenned Generation (RAG) RAG work- flows append retrieved chunks to large LM's
(b) Scaling parallel workloads on-device. How should we
2

---

MINIONS: Collaboration Between On-device and Cloud Language Models
The specific techniques used in MINIONS build upon sev- eral important ideas proposed in the literature:
gpt-4o (0.72)
07
Miiinons
Remote Alone
0.6
Orchestration for long-contexts Prior works have used "divide-and-conuuer strategies to process long docu- ments in smaller chunks (Zhang et al., 2024c; Zhou et al., 2024; Shankar et al., 2024). These methods define chunking and processing protocols, often with pipeline optimizations, but typically rely on a single large LM or symmetric LMs-unlike our asymmetric local-remote setup with cost constraints. They also don't explore parallel on-device tasks or multi-round cloud commu- nication. Other approaches improve single-LM han- dling of lengthy inputs by compressing, summarizing, or streaming data. Techniques like MemGPT (Packer et al., 2023), PRISM (Jayalath et al., 2024), and writing- mn-the-margins (Russak et al., 2024) store partial results or structured data across external memory. While such approaches reduce context overhead for single LM, they do not address distributing computation across a local-remote system with distinct cost models.
MMnion
0.5
Accuracy
ilama-3.1.8b (0.44)
0.4
0.3
[lama-3.2-3b (0.21)
AiocelAione
gw3n2.5-3b (0.14)
0.1
0.00
0.05
0.10
0:15
0.20
Remote Cost
Local Model
Protocol
1aam--.2-33
Local alone
qwen2.5-3b
Remote alone
11ama-3,1-8b
MINION
MINIONS
Figure 2. Cost-Aceuracy Tradcoff in Edge-Remote Systems. Macro-average accuracy (g-axis) vs. cost (z-axis) across Fi- NANCEBENCH (Islam et al., 2023), LONGHRALTH (Adams ct al., 2024), and QASPER (Dasigi et al., 2021). Accuracy represents the fraction of correct predictions, while cost is the average USD per query based on GPT-40 rates (Jan 2025: $2.50/IM input to- kens, $10.00/1M output tokens); see Section 3. The table compares MINION (Section 4) and MINIONS (Section 5) protocols against local-only and remote-only baselines. Points, colored by local LM, use GPT-40 as the remote LM. Exact metrics Table 1.
Decomposition techniques The decomposition tech- niques used in MINIONS are inspired by prior work showing how prompting for decomposition can improve small LM quality (Arora et al., 2022; Patel et al., 2022; Wu et al., 2022). The MINTONS protocol builds on using code to aid reasoning (Arora et al., 2023; Li et al., 2023).
Test-time sampling and verification In MINIONS we pair repeated test-time sampling on-device with veri- fication in the cloud. This technique is motivated by extensive literature demonstrating the promise of using test-time sampling and verification to improve reasoning capabilities (Brown et al., 2024; Song et al., 2024; Hassid et al., 2024; Snell et al., 2024; Wu et al., 2024).
prompt (Lewis et al., 2020; Karpukhin et al., 2020; Lee al., 2019), mitigating hallucinations and externaliz- ing knowledge. Instead of sharing raw text chunks with the remote LM, we provide the filtered output of a flex- ible local LM executing tasks on these chunks, further compressing the input.
Speculative decoding and collaborative inference. Speculative decoding (Leviathan et al., 2023; Zhang et al., 2024a; Chen et al., 2024b) accelerates inference by sampling from a small "draft" model while only using a larger "target" model for verifying tokens. Collaborative inference approaches selectively sample from the tar- get model to correct tokens identified as low-confidence ((Hao et al., 2024) places the small model on an edge device). In (Xia et al., 2023), the cloud model further retrieves external documents, generating summaries that inform the small model's generation. Our work differs by enabling the remote model to orchestrate the local model (in MINIONS, using code), extending beyond simple token evaluation or correction.
Multi-LLM collaboration and routing A growing body of work explores multi-agent or multi-LLM sys- tems (Guo et al., 2024; Wang et al., 2024a) and model- routing (Chen et al., 2023; 2024a; Zheng et al., 2025). Typically, these either combine multiple large models or choose one LM from a "menu" of models for the entire task. In contrast, we explicitly study a two-model collab- oration where the smaller local LM handles extensive on-device context, while the larger remote LM is called on selectively, reducing cloud inference costs.
Compound LM systems A broader line of research inte- grates LMs with retrieval modules, tool use, or orches- trators (Saad-Falcon et al., 2024; Khattab et al., 2023). While they optimize accuracy or adapt prompts, they do not usually focus on asymmetric edge-cloud costs or local parallelization.
Prompt compression One way to reduce cloud inference costs is to shorten input token sequences by removing tokens identified as less informative ((Jiang et al., 2023)).
3

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Informativeness is measured by token probabilities or learned classification heads (Pan et al., 2024). In contrast, we achieve compression by offloading all raw context processing to a small LM, which can summarize and execute tasks on the context.
Number of Chunks in Context
Number of Sub Tasks
0.600
0.7
0.575
0.6
Accuracy
0.550
Accurscy
0.5
0.525
0.4
0.3
Some recent works have explored aspects of the local- remote setting. Several study how local-remote systems can limit leakage of private information to a cloud-hosted LM API (Siyan et al., 2024; Zhang et al., 2024b). In this work, we do not address privacy concerns, though these privacy techniques can be used in conjunction with MIN- IONS. Other techniques partition LM layers between local and cloud devices (Jin& & Wu, 2024; Yang et al., 2024) with- out a multi-round dialogue. Our system is distinct in that the local LM and remote LM collaborate in natural lan- guage on tasks that draw on a large private context. This two-model interplay underlies our focus: reducing cloud inference costs while preserving performance.
0.475
32
64
128
Total Chunks In- In-Context (each 512 tokens)
Number of Sub Taske
Figure 3. Analysis of small LM limitations. Evaluation of LLAMA-3.2-31 on simple extraction tasks (see Section E.2). (Left) Performance drops significantly as context length increases. (Right) Increasing sub-task complexity reduces performance, with fewer sub-tasks yielding better results.
where a varies by provider (=1- 5) (Dubey et al., 2024; Anthropic, 2024). Decode tokens are more expensive since the decoding stage is 10-bound (Leviathan et al., 2023) and has lower GPU utilization. This is because generating each decode token requires loading the full model and KV cache into GPU SRAM.
3. Preliminaries
We first outline the problem setup and then provide details on how we measure accuracy and cost.
Although our focus is on monetary cost, we also measure end-to-en latency of local-remote systems. For these exper- iments, the Locall . is running on single consumer-grade GPU (e.g. RTX 4090, MSRP $1,599). We also include theoretical framework for estimating latency in Appendix C.
Problem setup We study language tasks that involve a context e (e.g. a long document), query q against that context, and ground-truth answer y (see (1) in Figure I).
Context (c):
The Fiscal Year 2015 1C-K report for Advanced Micro Devices, Inc.
Query (q): Compute the 2015 depreciation and amortization margin for AMD (in percentage).
4. MINION: A naïve communication protocol
Answer (y): US$394, 328 million
In this section, we describe MINION, a baseline communica- tion protocol, which implements multi-turn conversation between LocalLM and RemoteT.M. The former has exclu- sive access to the input context c, and the latter sees only the query q and potential metadata about the context.
A local-remote system S (see (2) in Figure 1), consists of two language models that must collaborate to solve the task-a small LM (LocalLM) running on on-device, and a large frontier LM (RemoleLM) running in the cloud, S ingests a context and a query, and applies both models in conjunction to output predicted answer: ÿ ~ S(c. q)
The exchange begins with system prompts: the RemoteL. is told it will collaborate with a smaller, local model that can answer context-related questions, while LocalLM is in- structed to answer questions based the provided context. The RemoteLM initiates the dialogue by asking a ques- tion and the LocalLM replies using the available context. This back-and-ford continues until RemoteLM signals it is ready to answer the query or a maximum number of steps is reached. At each turn, the RemoteLM produces a structured output with a mes ge (to pass to the LocalL.) and a decision (to continue or terminate the conversation loop). This setup lets RemoteLM reason over long contexts without accessing them directly, using LocalLM as re- trieval and summarization interface. See Appendix D. for detailed description of the MINION protocol.
Measuring quality We evaluate the performance of S on a dataset DD = {(C,, qi y. via a scoring metric yt). Here, s(:-)) is binary (correct/incorreet) and we report accuracy. As baselines, we compare S to Vremote ~ RemoteLM(c. q) and ýlocal LocalLM(c, q).
Measuring cost Monetary cost (in $USD) is our primary cost metric. We assume that RemoteLM calls incur a cost while LocalLM calls are free, ignoring the fixed cost of the hardware and marginal cost of energy consumption.
More concretely, the cost of calls to RemoteLM are propor- tional to weighted sum of the number of prefill (i.e. input) tokens and decode (i.e. output) tokens:
We compare MINION to a baseline where RemoteLM is given the full context and the query. Excitingly, MINION
Cremate (Mpretill, dedecdee) X Pereeill + a+ Nllecade:
4

---

MINIONS: Collaboration Between On-device and Cloud Language Models
reduces M costs by 38.,3x, 31.3x, and 20.9x on FINANCRBENCH, LONGHEALTH and QASPER, respec- tively (see Section 6.1 for dataset details). Averaged across these datasets, it closes 87.0% of the quality gap between the MemoteI.M versus the LocalLM operating alone.
Step I: Programmatic job preparation on remote. Our
goal it to create list of job specifications for LocalLM. Each job specification is a tuple including instruction and
chunk of context to run it on t(*) (q(*), ) The list
of job specifications is denoted with T = [t(4), (....
To further close the gap, we analyze MINION conversa- tions and find that in unconstrained chat, RemoteLM of- ten gives LocalLM complicated instructions over long con- texts. Appendix E.2 presents nicro-experimmets illustrating LocalLM's struggles with these instructions:
Extract the total revenue for FY2015, abstain if not present. Try to look for the income statement and make sure it is from 2015.
Instruction (q)):
Context (é "Operating income for North America for the years ended..."
Crucially, to create T, (at iteration t) without seeing the context c, we instead have RemoteLM generate a Python function, f(c. Tς-1), that accepts the full context c and previous job specifications TĄ-1 - both available only on- device and outputs a new list of jobs T,, Access T,-1 allows RemoteLM to previous job outputs to allocate work to specific chunks of context.
1. LocalLM struggles to handle multiple instructions at once. Using GPT-40, generate requests with varying numbers of instructions. We then show that processing each instruction separately leads to a 56 point performance improvement (see Figure 3).
2. LocalLM struggles to reason across long contexts. We show how increasing context length from < 1K to > K5K tokens can decrease performance by 13% on a simple extraction instruction (see Figure 3).
We illustrate an abbreviated version of a Python function that was generated by RemoteLM. It includes fourteen lines of code, but can generate hundreds of jobs when executed (for complete code example, see Appendix G).
Put simply, these smaller LMs are currently better equipped to answer simple queries on shorter contexts.
edataclass
instruction: str
5. MINIONS: A decomposition- communication protocol
chank: str
6 def (ctx: stc, last_jobs: List (Job]) -> List Job]:
Motivated by these observations, we design MINIONS, simple extension of MINION that uses iterative, divide- and-conquer approach. In each step, RemoteLM breaks the task into jobs-atomic instructions tied to context chunks-executed in parallel on-device (see (4) in Fig- ure 1). Rather than a single assistant, MINIONS runs a fleet handling smaller tasks. The key challenge: How can RemoteLM create jobs for unseen context? We solve this by having it generate code that runs locally where the con- text lives.
7
jobs - I:
instructions ["Extract the total revenue for...", "In the statement of cash flow...
"J
chanks chunk_on_pages (ctx) # chunk
context into pages
chunk in chunks:
11
for insts in instructiens: for range (5):
append (Job (instr, chunk))
zeturn jobs
To generate such function, we prompt RemoteLM with the query q and instruction prompt Pdecompose f(::)) ~ RemoteLM(q Palcompose)- The function is then executed locally to create the list of job specifications Tr,
5.1. Protucol description
Each iteration of MINIONS consists of three steps:
This strategy, which builds on work using LMS generate code for information extraction (Arora et al., 2023; Li et al., 2023), allows us to decouple the number of unique jobs from the number tokens generated by the cloud model.
1. On remote. MemoteLM generates code to create job specifications for LocalLM (see 4(a) in Figure 1). The code is sent to device.
On device. The generated code is executed locally to produce the job specifications. Jobs are then executed by LocalLM, and the job outputs are filtered locally for relevance (see 4(b) in Figure 1), and sent to remote.
Job execution and filtering on-device. In this step, we convert jobs T = (1) (?... into prompts and execute them in batch(es) locally. The prompt further guides LocalLM to produce structured output, with fields expla- nation (concisely stating its reasoning), citation (a direct snippet of text supporting the answer), and answer, setting the values to null if the instruction cannot be completed con- fidently on this chunk, We denote each LocalL. response
Step 2:
3. On remote. RemoteLM synthesizes the filtered outputs and decides whether to output an answer or continue to the next iteration (see 4(c) in Figure 1).
5

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Scaling Tasks
Scaling Samples
Scaling Chunks
0.50
de 10 851
0.70
6:65
0.55
0.52
36
0.50
0.45
0:30
6:45
0.45
0:45
0.40
1cc
0.02
0.03
3.04
0.05
0.00
0.05
0.04
0.87
Romote Cost
Remoto Cost 181
Bermtt Cost
Uamrdd 1-48- ratruct
Figure 4. Scaling parallel jobs on-device improves quality. The x-axis represents tokens processed by the remote model, and the y-axis shows macro- average accuracy across LONGHEALTH and QASPER. The cloud model is GPT-4o. Each plot varies different MINIONS hyperparameter affecting parallelism, with annotated values. (Left) Varying the number of unique instructions. (Middle) Varying the number of unique samples. (Right) Varying the chunking granularity in code f. See Section 5 for details.
with (*)
knobs for increasing the degree of task decomposition and
( LocalLM( Pworker)
thus, workload parallelization: (1) number of tasks per round (i.e. "Extract the ARR for Q1 of 2014"), (2) number of samples per task (i.e. number of repeated generations created with LocalLM, 2 1), and (3) chunk size (i.e. chunk by page, chunk by paragraph, etc; smaller chunks will send more information to cloud).
We discard any (0) for which the model abstained intu- itively, many instructions will be irrelevant to their paired chunks allowing us to avoid sending unnecessary infor- mation to the RemoteLM. The surviving instruction-chunk pairs concatenated into single string w.
Sequential communication protocol. In practice, it is important to cap the number of times MINIONS can loop. After the maximum number of rounds, the synthesis prompt is modified to force the model to produce final answer. The choice of this maximum affects accuracy and cost. The strategy for maintaining context between rounds (simple retries vs. scratchpads) is another important hyperparameter.
Step 3: Job synthesis on remote. RemoteLM receives w and synthesis prompt Psyntesizee that, like in MINION, asks it to generate a structured output a with decision and answer fields:
y ~ RemoteLM(W, Psynthesize
(2)
If the RemoteLM decides that more information is needed, the loop continues from Step 1.
We analyze these hyperparameters in Section 6.
6. Results
There are several ways to maintain context across MINIONS rounds. One is to keep the entire the conversation in con- text, yet, this incurs significant additional cost, even with caching. We experiment with two alternatives: (1) simple retries, in which only the RemoteL.Mss advice is carried over between rounds and (2) scratchpads, in which the RemoteLM can record what it learned from the round before proceeding to the next.
Here, we analyze how the design of MINIONS affects cost and quality. Our main takeaways are: (a) On average across three datasets, MINIONS can recover 97.99 77.9% of the perfor- mance of remote-only systems while spending 5.7x less; (b) We identify protocol hyper-parameters that let us flex- ibly trade-off cost and quality; (c) As local models grow stronger, MINIONS becomes increasingly cost-effective.
5.2. Protocol hyper-parameters
We structure our analysis around three core design choices:
MINIONS has three hyper-parameters: choice of RemoteLM and LocalLM (model choice), job preparation strategy (scale of parallel workloads on-device), and looping strategy (sequential communication protocol).
1. Model choice How does the choice of local and remote model effect cost and quality? We examine different model types and sizes for LocalLM in Section 6.2.
2. Scaling parallel workloads on-device How should we structure parallel workloads on the local device to maxi- mize performance and minimize cost? We highlight how scaling the local workloads can improve performance (Section 6.3) and study the effects on cost.
Model choice. Different model sizes (e.g. 3B vs. 8B), families (e.g. QWEN2.5 vs. LLAMA) can be used for both the LocalLM and the RemoteLM.
Scale of parallel workload on-device. MINIONS has three
6

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Protocol
Local Model
Remote Model
Macro
Avg.
BINANCEBENCH
LONGHEALTH
QASPER
Acc.
Cost
Acc.
Cost
Acc.
Cost
Acc.
Cost
Remote Only
GPT-4o
0.724
S0.233
0.826
$0.261
0.748
$0.301
0.598
$0.137
Local Only
LLAMA-8B
0.444
$0.000
0.326
$0.000
0.468
$0.000
0.538
$0,000
Local Only
LLAMA-1B
0.038
$0.000
0.000
$0.000
$0.000
$0.000
Local Only
LLAMA-3B 3B
0.213
50.000
0.130
$0.000
0.345
$0.000
0.164
$0.000
Local Only
QWRN-3R
0.140
50.000
0.087
50.000
0.177
$0.000
0.156
$0.000
MINION
LLAMA-80 8B
GPT
0.630
S0.008
0.804
$0.007
0.635
$0.010
0.450
$0.007
MINION
LLAMA-3B
GPT-4n
0.518
50.010
0.698
50.010
0.482
$0.009
0.372
$0.011
MINION
QWEN-3D
GPT-40
0.236
$0.028
0.217
$0.029
0.281
$0.021
0.210
$0.035
MINIONS
LLAMA-8B
GPT-4n
0.709
$0.042
0.804
$0.053
0.740
$0.054
0.582
$0.019
MINIONS
LLB
GPT-40
0.662
$0.052
0.726
$0.079
0.703
$0.057
0.558
$0.020
MINIONS
QWEN-3R
GPT-4n
0.676
50.039
0.783
50.059
0.645
$0.043
0.600
$0.015
Table Accuracy and cost of local-remote systems. Evaluation cost and accuracy on evaluations datasets. The table compares two edge-remote Pitocolss-MiDDE (Section 4) and MINIONS (Section agaanaistt edge-only and remote-only baselines. We assess local models and remote model. Cost (USD) is the average per-query expense, based on GPT-40 rates (Jan 2025: $2.50M/input tokens, $ $10.00M/output tokens). Local model execution is assumed free (see Section 3 for cost details).
3. Sequential communication protocol Can multiple rounds of communication improve quality? At what cost? We explore this trade-off in Section 6.4.
How does the capacity of LocalLM affect the cost- accuracy tradeoff? In our system, LocalLM implicitly acts as an information encoder, optimizing the Information Bottleneck objective (Tishby et al., 2000) by compressing input context while preserving predictive information (see Appendix D.2). To measure this, we analyze the tradeoff be- tween remote "prefill" tokens (fewer tokens indicate greater compression) and accuracy (higher accuracy means better retention). Figure 5 shows that as LocalLM size increases, representations become more compressed and accurate, im- proving Information Bottleneck values. Larger LocalLM models trade local FLOPPs for communication, with 7-8B models being 1.53x more token-efficient than IB models. Additionally, the QWEN2.5 family follows a different trade- off than LLAMA, yielding more compressed representations. This suggests that as small LMs improve, local-remote sys- tems will become increasingly cost-efficient.
Our findings are detailed in Sections 6.2, 6.3, and 6.4. Additionally, we analyze latency in Section 6.5.
6.1. Experimental setup
Datasets and models We evaluate MINIONS on three benchmarks that are well suited for data-intensive reason- ing: FINANCEBENCH, LONGHEALTH, and QASPER. Fi- NANCEBENCH tests financial document understanding with complex reasoning over reports. LONGHEALTH focuses on tracking and interpreting longitudinal health records. QASPER assesses question answering over dense scientific papers. See Appendix B.0.1 for details. We use two open- source model families (LLAMA, QWEN2.5) as LocalLM and GPT-40 RemoteLM (details in Appendix B.O.2).
Does the performance of MINTONS relative to the remote- only baseline change with increasing task complexity? In Tables 8 and 9, we stratify the queries by complexity. Our main finding is that the accuracy of MINIONS relative to the remote-only baseline actually improves with increasing task complexity. For example, on simple information extraction tasks in FINANCEBENCH, MINIONS with QWEN2.5 trails the remote only baseline by -22.7 accuracy points, but on tasks requiring numerical reasoning on top of extracted data, MINIONS outperforms the remote only baseline by +4.60 points. On LONGHEALTH, LLAMA-8B MINIONS trails by 6.2 points on questions requiring a single evidence span but outperforms the remote only baseline by +16.0 points questions requiring synthesis of three evidence spans. This pattern is consistent across local model sizes.
6.2. Model choice
This section explores the model requirements and general- ization capabilities of MINIONS, examining the local model sizes necessary for effective collaboration, the sensitivity of the communication protocol across different local-remote model pairings, and the longitudinal evolution of MINIONS performance with advances in model capabilities over time.
What size does LocalLM have be in order to be effective in MINIONS? Our results demonstrate that MINIONS starts being competitive with MemoleLM-only baseline at the 3B parameter model scale. When considering both the QWEN2.5 and LLAMA model families running locally, at 1B scale, MINIONS recovers 49.5% of the GPT-40-only baseline performance, 3B scale recovers 93.4% and 8B re- covers 97.9% accuracy (see Table for more details). We note that it is now feasible to run 1-8B parameter mod- els modern laptops (Apple M-Series) and workstations (NVIDIA DGX Spark) (Apple, 2024).
Is MINTONS sensitive to different local/remote pairs? We ask whether the communication protocol in MINIONS is invariant to changing the model types (i.e. LLAMA vs QWEN2.5 locally and LLAMA vs GPT-40 remotely). Our results indicate that MINTONS performs similarly with dif-
7

---

MINIONS: Collaboration Between On-device and Cloud Language Models
ferent local-remote LM combinations (see the Table I): varying the LocalLM from QWEN2.5 to LLAMA-3.2, re- sults in performances within + .05 performance points (see Table 1). Furthermore, we find that holding the LocalLM fixed as LLAMA-3. -.--3-B and varying RemoteLM from GPT-40 to LLAMA-3.3-700 leads to similar overall perfor- mances within + 0.07 points (see Table 3 in Appendix).
6.4. Scaling sequential communication
Both the MINION and MINIONS communication protocols feature sequential communication: they allow for multiple rounds of exchange between the local and remote models.
Does performance improve as we increase the maximum number of rounds? At what cost? We vary the maxi- mum communication rounds and find it is correlated with accuracy and cost (see Figure 5). By simply increasing the maximum number of rounds in MINION from 1 to 5, we enable a 8.5-point lift in average accuracy across the three tasks (with LLAMA-3.2 on-device). However, this accuracy improvement comes at a cost: each additional round of com- munication increases the cost by $0.006 per query while boosting accuracy by 4.2 points.
How have local remote model capabilities changed over time, and what effects do they have MINIONS? In Table 4, provide retrospective analysis demonstrating how the quality of MINIONS would have changed with model releases over time. From 2023 to 2025, the average performance of MINIONS with the best models available has improved from 0.26 to 0.66 (see Table 4 in Appendix). Interestingly, was only in July 2024- with the release of GPT4-TURBO and LLAMA-3. 1-8BBBBB could have come within 12% of the best frontier model performance at the time (see Table 4 in Appendix).
How should we maintain context between MINIONS rounds? We experiment with two sequential protocol strategies: (1) simple retries and (2) scratchpad. See Sec- tion for details of these strategies. As shown in Figure 7, both strategies show consistent increases in both accuracy and cost when increasing the maximum number of rounds, with the scratchpad strategy achieving a slightly better cost- accuracy tradcoff. Notably, each additional round of com- munication with the scratchpad strategy leads to a larger improvement in accuracy (6.1 accuracy points) which are mostly offset by larger increases in cost (8.6 dollars).
that MINIONS
6.3. Scaling parallel workloads on-device
In MINIONS, there are three levers for maximizing local compute resources through parallelized, batched processing: (I) number of tasks per round, (2) number of samples taken per task, and (3) number of chunks. We ablate each, showing their impact on performance. We find that (1) and (3) are more cost effective ways of increasing performance.
6.5. Latency of local-remote systems
How does the number of tasks per round affect perfor- mance? Increasing tasks per round proxies task decom- position, with more sub-tasks enhancing decomposition. Raising tasks from 1 to 16 boosts performance by up to 14 points but doubles RemoteLM prefill costs. Optimal task count varies, but exceeding 16 reduces performance.
Here, we compare the latency of running MINION and MIN- 1ONS with the latency of remote-only or local-only systems.
As reported in Table 2, on a single RTX 4090, a consumer- grade graphics card for gaming (MSRP $1 $1,599), MINION with LLAMA-3B runs only 3.5% slower than remote only system. With LLAMA-8B, MINION'S 's latency increases by only 44.4%. Although MINIONS processes far more tokens than MINION or the the remote only baselines (16x more in these experiments), with careful batching and hardware- aware prefix sharing (Juravsky et al., 2024), we show we can run MINIONS on single RTX 4090 2.15x and 2.89 the latency of the remote only system.
How does scaling local samples affect performance? We explore whether increased sampling at an individual [task, context] level improves performance. Increased sampling enables us to better utilize the available compute resources while improving task-level accuracy (Brown et al., 2024). Our results indicate that increasing the number samples from 1 to 32 can improve performance on average 7. 7.4 points, but comes at the cost of 5x the RemoteLM prefill costs. Though, increasing sampling beyond 16 starts hurting per- formance because the remote model struggles to parse many conflicting responses (Kuratov et al., 2024).
The measurements described above are imperfect for pro- jecting the latency that future users of local-remote systems would experience. For one, they depend on a number of unobserved variables on the remote side (e.g. remote hard- ware and user load) that are subject to change in the future. Furthermore, local hardware is improving rapidly is the software used for serving LLMs efficiently on edge devices.
What effect does chunk size have on downstream perfor- mance? We test whether increasing local utilization by using more chunks per task improves performance. Our
results indicate that increasing # of chunks per task (by de- creasing the number of "pages" per chunk from 100 to 5) leads to an 11.7 point accuracy lift. However, this lift comes with 2.41 x increase in RemoteLM prefill costs.
To address these limitations, in Appendix C, we include a thcoretical framework for estimating the latency overhead of local-remote system. This framework can be applied to any
8

---

MINIONS: Collaboration Between On-device and Cloud Language Models
gpt 4o [0:67]
5.66
Qwenz.5-78
gp (0.67)
0.63
0.65
Llama-3.1-08
Llama-3.2.3
0.60
0.55
0:33
0.55
Qwen2.5-33
Accuracy
Accuracy
0.50
0.50
0.45
0.45
0:40
0.35
Qwen2.5:1.58
0.31
Llama-3.244 3.2-18
0.30
0.30
12000 14000
18000 20000 22000
Model Size (Billion Parameters)
Remote Promet Takens
Local Model Family
Qwen
mata liama
Edgo Model
Owen meta-llama
Figure 5. Trade-off's in edge model performance, communication efficiency, and cost of sequential communication. (Left) Accuracy vs. local LM size, with the purple dashed line showing the GPT-40 model baseline. (Right) Communication efficiency of MINIONS with different local LMs, where larger LMs (7-8B) are more token-efficient.
Protecol
Local
Remote
Latmey (RTX 4000))
Laiency (RTX 6000 Ada)
our reliance on cloud API's. We take a preliminary step by extending the MINION framework to support local tool use (see Section E.5), showing that 3B models can complete multi-step tasks with light remote guidance-approachhnee
Toeal
Local
Remsoe
TTeal
Local
Kemote
Remote Unly
GP
11.00
0.00
11.00
11.00
11.00
Local
LLAMA-8R
21:13
0.00
19.25
19.25
0.00
Local Only
LLAMA-3B
11.33
0.00
11.01
11.01
0.00
MINION MINION
LB
GPT-40
6.17
29.19
15.03
GPT-40 performance while using far fewer remote tokens.
LLAMA-3B
GPT-40
11.39
5.11
5,88
15:32
5.51
9.81
MINIONS
LLAMA-SB
GPT-40
31.81
16.99
14,82
35.08
19.39
15.09
Local-remote model co-design There is great promise in going beyond orchestration and finetuning small models to be better minions. Our preliminary results show that MINION accuracy can be improved by simple supervised finetuning of LocalLM on the target domain (see G.2). Fu- ture work should train minions to more concisely and ac- curately communicate with remote models, and, eventually, jointly train two models to cooperate better. Furthermore, communication should not be limited to natural language.
MINIONS
L1AM4-3R
GPT-4n
23.65
9.23
14.42
26
79
10.76
16001
Table 2. Latency of local-remote systems. Latency is measured seconds on ten samples from the LONGHEALTH dataset. The LocalLM is run on a single consumer-gran grade GPU: Nvidia RTX 4090 (MSRP $1,599) or Nvidia RTX 6000 Ada (~$$,000)) See Appendix C for an accompanying theoretical analysis of latency.
local and remote configuration by simply plugging in model and hardware specifications (e.g. memory bandwidth, model size). For instance, we estimate that running MINIONS with LLAMA-8B on a single RTX 4090 locally and Llama 405B on full 8x H100 node remotely would incur a 4.75x latency increase compared to running the remote alone.
Energy efficient inference systems Local-remote frame- works like MINIONS can cut the energy footprint of AI by shifting compute from centralized data centers to efficient edge devices. Cloud inference is energy-intensive due to overheads like cooling and idle overprovisioning (Strubell et al., 2019; Henderson et al., 2020). Offloading to on-device models reduces reliance on power-hungry accelerators. As a first step, we quantify these savings in Section E.4, showing that MINIONS uses up 12x less energy than fully remote execution, highlighting hybrid deployment as promising path for sustainable AL.
7. Discussion
MINIONS offers insights into evolving workload distribu- tions and the expanding role of local compute. We explore two protocols-Moe MIINION and MINIONS - for collaboration between on-device and cloud LMs. Our results show that cloud compute costs can be reduced by 5-26x by enabling remote LMs to delegate work to local ones. We also discuss broader implications and future directions.
Privacy implications MINIONS offers a promising path to greater privacy by keeping user data on local devices during LLM inference. More broadly, local-remote systems raise important questions around privacy and security. Fu- ture work should establish formal guarantees against data leakage and secure the interface between local and remote components- for example, preventing reverse engineering or tampering of transmitted representations. Advancing adversarial robustness, secure protocols, and local model auditing will be key to hardening these systems.
User experience Soon, commodity hardware- haptoare-lappops, smartphones, and loT devices- will host powerful GPUs, enabling always-on local models for tasks like code refac- toring, document analysis, and anomaly detection. As LocalLM capabilities grow, offloading complex, tool- augmented tasks to local compute becomes viable, reducing
9

---

MINIONS:
Collaboration Between On-device and Cloud Language Models
Impact Statement
Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024.
MINIONS advances cost-efficient Al by demonstrating how small on-device language models can collaborate with pow- erful cloud-hoster models to perform data-intensive reason- ing. By reducing reliance expensive remote inference, MINIONS makes advanced Al more accessible and sus- tainable. This has broad socictal implications, including lowering barriers to Al adoption and enhancing data privacy by keeping more computations local. However, careful con- sideration must be given to potential biases in small models and the security risks of local code execution.
Chang, Lo, K., Goyal, T., and lyyer, M. Booookscore: A systematic exploration of book-length summarization in the era of llms. arXiv preprini arXiv:2310.007755 2023.
Chen, L.,, Zaharia, M., and Zou, Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05/76 2023.
Chen, L., Davis, J. Q.. Hanin, B., Bailis, P., Stoica, I. Za- haria, M., and Zou, J. Are more Ilm calls all you need? towards scaling laws of compound inference systems. arXiv preprint arXiv:2403.02419 2024a.
References
Adams, L., Busch, F., Han, T., Excoffier, J.-B., Ortala, M., Löser, A., Aerts, H. J., Kather, N., Truhn, D., and Bressem, K. Longhealth: A question answering benchmark with long clinical documents. arXiv preprint 1rXiv:2401114400 2024.
Chen, Z., May, A., Svirschevski, R., Huang. Y., Ryabinin, M., Jia, Z., and Chen, B. Sequoia: Scalable, robust, and hardware-aware speculative decoding. arXiv preprint urXiv:2402.123777 2024b.
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., and Gardner, M. A dataset of information iilormation-scekiiii sceking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4599-4610, 2021.
AI, E. How much energy does chatgpt use? https://epcch.al//rrdiien-----e
cov-muuhhennarcyy-dde-----ttogge 2024.
wtwrmmuchheeeeryy-ddooe chep-cyypttt use. Ac-
cessed: 2025-03-28.
Devroye, L. Nonuniform random variate generation. Hand- books in operations research and management science, 13:83-121, 2006.
Anthropic.
The Claude Model
Fam- URL
ily: Opus, Sonnet, Haiku.
2024.
/top:://wwwwcn// anthropic. com/
b
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A.. Fan, A., et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21788 2024. URL https:// www.//gg//b////oooo///////e..ea/ / 21783.
Apple. Apple introduces m4
pro
and
m4
max, October 30 2024.
URL https:
//www. wtt.:/cc.//oonneeeeeeee http////wwoooo////////e/
arxiv. orgiv.orrgaan abs
apple-introduces -ao-----
Gunter, T., Wang, Z., Wang, C., Pang, R., Narayanan, A., Zhang, A., Zhang, B., Chen, C., Chiu, C.-C., Qiu, D., al. Apple intelligence foundation language models. arXiv preprint arXiv:2407.2107. 2024.
Accessed: 2025-03-31.
Arora, S., Narayan, A., Chen, M. F., Orr, L., Guha, N., Bha- tia, K., Chami, I., Sala, F., and Ré, C. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02444 2022.
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:240..0168/ 2024.
Arora, S. Yang, B., Eyuboglu, S., Narayan, A., Hojel, A., Trummer, I, and Ré, C. Language models enable simple systems for generating structured views of heterogeneous data lakes. arXivv2304.09433 2023.
Guu, K., Lee, K., Tung, Z.,, Pasupat, P., and Chang, M.- W. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference Machine Learning, 2020.
Biderman, D., Portes, J., Ortiz, J. G., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J.. et al. Lora learns less and forgets less. Transactions on Machine Learning Research, 2024.
Hao, Z., Jiang, H., Jiang, S., Ren, J., and Cao, T. Hy- brid slm and Ilm for edge-cloud collaborative infer- ence. In Proceedings of the Workshop on Edge and Mobile Foundation Models, EdgeFM '24, pp. 36 36-41,
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys:
10

---

MINIONS: Collaboration Between On-device and Cloud Language Models
New York, NY, USA, 2024. Association for Comput- ing Machinery. ISBN 9798400706639, doi: 10.1145/ 1662006.3662066 URL pettps:/doi..omm org/10. 1145/3662006.
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., San- thanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. Dspy: Compiling declarative language model calls into impr-iipg pipelines. arXiv preprint arXiv:23 14, 2023.
Hassid, M., Remez, T., Gehring, J., Schwartz, R., and Adi, Y. The larger the better? improved IIm code-generation via budget reallocation. arXiv preprint arXiv:2404.00725, 2024.
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., and Burtsev, M. Babilong: Testing the limits of Ilms with long context arXiv preprint arXiv:2406. 10149, 2024.
Henderson, P. Hu, J.. Romoff, J., Brunskill, E., Jurafsky, D., and Pineau, J. Towards the systematic reporting of the energy and carbon footprints of machine learning. In Proceedings of the 37th International Conference on Machine Learning (ICML), pp. 4323 4332. PMLR, 2020.
Lee, K., Chang, M.-W., and Toutanova, K. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv: 1rX.000000000000,, 2019.
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Inter- national Conference on Machine Learning, pp. 19274- 19286. PMLR, 2023.
Hu, E. J., Shen, Y., Wallis, P., Allen-llnnie Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1,2)3,,000 2022.
Islam, P. Kannappan, A., Kiela, D., Qian, R., Scher- rer, N., and Vidgen, B. Financebench: A new bench- mark for financial question answering. arXiv preprint arXiv:2311.11944 2023.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H.,, Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmenned generation for knowledge- intensive nlp tasks. Advances in Neural Information Pro- cessing Systems, 33:9459-9477 2020.
Izacard, G. and Grave, E. Unsupervised dense information retrieval with contrastive learning. In Advances in Neural Information Processing Systems, 2021.
Li, C., Liang, J., Zeng, A., Chen, X., Hausman, K., Sadigh, D., Levine, S., Fei-Fei, L.,, Xia, F., and Ichter, B. Chain of code: Reasoning with language model-augmented code emulator. arXiv preprint arXiv:2312. 12.44444.4444,.. 2023.
Jayalath, D., Wendt, J. B., Monath, N., Tata, S., and Gunel, B. Long-range tasks using short-context Ilms: Incremen- tal reasoning with structured memories. arXiv preprini 1rXiv:2412.189144 2024.
Mehta, S., Sekhavat, M. H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, S. I., Najibi, M., Belenko, D., Zat- loukal, P., et al. Openelm: An efficient language model family with open training and inference framework. In Workshop on Efficient Systems for Foundation Models II@ ICML2024, 2024.
Jingg H., Wu, Q., Lin, C.-Y., Yang, Y., and Qiu, L. LLM- Lingua: Compressing prompts for accelerated inference of large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 13358-13377 13376, Singapore, December 2023. Association for Computational Linguistics. doi: 00.18633/v//023. emnlp- mainp-main.825. URL https:/ / /aclanthoooog.y.... org/2023,
Neelakantan, A., Xu, T., Puri, R., Radford, A., Han, J. M., Tworek, J., Yuan, Q., Tezak, N., Kim, J. W., Hallacy, C., et al. Text and code embeddings by contrastive pre- training. arXiv preprint arXiv:2201. 10005, 2022.
825//
Packer, C.. Wooders. S., Lin, K., Fang, V., Patil, S. G., Stoica, I., and Gonzalez, J. E. Memgpt: Towards llms operating systems. arXiv preprint arXiv:2310.08560 2023.
Jin, H. and Wu, Y. Ce-collm: Efficient and adaptive large language models through cloud-edge collaboration. arXiv preprint arXiv:2411.02822 2024.
Juravsky, J.. Brown, B., Ehrlich, R., Fu, D. Y., Ré, C., and Mirhoseini, A. Hydragen: High-th thghtthroughput I1m inference with shared prefixes. arXiv preprini 202.0994,,. 2024.
Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q. Rühle, V., Yang. Y., Lin, C.-Y., Zhao, H. V., Qiu, L., and Zhang, D. LLMLingua-2: Data distil- lation for efficient and faithful task-agnostic prompt compression. In Ku, L.-W., Martins, A., and Sriku- mar, V. (eds.), Findings of the Association for Computa- tional Linguistics: ACL 2024, pp. 963-981, Bangkok, Thailand, August 2024. Association for Computa- tional Linguisties. doi: 10. 116118/112222, 1/5S3//1/222..ndinns--aaa findings-acl.
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.t. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004. 04906 2020.

---

MINIONS: Collaboration Between On-device and Cloud Language Models
57. URL hwtph:ttp//// findings-acl.574.
Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. arXiv preprint physics/0004057, 2000.
Patel, P., Mishra, S., Parmar, M., and Baral, C. Is a ques- tion decomposition unit all we need? arXiv preprint arXiv:2205.1255:. 2022.
Wang. J., Wang, J., Athiwaratkun, B., Zhang, C., and Zou, J. Mixture-of-agemn enhances large language model capabilities. arXiv preprint urXiv:2406.04662 2024a.
Petroni, F,, Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019.
Wang. K., Pam, J., Shi, W., Lu, Z., Ren, H., Zhou, A., Zhan, M., and Li, H. Measuring multimodal mathematical reasoning with visiin dataset. Advances in Neu- ral Information Processing Systems, 37:95095-55169, 2024b.
Robertson, S. and Zaragoza, H. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3:333-389, 01 2009. doi: 10. 1561/150000001.
Wu, T., Terry, M., and Cai, C. J. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI conference on human human factors in computing systems, pp. 1-22, 2022.
Russak, M., Jamil, U., Bryant, C., Kamble, K., Magnuson, A., Russak, M., and AlShikh, W. Writing in the margins: Better inference pattern for long context retrieval. arXiv preprint Xivi:22408. 14906, 2024.
Wu, Y., Sun, Z.,, Li, S., Welleck, S., and Yang, Y. An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.000 2024.
Saad-Falcon, J., Lafuente, A. G., Natarajan, S., Maru, N., Todorov, H., Guha, E., Buchanan, E. K., Chen, M., Guha, N., Ré, C., et al. Archon: An architecture search framework for inference-time techniques. arXiv preprint MoXiv:2409.152555 2024.
Xia, M., Zhang, X., Couturier, C., Zheng, G., Rajmohan, S., and Ruhle, V. Hybrid-RACA: Hybrid Retrieval- Augmented Composition Assistance for Real-time Text Prediction. arXiv e-prints, art arXiv:2308.0423 August 2023. doi: (0.48550/a2Xiv.2338.00
Shankar, S., Parameswaran, A. G., and Wu, E. Docetl: Agen- tic query rewriting and evaluation for complex document processing, arXiv preprint arXiv:2410. 12189, 2024.
Xu, D.,. Zhang, H., Yang, L., Liu, R., Huang, G., Xu, M. and Liu, X. Empowering 1000 tokens/second on- device Ilm prefilling with mllm-npu. arXiv preprint 2024.
Shuster, K., Kiela, D., Perez, E., de Vries, H., Urbanek, J., Szlam, A., and Weston, J. Retrieval augmentation reduces hallucination in conversation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
Yang, Z., Yang. Y., Zhao, C., Guo, Q., He, W., and Ji, W. Per- llm: Personalized inference scheduling with edge-cloud collaboration for diverse Ilm services. arXiv preprint 2024.
Siyan, L., Raghuram, V. C., Khattab, O., Hirschberg, J., and Yu, Z. Papillon: Privacy preservation from internet-based and local language model ensembles. arXiv preprint 24X1v:2410.11127. 2024.
Yi, R., X., Xie, W., Lu, Z., Wang, C., Zhou, A., Wang, S., Zhang, X., and Xu, M. Phonelm: an efficient and capable small language model family through principled pre-training. arXiv preprint MxXi:24111.00000 2024.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.0333 03314, 2024.
Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., and Zou, J. Textgrad: Automatic" dif- ferentiation" via text. arXiv preprint arXiv:2406.0744 07496. 2024.
Song, Y., Wang, G., Li, S., and Lin, B. Y. The good, the bad, and the greedy: Evaluation of Iims should not ignore non- determinism. arXiv preprint arXiv:2407. 10457, 2024.
Zhang, K., Wang, J.,, Ding, N., Qi, B., Hua, F.,, Lv, X., and Zhou, B. Fast and slow generating: An empirical study on large and small language models collaborative decoding. arXiv preprint arXiv:24406 12295,22.52 2024a.
Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for deep learning in nlp. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6645-3650, Association for Computational Linguistics, 2019.
Zhang, K., Wang, J., Hua, E., Qi. B., Ding, N., and Zhou, B. Cogenesis: A framework collaborating large and small
12

---

MINIONS: Collaboration Between On-device and Cloud Language Models
language models for secure context-aware instruction following. arXiv preprint arXiv:2403.03/29, 2024b.
Zhang, Y., Sun, R., Chen, Y., Pfister, T., Zhang, R., and Arik, S. Ô. Chain of agents: Large language mod- els collaborating on long-context tasks, arXiv preprint vrxivv24466022815 2024c.
Zheng, L., Chiang, W.-L., Sheng. Y.. Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z.,, Li, 7.,, Li, D., Xing, E.,, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Zheng, W., Chen, Y., Zhang, W., Kundu, S., Li, Y., Liu, Z.,, Xing, E. P., Wang, H., and Yao, H. Citer: Collaborative in- ference for efficient large language model decoding with teven-level routing. arXiv preprint 1117: 2025.
Zhou, Z., Li, C.,, Chen, X., Wang, S., Chao, Y., Li, Z., Wang, H., An, R., Shi, Q. Tan, Z., al. Llm x mapreduce: Sim- plified long-sequence processing using large language models. arXiv preprint arXiv:2410.09342, 2024.
13

---

MINIONS: Collaboration Between On-device and Cloud Language Models
A. Extended Related Work
Orchestration of LMs Recent works attempt to improve long document processing by taking a divide-and-conquer approach akin to MINIONS. Instead of using single LM calls with the entire context, the task is decomposed into smaller tasks to be executed on chunks of context. (Zhang et al., 2024c: Zhou et al., 2024) use predefined protocol for chunk processing (defined by prompt). (Shankar et al., 2024) performs a more involved automated pipeline optimization (via agent-based rewrite directives). Crucially, none of the works study the cost-efficient interaction between small local LM and large remote LM and instead focus exclusively on larger LMs (70B parameters and above). Moreover, they do not explore multi-round communication patterns for document analysis.
Long-context management techniques These works aim to improve single LM accuracy in long context tasks. (Russak et al., 2024) prefill the context using chunks of the document, summarize each chunk (using a predefined prompt), and aggregate the results. This improves accuracy and requires marginal additional computation. PRISM similarly (Jayalath et al., 2024) processes the context as a stream of chunks, and writes important information into a typed data structure which can be amended as needed. MemGPT (Packer al., 2023) proposes a virtual memory paging system inspired by operating systems, where the LLM manages information across main context (akin RAM) and external storage (akin to disk). When approaching context limits, the system actively decides what information to preserve and can later retrieve this information through paginated function calls. Orthogonally, other methods explore the usage of code for context management (Arora et al., 2023).
Cost-efficient multi-LLM Systems A plethora of recent works show that multiple LMs can collaborate on a task to improve both accuracy and efficiency (Guo et al., 2024). The most similar work is perhaps (Wang et al., 2024a) which neither investigates LMs with with asymmetric capabilities nor optimizes for local compute efficiency.
Model routing techniques Our work studies a collaboration of LMs, and thus should be diffcrentiated from model routing techniques (Chen et al., 2024a; 2023) that route a prompt to the appropriate single LM that can completely answer it using the full context. This is often done for cost reduction, identifying that simple tasks can executed by smaller and cheaper LMs.
Compound LM systems Recent works explore the use of LMs as part of more elaborate pipelines that, retrieval models, tool use, and more. (Saad-Falcon et al., 2024; Khattab al., 2023; Yuksekgonul et al., 2024) seeks to optimize the pipeline architecture and prompts using different approaches, which we do not pursue on this work.
Retrieval-Augmeeneed Generation (RAG) RAG is hybrid approach that integrates information retrieval into the text generation process, leveraging external knowledge sources to enhance the output of language models (LMs). Instead of relying solely on parametric memory, RAG reduces the number of tokens processed by an LM by first retrieving subset of relevant documents or document chunks and appending them as context to the LM (Lewis et al., 2020; Karpukhin et al., 2020; Lee et al., 2019; Izacard & Grave, 2021; Guu et al., 2020). This retrieval step mitigates issues such as hallucination and knowledge staleness, which are common in traditional autoregressive models (Shuster et al., 2021; Petroni et al., 2019) 9). We differ in two ways: first, our local LM can perform tasks beyond information extraction, such as summarization or reasoning. Second, by performing arbitrary tasks on document chunks, the small LM communicates its compact answer instead of the raw document chunk, which amounts to sending fewer tokens to remote.
Speculative decoding Speculative decoding (Leviathan et al., 2023; Zhang et al., 2024a; Chen et al., 2024b) techniques addressing the different question of how effectively sample from the distribution of a large LM by only sampling from smaller LM and using the large LM for cheaper, likelihood evaluation (using the "acceptance-compleeen" algorithm" (Devroye, 2006)). It neither considers collaboration between two LMs, nor attempts to minimize the communication between them.
On-device language models for privacy Siyan et al. (2024); Zhang et al. (2024b) attempt to prevent leaks of private information to a cloud-hosted LM API by mediating the communication with local privacy-aware LM that removes private information from the prompt. While the local-remote LM setup bears resemblance to ours, we do not study the aspects of privacy, but rather focus on reducing cloud costs by delegating work to devices while maintaining accuracy. Moreover, we have additional focus on local runtime efficiency.
14

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Local-remote systems Recent work has explored efficient routing patterns between local and remote computation for LM workloads, albeit without two models communicating or collaborating LLM with early layers on the edge and later layers in the cloud, routing to the cloud when confidence is low. (Yang et al., 2024) propose complementary task scheduling framework that routes to cloud or local based on resource constraints and service requirements.
solution. (Jin & Wu, 2024) partition single
B. Extended Description of Experimental Setup
B.0.1. DATASET DETAILS
In this section we provide additional details dataset preparation. In order extend the context length of the problems in LONGHEALTH and QASPER, we make few modification to the dataset.
FINANCEBENCH We filter the original FINANCEBENCH to include only the numerical reasoning, resulting in dataset of length 64. Each sample has an average context length of 142.9 (22.9((=7922.33)) 79224.32).
LONGHEALTH In the original instantiation the LONGHEALTI dataset, each question is paired with set of medical documents corresponding to a single patient. To increase the complexity of the dataset, we include medical documents from 10 other patients in the context. We evaluate over the entire dataset (400 problems) for results reported in Table 1. Each sample has an average context length of 1 120.1K 237) tokens. For all ablations in Section 6, 128 problems.
a fixed subset of
QASPER Similarly, in the QASPER dataset, the original dataset provides questions that are associated with a single scientific paper. In order to increase complexity, we include 10 other papers in the context. We evaluate over a random subset 500 problems for results reported in Table 1. Each sample has an average context length of 54281 tokens (+2403). For all ablations Section 6, we use fixed subset of 128 problems.
B.0.2. MODEL DETAILS
Local Models. For QWEN2.5 we use the following models: QWEN2.5-L.5 QWEN2.5-3B-Instte QWEN2.5-7B- Instruct. For LLAMA, we use the following models: LLAMA-3 1---uunntttuct, LLAMA-3. 1.LAMA-..2-3B-Imm In LLAMA-3.1 3----B------- Instruct.
Remote Models. We use GPT-40 and LLAMA-3 -.2 Inn-ttutt,,,
All "local-only" and "remote-only" experiments are rum with temperature of 0.2. For all MINIONS experiments run in Table 1, the RemoteLM with temperature of 0.0 and LocalLM with temperature of 0.2 for FINANCEBENCH and 0.00001 for QASPER and LONGHEALTH. We run our local models A100 GPUs.
C. Extended Discussion of Cost Model
Here, we explain in detail the costs of the different communication protocols discussed in this renerr-rmmt-ooly.. MINION, and MINIONS- with a strong focus on the latency of these methods. This section is organized as follows:
Section C.I: We review background on language model inference, to motivate our and latency models.
Section C.2: We present mathematical models for the latency of the remote-only, MINION, and MINIONS protocols.
Section C.3: We present Proposition C.I, an upper bound on the total latency of MINTONS, relative to that of the remote-only model, demonstrating that MINIONS is not much slower than the naive approach of performing the full query in the cloud. As example, show that Llama-8B model GTX-4090 GPU collaborating via MINIONS with Llama-405B model 8xH100 server is at most x.75x slower than the remote-only protocol.
C.1. Background on language model inference
Language model inference consists of sequence of forward passes through a model, one for prefill (i.e. input) followed by one for each additional token generated (i.e. output). At low/medium batch sizes, each forward pass after prefill is I/O bound, meaning the time it takes to load weights from memory exceeds the time takes to actually compute the output. As the batch size increases, the computational cost of the forward pass eventually exceeds the I/O cost. Strikingly, for most
15

---

MINIONS: Collaboration Between On-device and Cloud Language Models
gpt-4o (0.72)
0.70
0.65
3
Accuracy
0.60
0.55
0.50
0.45
lama- (0.44)
0.040
0.045
0.050
0.055
0.060
0.065
Remote Cost (S)
Edge Model
llama-3.
llama-3.1
Figure 6. Exploring the trade-off between cost and quality through multiple rounds. The x-axis represents the remote model's 's token cost, while the y-axis shows accuracy. Point labels indicate communication rounds. The purple dashed line marks GPT-40 performance as a benchmark.
models and hardware, this happens batch size >> 100 (Leviathan et al., 2023; Chen al., 2024b). As result of this model (as is common in the literature) the cost of running
transition from being I/O bound to being compute bound, forward pass as piecewise linear function CM.=(n) max(A, or + (3) of the number of tokens being processed. This is because for small n, the IO cost dominates (and is roughly constant as n grows), whereas at larger the compute cost dominates and scales roughly linearly with n (assuming n is not too large).
In the cloud, the provider can batch generation requests from multiple users to keep hardware utilization high. Therefore, the cost of each output token is typically within small multiple the cost of each input token, and the total cost of processing the request scales as fill
Redcodd: for some small < 5.
On-device, we cannot assume we'll IIl have enough concurrent user requests to form a large enough batch to achieve high utilization. As a result, the latency of a request does not scale linearly with the number of tokens. A single request can occur similar latency to hundreds run in parallel. As a result, tokens are a poor proxy for cost on-device and we instead measure latency in micro experiments (see Section 6.3).
C.2. Latency models for all protocols: Remote-only, MINION, MINIONS
We now model the latency of each of these protocols (remote-only, MINION, MINIONS). We will then use these results in the following section to upper bound the latency of MINIONS by a scalar multiple of the latency of the remote-only protocol. First, we introduce the following assumptions and notation:
We assume we have a local GPU (e.g. RTX-4090) with peak compute Ft (flops/sec), and peak bandwidth Mj (bytes/sec), and remote GPU (e.g. H100) with peak compute F, (flops/sec), and peak bandwidth M, (bytes/sec),
We also assume for now simple transformer architectures for both the local and remote models:
LocalLM: L, layers, each with &df params MLP (Up/down projections each of size d₁ x 4d,, and 4df parameters in the WQ..vv. projections. The total memory required for the (non-embedding/LM, head) parameters is thus P. = 2 12L. For simplicity, assume the memory for the LM head is small relative to P.
RemoteLM: Equivalent architecture to the LocalLM, but with L, layers, d, hidden dimension, and P, total non-embedding/L.c MM-aad parameter memory (again assumed to be much greater than the number of LM head
16

---

MINIONS: Collaboration Between On-device and Cloud Language Models
parameters).
We model the number of input/output tokens of each protocol as follows, letting denote the number of tokens in the original document:
Remote-only: n prefill tokens and yuu decode tokens. Note that we assume-hhee here and below-that the number of effectively ignore the KV-cache load
tokens in the query is negligible relative We assume nou time for the output tokens.
MINION:
For LocalLM, we assume n prefill tokens and nou decode tokens. For RemoteLM, we assume n prefill tokens, and you decode tokens. the case of multiple rounds of communication, the KV cache for document can be stored avoid recomputation.
MINTONS: For LocalLM, assume n/e prefill tokens per chunk (c chunks total), and no decode tokens per job (though we assume only p fraction of output jobs do not abstain). For Remotel.M, we assume J noott prefill tokens, and nout decode tokens, letting J = cks denote the total number of jobs in MINIONS (c chunks, instructions, 8 samples). In the case of multiple rounds of communication, the KV cache for each document chunk can stored to avoid recomputation.
Throughout, we use the fact that (m ni] [n x k] matmul takes 2 mnn flops, and assume model parameters are stored half-precision (2 bytes/param).
We are now ready present the latency models for the three protocols (remote-only, MINION, MINIONS),
C.2.1. REMOTE-ONLY
We are compute bound, so time is approximately given by total cotal_flops//e Fy. We can break down total flops into the matmuls (MLP up/down projections, and QK operations) and attention operations.
Prefill:
Matmuls: 12nd per layer. Equivalent to (n x dz] [d, x 12d,) matmul.
Attention: 2 n² d, per layer. Equivalent to [n d,] [d, n] matmul.
Time: L, (24nd + 2n ) F, = (nP, + n
Decode: We are memory bound (batch size 1 for Minion), so time is approximately given by total motal_memory/ y/M, per decode step. can break down Lotal Lotal. memory into model parameters and KV cache.
Model parameters: 2 12d bytes per layer.
KV-cache: 2 2nd, bytes per layer (K and V are each [n x d] matrices).
Time: I, noat (24d. + 4nd,)/ //, mowa(P, + n.)
Total time is given by the sum of prefill and decode times:
Fremote nPr + 2L,d,n
Wona(Pr + dL
F,
M,
C.2.2. MINION
The latency of the LocalLM the MINION protocol can be modeled equivalently to the latency of the remote-only protocol, but replacing the remote parameters with the corresponding local ones. Thus, total local latency is:
nP, + 2L,dm2 + no (Pi + ALddi)
MINION
Fi
Mi
The total remote latency can also be expressed using these same equations. but with noue prefill tokens, and nowt decode tokens.
n/ P, 2I, (ni
+ nimue (Pr + 4T, dn
MINION
Fr
M,
17

---

MINIONS: Collaboration Between On-device and Cloud Language Models
C.2.3. MINIONS
The LocalLM latency of the MINIONS protocol has some important differences from the MINION protocol-the prefill computation avoids cross-chunk attention (which saves time), while the decode operations actually be compute bound if batching of the different jobs is done. We review these details below:
Prefill: We are compute bound, so time is approximately given by total_flops: F. We can break down total total_flopp flops into the matmuls (MLP up/down projections, and QKVO operations) and attention operations.
Matmuls: 2 12ndf per layer. Equivalent to [ne x d:) [d 12dj matmuls (where ne = n/c).
Attention: 2- cn 2 (n/c)²dh 2n d per layer. Equivalent [ne x di] [dl nel matmuls.
Time: L (24nd + 2n di/
di/ cc) = (nPi + 2Ljdm /c)/F.
Decode: We will now assume we are compute bound during decode, because we have many jobs (ks) per chunk, and many chunks (c) per document, which we can batch together. Thus, time approximately given by total flopl/ Ft per decode step. We can break down fotol_ into matmuls and attention. The flops below are per job, per output token (so for total flops we will multiply by nout pcks):
Matmuls: 12d per layer. Equivalent to [] x di] [d: x 12d] matmul.
Attention: 2- nedd 2d n/c per layer. Equivalent to [1 x d:] [d x nel matmul.
Time: L nuut peks (24d 2dm//c/ F nout= pcks (Pr + 2Lddn/c)/F..
The total local latency for MINIONS is given by the sum of prefill and decode times: MINIONS
nP. 2Tqder2 /e nout peks
(Pr + 2Tddm//c)
MINIONS
Fi
Fi
The total remote latency for MINIONS be expressed using the same equations MINION, but with poks moue prefill tokens, and not decode tokens.
(pcks moutt Pr + (pcks noue)² + nout (Pr + 4h, rlout))
T MINIONS
remote
F,
M,
C.3. MINIONS vs. remote-only comparison
Proposition C.1. Assume nout pokks an, for some 1, and that Fr,, dr and Ly, are all as defined in Appendix C.2. In this case, can show that the ratio of total latency of MINIONS vs. the remote-only protocol upper-bounded by the following expression:
MINIONS
MINIONS locol
F. Lads
remote
1+ (1 + aa
T Tremote
Fi
L,
Proof. Let's assume nime pcks = an, for some < 1.
nPi + 2Iqdin
an Mm--Pi + 2Tddm/c)
T MINIONS
Fi
F
(1+ nP: 2Lqdm2
Fi
(a (an)2 + nour (P, + (an))
I MINIONS
F,
Mr
nP, 2L,dm1
W nou )
No Pr M,
F.
Mr
nP, dy, F,
Wone (Pr dled
T Tremote
Mr
18

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Thus, it is easy to see that
Pemmats Trmmtt
<
1. Now let's look at
T Tremate" and show it is upper bounded by constant:
MINIONS
(1+ a)
nPt 1 22
Local
T remote
d2,
nPt + 24ddnn
(1 a)
Fi
nP, 22
Fr.
Pt Lidh Pr L Lidk Lidh
x (1 [1+0 a)
max
Fi
Fr
= (1+a)
max
Fi
d Lndic)
Fr
Lich
< (1+a)
Fi
L,dr
Thus, combining the above two results see that:
T MINIONS MINIONS
Fr
Lid
<
1+ (1 (1+a)
T remote
Fi L,d,
Real example: Let's assume that the local GPU is RTX 4090 (F) 160 TFLOPS), the remote server is full node of 8 H100s (F, = 8000 TFLOPS across full node), the local model is Llama-8B (L, = 32, d, = 4096), and the remote model is Llama-405B (L1 126, di = 16384). Furthermore, let's assume = 0.2, which is actually bit larger than we see in
practice. In this case:
Fr
Lid
8000
32 4096
1 + (1 + a)
R2
1 + 1.2
Fi L,d
160
126
16384
1+
1.2. 50
16
4.75.
Note that if we perform multiple rounds of MINIONS, this ratio gets multiplied by at most the number of rounds, though as mentioned previously, we can save time by only performing prefill on all the document chunks in the first round.
D. Extended discussion of methods
D.1. Extended description of MINION
In this section, we describe MINION, a baseline local-remote communication protocol. We ask whether we can reduce remote prefill tokens, and thus cost, by simply orchestrating a free-form conversation between the LocalLM and the RemoteLM in which only the LocalLM has direct access the context
The protocol proceeds with initialization step followed by simple correspondence between the two models, which terminates when the remote model can answer the question or maximum iteration limit is reached.
Iteration i 1: Initialize. The RemoteLM receives the task query along with a system prompt Premote that instructs to interact with small LM that has full access to context. It outputs first message mfll
(1) Premote RemoteLM(g Premote)
19

---

MINIONS: Collaboration Between On-device and Cloud Language Models
The message is then provided to LocalLM, along with the full context c, the query c, and minimal system prompt Plocal that instructs it answer questions the context:
LocalLM ( (M Plocal c)
local
Iteration 1. Step Message from remote to local. MemoteLM consumes the conversation history and outputs new messages:
remote
MemoteLMMm jemote
(:i-1) (3--1)
Iocal 9. Premote)
In its message, RemoteLM indicates whether it has sufficient information to terminate the loop and answer the question, or alternatively raises additional questions.
Step Message from local to remote LocalLM consumes the latest remote message and conversation history, and outputs (8) local
LocaLLM ((io))
(+--1) 1)
I0 local
m local
9, Plocal c)
We then increment the iteration i and loop back to Step until the break condition is met or we reach a maximum number of iterations.
D.2. Information Bottleneck Perspective
How does local model capacity affect the cost-accuracy tradcoff?
The Information Bottleneck (IB) principle (Tishby al., 2000) provides a useful analogy. One communication round of local-remote system does follows:
p(2/c) [Extract info. from context]
(>p(y|z) [Predict outcome from extracted info]
The IB principle seeks to find a p(2 c), our LocalLM, as follows:
min p(z|e)
(3)
f.e. find mapping that forces the latent representation to be maximally informative of the label I(Z; Y) and minimally informative of the input T(C; Z), with a tradeoff parameter B. Here, we do not optimize the mapping p(z c) but instead only get to choose by setting LocalLM.
Since we cannot compute these quantities in closed form for nonlinear distributions over tokens, we use (coarse) empirical proxies follows. As proxy for (C; compute the number of prefill tokens sent to RemoteLM, capturing the intuition that more tokens carry more information the input. I(Z;Y) is estimated the average accuracy of the local-remote system, quantifying the preservation of task-relevant information in z. While these proxies do not exactly match the underlying mutual informations, they capture the core tension of compressing the input vs. preserving predictive power.
We plot these quantities in Figure ??. We find that across both QWEN2.5 and LLAMA model families, as we increase LocalLM size, we send fewer tokens RemoteLM (se ((C 1), and improve accuracy (+e L(Z;Y) 4). We find that LLAMA has higher = I(C: Z) and higher z I(Z;Y).
E. Extended Results
F. Model Analysis
We include additional experiment results from Section 6.2. In Table we show the effects of varying RemoteLM on MINIONS, In Table 4, we show the performance of MINIONS using the best in-class models at the time (from late 2023 to late 2024).
20

---

MINIONS: Collaboration Between On-device and Cloud Language Models
FinanceBench
gpt-4o (0.83)
0.80
0.75
Strategy
Accuracy
0.70
3
Simple retry
Scratchpad
0.65
0.60
0.55
8000
10000
12000
14000
16000
Total Remote Tokens
Figure 7. Comparing strategies for maintaining context between MINIONS rounds. The x-axis represents the number of tokens processed by the remote model, while the y-axis shows the accuracy achieved.
Local Model
Remole Model
Release Date
Accuracy (Longhealth)
Accuracy (QASPER)
Accuracy (Vinance)
Iaaaa 3B
gpt-4a
May 2024
0.7025
0.598
0.7826
1amm-33
gpt-4-turho
April 2024
0.6247
0.614
0.6304
Uama-38
gpt-3.5-urbo-0125
Jan 2024
0.2157
0.4314
0.1707
Ulama-3
gpile-mini
July 2024
0.6275
0.568
0.6522
lama
70B
Turho
April 2024
0.1818
Hamm--3
1aama3. CooB--Istruut-Tuee
July 2024
0,6193
0.514
0.4348
Ulama-3B
llama3 1--to---strue---uri
December
0.6739
Table Accuracy Results for Longhealth, QASPER, and Finance across Various Models
F.2. MINTON LocalLM Analysis
We perform an empirical analysis evaluating the robustness of LocalLM. We perform experiments evaluate two axes of model capabilities: (1) ability reason over long contexts and (2) ability to solve multi- -tait queries. To test (1) and (2) curate synthetic dataset built over the FINANCEBENCH dataset wherein we use GPT-40 construct extraction based question-answere dataset over chunks (length 512 tokens) of documents in the FINANCEBENCI dataset. We then construct two settings evaluating over LLAMA-3 1-.2--BB-nne Instruct..
Long Context Reasoning: To evaluate long-context reasoning, we concatenate between {1,16,32,64,128, chunks to construct the context. At least one chunk the concatenated context contains the ground truth result. As seen Table increasing the context length from 512 65.5K tokens leads 13 point drop in accuracy.
Multi-step Queries To evaluate the ability of LocalLM to fulfill multi-step queries, we construct queries that have between {1,2,3,4) sub-tasks. Our results indicate increasing from to sub-tasks leads to 56.3 point drop in accuracy (see Table 6).
Local Model
Remote Model
Accuracy (Longhealth)
Accuracy (QASPER)
System Date
Ilaam 1.lama-2-7b-chan 2-7b--hht
gpt-4-1106 106-priiwere
0.340
0.178
November 2023
Llama-3.1-- I-8t-lnstruct
gpt-4-turbo
0.645
0.528
April 2024
Llama-3.1 --8-stunttt
gpt-4o
July
gpt turbo
0.768
0.391
April 2024
Table 4. Point in time results for MINIONS configurations with best-in-class LocalLM and RemoteLM
21

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Total Chunks In-Context
Accuracy
0.59375
16
0.53906
32
0,50000
64
0.48438
128
0.46094
Table Accuracy vs. Number of Chunks Context Each chunk has 512 tokens.
Number of Sub Tasks
Accuracy
0.70313
0.39844
0.19531
0.14844
Table 6. Accuracy vs. Number of Sub Tasks
E.3. Relationship with Retrieval-Augmented Generation
In this section, we discuss the relationship between local-remote collaboration and retrieval-augmentn generation (RAG), technique that reduces the number of tokens processed by an LM by retrieving subset of relevant documents or chunks LM (Lewis et al., 2020; Karpukhin al., 2020; Lee et al., 2019).
Retrieval-augmennt generation and local-remote collaboration (e.g. MINIONS) are complementary techniques. They both provide a means to reduce cost by providing an LLM with a partial view of a large context. But, as we discuss below, they also have different error profiles and can be used in conjunction to improve performance.
E. COMPARISON OF MINIONS AND RAG FINANCEBENCI
In Figure 8 (left), we plot the quality-cost trade-off FINANCEBENCH for local-remote systems (MINION and MINIONS) and RAG systems using BM25 and OpenAl's text- -d- 1 embeddings (Robertson & Zaragoza, 2009; Neelakantan et al., 2022). For RAG, we use chunk size of 1000 characters, which we found to be optimal for this dataset after sweeping over chunk sizes with the BM25 retriever (see Figure 8 (center)). We show how a simple hyperparameter (number of retrieved chunks provided to the remote model) allows us to trade off quality of the RAG system for remote cost. Furthermore, we note that when the BM25 RAG system provides 50 or more chunks of the document to the remote model, exceeds the performance of the remote model with the full context. This likely indicates that RAG helps minimizing distractions from the long context. For FINANCEBENCH, when compared to MINIONS, the RAG system with OpenAl embeddings reaches similar points in the quality trade-off space. Interestingly however, none of the RAG configurations are able to match the quality of MINION at the same low cost.
E.3.2. COMPARISON OF MINIONS AND RAG (EMBEDDINGS BM25) ON SUMMARIZATION TASKS
In this section, we discuss the relationship between local-remote collaboration (e.g. MINIONS) and retrieval-augmented generation (RAG). These are complementary techniques with different strengths, which can be combined.
We perform quantitative and qualitiateive compartive analysis demonstrating the different benefits of the two techniques. We analyze MINIONS, MINION, and RAG on two data intensive reasoning tasks: one that relies heavily on extraction (FINANCEBENCH) and another that requires long-documen documenttn summarization (BOOOKsCORE). Our results demonstrate that, on FINANCE ENCH (see App. F.3.1), 1). RAG can achieve similar, sometimes better, cost-accuracy tradeoffs compared to MINIONS, but is less cost-effective than MINION, On BOoOKSCORE (see App. E.3.2) - which requires integrating information dispersed across the document - we find that MINIONS generates summaries comparable to a GPT-40-only baseline when evaluated using an LLM audd (see App. Table 10), while RAG results in lower quality summaries that miss critical plotlines, characters and details (see App. Table 11).
22

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Protocol
Local Model
Remote Model
FINANCERENCH
LONGHEALTH
QASPER
Cost
Out
Acc.
Cost
In
Out
Acc.
Cost
In
Out
Tok.
Tok.
Tok.
Tok.
Tok.
Tok.
(IK)
(1k)
(1k)
(1k)
(1k)
Remote
GPT-40
0.826
S0.261
103.04
0.32
0.748
$0.301
130.10
0.07
0.598
$0.157
54.40
0.09
Only
Local
LLAMA-8B
0.326
50.000
0.00
0.00
0.468
$0.000
122.58
0.07
0.538
$0.000
54.41
0.06
Only
Local
LLAMA-1B
0.000
50.000
0.00
0.00
0.115
50.000
122.58
0.07
0.000
$0.000
54,41
0.10
Only
Local
LLAMA-3B
0.130
50.000
0.00
0.00
0.345
50.000
122.58
0.08
0.164
$0.000
54,41
0.08
Only
Local
QWDN-30
0.087
$0.000
0.00
0.00
0.177
50.000
31.24
0.08
0.156
$0.000
32.58
0.08
Only
MINION
LLAMA-8B
GPT-40
0.804
50.007
0.88
0.46
0.635
50.010
1.85
0.50
0.450
$0.007
0.92
0.42
MINION
I.AMA-3R
GPT-40
0.698
50.010
1.74
0.52
0.482
50.009
1.56
0.47
0.372
$0.011
2.26
0.53
MINION
QWEN-38
GPT-40
0.217
$0.029
8.28
0.82
0.281
50.021
5.70
0.68
0.210
$0.035
10.51
0.87
MINIONS
LLAMA-8B
GPT-40
0.804
50.053
1.29
0.740
50.054
$0.019
5.10
MINIONS
LLAMA-3B
GPT-40
0.726
$0.079
24.67
1.77
0,703
50.057
20.11
0.66
0.558
$0.020
5.62
0.60
MINIONS
QWEN-3B
GPT-40
0.783
$0.059
17.20
1.56
0.645
$0.043
14.43
0.65
MINIONS
QWEK-7B
GPT-40
0.600
$0.015
3.44
0.61
Table Accuracy and cost of local-remote systems. Evaluation of cost and accuracy on the FINANCEBENCH (Islam et al., 2023), LONGHEALTH (Adams et al., 2024), and QASPER (Dasigi et al., 2021). The table compares two edge-remote communication protocols Naive (Section 4) and MINIONS (Section 5) - alongside edge-only and remote-only baselines. Three different edge models are considered (ILAMA-8B. B., LLAMA-3B. QWEN-3B, and LLAMA-1 B) and remote model (GPT-40). Accuracy (Acc.) is the fraction of correct predictions across the dataset. Cost (USD) the average cost in USD per query in the dataset computed. Costs are incurred for any calls the remote model at GPT-40 rates (January 2025: $2.50 per million imput tokens and $10.00 per million output tokens). We assume that running the edge model is free; see Section 3 for details on the cost model. In tokens is the number of input (i.e. prefill) tokens sent the remote model. Out tokens is the number of output (i.e. decode) tokens generated from the remote model. Both values are shown in thousands.
Protocol
Local Model
Remote Model
Macro
Avg.
Information Extraction
Numerical Reasoning
Acc.
Cost
Acc.
Cost
Acc.
Cost
Remote Only
GPT-40
0.883
$0.259
1.000
$0.246
0.767
$0.271
Local Only
LLAMA-8B
0.367 $0.000 0.467
$0.000
0.267
$0.000
Local Only
LLAMA- 1B
0,000 $0.000 0.000
$0.000
0.000
$0.000
Local Only
LLAMA-3B
0.150 $0.000 0.200
$0.000
0.100
$0.000
Local Only
QWEN-3B
0,083 $0.000 0.067
$0.000
0.100
$0.000
MINION
LLAMA-8B
GPT-40
0.850
$0.007 0.933
$0.006
0.767
$0.007
MINION
LLAMA-3B
GPT-40
0.743
$0.010 0.831
$0.013
0.656
$0.007
MINION
QWEN-3B
GPT-40
0.233
$0.028 0.267
$0.025
0.200
$0.032
MINIONS
LLAMA-8B
GPT-40
0.840
$0.048 0.893
$0.031
0.787
$0.064
MINIONS
LLAMA-3B
GPT-40
0.730
$0.077 0.693
$0.067
0.767
$0.087
MINIONS
QWEN-3B
GPT-40
0.793
$0.055 0.773
$0.044
0.813
$0.067
Table 8. Stratified analysis of local-remote systems on FINANCEBENCH Evaluation of cost and accuracy the FINANCEBENCH (Islam et al., 2023) stratified by question type. We consider two types of questions: Information Extraction, questions that require extracting specific data or textual content from filings, and, Numerical Reasoning, which requires performing additional numerical reasoming and calculations on extracted data. The table compares two edge-remote communication protocols - MINION (Section 4) and MINIONS (Section 5)- alongside edge-only and remote-only baselines. Three different edge models considered (I.LAMA-8B, LLAMA-3B -BB QWEN-3B, and LLAMA- B) and a remote model (GPT-40). Accuracy (Acc.) is the fraction of correct predictions across the dataset. Cost (USD) the average cost USD per query in the dataset computed, Costs are incurred for any calls to the remote model at GPT-40 rates (January 2025: $2.50 per million input tokens and $10.00 per million output tokens). We assume that running the edge model free; see Section 3 for details on the cost model.
23

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Protocol
Local Model
Remote Model
Macro Avg.
evidence span
2 evidence spans
evidence spans
4+ evidence spans
Acc.
Cost
Acc.
Cost
Acc.
Cost
Acc.
Cost
Acc.
Cost
Remote Only
GPT-40
0.668
$0.301
0.815
$0.301
0.661
$0.302
0.600
$0.302
0.596
$0.301
Local Only
LLAMA-8B
0.448 50,000
0.498
$0,000
0.390
$0.000
0.520
$0.000
0.386
$0.000
Local Only
LLAMA-IB
0.097 $0.000
0.124
$0,000
0.102
$0.000
0.040
$0.000
0.123
$0.000
Local Only
LLAMA-3B
0.327
$0.000
0.375
$0.000
0.271
$0.000
0.400
$0.000
0.263
$0.000
Local Only
QWEN-3B
0,174 $0.000 0.185
$0.000
0.186
$0.000
0.200
$0.000
0.123
$0.000
MINION
LLAMA-SB
GPT-40
0.582
$0.011
0.683
$0.009
0.627
$0.009
0.560
$0.017
0,456
$0.011
MINION
LB
GPT-40
0,436
$0.009 0.529
$0.008
0.390
$0.009
0,440
$0.008
0,386
$0.010
MINION
QWEN-3B 3B
GPT-40
0,259
$0.019 0.298
$0.023
0.288
$0.016
0.240
$0.017
0.211
$0.018
MINIONS
LLAMA-&B
GPT-40
0.731
$0.056
0.753
$0.053
0.763
$0.051
0,760
$0.060
0.649
$0.060
MINIONS
LB
GPT-40
0.665
$0,058 0.741
$0,056
0.661
$0.055
0.680
$0.061
0.579
$0.062
MINIONS
QWEN-3B
GPT-40
0.635
$0.045 0.649
$0.041
0.678
$0.040
0.600
$0.050
0.614
$0.048
Table 9. Stratified analysis of local-remote systems on LONGHEALTH. Evaluation of cost and accuracy on LONGHEALTH (Adams al., 2024) stratified by the dispersion of the supporting evidence for each question. We measure dispersion with the numher of distinct spans in the context which contain information relevant to the question. We include performance for questions with 1, 2, 3, and 4+ distinct evidence spans. The table compares two edge-remote communication protocols- MINION (Section 4) and MINIONS (Section 5) alongside edge-only and remote-only baselines. Three different edge models are considered (LLAMA-SB) -BB, LLAMA-3B, QWEN-3B, and LLAMA-ID B) and remote model (GPT-40). Accuracy (Acc.) is the fraction of correct predictions across the dataset. Cost (USD) is the average cost in USD per query in the dataset computed. Costs are incurred for any calls the remote model at GPT-4o rates (January 2025: $2.50 per million input tokens and $10.00 per million output tokens). We assume that running the edge model is free; see Section 3 for details on the cost model.
FinanceBanchh
FinanceBench
gpb-4o (0,83)
0.8
0.7
Edge Model
0.8
opt-co
Chunk Size
lama 3.2
500
Accuracy
0.6
0.5
10
10
Open-A Embedding
BM25
3am9-3.1-4b
Accuracy
0.6
5000
2000
1000
Protocol
10000
0.4
RAG
0.4
Protocol
0.3
(lama 3.1 8b (0.33)
Remote On
Minions
(0.33)
Edge On
Only
0.2
Mirion
Edge Only
RAG
-lama-3.233 (0.13)
(1ama-3.2-33 (0.15)
0.1
0.00
0.15
0.20
0.25
0.15
0.20
Remote Cost
Remote Cost
Figure 8. Relationship with retrieval-augmmented generation.
RAG is a very suitable approach for FINANCEBENCH, since all of the questions heavily rely on information extraction from specific sections of financial statements. However, RAG will not be suitable for a summarization task, unlike small LMs. Therefore, we use the long-document summarization dataset, BooooKSCORE (Chang et al., 2023). BoooOkScORF which contains set of 400 books published between 2023-2024. The average story length in BooookscoRE is 128179 tokens with max of 401486 tokens and minimum of 26926 tokens. We utilize both MINIONS, RAG (w/Embeddings + BM25), and GPT-40 only to complete the task. We describe the set-up for all three approaches next.
MINIONS for summarization In applying MINIONS to the task, the LocalLM (LLAMA-3. 1.-MA--32-3B-Insttucc) provides summaries on chunks of the original text, passing a list of chunk summaries the RemoteLM (GPT-40). RemoteLM produces the final summary.
RAG (Embedding) for summarization In our embedding- RAG approach, we use the OpenAl TEXT-EMBEDDING-3: SMALL to embed chunks of the original text (of length 5000 characters) and we retrieve the t5p- most relevant chunks using the query "Summarize the provided text". We then prompt GPT-40 to generate a complete summary over the retrieved chunks.
RAG (BM25) for summarization In our BM25-based RAG approach, we use the BM25 to retrieve chunks of the original
24

---

MINIONS: Collaboration Between On-device and Cloud Language Models
text (of length 5000 characters) based the query: "Summarize the provided text". We retrieve the top- 15 most relevant chunks and prompt GPT-4o to produce final summary over the retrieved chunks. We choose top-15 to ensure the number of tokens passed up by the baseline is comparable with those passed up by MINIONS.
GPT-4o In our final baseline, we use GPT-40 alone to create the story summaries. For texts that extend beyond the 128K context length window, we truncate the stories.
Evaluation
Qualitative In Table I we provide samples outputs from each of the 4 methods described above. We highlight major events in red, themes in green, locations in blue and names in indigo. The samples demonstrate that amongst all the methods, MINIONS outputs contain the most entity mentions and story specific details. Moreover, when compared to GPT-40-only RemoteLM, MINIONS is 9.3x more efficient 11,500 versus the full 108,185 prefill tokens.
The summaries from MINIONS are generally 1.3x longer and more verbose than the RAG systems' summaries, likely indicating that the former is more effective at "passing forward" salient information, Morcover, RAG systems' summaries are missing the main arc of the narrative in favor of what seems an assortment of facts.
Quantitative We additionally perform quantitative analysis of the generated summaries using a LLM-as-a-judge frame- work. As an evaluator, we use the CLAUDE-3.5 -.-NSNET model, to avoid any biases between the evaluator and the supervisor model. We prompt the model with the generated summary, ground truth summary (gpt4-4096-ie-cllannd) provided from the original BooooKSCoRE generations, and grading rubric (see Figure 9). The rubric evaluates criteria: coherence, relevance, conciseness, comprehensiveness, engagement readability, accuracy, and thematic depth. We prompt -.-DD-- to generate score (1-5) for each of the criteria and average the scores. We find that summaries generated by MINIONS score comparably with GPT4O-only generated summaries, while RAG based baselines perform worse. Our results can be found in Table 10.
Evaluation Rubric for Summaries
1. Coherence (1-5): Summary is logically structured, with clear connections between events, avoiding abrupt jumps or inconsistencies.
2. Relevance (1-5): Accurately reflects key themes, events, and characters, focusing essential details without unnecessary plot points.
Conciseness (1-5): Thorough yet avoids excessive detail, presenting necessary information without redundancy.
4. Comprehensiveness (1-5): Covers all major characters, events, and themes, ensuring complete overview without omissions.
Engagement & Readability (1-5): Engaging and easy read, with well-constructed sentences and clear, precise language.
Accuracy (1-5); Stays true to the book's storyline, themes, and tone, with correct details, names, and events.
7. Thematic Depth (1-5): Identifies underlying themes and messages, providing insights into conflicts, motivations, and resolutions.
Figure 9. Evaluation Rubric for Summaries
25

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Method
Score
MINIONS
3.01
GPT4o
3,06
RAG (BM25)
2.48
RAG (Embedding)
2.38
Table 10. Comparison of Methods and Rubric Scores
Table 11: Comparative Analysis of Story Summaries Across Different Methods
Story
Minions
GPT-4o
BM25
Embedding
te
A Likely
The novel follows the complex dynamics of the Manning fam- ily, locusing on lsubelle Man- ning, struggling writer grap- pling with the legacy of her fa- mous father; Ward Manning, and the recent death of her mother, Claire. As Isabelle nav- igates her career and personal life, she uncuvers family se- crets, including her father's is pla- giarisan and her mother's hid- den manuscript, which chal-
In narrative spanning New York and Sag Harbor, Claire Cunninghan grapples with ber identity beyond heing Vassar graduste, while her daughter Is- shelle faces personal and pro- fessional challenges, including her father's illness and her own writing struggles.
Story
Isabelle Manning, daughter of famous author, discovers unfinished mamuscript by her late mother. Claire. She pletes and publishes as ber own, leading success. but also guilt and exposure. Her father, Ward, reveals he sabu- taged her first book to teach her resilience, deepening her sense of betzayal. Ultimately. Isabelle confesses her actions. seeks redemption, and begins to redefine her identity beyond her father's 's shadow.
Isabelle Manning, overshad- owed by her famous father Ward, receives a threatening text, "I KNOW WHAT YOU DID," hinting al a secret tied book. "Underpaint- ing" Meanwhile, her mother Claire's 's pusi support and her fa- ther's 's hiddlen illness add layers to Isabelle's is challenges.
lenge her identity and ambi- tions. The narrative explores themes fame, family expec- tations, and personal growth, culminating in Isabelle's is jour- mey lowards self discovery and independence. Set against the backdrop of New York City and Sag Harbor. the story the intricucies milial relationships and the pur- suit creative fulfillment.
All the
The narrative follows Isabelle, dealing with Mason's disup- pearance. She works with podcaster Waylon, uncovering links to Ben's deceased wife. Allison.
Dan-
Isabelle Drake, woman grap- pling with the traumatic dis- appeurance of her Mason, navigates a complex web of grief, guill, and suspicion. As she becomes entangled with true crime eathusinsts and in- vestigators, including podcast bost Waylon and Delective Dazier, Isahelle's quest for truth reveals unsettling fam- ily secrets and personal betray- als. Her journey is marked by strained relationships. pas- ucularly with her ex ex-husband Ben and his connections to other women, including Valerie and Allison. Throughout the narrative, themes of mather- hood, mental health, and so- cictal judgment are explored, culminating in deeper under- standing of the pressures and expectations faced by women.
Isabelle Drake, plagued by in- sommia and guilt, desperate to find her missing son, Ma- son. She suspects her husband, Ben, and new purtner, Va- lerie. With Waylon's is help, she discovers Abigail Fisher, ma- nipulated by Valerie, took Ma- son believing she was rescuing him.
Isabelle, strugeling with grief and insomnia, joins grief counseling group. She meets Valerie and collaborates with Waylon, becomes wary ter finding unsettling informa- tion on his laptop.
gerous
Things
Continued page
26

---

energy MINIONS: Collaboration Between On-device and Cloud Language Models
Continund from previous page
Story Ti-
Minions
GPT-40
BM25
Embedding
tle
A Laving Remedy:
The protagonist struggles with visiting her dying mother dur- ing the COVID-19 pandemic. The story explores grief, fam- ily responsibility, and cherish- ing life amids adversity.
A woman reflects her par- cnts' illnesses and deaths, bal- ancing her as a daughter and mother. She finds solace in childhood memories and the legacy of her purents' love.
Nicole Chung, Korean Amer- ican udopiee, reflects on her complex relarionships with her adoptive parcots, her ideatity, and the challenges ing life as a minority in pre- dominanty white community in Oregon. Her memoir ex- plores themes of family, loss, and resilience, she recounts her father's death from kidacy failure, and her mother's 's battle with cancer. Amidst these per- sonal challenges, Chung grap- ples with her own grief. fi- nancial struggles, and the im- pact of the COVID-19 pan- demic, while finding solace in her family. faith, and writing. Her journey is marked by deep appreciation for her par- cuts' sacrifioes, the support of her hushand and children, and the codoring legacy of love and lorgiveness instilled by her muther.
Nicole Chung's memoir plores her joumnyy after the loss of her adoptive parents, As Korcan adoptec, she reflects on family's financial struggles. parents" health battles, and their deaths' impact on her identity. She finds solace in writing and her own family.
A Memoir
navigal-
A House with Good Bones
The protagonist returns their grandmother" is unchanged gar- den. filled with roses myste- riously devoid of insects. They uncover unsettling truths about their graddmother' 's past and their mother's sss current state of mind, The narrative explores themes of family legacy and the passage time.
Samanthu, a 32 yeay---oll ar- chaecentormlogist, returns to her childhood home Lam- mergeier Lane in North Car- olina, where she confrants her family's ss dark past, in- cluding her grandmother Gran Mae's mysterious and malevo- lent logacy, As Samantha nav- igates her mother's is strange be- havior und the eerie presence of vultures, she uncovers se- crets involving ritual magic, jar human teeth, and the su- pernatural "underground chil- dren." With the help of her friend Gail and handyman Phil, Samunthu faces the haunting manifestations of her family's history. The novel explores themes family, memory, and the supematural, blending ele- ments of horror and funtasy.
Samantha Montgomery returns home to find her mother act- ing strangely and the house de- void of insects. She uncov- dark history involving her great-grand grandfather. a sorcerer, and her grandmotther, who used roses to wield power. With help from Gail and Phil, she confronts the terrifying "under- ground children." using rose power to hanish threats.
Samantu, an archaeoentomol. ogist, retums to her childhood home and finds herself investi- gating insect collections. Deal- ing with sleep paralysis and memories of her grandmother,
peculiar ab- sence of insects in the garden. She navigates family dynamics and her mother's anxiety amid an cerie atmosphere.
discovers
E.4. Energy Consumption Analysis
We perform energy analysis of MINIONS worklonds by measuring the lotal energy consumed by LocalLM and estimating the energy consumed by RemoteLM. using Jooles (I) as the primary unit of measurement. For our Remotel. energy consumption estimates, we use the inference time energy estimate calculations provided hy local worklonds two types hardware: A100 Nvidia GPU and Apple MI Max GPU. For measuring energy
Epoch AT (AI, 2024), We perform analysis running consumption on these hardware set-ups, we use off the shelf packages such as powernetrics and avidia-smi. We sweep our experiments across two LocalLM variants use GPT-40
(L..AM--322 2-1B and AMA-3.2-3R) estimating the total energy consumption that GPT-40 would have if it performed the equivalent work (as measured by input output tokens) of the LocalLM. We compare energy consumption between MINIONS and remote-only setups, varying the length of the input context length. Our results indicate that we can achieve up to 12x energy consumption savings when using a 1 parameter local model and up to 6x savings with 3B parameler mudel running local workloads A100. Similarly, consumption reductions when ninning workloads on the M- M-series hardware of up to 8x. Plense see Figure 10 for more details.
Remotel. M. As baseline, we estimate cost entirely running MINIONS workload Remotel.M
E.5. Local LM Agentic Tool Use
We lake preliminary step in extending MINIONS to support LocalLM tool calling. This configuration enables the LocalLM to automemoussly execule tools necessary fore task completion, guided the M awareness of available local tools (i.e., the Remotel.M might request the LocalLM describe the contents the local filesystem. further suggesting it use the tool that lists all local files). We evaluate this extended setup on a hand-curated set of queries (see Table 13 for sample queries) against a local folder containing reciepts. giving the LocalLM access the following tools: lis:Directte Dirtteeyyyy re readFile readMul www.ltippeellic eieeeieeeseee MeitePile searcht's i Les. For the purposes of our evaluntion, we use OWEN2,5-38 3BB and 1WEN2.5-1B as our LocalLM's and GPT-40 as the RemoteLM. We compare performunce against a remote-on only baseline which answers the queries given all the documents context. We find when using QWEN2.5 5B3 locally, MINIONS performance comparable to the GPT-4o baseline, will only using < 28% of remote token usage (see Listing for a sample Minion trace with tool calling enabled). However, we find that QWEN2.5 5. 1B unable to achieve parity in performance and lacks the ability to ellectively ulilize lools. See Table 12 for complete results.
cwealelireee weptrrectoo o ryy and
Peask": "Do have rwoe Spt for Guagas? Tf yes, pi ease tell me the botal amount.". "
27

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Energy Savings of Minions Cloud Only Workload
14x
Energy" Consumption Seving fector
10x
Ax
10000
20000
30000
40000
50000 60000
Input Context Length (cakens)
Figure 10. Energy savings of Minions vs. cloud-only workload across varying input context lengths We compare the energy consumption of local execution of Minion using two hardware configurations- baseline powered by GPT-4o. Local workloads are using LLAMA-3.. 2-IB and LLAMA-3.2-2 models. with energy measured via nvidia-smi and powermetrics, while remote energy use estimated following methodology from Epoch (AI, 2024). Across both hardware types, local execution yields substantial energy savings, reaching up to 12x with the 1B model on A A100, and consistent gains across other configurations.
AiViiIiraioos-NNVIIIA NVIDIA A100 and Apple M1 Max Max-against cloud-only
Local Model
Remote Model
Task Completion Rate
Total Remote Tokens Processed
QWEN2.5-1.5B QWEN2.5-3B
GPT-40
0.10
29697.6
GPT-40
0.70
8741.1
GPT-40
0.70
32498.3
Table Comparison task completion rates and token costs: MINIONS set-up performs comparable with the GPT-40-only baseline while reducing RemoteL.M M prefill costs by 3.7x.
Question
Answer
Tell me all the dates ordered sprouts
3/5, 1/26, 1/29, 2/10, 9/24 Yes
am tracking my calories. Did order tiramisu Terun?
How many times did eat at Sultana?
3
What is the most common menu item ordered across all my
The Workout
Asian Box receipts?
Table 13. Queries for agentic tool use analysis: sample question/answer pairs for the reciept history question-answeriig task used in Section F.5.
"stops" :
"usozf: "ranote", "prosgt"; "Me
felloving task, Anko4ed Teskinnto receipt Guages? yea, tell not have direct access to the context, but can chat with snall language model
Lotal anount. valns4s Instrugtions has access to local concext via the following An. *oulpul " "iVenaaaee (f search for flles selated to puclude_patteriee Jpg', "rapng', (x
Fllea (pather, pel Guspass
10
"uaozf: Flocal",
"prontt"; fsearch, Filesdpethef 2 "Buspass", patternae("* ipg""" (t.pngf, ("g" "oulpat";
13
1
14
fuawsf; FLoul",
"coolf; fsearci_fllea",
17
"argurents"; 4
28

---

MINIONS: Collaboration Between On-device and Cloud Language Models
"exclude, "excluddeppatserne" pat:erns"; ("hdd90", "rigngt, "rigit"),
19
"path"; ".",
20
"pattern"; "vSuapse*"
21
"output"; "Found "quapos.pd"" in the cursent cirestery, Net aecersarily a reseige." })
22
23
24
"usort: "rsnote",
26
"prorpt": "Aralyze the response tron the local Language model...", "output"; "i. Ne found "quapss.pe"" An2. Not sufficient. Nead to read the file. And, Kissing content. \a4, Next tool: rile_ Anint Mneesapee,, Vraacttioofteee.ts "uapas.ppr"
27
29
30
"user": "rsmote",
31
"prorpt";
after step-by-step thicklog...', "output"; "/>nnesaage""" V"road_ pacha_ (wwppaaaaaaaapppe11 quapas.pdE")
33
34
35
"userf:
"ranote",
"prorpt": Lask complate, provide final ansves. Otlerwise, eequest additional info.", "outpat"; "I ("dacision"", Vrequest aaddest__dddiinne Lnaee V"massgee ("Please read the file "quapas.pdE" to daternine if is receipt and, if so, extract the total emount menticned in it.""
37
it
40
"esezf: "locai",
41
"prurpt"; " "Please sead the Cile "guapas.pdE" Lo deternion if it is recwipl and, if so, exicact Che Lotal anouns mantioned in it.",
"output": 1
43
44
45
"uses"; "Loul",
"cool"; "read_file""
47
"arqumenzs";
48
"ath" "guspax.pdE"
49
"output"; "Tha contains Doorbash secsipt Quapas Mexican Grill. Jotal Anount: $639.37"
51
52
53
"usenf;
Frenote",
"prorgt"; "Analyas the Lool pulpal and deternine if sufficient.
35
"outpat"; "I. "quapas.pd2" is a receipt.1n2. Information is acfficlenc An3, No miasing info And, Final Answer: AMAGEA? Aeciaionn's Mprovide, dineleeffiiiinii finel, snewer'", Vansver)"; (Yess, you have receigt for Snapee. The total amoont is $7,9.57.
38
"usenff "renote",
59
"prurpi"; "Binal confirnation ion step",
"output"; "(\"decision\"; "provide final, ansver)", Vannwerr""; ("Yes, you receipt Guapes,
is
$559.67.1*11
61
62
Listing Full Minion trace with local LLM tool calling
E.6. Analysis of Network latency
We measure network latency in the MINION setup. In evaluation. the MINION configuration uses LocalLM as LLAMA-3.2-1B and RemoteLM GPT-40. For each round of communication, we record the time spent on calls to LocalLM and Remote LM, as well as the time spent on communication between the local and remote clients. queries, cach an input context longth of 12,234
find communication
less than 0.002% of the total per round.
E.7. Extension to Multimodal Tasks
We demonstrate the extension MINIONS to multimodal settings. Specifically, evaluate multimodal math question-answering lask: MATH-VISION (Wang al.. 2024b). MATH VISION is challenging dataset that tests wide range of mathematical skills, ranging from algebru to geometry to combinatories, which require reasoming over visual input diagrams). our setup, we LocalLM be GEMMA3-4P 4BB image stays local and is not seen by RemoteL.M. For our remote only buseline, the image and question are passed directly to GPT-4o On sample of the test set, we find that MINIONS
SOTA
use GPI-40 RemoteLM.
MINIONS implementation.
points -0---y-
18.2% vs 87.8% accuracy.
F. Prompts
F.1. MINION RemoteL.M
29

---

MINIONS: Collaboration Between On-device and Cloud Language Models
need the foll ogting question based [doc_type]]
#+ Qoastion (query)
#+# instructions
You will not have direct access to the [doc_type), but can chat with a small Language model which has entire thing.
Feel free
step-by-step, but eventually you must provide an octpot
10 in the Format below:
11
<think step by step here>
13 "jaon
H
15
"massage": "syour message to tha small language model>"
13
17 10
LocalLM
will help the follewing question based (doc_type).
Read the [doc_type] bollww and propare to answer quastions from expert user. #4# [doc_type)
(context)
Quastion
(query)
Conversation
Sere is the response from the small language model:
#4# Response
(response]
instructions
Analyze the resporse and think-stap-byy-stop determine if you have enough information to answer the question.
10
11 Tf you have enough information ton, provide final mimeric anawer in the formst below.
13
step by step
15 *
11
17
"decision"; "provide_ fiieed_sweelle_sser""
18 "answe:*; #eyour answers*
1:
20
21
22 Otherwise, request additional information from the small language model
23 outputting the following:
25 <think step by step here> *jjon
27 11
"deciston"; seddeest_addttoon
29
"message"; "eyouz messaçe to the seall language model>"
39 13
31 14
E2. MINIONS
MINIONS: FINANCEBENCH
Derompose:
30

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Recorposition Round [sttep_number]
not have accesa to the docurent (s), but
assign tasks to
capable
language models that read the document (8).
Note that the document (s) cen be very long, 20 each task should be performed only over a small chunk
Write Python furctior
oucput tormatted
language model.
Make sure that NESE of the tasks require cslculations or complicated reasoning. Any information you mentioned your task ahould given an extract taak.
10 Flease use chunks of ipages_per_chunde pages using the 'chunk_on wonnkkoo_misiipeeeppeee. Idoc context, pagea_per_chunk, =lpages_per_chunk
funstion.
11
you multiple Lasks, conaider using nested for- 1oops apply Though it's not required to have more than one task,
14
15
Agsume Pydantic modal colled NobMani fast (BaseKodel) is already glabel scope. your reference, here la the model:
...
18
Assume Pydantic model called Vobbutput (BaseNodel) is already in global scope. For reference, here is the model:
21 ...
22 [outpat_source)
24 DU NUT rewrite import the madel in your code.
25
The function signature will look like:
27
gnature_sourcel
29
31 You can assume you have access to the following chunking function (s). Do not reimplement the function, just use it.
...
38 Ichunking sg_source)
...
Worker
Your job is to complete the following task using only the context below. The context is chunk of text taken acbitrarily from document, it might might consain relevant information to the task.
#+ Document [context]
#4 Task [task]
ladvice]
11 Return your reault in JSON with the following keys: "explanation", "citation", and "answer",
12
"explanation"; concise statement of ressoning how concluded your
14 "citation"; direst anippet of the text that supports your ansver. If nothing is found, put "Nona". Tr nothing found, put "None".
"answer"; The extracted 16
Re certain to on's y rely the provided text. you camat determine information confidently from this chunk, zeapond with "None" for all Iields.
Synthesime
31

---

missing MINIONS: Collaboration Between On-device and Cloud Language Models
Now synthesize the findings from mulziple junior workers (LUKS) an onswer tha questian below only you auf fcceet, reliable information.
task
Otherwiser you muet zequest additicnal work.
#4 Inputs
1. question to answer:
(queation]
10
11 2. Collected Jeb Outouts (Erom junior models):
[extractions]
13
15 First think step-by-step and then answer the question using the exact formst belew.
17 #4 ANSKER GUIDELINES
18 1. .*Jetermine the collected Job Outputs provide enough trustworthy, consistent evideace to confidently
the question.**
19
-It the cata is incomplete or contradictory, do Not quess. Iratead, specity what is mlasing. sufficient, provide final
21
22 2.
<B conservative. When in doub., ask for more information.
23
24 3. **ddéress conflicts... if multiple jobs give different answers, roly on whichever is best supported
valid "explanation" and tattattan"
25
II you need more information. from the conflicting jobs you could request additional work Irom your additions field). Then, in the next round can make smaller set of jobs to deternine which answer is correct.
specific jobs sure meation specific job
26
27
output 29 "decision"; Must be aither serovidd_fina__aee
4. Requized
Output**:
object with these keys:
0s "roguest_aaddtioon___.m tse "provide_tira aroovddetiral_araaerr if you enougn information.
31 Use "requeet_adee inweee___ddiiiialiiiieie if you caanet conclusively answer.
"explenation";
stazement about you arrived at your conclusion
33 "answer": The final answer string if "pirrrieei
or mull etherwise. Should without additional salculatiens explanations.
contain ONLY the final
34
is
your 35ON responae (with
outside the
36
37 <think step-by-step
38 3**
39 1f
"decision": "...",
41 "explanation "...",
"answer": "... null", Good formal: "0.56"; format: ratio is calculated
1-0.27+2 = 0.55"
1)
44
45
Importantt=
47 If there is not enough information, set "answer" to null, set "decision" to "request_adee" addaliooallinn specify exectly you need "missing_infor
job
48
Now, corafully inspest question, think step-by-stop perform any caleulations outputting the J5ON oblect.
MINIONS: LONGHEALTH
Derompose:
32

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Recorposttion Round [sttep_number)
not have accesa to the docurent (s), but
assign tasks to
capable
language models that read the document (8).
Note that the document (s) cen be very long, 20 each task should be performed only over a small chunk
Write Python furctior:
oucput tormatted
language model.
Maka sure that NENE of the tasks require miltiple steps. Each task should he atomic! Consider usting nested for-loops to apply a aet tasks to set of chunka.
The same 'Laek_id" should be applied to aultiple chunke. DO NOT instantiate new "Lask_id" for each combinetion of task and chunk.
the conversational history inform what chunking strategy has already been applied.
11
(ADVANCED STNIEEDD_STEPSSSSTTCCTTTIIC
13
Assume a Pydantic model called NobManii (BaseModel) is already in global scope. For your reference, 15 here is the model:
[ranifest_ccuress
17
Pydantic model called NeboOtput (DaseModell) is already in glebal scope. your reference, here is the model:
19
20 [outpat_source)
21
rew-ite import model your code.
23
function signature will look like:
25
26 [signature_sourcel]
28
30 You can aasume you have access to the following chuaking (usetien Do not reinglement the Sunct Lon, just use
31
32 [chunking_souree:
34
exangle
36
Jnique identifier 38 for doc_id, document it. enumerste (context)
task_id
39
$ if you need to chunk the document into sections chunks chunk_by_secctio (documant)
41
$ or if you need to churk the docurent into pages chunk_by__age (document)
43
chunk_id,
enumerata ( Create taak for extracting mantionn of spect fic. keywords
45
46
task = "Extract all mantions of the following keyworda:
48
"'aa"-9',, 'tunor marker', "September 2021', <U/m1', "Mra, Anderson'
50
job job_manifest = JcbNanifest(
51
52
taak_id-task_ld
53
chunk=chunk,
task=task,
35
advice="Focus on ex:racting the specific keywords related Lo Mrs. Anderaon's 8 Lumoz marker levele."
56
job_man mont fappad (japapdddobbppaepeepppppppppppap [jab_rani fest)
58
Poooker
33

---

MINIONS: Collaboration Between On-device and Cloud Language Models
compl complete the lowing task using only the context lowow. 1s chunk of text taken acbitrarily from document, it might or might not concain relevant information to the task.
#4 Document
[cen:ext)
### Quastion you are trying to arawar: lqueation]
have been trstructed extract information pertaining the fowlowing concepti::
10 \*Date of viait\", (Lask)
11
Format your respense follows:
13 1f
visit" : dddrect quote extracted Lexi"", 15 "ckayword_13 1>" ddirect quote extracted text "", "c ddiieett quote extracted text
17
18 1:
20 Can you please extract the relevant aectiona from the document that are related to the concepts provided? direct quctes sentences. concept mentioned, it
21
22 Your Answe::
Psyntheiize
following by synchesizing findings multiple junior (LLMs)
# Inputs
Question
[queation)
2. collecsed Job Outputs (from junior models):
(extractions)
12
think step-by-step
question vaing
below.
14
16
Reequ red OutputAs: You must output exactly one object with these keys:
"decision"; Must be "provide_finas seoride_finn__oe final__nnwwerr
19
"explanation"; short stacemens about how you arrived at your conal usion or what is atill mi missing
20 "answer"; The final answer string (thet matches are of the provided optioas) 15 "decisioa"""
provide_fii finel answer", null otheswise.
21
22
is the tomplate for your J8ON response:
24
<Lhink step by step heres
26
27
28 14
29 "decision"; "...",
lexplanation":: "..."
31 "answer"; "...", 1;
33
35 Now, carefully inspect the question, thinx step-by-step perform calculationa betore outputting the JBON objest. answer choices are provided, your answer must mexacttye natch one of the choices.
36
Quession:
38 qqasstoon)
40 Your Answes:
MINIONS: QASPER
34

---

MINIONS:
Collaboration Between On-device and Cloud Language Models
Paccompott
Racomposition Round (sstep_numba)
not access
document (s),
assign small capable
language modela that read the document (8)
Note that the document can be very long, so each Lask should performed only over a small chunk
Write Python furctior
will
formatted taaka
amall language model,
Make sure that NONE of the Laska require multiple slepe, Each Lask should be atomic! chunks.
Consider using nested for-loops
to The samee "task_id' should be applied to aultiple chunks. DO NOT instantiate new "Laak_id" for each combinetion of task and chunk.
apply set
the conversationsl history inform what chunking strategy has already been applied.
11
(ADVANCED DITANGE__STPPSLSNTTTTTTTTTL
13
Assume a sydantic model called NobManifest (BaseXodel) already globel scope. Fer your reference, 15 here is the model:
[ranifest__coure)
17
Pydantic model called NobOutput (BaseModell is already in global scope. your reference, here is the model:
...
20 [outpat_source)
21
rewrite import the model your code.
23
function signature will 1ook like:
25
[aiggaturo__ouraal
27
28
30
You can aasume you have access to the following chuaking Iusction (8), Do not reinglement the Cunct Lon, just use
31
|ehunking_sourree
33
34
exangie
36
task_id
Unique identifier
38 for doc_id, document in enumeraze (context)
if you need chunk the document sections
40
chunks chunk_by_sect.ine (document)
41
$ or if you need to churk the docurent into pages chunk_by_page (document)
43
chunk_id,
(chankke
45
Create task for extracting mantions of speci fic keywords
task
47
"Extract all mentions of the following keywords:
48
"'Ca"--r,, 'tunor marker', SSeptemeer 2321', <U/m1', "Mra, Anderson'
50
job cob_manifest = JobNanifest( chunk_idee id=-iid==ddddd"" id___ink_i___,
51
52
bask_id-task_dd,
53
chunk=chunk,
54
task=task,
55
advice="Focus on excracting the specific keywords related to Mrs. Anderaon's 8 tumoz marker levele."
56
job mont feats.appand jjab_rani
58
Powoker
35

---

MINIONS: Collaboration Between On-device and Cloud Language Models
compl complete the lowing using only the context lowow. 1s chunk of text taken arbitrarily from document, it might or might not concain relevant information to the task.
#4 Document
[cen:ext)
### Quastion you are trying to arawar: lqueation]
have been trstructed extract information pertaining the following concepti::
10 \*Date of viait\", (Lask)
11
Format your respense follows:
13 1f
visit" : dddrect quote extracted Lexi"", 15 "ckaysord_13 1>" ddirect quote extracted text "", "c ddiieett quote extracted text
17
18 1:
20 Can you please extract the relevant aectiona from the document that are related to the concepts provided? direct quctes sentences. concept mentioned, it out.
21
22 Your Anewe::
Poyntheiie
Answez the following by the aynchesizing findings multipla junior (LLM5)
# Inputs
Question
(question)
2. Collected Joa Outputs (Crom junior models)
10 (extractions]
12
First think step-by-step
question vaing
below.
14
# GUTDRA-NNR
16
17
Required JeoN OutputAs: You must output exactly one JSON object with these keys:
"decision": Must be "provide_fise fins___ssse____rrrrrr_r answer" "need more information"
19 "explanation"; short statemens about how you arrived at your conal usion or what is atill mi missing
20 "answer": final ansker that is text span pulled directly from the job output citations.
21
22
23 sere is the template for your JSON response:
25 <think steg-by-step here>
27 14
28 "decision"; "...", "explanation": " "..."
39 "answer": "..,"
32
Now, carefully inspect the quessien, step-by-ste perform any calculations outputting the JSON ab'ect.
34 If answez choices are provided, your answer must Axxxactyysa match one of the answer choices. directly from the document previous job
Don't paraphrase the final outputs.
37 Ques: Ques_ioa: ion: [question]
39
10 Your Answes:
36

---

MINIONS: Collaboration Between On-device and Cloud Language Models
G. Generated code
G.1. Minions: Example generated code for task decomposition
The document is statement Amazon. The question is: "What the breaklown of Amazon operating expenses as a percentage of nee sales for 2019?", The code generated the remote model for creating lask specification (lisst round) appears below. The code includes functions, prepare the list of job specifications, and aggregale the oulputs returned from executing these jobs, formatting them inlo single string.
def prepare_jobst
context: List [stt)
Optionsl [ebManifest]] None,
Optionel (List [JobCutput]] None,
-> List [JabMani fest
t Chunk the document into manageable sections chunks chunk_by_sect_on (contex: max_chunk_size=3000 overlag=201
t Define the task to extract operating expenses and not sales
"Ex:ract operating expenses and seles figures percentagas if available."
this chunk. Include any
12.
Create Joa manifests for each chunk bab_manifests
13
14
for chank in chunks: job_mani fests (JobMani feat (chunkschunk,, tosketosk, advice="Fouus on financisl data and percentages.
15
16
17
return job_mani reats
18
19 def caanssformouutuuze johs: [Jab),
21 } -> str:
t initialize dictionary store excracted datracted_data = [)
23
24
25
Process each output
26
for job in jobs:
output job.autput
28
if autpat.answer:
29
Aggragate the enswers by chunk_id
30
chunk_ld jeo.manifest costt.chunkkke
31
chunk_id not in extracted data:
32
extracted__atad
33
excracted_dese wtt.://ddttta.ttt____eaee
.appand lou:pot.annwwer)
35
Combine all extracted data into single string for turther analysls aggregated_outpe
37
for chunk_id, answera in patrractee_data.ttemeee gggregated_outpur ["Chuak (chunk_id):\a" in
38
for aggrogated_output +s (anaswwrr] \n
40
42
return aggregated_out.pu
For this example query, the model decides chunk the context with maximum 3000 characters per chunk (line It then assigns single extraction task be performed chunks (line 10).
Note that generie character-based chunking function specified made available global workspace during code exccution:
def chunk chunk_by_sse by_sttttionn
str, fin__chunk_sisee siser 3C00, overlaps
3 -> List [ste]:
sections []]
stert
while start < (doc)
and
j___chun__iiss
sect.ions.append Idoo sear]: start mrax_chuakkeeee overlap return sections
10
Our framework enables providing library chunking functions select from. Future work should have the model up with chunking function itself.
G.2. Improving Minions with Supervised Finetuning
Con we make better minion through finetining? There are at least possibilities for doing
Training the domsin, Having the small model specialize the target domain, by training it to solve problems and answer questions documents from the new domain.
Training to be a Minion. Training the small model to produce correct and concise responses to a supervisor model, arrive correct responses quickly.
37

---

MINIONS: Collaboration Between On-device and Cloud Language Models
Minion Supervised Finetuning (SFT) on LongHealth Task SFT Done on Question-Answer (QA) pairs
Liama 3.2-1B Instruct Variants
Llama 3.2-3B Instruct Variants
Type
1.000
1.0
Baseline
0.941
SFT
0.824
0.824
Accuracy (LLM as Judge)
0.765
0.706
0.706
0.6
0.4
0.353
0.2
0.0
QA
Minion
QA
Minion
Task
Task
Figure 11. Supervised finetuning improves minion performance LongHealth. We compare baseline QA (answering query document) and MINION protocols, with and without finetuning (blue and gray, respectively), using LLM-as-a-judge accuracy (y-axis). Left: IB LocalLM. Right 3B L.calLM.
Here we present prefiminary results on first possibility.
We finetune small models (LLAMA-3.22 1-.--11--Inntaa and LLAMA-3,2-38 Instruct)) the LONGHEALTH dataset (Adams et 2024). on 317 quesstions and lest on 17 held out questions. finetune models using Low Rank Adapiaation (LoRA; (Ilu el al., 2022)), following best pructices for hyperparameter selection 16, LR for 1B was le and 3B 2e - 4).
Experimental Setup
d5pout0.05, targeting all modules transfommer and training for four epochs; (Biderman et 2024)), and sweeping learning rites (best
Evaluation We LLM-as-a-Judge (Zheng 2023) with GPT-4o dotermine the prodicted response matches the ground-truth response, avoiding string matching.
For each model size, analyze accuracy four conditions:
Baseline QA asking model answer test question context, without any linetuning local-remote protocols
SFT QA asking a linetured small model to answer test question on the context
Baseline Minion without any finetuning, using the small model as part of the MINION pmotncll
SFT Minion - using fiactuned model as part of MINION
See Figure II. dataset regimes, find that Minion superior vanilla that the small model does when guided frontier model that asks one question at time until reaching solution. For IB but not BB, SFT improves the baseline QA performance. Using the SPT' checkpoints as part of the Minion protocal results further gains
Results
model sizes.
We caveat that this is relatively small finetuning dataset, and that future work is needed to reproduce the results on larger scales.
38


Reviews:
[{"review_id": "bdppmRRbyh", "reviewer": "Reviewer_vVmz", "summary": "The paper proposes an agentic pattern for collaborative modeling between a cloud-based large LM and a client-side small LM to reduce cloud inference costs. The authors propose two approaches:\n1. MINION: A simple communication protocol where the small model summarizes and interacts with the cloud model. However, it struggles with long contexts and following multi-step instructions.\n2. MINIONS: An improved systems approach that decomposes tasks into smaller chunks, enabling more efficient execution and parallelization.\n\nThe authors show that MINIONS recovers 97.9% of the accuracy of a cloud-only model while reducing costs by 5.7×. They also discuss ways to optimize the client LM, such as parallelizing subtasks to improve efficiency.\nThe study evaluates a range of client models and finds that this approach is effective for models above 3B parameters, with performance improving as model size increases.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback! We include a Common Response, followed by an Individual Response.\n\nPlease see the revised paper at this anonymous link: https://storage.googleapis.com/anonymous-files/minions.pdf\n\n## Common Response\n\nWe appreciate the positive feedback from all the reviewers:\n\n- Local-remote systems are a “compelling” \\[xykf] and “underexplored direction of research” \\[vVmz], making our submission a “strong contribution to the conference” \\[5xc4]. \n\n- Our central claim – that such systems offer attractive cost-accuracy tradeoffs – is substantiated \\[xykf], “well-supported by evidence” \\[5xc4], and “backed by experiments” \\[vVmz].\n\n- The experimental setup is “thorough” \\[vVmz], “well-planned” \\[5xc4], and “comprehensive” \\[xykf], and forms the “core strength of the paper” \\[5xc4]. \n\nThe reviewers’ feedback motivated the following updates:\n\n- **Latency \\[vVmz,xykf]:** We added latency benchmarks on a single consumer GPU (_e.g._ RTX 4090) and found Minion/MinionS are only 1.44× and 2.89× slower than remote-only, while yielding 30× and \\~5× cost savings (See §6.5)\n\n* **Adapting LocalLM \\[vVmz]:** We show that Minion accuracy can be improved by supervised finetuning (SFT) of the minion (1B & 3B scales) on the target domain (§G.2). In §7, we highlight further opportunities for co-adaptation.  \n\n* **Agentic Tool Use \\[vVmz]:** Introduced a tool-augmented version of Minions, where the local model can use local tools, matching GPT4o-only accuracy (0.7) while cutting prefill cost by \\~3.7× (See §E.5)\n\n* **Energy savings \\[xykf]:** Added energy consumption analysis showing Minions consumes just 1/12th (1B LocalLM) and 1/6th (3B LocalLM) the energy of GPT-4o alone (see §E.4).\n\n* **Privacy implications \\[xykf,5xc4]:** We highlight this important direction in §7. While privacy merits a more careful treatment in future work, our preliminary results show that local LLM-based PII filtering reduces PII leakage from 22% to 4.5%\n\n---\n\n## Individual Response\n\n**Latency analysis**\n\n> There is no evaluation of latency in the paper. I would expect that this approach would increase latency.\n\nYour suggestion led to new latency experiments to complement the theoretical analysis in the original submission. We benchmarked Minion/MinionS on 2 consumer-grade GPUs commonly used in local workstations (e.g., RTX 4090, MSRP 1,599 USD on 03/26), finding they are only $1.44×$ and $2.89×$ slower than remote-only, while offering \\~30× and \\~5× cost savings. See Tab. 2 in the revised manuscript for more details. \n\nWe note that these empirical latency measurements depend on point-in-time factors like local hardware and cloud load. Thus, in §C.2 we provide a theoretical framework for estimating the latency overhead of any local-remote system given model and hardware specifications like memory bandwidth.\n\n**Minion + agentic tool use.**\n\n> Given the setup for cloud/client, agentic tasks such as automatically perform certain actions would have been relevant to include.\n\nTo support agentic tasks—where the model autonomously performs actions—we extend the Minions framework to enable local tool use (see §E.5). In this setup, the local LM executes actions guided by the remote LM. We evaluate this on filesystem queries with 5 tools and find that using Qwen2.5 locally with GPT-4o as the remote LM, matches the performance of GPT-4o-only while using less than 28% of the remote tokens (see Tab. 12).\n\n**Expansion of related works.**\n\nWe now cite several of the works you highlighted including speculative decoding, collaborative inference, and prompt compression. \n\n**Adapting LocalLM**\n\n> Adapting the small model to this interaction pattern would be an interesting ml aspect and extension.\n\nSee common response. The revised manuscript now takes a step in this direction by finetuning small models on the target domain and demonstrating improvements in Minion accuracy (§G.2). The revised discussion (see “Local-remote model co-design”) spells out a number of extensions we are excited about, including multi-agent co-training.\n\n**Stratifying by task complexity**\n\n> It would have been desirable to include evaluations that more explicitly span tasks of varying complexity\n\nWe agree and therefore stratify Financebench (FIN) and Longhealth (LH) problems by complexity. Surprisingly, we find that the Minions protocol outperforms the remote-only condition on harder problems. For example, on simple info. extraction tasks in FIN, Minions (with Qwen2.5) trails by 22.7 points, but on complex extraction & numerical reasoning tasks, it outperforms the remote-only by +4.6. Same holds for LH, where Minions (with Llama-8B) is -6.2 pts. on single-span questions but leads by +16.0 on multi-span synthesis. This trend holds across model sizes.\n\n**Expanded discussion of cloud/client setup** \n\n> Alternatively, the author should double down more on the benefits of cloud/client setup…\n\nWe’ve expanded §7 (Discussion) to better highlight the benefits of the cloud/client setup.", "claims_and_evidence": "Claims are reasonably well supported and backed by experiments.", "methods_and_evaluation_criteria": "The selected datasets cover a diverse set of domains domains in finance, healthcare, and science. It would have been desirable to include and discuss evaluation methods that more explicitly span tasks of varying complexity (e..g. simple Q&A retrieval tasks, more complex reasoning, ...). Also, given the setup for cloud/client, agentic tasks such as automatically perform certain actions would have been relevant to include.\n\nLastly, the main evaluation is cost and performance. One major reason for client side inference is closeness to user, and providing a snappy experience. Beside a little mention at the beginning, there is no evaluation of latency across the paper.I would expect that this approach would significantly increase latency and diminish value of this approach.", "theoretical_claims": "I did not check correctness of theoretical claims", "experimental_designs_or_analyses": "The authors perform a comprehensive set of experiments. Their setup and analysis seems sound, but there is a definitve gap in evaluating latency as pointed out above.", "relation_to_broader_scientific_literature": "The paper makes a contribution in the very crowded space of LLM agents. It's angle on splitting agents across cloud and client is generally an underexplored area and interesting direction of research.\n\nThe papers main positioning seems to be reducing cloud inferencing cost. As such, a more detailed comparison with alternative approaches such as prompt compression, speculative decoding may be desirable. Alternatively, the author should double down more on the benefits of cloud/client setup, and what type of experiences this could enable.", "essential_references_not_discussed": "There is other work leveraging a smaller model combined with a big one. Consider citing the following:\nPrompt compression using a smaller model. This will reduce cloud efficiency and potentially also could run on client\nJiang et al. (2023). LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP 2023.\nPan et al. (2024). LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. ACL 2024.\n\nConsider referencing for collaborative cloud/edge modeling (and related references):\nHao et al (2024) Hybrid SLM and LLM for Edge-Cloud Collaborative Inference. EdgeFM 2024\nXia et al. (2024). Hybrid Retrieval-Augmented Generation for Real-time Composition Assistance, EMNLP Industry Track 2024", "other_strengths_and_weaknesses": "Strengths:\n- Interesting take on cloud/client models of agent scenarios\n- thorough experimental study (despite lack broader benchmarks)\n\nWeakness:\n- Mainly experimental with little theoretical backing\n- Main ML aspect is prompt engineering, and combination of multiple agents. Adapting (or at least the discussion of it) the small model to this interaction pattern would be an interesting ml aspect and extension.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MQPR8kTjDV", "reviewer": "Reviewer_xykf", "summary": "This paper presents MINION and MINIONS, novel frameworks for cost-efficient collaboration between small on-device and cloud-based language models. MINION enables asymmetric collaborative communication between LocalLM (Reading) and RemoteLM (Reasoning), achieving a 30.4× cost reduction while recovering 87% of remote-only performance. However, it struggles with multi-step instructions and long contexts. To address this, MINIONS introduces task decomposition and parallelized subtasks in LocalLM, reducing costs by 5.7× while recovering 97.9% of remote-only performance. Experimental validation on multiple benchmarks highlights the trade-off between cost and accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback! We include a Common Response, followed by an Individual Response.\n\nPlease see the revised paper at this anonymous link: https://storage.googleapis.com/anonymous-files/minions.pdf\n\n## Common Response\n\nWe appreciate the positive feedback from all the reviewers:\n\n- Local-remote systems are a “compelling” \\[xykf] and “underexplored direction of research” \\[vVmz], making our submission a “strong contribution to the conference” \\[5xc4]. \n\n- Our central claim – that such systems offer attractive cost-accuracy tradeoffs – is substantiated \\[xykf], “well-supported by evidence” \\[5xc4], and “backed by experiments” \\[vVmz].\n\n- The experimental setup is “thorough” \\[vVmz], “well-planned” \\[5xc4], and “comprehensive” \\[xykf], and forms the “core strength of the paper” \\[5xc4]. \n\nThe reviewers’ feedback motivated the following updates:\n\n- **Latency \\[vVmz,xykf]:** We added latency benchmarks on a single consumer GPU (_e.g._ RTX 4090) and found Minion/MinionS are only 1.44× and 2.89× slower than remote-only, while yielding 30× and \\~5× cost savings (See §6.5)\n\n* **Adapting LocalLM \\[vVmz]:** We show that Minion accuracy can be improved by supervised finetuning (SFT) of the minion (1B & 3B scales) on the target domain (§G.2). In §7, we highlight further opportunities for co-adaptation.  \n\n* **Agentic Tool Use \\[vVmz]:** Introduced a tool-augmented version of Minions, where the local model can use local tools, matching GPT4o-only accuracy (0.7) while cutting prefill cost by \\~3.7× (See §E.5)\n\n* **Energy savings \\[xykf]:** Added energy consumption analysis showing Minions consumes just 1/12th (1B LocalLM) and 1/6th (3B LocalLM) the energy of GPT-4o alone (see §E.4).\n\n* **Privacy implications \\[xykf,5xc4]:** We highlight this important direction in §7. While privacy merits a more careful treatment in future work, our preliminary results show that local LLM-based PII filtering reduces PII leakage from 22% to 4.5%\n\n---\n\n## Individual Response\n\n**Latency Analysis**\n\n> A more detailed discussion of latency optimization would be helpful.\n\nThe updated §6 includes comprehensive latency experiments. See §6, the Common Response and the individual response to vVmz for more details. \n\n**Feasibility**\n\n> The size of Local LM might be too large for practical on-device deployment.\n\n§6.2 now discusses the feasibility of running LLMS on modern laptops and workstations¹. Devices like the d MacBook Pro support up to 600B and 200B models with quantization², and even iPhone 15 Pro handles \\~3B³. Our latency experiments on consumer grade hardware show Minion and Minions are only 1.44× and 2.89× slower than remote-only (see §6).\n\n\\[1] <https://ollama.com/library>\n\n\\[2] <https://www.apple.com/newsroom/2024/10>\n\n\\[3] <https://machinelearning.apple.com/research/introducing-apple-foundation-models>\n\n**Model size tradeoffs**\n\n> Discussing the trade-offs between model size and performance in more detail would provide valuable insights into the feasibility.\n\nIn our revised manuscript, §6.2 (Model Choice) and Figure 4 detail how performance and communication efficiency vary with local model size.\n\n**Energy consumption analysis** \n\n> The paper does not extensively discuss energy consumption.\n\nYour point led to a new analysis, showing major energy savings by Minions (see  §E.4 for analysis). As we do not have access to the hardware running GPT-4o, we use energy consumption estimates from Epoch AI [1].  We benchmark local energy use for 1B & 3B models on an M1 Max and A100 GPU. We compare to GPT-4o-only execution, and find 12$\\times$ energy savings with the 1B LM and 6$\\times$ with the 3B model on A100, with similar gains on M1 Max. \n\n\\[1] https\\://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use\n\n> Might be useful to explore privacy-aware chunk extraction to enhance secure collaboration.\n\nWe agree! While a full treatment of data privacy is beyond the scope of this paper, we report preliminary experiments on leakage mitigation within Minions (see Common Response and response to 5xc4).\n\n**Clarity and Presentation**\n\n- Experimental Setup: We added hardware details for local models (see §B).\n\n- Protocol Description: We rewrote the Methods accordingly; added an example of remote chunking (see §4,5,G.1).\n\n- Consistent Terminology\n\n  - The revision uses a more consistent terminology for the names of the local and remote models.\n\n  - Consolidated \"multi-step\" and \"multi-part\" language.\n\n**Other Questions**\n\n> Does this work with small models (<1B)?\n\nNot well—performance improves significantly with local model sizes >=3B (see §6 + Fig. 4)\n\n> Is network latency a bottleneck?\n\nNo, communication time is negligible (<0.002%) compared to inference (see §E.6)\n\n> Are there security concerns?\n\nYes! There are works that study this extensively [1].\n\n> Can this support multimodal inputs?\n\nYes, using a VLM as the local LM enables image-text processing (see §E.7)\n\n\\[1] <https://arxiv.org/abs/2310.06845>", "claims_and_evidence": "This paper asserts that the MINIONS protocol significantly reduces cloud inference costs while maintaining accuracy comparable to cloud-based models. Additionally, it claims that by adjusting the protocol’s hyperparameters, a flexible trade-off between cost and performance can be achieved. To support this claim, the authors conduct extensive evaluations across various domain benchmarks and different model sizes and types, reinforcing the validity of their argument. Notably, the analysis of performance recovery based on model size substantiates the claim regarding the cost-performance trade-off.", "methods_and_evaluation_criteria": "The proposed MINIONS framework is well-suited to the problem and highly plausible. The explicit role definition for both language models and the protocol designed to compensate for the limitations of small LMs are particularly compelling. However, the explanation of how the method operates in each iteration is located in the appendix, making it difficult to find and understand the exact workings of the loop. It would be beneficial to include a brief explanation in the main text or provide a hyperlink to the appendix. The evaluation criteria focus on accuracy and cost, both of which are highly appropriate for assessing the proposed protocol’s cost-performance trade-off.", "theoretical_claims": "Rather than relying on theoretical claims, this study is primarily supported by experimental evidence, which is appropriate for the research context. However, a comparative analysis of communication latency between the two language models alongside real experimental data would have enhanced credibility.", "experimental_designs_or_analyses": "The experimental design and analysis are conducted using appropriate methodologies. However, there is a lack of information on the experimental setup, such as the specific GPU used, which should be supplemented. Additionally, the comparative analysis of different hyperparameters effectively identifies the key factors influencing the protocol’s performance.", "relation_to_broader_scientific_literature": "This study builds upon prior research in multi-agent systems, retrieval-augmented generation (RAG), and cost-efficient LLM routing. Notably, it differentiates itself by addressing asymmetric roles between small on-device models and large cloud-based models, as well as exploring inter-model interactions. Citations to relevant literature are sufficient, and comparative experiments with RAG models effectively demonstrate the value of MINIONS.", "essential_references_not_discussed": "This paper leverages the appendix to cite all relevant studies comprehensively.", "other_strengths_and_weaknesses": "Strength\n- The multi-round communication approach introduced in the framework allows for iterative improvements in performance, which is a promising direction.\n- The paper provides an insightful cost analysis, demonstrating how task decomposition and parallelization contribute to cost reduction. Also, the study does well in quantifying the trade-offs between cost and accuracy across multiple benchmarks.\n- The inclusion of various hyperparameter evaluations strengthens the reliability of the findings.\n- Sharing prompts used in experiments, along with example responses, enhances reproducibility and improves clarity.\n\nWeakness\n- The size of Local LM might be too large for practical on-device deployment in resource-constrained environments. Discussing the trade-offs between model size and performance in more detail would provide valuable insights into the feasibility of different approaches.\n- While cost reductions are well-analyzed, a more detailed discussion of latency optimization would be helpful. Measuring actual latency could provide clearer performance.\n- The paper does not extensively discuss energy consumption or the impact on local resources (e.g., memory and computational overhead). A deeper analysis of these aspects would be valuable.\n- The framework would benefit from a clearer pipeline structure, particularly in explaining how the loop functions, as described in Section D. The current explanation is somewhat unclear, making it difficult to fully understand the method and integrate it into existing workflows. Additionally, providing example data in the appendix to illustrate how the Remote LM applies its data chunking strategy would be helpful. The inconsistent use of terminology also makes the paper harder to follow on the first read. If these aspects were clarified and the protocol were made more explicit, I would be willing to reconsider my rating.", "other_comments_or_suggestions": "- It might be useful to explore privacy-aware chunk extraction to enhance secure collaboration between LocalLM and RemoteLM.\n- Maintaining consistent terminology (e.g., ensuring \"multi-step\" and \"multi-part\" are clearly defined) would help avoid potential confusion.\n- In line 246, clarifying whether \"cloud model\" refers to RemoteLM in this study would improve clarity.", "questions_for_authors": "- Does the framework remain effective for small language models under 1B parameters?\n- Has the impact of network latency on local-remote communication performance been analyzed?\n- Are there potential security risks, such as adversarial attacks, in local-remote collaboration?\n- Can this approach be extended to other multimodal inputs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cAW5qRl1i6", "reviewer": "Reviewer_5xc4", "summary": "The paper presents a setting where a small model having access to local data collaborates with a state-of-the-art LLM cloud-hosted (without access to the data) to solve real tasks. To improve over an initial naive protocol (with back and forth chats between the two models), the paper introduces Minions, where the cloud model creates sub-tasks for the local model to execute. The paper presents clear reduction in costs while maintaining high performances.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback! We include a Common Response, followed by an Individual Response.\n\nPlease see the revised paper at this anonymous link: https://storage.googleapis.com/anonymous-files/minions.pdf\n\n## Common Response\n\nWe appreciate the positive feedback from all the reviewers:\n\n- Local-remote systems are a “compelling” \\[xykf] and “underexplored direction of research” \\[vVmz], making our submission a “strong contribution to the conference” \\[5xc4]. \n\n- Our central claim – that such systems offer attractive cost-accuracy tradeoffs – is substantiated \\[xykf], “well-supported by evidence” \\[5xc4], and “backed by experiments” \\[vVmz].\n\n- The experimental setup is “thorough” \\[vVmz], “well-planned” \\[5xc4], and “comprehensive” \\[xykf], and forms the “core strength of the paper” \\[5xc4]. \n\nThe reviewers’ feedback motivated the following updates:\n\n- **Latency \\[vVmz,xykf]:** We added latency benchmarks on a single consumer GPU (_e.g._ RTX 4090) and found Minion/MinionS are only 1.44× and 2.89× slower than remote-only, while yielding 30× and \\~5× cost savings (See §6.5)\n\n* **Adapting LocalLM \\[vVmz]:** We show that Minion accuracy can be improved by supervised finetuning (SFT) of the minion (1B & 3B scales) on the target domain (§G.2). In §7, we highlight further opportunities for co-adaptation.  \n\n* **Agentic Tool Use \\[vVmz]:** Introduced a tool-augmented version of Minions, where the local model can use local tools, matching GPT4o-only accuracy (0.7) while cutting prefill cost by \\~3.7× (See §E.5)\n\n* **Energy savings \\[xykf]:** Added energy consumption analysis showing Minions consumes just 1/12th (1B LocalLM) and 1/6th (3B LocalLM) the energy of GPT-4o alone (see §E.4).\n\n* **Privacy implications \\[xykf,5xc4]:** We highlight this important direction in §7. While privacy merits a more careful treatment in future work, our preliminary results show that local LLM-based PII filtering reduces PII leakage from 22% to 4.5%\n\n---\n\n## Individual Response\n**Exploration on sensitive data.**  \n\n> _“The main point which I would like the author to discuss more in details is a setting where the data on premise is sensitive and should absolutely not be shared with the cloud model. How would you guarantee that that the local LLM would not leak, even by mistake, any piece of information to the cloud model? Have you done any experiment on this specific topic?_\n\nThank you for highlighting this important setting! While a comprehensive treatment of data privacy  is beyond the scope of this paper, we agree that the topic merits a preliminary exploration. We thus present an initial analysis of data leakage mitigation within the Minions framework using  contemporary privacy-preserving techniques, including controlled prompting [1,2] and PII filtering. Methodologically, we use a prompt-based filtering layer—a secondary LLM call applied to local LMs output—to remove sensitive information. We evaluate this method on a QA dataset over a local filesystem of purchase receipts containing emails, addresses, phone numbers, credit card details, and more. With the privacy filter enabled, PII leakage rate drops from 22% to 4.5%. While preliminary and imperfect, these results highlight the importance of further research in this area.\n\n**_Clarity and Presentation._**\n\nAs per your feedback, we have improved the clarity of Figure 2.\n\n**\\[1]** [**https://arxiv.org/abs/2410.17127**](https://arxiv.org/abs/2410.17127)\n\n**\\[2]** [**https://arxiv.org/pdf/2403.03129**](https://arxiv.org/pdf/2403.03129)", "claims_and_evidence": "The core of the paper is focused around presenting processes that reduce cost while maintaining high performance (see for instance Figure 2). Overall the claims of cost reduction while maintaining high performance are convincing and well supported by evidence (e.g. Fig 5-6 and Table 1) and overall the paper is very well structured making it a strong contribution to the conference.", "methods_and_evaluation_criteria": "The method and evaluation criteria are robust (see Table 1) and especially the benchmark across different type of datasets (finance, health and scientific QA) gives a clear overview of benefits of this approach.", "theoretical_claims": "The paper doesn't rely on theoretical claims", "experimental_designs_or_analyses": "The experimental design is the core strength of this paper: it is well planned and shows the advantage of Minions over the initial naive protocol and in comparison to the state of the art performance of a frontier model (GPT-4o)", "relation_to_broader_scientific_literature": "The paper is well positioned in the scientific literature (however this is mostly presented in the appendix and not directly in the main content of the paper)", "essential_references_not_discussed": "I haven't noticed anything missing", "other_strengths_and_weaknesses": "The main point which I would like the author to discuss more in details is a setting where the data on premise is sensitive and should absolutely not be shared with the cloud model. How would you guarantee that that the local LLM would not leak, even by mistake, any piece of information to the cloud model? Have you done any experiment on this specific topic?", "other_comments_or_suggestions": "I would improve Figure 2, which I think it's very important for the narrative of the paper, but I found a bit hard to read as many layers of information are embedded in it", "questions_for_authors": "Nothing else", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a framework designed to enable efficient collaboration between a small client-side language model and a large cloud-based model, with the primary goal of significantly reducing cloud inference costs while maintaining competitive performance. All reviewers appreciate the paper’s novel approach and rigorous experimental validation, noting its relevance in the rapidly evolving domain of collaborative LLM agents. Strengths highlighted include comprehensive experimentation across diverse domains, clearly demonstrated cost-performance trade-offs, insightful analyses on hyperparameter tuning, and robust supplementary materials for reproducibility. The reviewers find the paper's claims well-supported by extensive empirical evidence.

The authors and reviewers have a good rebuttal discussion. The authors are encouraged to incorporate these discussion and findings in the camera ready revision.
Decision: Accept (poster)