Meta-Learning for Speeding Up Large Model Inference in Decentralized Environments
Authors: Yipeng Du
Zihao Wang
Ahmad Farhan
Claudio Angione
Harry Yang
Fielding Johnston
James P. Buban
Yue Zhao
Yuzhe Yang
Venue: COLM 2025 (2025)
Keywords: Fast inference
LLMs
meta learning
decentralized systems
TL;DR: We propose a meta-learning framework to optimize inference acceleration in decentralized AI systems based on task-specific data, promoting more cost-effective and scalable AI deployment.

Abstract: The deployment of large-scale models, such as large language models (LLMs), incurs substantial costs due to their computational demands. To mitigate these costs and address challenges related to scalability and data security, there is a growing shift towards decentralized systems for model deployment, where choosing efficient inference acceleration schemes become crucial to manage computational resources effectively and enhance system responsiveness. In this work, we address the challenge of selecting optimal acceleration methods in decentralized systems by introducing a meta-learning-based framework. This framework automates the selection process by learning from historical performance data of various acceleration techniques across different tasks. Unlike traditional methods that rely on random selection or expert intuition, our approach systematically identifies the best acceleration strategies based on the specific characteristics of each task. We demonstrate that our meta-learning framework not only streamlines the decision-making process but also consistently outperforms conventional methods in terms of efficiency and performance. Our results highlight the potential of inference acceleration in decentralized AI systems, offering a path towards more democratic and economically feasible artificial intelligence solutions. Our code and data will be released later.

Published as a conference paper at COLM 2025
Meta-Learning for Speeding Up Large Model Inference in Decentralized Environments
Yipeng Du", Zihao Wang" Ahmad Farhan, Claudio Angione, Harry Yang',
Fielding Johnston, James P. Buban, Patrick Colangelo, Yue Zhao, Yuzhe Yang Nesa Research
*zwangin@connect.uss ust. 'yangharry@ust.hk
researchenesa.ar
Abstract
The deployment of large-scale models, such as large language models (LLMs), incurs substantial costs due their computational demands. To mitigate these costs and address challenges related to scalability and data security, there is growing shift towards decentralized systems for model deployment, where choosing efficient inference acceleration schemes be- come crucial to manage computational resources effectively and enhance system responsiveness. In this work, we address the challenge of selecting optimal acceleration methods in decentralized systems by introducing a meta-learning-basee based framework. This framework automates the selection process by learning from historical performance data of various accelera- tion techniques across different tasks. Unlike traditional methods that rely on random selection or expert intuition, our approach systematically iden- tifies the best acceleration strategies based on the specific characteristics of each task. We demonstrate that our meta-learning framework not only streamlines the decision-making process but also consistently outperforms conventional methods in terms of efficiency and performance. Our results highlight the potential of inference acceleration in decentralized Al sys- tems, offering path towards more democratic and economically feasible artificial intelligence solutions.
1
Introduction
The growing demand for large-scale models such as language models and image generation systems has significantly increased computational requirements. Although centralized systems remain capable, they often face challenges related to scalability, data security, and cost. These limitations have sparked interest in decentralized architectures, which distribute computation across multiple nodes to improve efficiency, reduce latency, and enhance privacy. (Brown et al., 2020; Ramesh et al., 2022; Zhang et al., 2025). Traditional centralized systems, though powerful, face key limitations in scalability, data security, and cost (Li et al., 2022; Dong et al., 2025). These constraints have driven interest in decentralized architectures that distribute computation across nodes to enhance efficiency, reduce latency, and protect data privacy (Belotti al., 2019; Zeng et al., 2025; Zhang al., 2024).
Deploying Al in distributed settings introduces distinct challenges from centralized systems, due to heterogeneous hardware, variable network latency, and dynamic load balancing requirements (Borzunov et al., 2024; Biran & Kissos, 2025; Balseiro et al., 2025). Unlike ho- mogeneous data centers, decentralized systems span diverse devices with varying compute, memory, and communication capabilities. This heterogeneity complicates workload dis- tribution and calls for adaptive strategies responsive hardware availability and runtime constraints(Shen et al., 2025).
Motivated by these challenges, we evaluate fast inference techniques, such as continuous batching, prefix caching, and chunked prefill-under varying batch sizes and hardware
"qqual contribution.

---

Published as a conference paper at COLM 2025
Counted Pret#
Contirunus Sarching
Fretis Caching
1300
Duration (seconds)
200
17.
27.8
13.7K
50
16
64
128
256 512
Batch
Figure 1: Performance comparison of acceleration strategies across varying batch sizes. We compare the performance different fast inference techniques on the LLaMA 3.1 8B model. The combined strategy outperforms all individual techniques for small batch sizes, while prefix caching performs best for larger batches, This highlights the need for adaptive selection mechanisms tailored to workload scale.
settings. While effective uniform, high- thghtthroughput settings, their utility in heteroge- neous systems is less predictable. For instance, prefix caching may underperform on monsry-constrained nodes, and chunked prefill can incur communication overhead in low-bandwidth environments. Performance is further modulated by interactions between model architecture, input characteristics, and system state. shown Figure combining all techniques excels in small--btch regimes, whereas individual strategies fare better with large batches. This reinforces a central insight: no single inference strategy is universally optimal. We further demonstrate this in where Phi-2 performs substantially worse with prefix-caching turned on while other models like Baichuan2 performs significantly better. This demonstrates the need for predictor that understands the different attributes and can thus pick the correct method for each model and particular hardware.
Hence, achieving efficient inference in decentralized systems requires adaptive scheduling that reasons over models, inputs, and hardware in real time.
Model
Chunked Prefill
Prefix Caching
All Methods
Baichuan2-7B-Chat
+3.82%
+37.63%
+7.96%
Qween2.577-Innstruct-11
+4.15%
-10.66%
-7.20%
Phi-2
-0.90%
-56.79%
-11.39%
Meta--Llama-3.1-8B-Instuue
+5.40%
-4.03%
+0.33%
Table 1: Additional experiments on NVIDIA L4 GPUs, with continuous batching turned on by default with fixed batch size of 1024
To this end, introduce learnable meta-scheduler that predicts optimal inference strategies and hardware configurations from real-time input and system signals. We cast inference op- timization as a meta-learning problem, where rich embeddings of workloads and hardware states guide deployment decisions. This flexible framework, detailed in Section 3, addresses the need for robust inference across diverse conditions.
Our contributions are as follows:
We introduce a principled formulation of inference strategy selection as budget- constrained meta-learning problem, where acceleration methods are selected based on learned mappings from task, model, and hardware representations to performance outcomes. This framing enables cost-aware deployment of large models under heterogeneous and decentralized constraints.
We propose Metalnf, a lightweight meta-scheduling framework that leverages LLM- derived semantic embeddings and historical inference metadata to predict optimal
2

---

Published as a conference paper at COLM 2025
acceleration strategies under deployment constraints. By decoupling selection from online measurement, Metalnf enables zero-shot generalization to unseen bonuell-hrrdaarewwokkaan combinations and maintains high accuracy even under strict cost budgets.
We develop unified embedding architecture that encodes heterogeneous figuration comprisin model architecture, hardware profiles, and task descriptions-uuing language model-derived representations. These embeddings capture latent compatibility across system components and support robust general- ization under distribution shift.
Related Works
Inference for Large Systems. Fast inference is essential for deploying large-scale models under constraints of real-time performance, responsiveness, and cost-efficiency (Brown et al., 2020; Rajbhandari et al., 2020). While much prior work targets acceleration in centralized, high-performance environments, an emerging body of research addresses the distinct challenges of distributed and heterogeneous systems.
Inference in Centralized Systems. Key techniques have been developed to accelerate inference in centralized settings. Large batch sizes improve GPU utilization via request aggregation (Shoeybi et al., 2019; Henighan et al., 2020). Continuous batching interleaves requests to reduce idle compute and boost throughput (Yu et al., 2022; Kwon al., 2023). Prefilling and prefix caching exploit autoregressive structure to reuse cached key-value pairs, avoiding redundant computation (Pope et al., 2023; Rajbhandari al., 2020). Prompt caching generalizes this reuse by detecting shared prefixes across requests (Brown et al., 2020). Recent advances such as speculative decoding employ lightweight draft models to propose candidate outputs, later verified by larger models. This technique accelerates generation while preserving output quality (Leviathan et al., 2023; Cai et al., 2024; Li et al., 2024), and has been widely adopted in fixed, high-throughput inference pipelines.
Inference Distributed and Heterogeneous Systems. While these techniques yield gains in homogeneous clusters, distributed and heterogeneous environments introduce new challenges- such such as diverse compute capabilities, memory limits, and variable network latency (Borzunov et al., 2024; Xu et al., 2019). In edge and decentralized deployments, assumptions valid in centralized systems often break down due to dynamic conditions. Several works address these challenges via expert routing (Lepikhin et al., 2020), model parallelism, and adaptive scheduling (Ren et al., 2021). However, many rely on fixed routing strategies or assume stable infrastructure, limiting adaptability in real-world, non-stationary settings.
Towards Adaptive Distributed Inference. Our work frames inference optimization as dynamic decision-making problem. Instead of static heuristics, we propose learnable meta-scheduler that adapts in real time to hardware profiles, input complexity, and system conditions. This approach bridges fine-grained acceleration with system-level adaptability, enabling scalable, robust AT deployment.
3 Method
3.1 Motivating Example: Performance Variability in Inference Acceleration
To motivate the need for a meta-learnnng-basee scheduler, we first conduct a comprehensive evaluation of popular inference acceleration techniques across heterogeneous hardware and model configurations. Our goal is to highlight the variability and unpredictability of method performance, thereby justifying the need for adaptive selection strategies.
3

---

Published as a conference paper at COLM 2025
Method
BS=16
BS=256
Method
4 GPUs
8 GPUs
None
1435.27
1424.99
CP
120.89 (5.43)
117.21 (5.15)
CP (Chunked Prefill)
128.70
114.44
CB
122.73 (12.75)
118.19 (13.87)
CB (Cont. Batch)
146.44
107.40
PC
79.18 (11.86)
76.33 (11.92)
PC (Prefix Cache)
101.10
68.46
All
85.07 (6.04)
82.90 (5.96)
All
96.21
80.65
Table 3: Scaling with GPUs, Runtime (s, var.) for 4 vs. 8 GPUss Diminishing re- turns appear for CP and CB due to comm. overhead.
Table 2: Latency batch size. Infer- ence latency (s) for each strategy at BS 16 and 256. PC excels at large BS, while combining all works best at small BS.
3.1.1 Evaluation Setup
We implemented continuous batching, prefix caching, and chunked prefill, as well their combined use, within a unified inference engine. All experiments were conducted using the ShareGPT dataset (Kwon al., 2023).
Models. We evaluated widely used open-source instruction-tuned LLMs, primarily Meta-Llama-3.1 1----IInttrutt since many open-source variants adopt similar architectural patterns.
Hardware. Inference was executed on NVIDIA L4 GPUs using tensor parallelism across and 8 GPU configurations.
Configuration. All models were capped a maximum output length of 2048 tokens. Quantization precision was maintained at float16.
3.2 Findings and Insights
Inference acceleration methods substantially improve throughput. Continuous batch- ing and prefix caching yielded up to 13x and 20 speedups respectively compared to unoptimized baselines (Table 2).
No method universally optimal. As shown in Figure 1, prefix caching performed best for large batches, while continuous batching excelled in low-latency settings. The "All" configuration performed well in some setups but introduced overhead others.
Hardware scaling shows diminishing returns. Scaling from 4 to GPUs led to only modest throughput gains suggesting that communication overhead and bandwidth constraints limit performance.
These observations reveal that acceleration performance depends not only on the method itself but also on contextual factors such as model architecture, batch size, and hardware topology. This motivates our proposed solution: a learnable meta-scheduler that can adaptively select acceleration strategies given target system state.
3.3 Problem Statement and Framework Overview
Given new model, dataset, and hardware environment for distributed inference, our goal is to select the best acceleration method such as quantization, model compression, or parallelism-wiie
costly empirical evaluations. This selection must account for hardware- specific constraints and computational capabilities.
We leverage meta-learning to transfer performance knowledge from prior inference tasks. The key insight is that methods effective in similar historical contexts likely to perform well on new configurations especially when immediate evaluations are infeasible due to deployment urgency resource constraints.
Our proposed meta-learner, Metalnf, optimizes inference acceleration across diverse envi- ronments by learning from historical data. It comprises the following components:
4

---

Published as a conference paper at COLM 2025
Offline Meta-learner Training
i {1,
72
R
gdataset
/ E (1, [1,,,, m}
Dataset
D
k {1, ...
Pre-trained language model
gaccel
77L
Accel.
Mj
R hardwaare
Hardware
H
k
Pi,j
Online Acceleration Method Selection
Pre-trained language model
Daraset D new
M arg max Pnew.jA j.k for Mj E M and H2 Et Output the method w/ highest pred. perf. under budget
Figure 2: Metalnf overview ($3.3). The top illustrates offline meta-training ($3.4), where predictor f (shown as @) learns to map embeddings of datasets and models to performance outcomes P. The bottom shows online selection ($3.5), where f is used to predict perfor- mance for new tasks and hardware settings.
A database of historical inference tasks, Dtrain = [DI,..,, .0i,...,Dr. Dr}, defines a broad experience base. Each task corresponds to distinct model input, e.g., prompts for LLMs.
A catalog of hardware configurations, H. = (H,,..., H4,,, captures how different plat- forms influence method effectiveness and cost.
A performance matrix P E RR"xmxh records the metrics of predefined methods M = [M,,.., Mm) across tasks and hardware. Each entry Pij.k quantifies the performance of method Mj on task D; using hardware HІ-
We aim to select an acceleration method M E M for a new task that maximizes performance while satisfying a deployment budget b. The deployment cost hardware H₃ is defined as the product of runtime and hardware cost, which must not exceed b.
Problem (Cost-Constrained) Acceleration Method Selection) Given new task Dnew H and a cost budget b, select an acceleration method M E M such that the cost of using M on hardware Hg for Duew is minimized while not exceeding b.
[Xmew Xnew
lest
Metalnf operates in two phases: (i) offline meta- meta-training on D uain to learn performance map- pings, and (ii) online method selection, where these mappings guide efficient deployment of new tasks. Figure summarizes this workflow, with full details in Sections 3.4 and 3.5.
3.4 Offline Meta-Training
In the offline phase, we generate embeddings that encode dataset characteristics, acceleration methods, and hardware environments. These representations are used to learn a mapping to performance metrics P.
f F. Edata E pmodel phardware P Pijk
(1)
iE n), je (1,..,,,,) m). [E
Embeddings Edata, Emodel and ghardware represent factors influencing the effectiveness of acceleration methods under resource constraints. A regression-based meta-predictor

---

Published as a conference paper at COLM 2025
learns to map these embeddings to predicted performance. While one-hot encodings considered in ablation, we primarily use LLM-derived embeddings, reduced via truncated Single Value Decomposition (SVD), to capture semantic relationships efficiently across tasks and systems.
Dataset Embedding (Edata): Encodes data volume, distribution, and complexity- key factors in method suitability.
Method Embedding (Emodel): Captures computeenergy cost, accuracy trade-offs, and method-specific traits (e.g., prefix caching).
Hardware Embedding (Ehardware):: Encodes processor type, memory, and power constraints, all critical for feasibility.
We train f using historical records spanning datasets, methods, and hardware settings. For this, we adopt XGBoost (Chen & high-dimensional inputs and interpretability via feature importance.
2016) due to its robustness to heterogeneous,
The goal to learn reliable function that maps (Edata, Emodel ghardware) to p-which may include runtime, energy, or accuracy-enabling accurate prediction of the best method for new task Dew-
Objective:
The complete pipeline is shown in Figure 2 and formalized in Algorithm 1.
3.5 Online Model Selection
Given a new task Dew, we compute embeddings for its dataset, the available accelera- tion methods, and the target hardware environment. These embeddings are input to the pretrained neta-performance predictor f, which estimates the performance of each acceler- ation method under current constraints. The optimal method M is selected to maximize predicted performance while satisfying a cost budget b.
M
arg max Pnew.jkr M,E.MGisSh
(2)
P new,jk f(Edete Emodel ghardware)
Cj& represents the cost of running method Mj on hardware Hş and must not exceed the budget b. It is calculated as the product of the cost associated with Hg and the expected runtime of Mj on Hg.
Thus, for each new task defined by Dnew [Xeew X Hnew), embeddings for the dataset (Edata), the applicable model (Emodel), and the hardware environment
(E (Ehardware) are computed. The trained meta-predictor f is then used evaluate the performance of each available acceleration method under the specific hardware settings of Hnew, ensuring that the total cost remains within the budget.
These embeddings are derived from LLM-based textual representations and reduced using SVD (see $4.2). No retraining or empirical evaluation required inference time selecionn is performed in zero-shot manner using the pretrained meta-learner f, enabling rapid, scalable meployment across novel settings.
The algorithm applies constrained maximization strategy to select the highest-performing method under budget b. This balances inference quality, latency, and hardware cost, tailored to the unique properties of each task and deployment environment. Figure 2 (bottom) illustrates this process in the online phase. Algorithm formally describes the selection procedure.

---

Published as a conference paper at COLM 2025
4 Experiments
We assess the accuracy and efficiency of our meta-scheduler across multiple LLM archi- tectures, hardware platforms, and real-world workloads. For each test instance, the meta- learner selects acceleration method conditioned on the model and GPU configuration. We then compare the predicted choice against the empirically optimal configuration, deter- mined via full evaluation of all candidate strategies. Performance metrics include selection accuracy, Fl score, and average acceleration ratio. Full details of the experimental protocol are provided in Appendix A.2.
4.1 Experimental Settings
Model Set. We evaluate four widely adopted generative LLMs: Baichuan2-7B-Chats
Qwen2.5-7B 1-7-----t--nt phi-2, and Meta-Llama-3, 1-8B-Instruct. These models span diverse architectures and training objectives, enabling us to assess acceleration strategies across heterogeneous LLM backbones.
Dataset. We use two representative benchmarks: (1) reasoning-vi -20m, a synthetic dataset for multi-domain and symbolic reasoning; and (2) ShareGPT, a real-world conversational dataset reflecting broad user interaction patterns. This pairing covers both structured and open-ended inference regimes.
Experiments span NVIDIA T4 (16GB), L4 (24GB), and A100 (40GB) GPUs, capturing deployment on both edge and datacenter-class hardware.
Hardware Platform.
Acceleration Methods. We assess three fast inference techniques:
Prefix Caching. Reuses key-value pairs from input prefixes to avoid redundant decoding computation; ideal for long or multi-turn contexts.
Chunked Prefill. Splits the prefill stage into smaller segments to lower memory overhead and reduce initial latency.
Continuous Batching. Dynamically batches incoming requests maximize GPU utiliza- tion without relying on fixed batch sizes.
Baselines. We compare proposed Metalnf scheduler against diverse set of baselines spanning heuristic algorithm selectors and standard machine learning models. Specifically, evaluate:
ISAC (Kerschke et al., 2018), Global Best, ARGOSMART (Agakov et al., 2004), and ALORS (Lindauer & Hutter, 2015), which have been widely adopted in algorithm configuration and meta-scheduling contexts.
Heuristic methods:
Learning-based models: Ridge Regression (Hoerl & Kennard, 1970), Random Forests (Breiman, 2001), Gradient Boosting (Friedman, 2001), LightGBM (Ke et al., 2017), Support Vector Machines (Cortes Vapnik, 1995), and Multi-Layer Perceptrons (MLP).
These baselines balance interpretability and model capacity, supporting a comprehensive evaluation of both inference performance and selection accuracy.
4.2 Data, Model, and Hardware Embeddings
To enable generalization across heterogeneous settings, we use structured embeddings for datasets, models, and hardware. While we conduct ablation studies using one-hot encodings to assess the impact of embedding choice, our default approach employs LLM- generated embeddings that encode rich semantic priors. These embeddings leverage the pretrained knowledge in large models, providing meaningful representations even in shot configurations. To manage the dimensionality these dense representations, we apply truncated SVD, preserving salient information while improving computational efficiency.
Dataset Example: "This dataset has 10,000 000 samples with 20 features for regression. High variability adds fitting di difficulty."
7

---

Published as a conference paper at COLM 2025
Moan Rank of True Best Contiguraton
Method
Acc.
Accel.
10
ISAC
0.578
0.60
1.10
Global Best
0.629
0.63
1.16
10
1.201
ARGOSMART
0.66
0.62
1.17
Moan
16
1.51
1335
ALORS
0.725
0.71
1.20
1773
SVM
0.802
0.79
1.28
2066
MLP
0.783
1.24
24
Random Forest
0.742
0.69
1.25
Mekated Cradived Browting
Pldge Regression
MEP
ALOWS
ARG ARGPOSRARR
LightGBM
0.737
0.68
1.19
Revern
Gental
Ridge Regression
0.784
0.73
1.27
Gradient Boosting
0.815
0.78
1.30
Methods
Metalnf (Ours)
0.898
0.85
1.55
Figure 3: Performance of our Metalnf and base- Table 4: Performance of our Metalnf and line methods on select the berf-onrno ac- baselines methods. Our Metalnf com- celeration combination. This bar plot shows pared with other prediction methods on the average rank of each method when choos- selection accuracy (Acc.), F1 score, and ac- ing the true best configuration across all tasks. celeration ratio (Accel.) The results show Lower values indicate more reliable and con- that Metalnf has the best performance com- sistent performance; Metalnf (ours) ranks first pared to other prediction methods.
with the best performance.
Model Example: "Transformer decoder with 8 billion parameters, instruction-tuned for dialogue and instruction following, optimized for inference."
Hardware Example: "NVIDIA A100 for high-throughput tasks; additional CPU-only machines for lightweight jobs."
4.3 Method Selection Performance
Table presents empirical results comparing our meta-learning--asee method, MetaInf, for selecting fast inference techniques against existing approaches. Our method demonstrably outperforms baseline techniques, achieving significantly higher accuracy in selecting the optimal acceleration scheme. The Average Acceleration Ratio, defined here as the ratio of inference time using the prediction model's selected solution to the average inference time across all solutions, quantifies the inference time and cost savings achieved by each method. MetaInf exhibits superior Average Acceleration Ratio compared to alternative methods, indicating more substantial reductions in inference time and cost. These findings under- score the effectiveness of adaptive selection, implemented in Metalnf, for accelerating inference large language models, offering a pathway towards more efficient and scalable deployment.
4.4 Performance Trade-off
Figure visualizes the trade-off between prediction accuracy and inference time across different methods. Metalnf emerges as the most effective approach, exhibiting the highest accuracy with fast inference time. While other methods prioritize either speed or accuracy, Metalnf effectively balances both, demonstrating its practical advantage in achieving high performance with efficiency.
4.5 Mean rank of the true best configuration
To evaluate the methods' ability to identify optimal configurations, Figure 3 presents the mean rank of the true best configuration. This metric quantifies how well each method ranks the empirically best configuration amongst its predictions. The process involved, for each model-GPU pair, predicting inference durations for all configurations and ranking them by predicted duration. Then, for each pair, we recorded the rank of the actual best
8

---

Published as a conference paper at COLM 2025
Model
GPU
Metalnf Time (s)
Continuous Batching (s)
Time Saved
Mistral-7B
A100
179.41
202.90
-11.6%
Mistral-7B
H200
31.47
38.20
-17.6%
Mixtral-8x7B
A100
215.86
241.92
-10.8
Mixtral-8x7B
H200
39.85
37.42
+6.5%
LLaMA-3.1-70B
A100
895.24
1012.66
-11.6%
LLaMA-3.1-70B
H200
150.52
168.71
-10.8%
Table 5: Evaluation of Metalnf in zero-shot setting on unseen model-GPU pairs. While A100 appears in the training data, H200 does not and was also absent from the embedding model's training corpus. No retrievl-aaummente generation (RAG) was used to avoid latency overhead, preserving real-time deployability. The meta-learner selected acceleration strategies which are compared to baseline continuous batching times. Metalnf shows consistent speedups across most settings, even for large dense models like LLaMA-3.1-70B.
configuration within this predicted ranking. Averaging these ranks, where lower mean rank indicates superior performance in prioritizing truly optimal configurations. The results show Metalnf achieving the lowest mean rank, demonstrating its effective identification of best configurations. These findings coherently underscore Metalnf's enhanced ranking capability, leading to more effective configuration selection for fast inference.
4.6 Ablation Study
To evaluate the impact of prompt engineering on the downstream performance of meta- learning-basee configuration selection, and whether the model is able to utilize knowledge within the LLM, we conduct an ablation study comparing three levels of prompt complexity as well different levels of SVD dimensionality reduction:
Basic Prompt: Minimal keyword-based descriptors, e.g., 'Model: LLaMA-7B''.
Rich Prompt: Structured templates containing architectural and operational context (e.g., compute intensity, model family, memory constraints).
ohain-of-Thoughe (CoT) Prompt: Extended, geasoning-driven prompts that encode performaance-rllevarr dependencies and conditional reasoning.
SVD Dimensionality: We vary the number of latent dimensions per embedded feature between 64, and 256.
Each prompt style is processed by an instruction-tuned LLM (LLaMA-3.2B-Instrruc) to pro- duce dense semantic embeddings for five configuration axes: model name, GPU architecture, and three boolean inference flags.
Figure 4 reports the evaluation results averaged over 1000 random (model, GPU) trials. We observe that higher SVD dimensionality improves accuracy and true improvement up to k = 256, beyond which performance may plateau.
5 Conclusion
In this paper, we introduced novel meta-learning framework designed to optimize in- ference acceleration within decentralized systems. By addressing the unique challenges associated with deploying large-scale models such as L.MSs, our framework significantly enhances the adaptability and effectiveness of acceleration techniques. Our results demon- strate its superiority over traditional acceleration methods. We also note that by simply switching out our optimization goal, we can turn this framework into assessing more attributes in compute resources other than inference time and cost, such voltage and power requirements and thus may have significant impact on our environment as our data centers for AI inference is growing rapidly in today's 's day and age.
The successful application of our framework highlights its potential to facilitate rapid, efficient inference processes, thereby supporting the broader adoption and democratization

---

Published as a conference paper at COLM 2025
Configuration Selection Accuracy (5VD-k=256)
muiddigg Kyp
Prentag Spa
Basic
Basic
(Dunt
649
One-hot
One-hot
130
CoT
Benchmark
075
40
6
Notel
Model
Figure 4: Performance comparison of Metalnf and baseline methods across different embedding schemes. We evaluate model selectors under four embedding schemes-One hot, Basic, Rich, and Chain-of-Thought (CoT)-using configuration selection accuracy (left) and average speedup improvement (right). Metalnf consistently outperforms both heuristic and learning-based baselines across all encoding schemes, demonstrating strong generalization and adaptability under embedding variation.
of advanced Al technologies in decentralized settings. Looking forward, this work sets solid foundation for future research focused on refining and expanding the capabilities of Al systems to meet the increasing demands of real-world applications. We believe that the methodologies developed here will inspire further innovations in the field, particularly in enhancing the operational efficiency of large model infrastructures.
References
Fedor Agakov, Edwin Bonilla, John Cavazos, and et al. Auto-tuning performance-critical library routines by annotation. Proceedings of the ACM/IEEE Conference on Supercomputing, 2004.
Santiago Balseiro, Vahab S. Mirrokni, and Bartek Wydrowski. Load balancing with network latencies via distributed gradient descent, 2025. URL https://arxiv wwtprriiivooog///.//sss// org/abs/ 2504. 10693.
Michele Belotti, Nikola Božić, Guy Pujolle, and Stefano Secci. Vademecum: A decentralized peer-to-peer network for cloud and edge computing. Future Generation Computer Systems, 94:868-878, 2019.
Yahav Biran and Imry Kissos. Adaptive orchestration for large-scale inference on hetero- geneous accelerator systems balancing cost, performance, and resilience, 2025. URL https://arxix org/ 20074.
Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, and Colin A Raffel. Distributed inference and fine-tuning of large language models over the internet. Advances in Neural Information Processing Systems, 36, 2024.
Leo Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.
Tom Brown et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. ar Xiv preprint 2xXi::441.100774 2024.
10

---

Published as a conference paper at COLM 2025
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, PP- 785-794, 2016.
Corinna Cortes and Vladimir Vapnik. Support-vecton networks. Machine Learning, 20(3): 273-297, 1995.
Haotian Dong, Jingyan Jiang, Rongwei Lu, Jiajun Luo, Jiajun Song, Bowen Li, Ying Shen, and Zhi Wang. Beyond single ai cluster: A survey of decentralized I1m training, 2025. URL httpp:://aran org hwtp:/orggg/b//////e/e 11023.
Jerome H Friedman. Greedy function approximation: A gradient boosting machine. Amnals of Statistics, 29(5) 2001.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autore- gressive generative modeling. ar preprint 201::2010.14701, 2020.
Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthog
onal problems. Technometrics, 12(1):55-67, 19700.
Guolin Ke, Qi Meng, Thomas Finley, and et al. Lightgbm: highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems (NeurlPS), pp. 3146-3154, 2017.
Pascal Kerschke, Holger H Hoos, Kevin Leyton-Brown, and Heike Trautmann. Algorithm selection for combinatorial search problems: A survey. arv preprint ar Xrie:1811.04563, 2018.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-626, 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant mod- els with conditional computation and automatic sharding. In Proceedings of the 37th International Conference on Machine Learning (ICML), pp. 5893-5904. PMLR, 2020.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Qinbin Li, Zeyi Wen, Zhaomin Wu, Shuxin Hu, Ning Wang, Yuan Li, Liu, and Bingsheng He. A survey on distributed machine learning. ACM Computing Surveys, 54(2):1-33, 2022.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling
requires rethinking feature uncertainty. arXiv preprint ar 27X1:22401.1507,, 2024.
Marius Lindauer and Frank Hutter. Collaborative filtering for automatic configuration of algorithms. AAAI, 2015.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling trans- former inference. Proceedings of Machine Learning and Systems, 5:606-624, 2023.
Samyam Rajbhandari, Jordan Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. International Conference for High Performance Computing, Networking, Storage and Analysis, 2020.
Aditya Ramesh et al. Hierarchical text-conditional image generation with clip latents. Xiv preprint ar Xrx::22204001 2006., 2022.
11

---

Published as a conference paper at COLM 2025
Shaoduo Ren, Xinyuan Zeng, Shuxiong Liu, Fan Yang, Olatunji Ruwase, Amar Phanishayee Rao, and Yuxiong Zhang. ZeRO-Offload: Democratizing billion-scale model training. arv preprint ar arXiv:2101.06840, 2021.
Tao Shen, Didi Zhu, Ziyu Zhao, Zexi Li, Chao Wu, and Fei Wu. Will Ilms scaling hit the wall? breaking barriers via distributed resources on massive edge devices, 2025. URL https://arxiv. wttp://...rrrabbb/5....222. 08223.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. Xiv preprint Xrix:1999.0000 2019.
Hao Xu, Cheuk Yiu Ho, Ahmed M Abdelmoniem, Aritra Dutta, El Houcine Bergou, Kon- stantinos Karatsenidis, Marco Canini, and Panos Kalnis. Geeps: Scalable deep learning distributed gpus with a gpu-specialized parameter server. In Proceedings of the Fourteenth EuroSys Conference 2019, pp. 1-16, 2019.
Gyeong-In Yu, Joo Seong Jeong, Geon- Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for (Trrasformer-Baseee generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538, 2022.
Ziqian Zeng, Jianwei Wang, Junyao Yang, Zhengdong Lu, Haoran Li, Huiping Zhuang, and inference large language models via
Cen Chen. Privacyrestore: privacy removal and restoration, 2025. URL https://arxiv org/ abs//406. 394..
Mingjin Zhang, Xiaoming Shen, Jiannong Cao, Zeyang Cui, and Shan Jiang. Edgeshard: Efficient Ilm inference via collaborative edge computing. IEEE Internet of Things Journal, 2024.
Zhen-Xing Zhang, Yuan-Bo Wen, Han-Qi Lyu, Chang Liu, Rui Zhang, Xia-Qing Li, Chao Wang, Zi-Dong Du, Qi Guo, Ling Li, et al. Ai computing systems for large language models training. Journal of Computer Science and Technology, 40(1):6 2025.
12

---

Published as a conference paper at COLM 2025
A
Appendix
A.1 Pseudo-code for Meta-train and Online Model Selection
Algorithm 1 Offline Meta-Learner Training for Inference Acceleration
Input: Meta-train train, model set M, hardware set H Output: Meta-learner f for acceleration method selection
1: Train and evaluate M across H. on Dtrain to get performance tensor P
2: for (1,..., do
3: Extract data embedding E (D;))
4: for j E mm) do
5: Encode method set Emooeel M,M, for E hI) do
7:
Encode hardware setup E. E.hardware 0(Hs)
8: Train by Eq. (1) using (Fglata, F.moodel F phardware)) predict Pafk
end for 10: end for
11:
end for
12:
return the meta-learner f
Algorithm 2 Online Acceleration Method Selection
Input: meta-learner f, dataset Xnew, hardware Hnew, budget b Output: Selected acceleration method Xnew
1: Lnew == ((Xnew)
Extract data embedding
2: chardware = ((
Extract hardware embedding
nnww
3: for to do
4: Emodel (M,)
Encode method D Predict performance
5:
P new. == f( (Erleta E.model, Ehardwaree
6:
Gjnew cost cost(Hneww (new) runtime( M,, Hnew)
D Calculate cost
7:
if Cinew 5 b then Consider Mj
8:
9:
else
10:
Exclude Mj
11: end if
12:
end for
13: M
arg max M,E. and Gleew <b
P Pnew.j
14:
return M
b Return method with highest performance within budget
A.2 Additional Experiment Details
The experimental procedure involves using untrained data from the reasoning-vl- -2-m and ShareGPT datasets to generate 1,000 random inputs. These inputs are then evaluated using the pretrained prediction methods. Both the reasoning model and GPU platform are randomly selected, while the prediction method identifies the combination of acceleration strategies expected to result in the shortest reasoning time. Following this, the actual reasoning is performed for all acceleration schemes using the previously randomly selected model and GPU configuration, with the same prompt as input. The fastest reasoning configuration, as verified through empirical testing, used calculate the accuracy of the acceleration scheme selected by the prediction method.
13

---

Published as a conference paper at COLM 2025
Performance Trade-off: Accuracy & Inference Time
0.95
Metalnf (Ours)
0.90
0.85
Gradient Boosting
Selected Accuracy
SVM
0.80
MLP
Ridge Regression
0.75
LightGBM
Random Forest
ALORS
0.70
ARGOSMART
0.65
Global Best
0.60
JSAC
0.55
0.50
0.00
500.00
1000.00
1500.00
2000.00
2500.00
Inference Time per Prediction (s)
Figure 5: Efficienc-eeffettiveenss trade-off of prediction methods. We compare the Efficiency-effectiveeee between the accuracy of the tested prediction methods and the inference time of each prediction to measure the performance of our proposed MetaInf compared to other methods. Metalnf shows highest accuracy and lower inference cost.
14


Reviews:
[{"review_id": "bjBGZbBGyz", "reviewer": "Reviewer_d5hq", "summary": "This paper highlights the challenges in choosing the correct acceleration in decentralized model deployment systems. The paper proposes to resolve these issues with a meta-learning-based framework that can automate the choice of acceleration based on the historical performance data of various acceleration techniques across different tasks", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The approach has been evaluated on multiple LLMs and datasets and also includes an ablation study.\n\n2. The paper has considered performance acceleration tradeoffs, which is appreciated.\n\n3. The figures are well-presented.\n\n4. The design includes theoretical details of the proposed approach.", "reasons_to_reject": "1. The paper does not provide sufficient motivational evidence for using meta-learning for acceleration choice; it seems that it could be resolved using a heuristics-based approach, as clear trends can be observed. For example, Prefix Caching is performing best overall, followed by Continuous Batching and Chunked Prefill for larger batch sizes, and the other way around (Chunked Prefill and Continuous Batching) for smaller batch sizes. So it would be much simpler to develop a heursitics based solution.\n2. The text in the paper seems AI-generated, including too many dashes, which gives an indication of typically AI-generated text.\n3. The paper is not well written. Figures are not discussed in order; Figure 5 is discussed before Figure 3 in the text.\n\nMinor:\n\nIn Tables 1 and 2, \"Chunked\" is written as \"Chucked\".\n\nThe term \"SVD\" is not explained.", "questions_to_authors": "1. Could you provide stronger motivation for using meta learning for acceleration selection instead of a simpler heuristic based solution given clear performance trends among Prefix Caching, Continuous Batching and Chunked Prefill across batch sizes?\n\n2. Can you revise the writing to remove the excessive use of dashes that suggests AI generated text?\n\n3. Can you ensure that figures are discussed in numerical order so that Figure 3 appears in the text before Figure 5?\n\n4. Can you correct the typo in Tables 1 and 2 where “Chunked” is written as “Chucked”?\n\n5. Can you add an explanation of the term SVD when it is first mentioned in the manuscript and add some background for it?\n\n6. Have you considered extending the framework to multi-objective optimization, for example balancing latency against energy consumption?", "extra_scores": {}}, {"review_id": "btAdL5H3BN", "reviewer": "Reviewer_bZwy", "summary": "This paper introduces a meta-learning-based framework for optimizing inference acceleration in decentralized environments, focusing on large-scale models like LLMs. The authors propose MetaInf, a meta-scheduler that selects the most suitable acceleration strategies based on task, model, and hardware profiles. Unlike traditional methods, which rely on heuristics or expert intuition, MetaInf uses historical performance data and learned embeddings to predict optimal acceleration techniques. The proposed method is evaluated against conventional methods, showing its superior performance in terms of efficiency and adaptability across different hardware and task configurations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Good writing and clear motivation: this paper clearly explains the need for adaptive scheduling in decentralized environments, where a one-size-fits-all approach doesn't work.\n\n2. This paper presents thorough experiments comparing MetaInf with various baselines, demonstrating its superior performance across multiple models, hardware platforms, and datasets.", "reasons_to_reject": "1. Limited Real-World Application Details: While the experiments are comprehensive, they primarily focus on synthetic or controlled datasets and setups. The paper could benefit from a deeper exploration of real-world deployment scenarios or edge cases.\n\n2. Computational Overhead: Although the method promises improved efficiency, the process of embedding generation, meta-learning, and online selection could introduce computational overhead, particularly in real-time systems. The paper could discuss potential performance trade-offs in more detail.\n\n3. Lack of discussion of latest works: the Related Work and Introduction sections primarily reference techniques and studies from 2022 and 2023, with limited inclusion of research from 2024. This lack of citation and discussion of the most recent advancements may leave out relevant state-of-the-art developments in inference acceleration and decentralized AI systems.", "questions_to_authors": "1. How does the proposed MetaInf framework handle real-time adaptation to new tasks or hardware when no prior performance data is available?\n\n2. What are the potential limitations of using historical data for selecting inference methods, and how can the system ensure robustness in the absence of such data?\n\n3. Could there be scenarios where the meta-learning approach might fail to select the optimal method, especially when the system is under heavy load or constrained resources?", "extra_scores": {}}, {"review_id": "S5SQqR30ee", "reviewer": "Reviewer_pGCu", "summary": "This work proposes a novel meta-learning framework used for selecting the optimal acceleration method when deploying LLM in decentralized environments. The clear presentation of paper clarifies their method and experiments to demonstrate their method's effectiveness in terms of efficiency and efficacy. The originality and significance of this work is good enough to accept. And for quality of their work, even though they did not provide lots of math proofs, the clear method and effective experimental results have shown the good quality.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Authors provide their method and describe the experimental part clearly and show an effective method to solve the deployment in decentralized environments in terms of performance and efficiency. They solve a practical problem considering both accuracy and efficiency by combining LLM. The whole paper also shows their work's social impact and practical meaning.", "reasons_to_reject": "The only reason to reject is that they only used a pretrained LLM to generate embeddings for thei work, which leads to they did not directly contribute to the development of LLMs: they simply applied them in a straightforward manner.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a meta-learning framework to select the optimal acceleration strategies (e.g., prefix caching and continuous batching) for large AI models in the distributed settings. The proposed framework outperforms multiple baselines, and all reviewers are leaning toward accepting the submission. The authors provided additional experimental results and discussion during the rebuttal, which should be incorporated to the final version of the submission.
Decision: Accept