Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference
Authors: Andrii Skliar
Ties van Rozendaal
Romain Lepert
Todor Boinovski
Mart Van Baalen
Markus Nagel
Paul N. Whatmough
Babak Ehteshami Bejnordi
Venue: Accepted by TMLR ()

Abstract: Mixture of Experts (MoE) LLMs enhance performance by selectively activating specialized subnetworks ("experts") per input. While MoEs offer efficiency benefits through distributed inference in typical high-throughput settings, deploying them on memory-constrained devices remains challenging, particularly for sequential token generation with batch size one. In this work, we optimize MoE for such constrained environments, where only a subset of expert weights fit into DRAM.
Through empirical analysis, we show MoEs can tolerate careful deviations in expert selection with minimal predictive performance loss. Inspired by this observation, we propose a novel cache-aware routing strategy that leverages expert reuse during token generation to significantly improve cache locality.
Evaluating on language modeling, MMLU, and GSM8K benchmarks, our method reduces cache miss rates by over 50%, with negligible impact on perplexity (0.1%–3%) and downstream task accuracy (<0.1%). Unlike prior methods limited by the optimal oracle cache bound, our approach surpasses this theoretical limit by allowing slight flexibility in expert selection. Finally, we present on-device results demonstrating 2$\times$ speedups on mobile hardware, offering a flexible and training-free solution to extend MoE's applicability across real-world applications.

Published in Transactions on Machine Learning Research (06/2025)
Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference
Andrii Skliar *f Conteztual AT
andri@contertuul.ce
Ties Rozendaal Qualcomm AT Research
tor ml
Romain Lepert
romain@@gii www.saggglaggcooommmommmmm
Qualcomm AT Research
Todor Boinovski
todorb @qti. qualcomom.com
Qualcomm AT Research
Mart van Baalen Qualcomm AI Research
Markus Nagel
markusn@qti, wttpbbaswn@ti.qquulcooom.mmm www.couw..ccooooommmomoo
Qualcomm AI Research
Paul Whatmough Qualcomm AT Research
muhhtmmou@@gs..quallmmm..
Babak Ehteshami Bejnordi Qualcoman AT Research"
behtesho wwt.::t..aaaaiooooooooo
Reviewed on OpenReview: https: openreview. net/ forum? id= WWW6KKKKK
Abstract
Mixture of Experts (MoE) LLMs enhance performance by selectively activating specialized subnetworks ("experts") per imput. While MoEs offer efficiency benefits through distributed inference in typical high-throughapu settings, deploying them on menory-constrained devices remains challenging, particularly for sequential token generation with batch size onc. In this work, we oplimize MoE for such constrained environments, where only subset of expert weights fit into DRAM. Through empirical analysis, we show MoEs can tolerate careful deviations in expert selection with minimal predictive performance loss. Inspired by this observation, we propose a novel cache-aware routing strategy that leverages expert reuse during token generation to significantly improve cache locality. Evaluating on language modeling, MMLU, and GSMSK benchmarks, our method reduces cache miss rates by over 50%, with negligible impact perplexity (0.1%-3%) and downstream task accuracy (<0.1%) Unlike prior methods limited by the optimal oracle cache bound, approach surpasses this theoretical limit by allowing slight flexibility in expert selection. Finally, we present on-device results demonstrating 2x speedups on mobile hardware, offering a flexible and training-Fe freee solution to extend MoE's applicability across real-world applications.
"Equal contribution
Work conducted while employed Qualcomm. :Qualcomm AT Research is initiative Qualcomm Technologies, Inc.

---

Published in Transactions on Machine Learning Research (06/2025)
1 Introduction
Mixture of Experts (MoE) models have emerged as powerful approach in large language mod- els (LLMs), offering advantages in scalability and efficiency (Fedus et al. 2022b ap. By selectively activating subset of experts for each input, MoEs handle diverse data distributions and capture complex patterns more effectively than traditional dense models. Recent models such as DeepSeek- V3 (DeepSeek-A) et al. 2024). Qwen2.5-Max (Yang et al. 2024), GPT-4 (OpenAI et al. 2024).
Gemini (Team et al. 2024), and Mixtral (Jiang et al., 2024) provide evidence for MoEs' success in leveraging specialized sub-ns to achieve superior performance. Simultaneously, small language models (SLMs) with MoE architectures, such OLMoB (Muennighoff et al. 2024), Phi-3.5-MoE (Abdin et al. 2024), and Qwen-MoE (Qwen Team 2024), have shown promisc for server deployment due to their ability to maintain high performance with fewer aclive parameters per Loken.
On On-Device Taken Generation Traughput
2.25
Flash
2.00
Brdre mocel parametors
Relative Throughout
F1
F2
F3
F
Ex
Low Bondwicthh Communiczinnic G1 GBIs")
4.50
3.25
CPU
GPU
NPU
XPU
Router
Cache Prto
High Bondwidrh
0.75
Communication 67 Gais
Qwen McE 4bit @12GB device
Owen MoE Shit @16GB device
5.50
cachool ouperta to and vis model
DRAM
Original
Cache Priai
Cache Prof k
Cache Prier bab
Original
Cach Prov
Caches Priar
Cache Prior
paramenens (egg. Rauner enighes ghee
Routing
Aed
Routing
Aeil
Uneashed Experts lrash
Par Farameters
Stado model parmmaters
Overview of our proposed cache-aware routing method. (Left) MoE models are hosted in slower Flash storage due to their size, with only a subset of expert weights cached in faster DRAM. Our cache prior method is cache-aware and adjusts expert selection to promole experts already in DRAM, siguificantly reducing cache misses and improving inference efficiency. (Right) Throughput of our routing method for models deployed on two mobile devices with 12GB and 16GB
Figure 1:
the 4-bit and 8-bit quantized @owel..5-Men available memory and cache size of 30 and 45 experts, respectively. Our proposed Cache-Aware Aware Routing method significantly enhances the token generation throughput compared to the baseline leveraging a Least Recently Used (LRU) Cache. https://enn.... sttpss://an.wikipdddss
Despite their advantages, deploying MoE models on memory-consttrineed devices like smartphones and laptops presents significant challenges. One primary issue is that the parallelism techniques enhancing efficiency server deployments of MoEs are less applicable in on-device scenarios, where tokens are generated sequentially with a batch size of onc. This limitation prevents batching or parallel processing through expert sharding, leading to increased latency and inefficiencies during inference. Additionally, MoE. models have a significantly larger memory footprint than dense architectures, complicating deployment on memory-constrained devices.
These models often exceed available DRAM, and loading model weights directly from flash storage is detrimental to latency. To mitigate this, DRAM can be used as cache for expert weights, but this approach is only effective if the hit rate is high. Idcally, MoE models should have strong cache locality, with a small sot of experts being accessed repeatedly over long token sequences. This requires better strategies for expert selection and cache optimization during inference.
Previous work has tried to improve MoE efficiency by pre-fetching experts based on hidden states from carlier layers and using LRU caching to store recently used experts (Eliseev & Mazur 2023 Yuan et al. 2024a). However, these methods assume that expert selection follows predictable patterns and has a strong locality. Temporal locality, in the context of MoE caching, refers to the principle that recently accessed experts are likely be accessed again subsequent token generations. A key challenge, which we analyze empirically in Section 4.6 is that state-of-the-art MoEs often lack strong temporal locality in expert selection, leading to
2

---

Published in Transactions on Machine Learning Research (06/2025)
inefficient Least Recently Used (LRU) caching strategies due to frequent evictions and reloads, which results in poor cache hit rate.
In this work, we aim to utilize already-trained, high-performing MoE models by incorporating cache locality priors to make them more efficient for memory-limited devices. Our method increases expert reuse during token generation, improving the cache hit rate. This approach is training-free and can be applied directly off tff-the-shelf MoE models, enhancing their efficiency for on-device deployment.
To enable efficient inference memory- constriedd devices, we first analyze the temporal locality in expert selection across four SoTA MoEs. We demonstrate their sensitivity to expert dropping and random expert swapping in Section [2.3] which shows the potential for exploring alternative routing strategies. Motivated by these observations, we propose method that manipulates router logits to increase the probability of selecting cached experts, all without the need for additional training. In summary, this work makes the following contributions:
We demonstrate that state-of-the-art MoE models lack temporal consistency expert selection, which leads suboptimal latency performance when caching strategies are employed. However, our findings indicate that these models exhibit low sensitivity to variations in the selection of lower-weighted experts, suggesting that lossy routing stralegies can be implemented without substantial degradation in overall performance.
We propose method to significantly improve the cache hit rate of experts, enhancing throughput for token generation in existing MoEs without requiring additional training. Our approach strikes a balance between cache hit rate (and token generation latency) and model accuracy under a fixed memory budget.
We report the performance of the proposed routing strategy for language modcling and present evaluation results on MMLU and GSM8K benchmarks. Surprisingly, we show that imposing some degree of routing consistency enhances benchmark scores while making these models more on-device friendly.
We present on-device results of our proposed approach on two different mobile devices with various memory constraints, demonstrating a consistent speed-up of up to 2x compared to the original routing with LRU caching. Our training- free method is adaptable and can be deployed across a variety real-world sconarios and diverse hardware configurations, making suitable for devices with different memory linitations.
2 Background and Motivation
In this section, give overview of MoE models, followed by sensitivity analysis that looks at how expert selection affects model performance, showing the potential for alternative routing strategies explored in the next iontion.
2.1 Preliminaries of MoE
The sparse MoE layer consists of N expert networks E1, Ez,..., EN and a routing network G that selects a subset of experts based on their relevance the input token x Rdd. The output of the MoE layer during inference given by:
y = Σ (x), KEF[:K]
(1)
argsort
(2)
w = (
(3)
Here, z = G(x) represents the logits assigned by the router the experts, is the softmax function, and is a ranking vector of experts based on the weights, from which only the top-K experts are selected. The weights assigned to each expert may be re-normalized after the top-K selection in Equation 1
3

---

Published in Transactions on Machine Learning Research (06/2025)
In conventional MoEs, the top-K routing mechanism for selecting experts is dynamic and input-dependent, which can cause significant variability in the experts chosen for each token. This inconsistency makes it difficult to achieve high cache hit rates, as it reduces the chance that selected experts are already in the cache. As shown in 2 cache lifotimes are short, and cache miss rates are high for all models using a simple LRU caching policy.
2.2 MoE Deployment on Memory-Constrained Constrained Devices
Deploying MoEs on memory- constrycon devices, like mobile phones, is challenging due to limited memory resources. As shown in Figure [ [Left]] these devices typically feature a combination of DRAM and flash storage, each with different speed and capacity characteristics. DRAM offers high bandwidth but limited capacity, while flash storage provides larger capacity at the cost of slower access speeds.
MoE models are typically larger than dense models, therefore, not all parameters may fit into the limited DRAM available on many devices. Static parameters, such as attention weights that do not change during inference, can be stored permanently in DRAM. In contrast, due to the dynamic nature of expert selection, only a subset of experts is loaded into DRAM at a time. To improve efficiency, recently used experts can be retained in a cache, reducing the need to reload them from slower storage. For an effective caching strategy, a high cache hit rate is important because it means experts are quickly available from DRAM, reducing the overall latency. The cache hit rate defined as the proportion of selected experts that are already in the cache at the time of their selection. Let the cache set C denote the indices of experts currently stored in DRAM, and rf: K] be the experts selected by the routing network. The cache hit rate can be expressed as:
hit rate count(r(: K] K]nC) K
(4)
The máss rate is simply hit rate.
The goal of this work is to improve the expert cache hit rate without affecting the model's predictive performance. To achieve this, we propose functions that modify the ranking vector r used to select the top- K experts without making any other changes to the MoE forward pass as shown in [Eqution|]] [ Our objective is to create new ranking vectors that prioritize experts already in cache while minimizing any negative impact on model performance. This allows for a model-agnostic method that can be universally applied.
2.3 MoEs Are Less Sensitive to Dropping Experts with Lower Scores
A key opportunity for making MoE models more fache-friendly without compromising task performance is understanding how sensitive the models are to changes in expert selection. To explore this, we analyze four different MoE architectures: DeepSeek- V2-Lite (DeepSeek-Al Al et al. 2024), Qwen1.5-MoB-A2.78 (Qwen) Team 2024), Phi-3.5-MoF. (Abdin al.) 2024), and Mixtral-8x7B (Jiang et al. (0044). Our results show that these models can tolerate deviations in the expert selection, as long as the experts with the highest weights are retained.
The left plot in F Figure 2 illustrates the impact of completely removing (pruning) experts ranked at or above a specified index on Wikitext perplexity. Removing the second highest ranked expert leads to a performance drop across all four models. However, models with more active experts, like Qwen and DeepSeek, recover quickly, allowing higher-ranked experts to be pruned with minimal performance loss.
To examine how the rank of an expert impacts performance while keeping the total number of active experts constant, we replaced the expert ranked & with randomly selected expert, shown in the right plot of The results show that swapping the top-ranked expert severely compromises model performance. Replacing the sccond-rankod expert also causes noticcable degradation, but the model remains functional. Beyond this, in granular MoEs (with a larger number of smaller active experts), the model becomes highly resilient to expert selection changes. This suggests that while top-1 is critical for all MoEs, the top-2 expert in standard MoEs such as Mixtral and Phi-MoE can be swapped without major loss perplexity. Granular models maintain flexibility in expert selection and incur minimal performance loss by swapping the 3rd highest ranked expert and beyond.
Figure 2
4

---

Published in Transactions on Machine Learning Research (06/2025)
Expert Pruning
Expert Resampling
14
Model
12
Qwen1.5-MoE-A2.777
Wikitext Perplexity
DeepSeek-V2-Lite
6
4
Prune expert with rank
k
Randomly replace expert with rank
Figure 2: Expert sensitivity analysis. We show the effect of dropping or replacing experts as selected by the router. The x-axis represents the expert rank (ordered by their scores) and the y-axis shows the Wikitext validation perplexity (lower values indicate better performance). The dashed lines represent the baseline perplexity of the MoF models.
The left figure illustrates the effect of dropping all experts ranked higher than k, whereas the right figure depicts the impact of randomly replacing the expert at rank k.
using a greedy search to find the optimal expert combination suggests that the router's predictions are often suboptimal. For Mixtral-8x7B, the router's predicted top-2 experts yield the best performance only 28% of the time on average with a maximum of 38% the last layer. This supports the idca that lowor-ranked experts can be replaced with little impact on performance. Taken together, these findings highlight that while preserving the nighest-ranked experts crucial, MoE models offer flexibility in expert selection an insight we leverage throughout this work.
Interestingly, further analysis AAppenixx A
3 Cache-aware Expert Routing
In this section, introduce our main method along with two simple baselines: Max Rank and Cumulative Probability Threshold. These methods, progressively more refined, balance cache hit rate with downstream task performance, improving model throughput and efficiency. All of our methods are general and training-free, making them easily applicable to off-the-shelf MoE models for on-device deployment.
3.1
Max Rank Approach
In /eccionn 2.3 we demonstrated that MoEs exhibit some robustness to the swapping of their experts with random ones. Building on this insight, we aim to deviate from the standard top-K set of experts r[: K], selected by the router and instead promote experts that are already present in the cache C. We first define a general promotion operation that elevates the ranking of all experts in an ordered subset poubset,
promote (rrnoteerssubeet.. subsett x =- subset @ (Iaal \ subbett..
(5)
Here denotes the concalenation operation and the subtraction \ preserves the order of it's left operand. Importantly, all sets in this context are ordered sets and maintain the order of elements in their original ranking. This ensures that the promotion operation is well-defined, as it preserves the relative ordering of experts throughout.
A naive solution would be to simply promote all experts currently the cache, but this approach does not take into account the expert probability assigned by the router. Instead, we limit the promotion of elements in the cache by maximum allowed rank M such that the experts with the lowest router probabilities are not chosen:
5

---

Published in Transactions on Machine Learning Research (06/2025)
max-rank promote(rf: MJ nC;r).
(6)
As observed in Section 2.3 certain top-J experts are crucial for model performance. To ensure these experts not overwritten by others in the cache, always select them, even if they not in the cache. This be achieved by adding second promotion operation:
promote (rl: J); (ro]mtt(r(:M]]]Crr]
Finally, the pscudocode for the max-rank algorithm with always keeping the top-J experts is shown in Additionally, we provide an intuitive explanation of the algorithm with an example in Appendix
A Aggrithmm]
B
Algorithm Max Rank
Require:
max-rank M, minimal-rank J, and original ranking promote(r]: M] C;r)
r' + promote(r|: J):r')
return r'
3.2 Cumulative Probability Threshold Approach
A limitation of the max-rank routing strategy is that does not take into account the distribution of all expert probabilities G(x). For instance, input where the most probable expert has very high probability will likely require very low max-rank M to maintain good model performance. Conversely, for input with uniformly distributed router probabilities, there is no strong expert preference, allowing for very high max-rank M for better cache hit rates.
We address this issue in the cumulative probability threshold approach by dynamically choosing the max-rank M for every layer and every input x. The process conceptually similar the sampling approach by Holtzman et al. (220)) in which tokens are sampled from a dynamic set containing the vast majority of the probability mass. We determine M by summing the sorted probabilities of the router outputs G(x) from highest to lowest until a predefined cumulative probability threshold p is reached:
M mini s.t. Σ G(x)[r,]22
(8)
Using the set of M values found for each layer and each input, we apply the max-rank strategy from Equation to select a new set of experts. We provide an schematic explanation this approach in Appendix [3] The
pseudocode for the cumulative probability threshold approach can be found in Algorith 2 This method is
more dynamic as it accounts for the confidence in the router's predictions. When the set of experts required to reach the threshold is larger, it indicates less confidence from the router (flat distribution), allowing for more flexibility in replacing a non-cached expert with the highest probability expert available within this set that is already in the cache.
3.3
Reranking
Although the cumulative threshold routing strategy dynamically adjusts the max-rank, it still imposes a hard limit beyond which experts are not promoted. This may be suboptimal, especially for router distributions with long tails, which may be better suited for making cache faabe-friendly decisions.
To address this, we use a Cache-Prior that directly manipulates the router logits z = G(x) increase the probability of selecting experts already present in cache. Importantly, the manipulated logits x! only

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm Cunsum Threshold
Require: probability threshold p, minimal-rank J, original ranking r, and expert weights w
1: Peum t
2: M <-0
3: while Poum < do
M + M +
Peum f Poum w[r[M]]
end while
r' < promote(r|: Mo n rr
8: r' + promote(r(: J}:r))
9: return
Model
Params
Experts
Footprint (int4)
Active
Total
Shared
Total
Top-k
Params
min
Mixtral-8x7B
13B
46.7B
8
2
176M
6,5 GB
23,4 GB
Phi-3.5-MoF:
6.6B
41.9B
16
2
79M
3,3 GB
21,0 GB
DeepSeek-Ve V2
2.8B
15.9B
2
(64+2)
(6+2)
8.6M
1,4 GB
8,0 GB
Qwenl.5-MoE
2.7B
14.3B
4
(60+-4)
(4+4)
8.6M
1,4 GB
7,2 GB
Table 1: The MoE architectures used in our experiments. Column "Experts" "Params" indicates the number of parameters per export. Column "Footprint" represents the total size of all model parameters and the expert cache under int4 quantization, with the cache size bounded by k (minimum) and N (maximum).
to find new ranking vector r' defined in Equation 2 We still use the unmodified router logits to compute the expert weights used in Equation
Figure provides an overview of our proposed cache-aware routing method. Let the bitmask E {0, 1 represent the state of the cache following the generation of the (t 1)-th token, where each bit indicates whether an expert in the cache or not.
To ensure the top-J experts are always selected, regardless of their presence in the cache, we can optionally add them the bitmask m,, This results in an updated bitmask, m. Using this updated bitmask, we then boost the logits for the corresponding experts follows:
z' z A Aavg m.:
where X E [o, 1] is a scaling factor that determincs the influence of the cache state on the logits. This scaling lactor allows our method to interpolate between the original routing (A = 0, equivalent to baseline cache hit rate) and fully cache-driven selection (X 1, potentially zero cache misses, but higher perplexity). Davg is defined as:
Aavg
E E (max(z) min(2))] XEX1EE....
(10)
which represents the range in logits for this layer that we estimate using a running average over sequences and tokens. In essence, our Cache-Prior method involves adding fraction of the average logit range to the experts that are already in the cache. By utilizing the true range weights, our Cache-Prior becomes adaptive, allowing single hyperparameter to yield distinct effects across different layers and tokens. We provide an ablation on the choice of A, including approaches for predicting it directly for each token, in
Appendix D and
4 Experiments
In this section, we evaluate the effectiveness of our Cacho-Prior routing method and compare out Pruning, Max-Ranks and Cumulative Sum Thresholding basclines, as described in [Section We implement all methods four state-of-the-art MoF. models: PeepSeck-V2-T.in (DeepSeek-A) et al. 2024 Qwen1.5-

---

Published in Transactions on Machine Learning Research (06/2025)
Expert mixing weights
Next cache state
++1
OOooOoOa
10000000
LRU cache
(-)
Top-k
Updated Ranking Logits
Davg
m
Top-j
00000000 mi
Experts cache
Uncached Experts Experts
Router G(-)
m.
LLM hidden
Binary cache
Figure 3: Our proposed Cache-Prine Prior routing method adds a bias to the logits only for in-cache experts my encouraging their selection. The magnitude of the bias determined by the average logit range, Aavg: and the tradcoff parameter X. The updatod logits, z are used only for re-ranking experts, while the export weights, Wt, remain unchanged.
Qwen1.5-Mee Swenn.5-MoE-A22. 5-M.7-22.77 7B with Cache Size: 30/60 experts
DeepSeek-V2-Lite with Cache Size: 32/64 experts
Paramro fetched from flash token)
Perann fetched from flash IM token!
100
150
200
250
300
50
100
150
200
250
300
350
Routing strategy
Original
8.20
7.30
Pruning
Wilttett Perplexty
0.18
Max-rank
7.25
tiinsum-trreshold Cache Prior
8.16
topj-00
top-j=2
8.12
7.15
8.10
8.08
10
35
Rate (96)
Phi-3,5- 1..5-ooEinnstrcct with Cache Size: 8/16 experts
Pittral-8x7B--O... 1 with Cache Size: 4/8 experts Params from flash taken)
tetched flash token)
Routing stratagy
Original
5.3
Pruning
Wiittxt Perplesity
Max-rank
5.2
Cumsum
Cache- Prior
5.1
top-j-0
top-j=1
5.0
better
10
Rate (56)
Figure 4: Trade-off curves between Wikitext perplexity and cache miss rate for four MoE models with a cache size set to half the total number of experts.

---

Published in Transactions on Machine Learning Research (06/2025)
Qwen1 5-MoE-22. 78 with cache size: 30/60 experts Params tetched from tolken)
DeepSeek-V2-Litee with cache size: 32/64 experts
Params flash (M token) 250
240
280
150
200
300
0.615
betee
0.566
better
0.614
0.564
0.611
0.560
Routing strategy
0.630
0.558
Original
0.609
0.536
breshhl
15.0
17.5
20.0
22.5
25.0
27.5
30.0
32.5
35.0
10
20
Cache Miss Rate (%)
Cache Miss Rate 199
with cache size: 8/16 experts fetched from token)
Mixtral-8x7B-vdl 1 with cache size: 4/8 experts Params fetched flash token)) 3.5
400
500
680
700
800
900
2.5
3.0
4.0
4.5
0.785
better
better
0.708
0.784
0.706
0.782
0.702
0.700
0.698
10
14
16
20
22
20
25
30
35
Cache Miss Rate 1%6
Cache Miss Rate 1%6
Figure 5: The trade-off between MMLU (5 shots) task accuracy and cache miss rate. For each method, points along the curve form the Pareto front, showcasing the best achievable accuracy for a given cache miss rate.
MoF-A2.7B Qwenn Team 2024), Phi-3.5-MoF, (Abdin al. 2024). and Mixtral-8x7B (Jiang et al. 2024). with architectural details summarized in [ablell
4.1 Experimental Setup
We conduct experiments three tasks: language modeling (using the WikiText--2raw-vi dataset), MMLU, and GSM8K. For WikiTeex, report perplexity and cache miss rate, while for MMLU and GSM8K, we report accuracy and cache miss rate. The MMLU dataset consists of multiple-choice questions across 57 subjects, and GSM8K evaluates multi-step reasoning for math problems. For all experiments, the cache miss rate is computed using the Least Recently Used (LRU) eviction policy, unless stated otherwise ablations.
assess the trade-offs between model performance and cache miss rale, we vary DRAM cache size limits and sweep hyperparameters for each method generate Pareto fronts. All reported results in the trade-off plots are point cstimates, obtained from single pass over the entire datasct. As our algorithm and basclines are deterministie, repeated runs yield identical results. Each plot also includes the performance of the "Original Routing" baseline, where no re-ranking strategy is applied, maintaining accuracy without trade-offs. In contrast, our cache-aware routing methods balance accuracy with cache utilization.
4.2 Implementation Details
All routing strategies in this work use the LRU eviction policy for expert removal, where we impose an eviction order by removing experts with higher router weights first, as MoE layers select top- experts parallel. For the pruning baseline, experts ranked 2 h (with h < k) are discarded, and the cache miss rate normalized using for fair comparison with other methods.
9

---

Published in Transactions on Machine Learning Research (06/2025)
Owen1.5-MoE-Ae with ca cache size: 30/60 experts retched flash tokken)
DeepSeek-Vz-Uli with cache size: 32/64 experts
Params fetched taken))
150
200
250
300
350
200
300
0.625
better
better
0.615
0.610
Routing strategy
0.600
0.36
Original
Cursum-treshool
0.595
0.35
Cache-Prion
20
30
10
25
Cache Miss Rate 1%)
Cache Miss Rate 1%)
Phi-3. 1il--..--M----struccc with cache size: 8/16 experts
Mixtral-8x 8x. 78-v0.1 with cache size: 4/8 experts Params tetched from flash 3.0
Params fetched from flash tolken)
40.0
500
800
700
800
900
2.5
3.5
4.0
4.5
better
better
0.60
0.880
0.59
0.875
0.58
0.57
0.865
0.56
10
12
10
20
30
40
Cache Miss Rate 1%)
Cache Miss Rate 1%)
Figure 6: The trade-off between GSM8K (8 shots) task accuracy and cache miss rate. For each method, points along the curve form the Pareto front, showcasing the best achievable accuracy for given cache miss rale.
Each cache-aware routing strategy has a hyperparameter to balance cache miss rate and task performance. We use the following values to generate Pareto curves: Pruning and Max-Rank use K, while Cumulative Sum Thresholding and Cache-Prior use 50 equidistant points in [0, 1]. For guaranteed top-/ loading, we set 1 for Mixtral and Phi-MoE models and J 2 for the granular Qwen-MoE and DeepSeek-Mol architectures.
For dataset preprocessing, concatenate Wexi text into single blob, split by "nn', and chunk it into context lengths of 1024. For MMLU and GSM8K, apply few-shot approach (5 shots for MMLU and 8 shots with chain-of-thought for GSM8K). Our method is applied to the entire sequence for WixiText and MMLU, and only during autoregressive generation for GSM8K.
4.3 Language Modeling Evaluation
We start with assessing the effect of our proposed cache-aware routing method on the language modeling capabilities of MoF. models. Figure 4 presents trade-off curves for perplexity versus cache miss rate of our Cacho-Prior approach compared to the Pruning, Max-Rank, and Cunsum-threshold methods with cache size equal to half the number of experts. The results show that our approach consistently outperforms all baselines across all ranges of perplexity and cache miss rate. Among the baseline methods, Pruning performs the worst, indicating that cache-informed expert replacement is essential for maintaining accuracy. Max-Rank consistently surpasses Pruning, while Humsum-Threshuu consistently outperforms Max-Rank. Ultimately, our Cache-Prior method Pareto-dominates all other approaches.
We also evaluate language modeling performance with only a quarter of experts cachod, as shown Figure 14 in Appendix Our method's robustness different cache sizes makes appealing for deployment across range of devices.
10

---

Published in Transactions on Machine Learning Research (06/2025)
Phi-3.5 MoE instruct (layer 26) with Original Routing
15.0
12.5
10.0
7.5
WM-3.5-MoE-instie (layer 26) with Cache-Prior(A-O.55)
15.0
12.5
10.0
0.0
Cumulative
Rhi-3.5-MoE-instiic loyer Cache-Priarit-0.55
Mni-J.--MoE-insruut layer 26 Original Routing
Cathe Mioi
30
0
1000
1020
1040
1060
1080
1100
Token
Figure 7: Expert selection during token generation GSM8K sample (green: cache hit, red: cache miss, blue: in cache).
4.4 Downstream Tasks Evaluation
Figure and Figure 6 present results on the MMLU and GSM8K benchmarks, respectively. Our Cache- Prior method's Pareto curve consistently outperforms other methods, often showing significant cache miss rate roductions with no accuracy loss compared to original routing. Inereasing the cache bias À improves cache miss rates until minimum reached due to the top-J selection guarantee. We also observe that GSM8K has noisier accuracy results, as can be expected for generative task. Importantly, while accuracy may fluctuate, adjustments to the hyperparamcters for each routing method -namely M, p, and X- consistently leads to predictable effect cache miss rates.
Interestingly, we observe that improved cache consistency (reduced miss rate) can lead to increased accuracy on both benchmarks. We hypothesize this may stem from an implicit regularization effect, where greater temporal consistency in expert selection stabilizes predictions, although other factors likely contribute.
4.5 On-Device Deployment
To evaluate the effectiveness of our cache-aware routing technique in real-world scenarios, we deployed the Owwnl.5-MoF-A2... model with our cache-aware routing two mobile devices (12GB and 16GB RAM equipped with Qualcomm Snapdragon processors running Android 14. Using llama-cpp (Gerganov 2023 for CPU-based deployment, we modified the implementation to add both LRU caching for experts and our Mache-Prion algorithm. Additionally, we enabled memory locking (mlock) to prevent the Android OS from offloading expert weights from memory.
We tested our approach in two settings: (1) deploying a 4-bit quantized model on device with 12GB RAM and (2) deploying an 8-bit quantized model on device with 16GB RAM. In both cases, method was
Snapdragon branded products are products of Qualcomm Technologies, Inc. and subsidiaries.
11

---

Published in Transactions on Machine Learning Research (06/2025)
Qwen1 5 MoE-A2.78
Qwen1.5 Dwen1.5-MoE-A2.77 MoE-A2.78 with cache size: 45/60 experts
1.6
Cache Size
Shart Prompts
30/60 experts
1.4
Long Prompts
45/60 experts
Caching Routing
1.2
Relative Threuphout
LRU
Original
Reletive Throughput
1.4
LRU
+ Cache Prior
1.0
1.3
122
0.4
11
0.2
100
0.0
65
70
75
80
85
90
95
100
Cache Prior (A-0.1)
Cache Prior (A=0.3)
Cache Prior (A=0.5)
Cache (A-0.71
Cache Priar (h=0.9)
Cache Hit Rate
Figure 8: of cache hit rate and prompt length token generation throughput for the 4-bit quantized Owenl.5-MoB-A2..11 model on the 16GB device. (Left) Relationship between cache hit rate and relative throughput (normalized to the LRU bascline of 1.0 for cach cache size) for cache sizes of 30 and 45 experts (out of 60), across varying À values 0.1 - 0.9. (Right) Influence of prompt. length on relative throughput across varying À values, with a cache size of 45 experts.
applied only during autoregressive generation, allowing the model to benefit from optimizations designed for prompt processing.
In the first setting, we reserved 2GB of memory the 12GB device, limiting available memory to 10GB (shared with the OS). Here, the cache size limited to 30 experts per layer (out of 60). In the second setting, the 16GB device hosted the 8-bit quantized Qwen Model with a cache size limit of 45 experts per layer. To determine these optimal cache sizes for each setting, we initially deployed the model on both devices using the LRU caching policy across various cache sizes. Results, shown in Figure 13 indicate that exceeding certain cache size reduces available memory, causing the OS to offload uncached components (e.g., KV-cache, activations) for each token, requiring reloading them from flash which significantly slows down inference. We used the highest LRU throughput from method independently of wall clock variations due implementation specifics.
reference point (1x) demonstrate the speedup
The quantitative results for both settings are presented in Figure (Right) Box plots, computed over 10 runs per experiment, show median, minimum, and maximum values. Our approach provides over 2x speedup on-device. Additionally, qualitative results in TTble 3 in the Appendix demonstrate that this speedup has minimal impact on the quality of generated text.
To quantify the relationship between cache hit rate and token generation throughput, we conducted an experiment, deploying the 4-bit quantized Qwen model the 16GB device and performed sweep over the Cache-Prior trade-off parameter X {0.1, 0.3, 0.5, 0.7, 0.9). A random subset of the MMLU dataset, comprising both short (40-60 tokens) and long (300-400 tokens) prompts, was used for these measurements.
Figure (left) illustrates the relation between cache hit rate and relative token generation throughput for two cache configurations: 30 experts and experts cached per layer (out of 60 experts). For each configuration, the throughput achieved with standard LRU caching and original routing (cquivalent to 0 where no expert swapping occurs) serves as the baseline (relative throughput of 1.0). The results demonstrate a increases, the cache hit rate improves, leading to a corresponding
near-linear positive relationship: increase in throughput. This observed trend is logical as cache-hit rate lincarly correlates with the number of flash reads. As loading from flash is the major performance bottleneck, the number of flash reads should correlate roughly linearly with the throughput.
Furthermore, Figure (right) examines the influence of prompt length on throughput for cache size of experts. Across all values of X, longer input prompts yield higher throughput (Similar trend is observed for the cache size of 30 in Appendix FF. This is likely attributable the inationaaiinn of initial overheads (such
12

---

Published in Transactions on Machine Learning Research (06/2025)
model loading and initial cache fills) over greater number of generated tokens, and potentially more stable expert usage patterns for longer sequences.
4.6 Temporal Consistency of Expert Selections
Figure shows a qualitative visualization of cache hits/mm misses for a GSM8K sample (tokens 1000-1100), comparing original routing and Cache-Prior (A=0.5) methods, with cache miss rates ~3% and ~15% respectively, for Phi-3.5-MoE-instruct It is evident that the Cache-Prior method results fewer cache misses and longer eache lifetimes compared to the original routing baseline.
These observations are further supported by Table 2 which presents average cache miss rates and cache lifetimes (the average number of time steps an expert remains in memory) for the Wikitext dataset. Longer cache lifetimes indicate better memory efficiency.
Using the original routing, each expert in Qwen and DeepSeek models stays memory for only 22 tokens. With our method, experts remain in memory for 9x longer, reducing weight loading from Flash and improving throughput. MoE models with fine-grained implementation (Ludziejewski al. 2024) where the hidden dimension of each expert is smaller than standard foed-forward layer (c.g., Qwen-MoE and Dee)Seek-MoE) are more cache-friendle compared to conventional MoE architectures like Phi-MoE and Mixtral.
8.0
Model
Model
Cache Size
Routing
Lifetime
Miss Rate
Mixtral-8x7B-vo.11
Wikitext Terlexxity
7.5
DeepSeek-V2-Lig
Original
26 (1:11)
35%
Pnl-3.5-MoE-Instruut
Qwenl.
30 / 60
7.0
Cache-Prior
58 (167)
16%
Qwen1.5.MoE-44
6.5
Caching + Routing
DeepSeek-V2
32 / 64
Original
19 (+:77
28%
Optimal + Original
Cache-Prior
76 (1:90
7%
6.0
* LRU + Original
Original
22 (131)
22%
5.5
LRU Cache-Prior
Phi-3.5-MoE
8 16
Cacho-Prior
55 (194)
0%
Mixtral-8x7B
4
Original
(+:)
40%
4.5
better
Cache-Prior
10 (1:00
21%
100M
500M
5B
fetched from (parameters token)
Table 2:
Cache sizes (number of experts that fit in cache / total number of experts in model), average cache lifetimes (in number of tokens), and cache miss rates for LRU and prior caching strategies with 0.5 on the Wikitext validation set.
Figure 9: Tradeoff between language modeling performance and the number of parameters loaded from Flash memory " "Optimal" represents the oracle cache eviction bound.
4.7 Impact of MoE Design Choices on Caching Performance
While retraining MoE models with varying design choices is beyond this paper's scope, our experiments offer insights into how architectural choices in pre-trained MoEs aflect deployment-lime caching. The evaluated MoE models [Table represent diverse architectures, differing in granularity, routing mechanisms, expansion rates, and the use of shared experts.
Granularity Our results suggest that model granularity (Ludziejewski et al., 2024 influences cache efficiency, particularly when combined with our cache-prior reranking method. We observe that granular MoEs (Qwen-MoE, DecpScck-MMo.) tend to be more resilient to the approximate routing introduced by expert swapping compared lo non-granular models like Mixtral. For instance, while Qwen-MoE and Mixtral exhibit comparable initial LRU cache miss rates (35% and 40%, respectively) in Figure 4 our cache-prior method halves Qwen-MoE's miss rate with only 0.5% perplexity increase. Achicving similar miss rate reduction for Mixtral incurs a much higher perplexity penalty (2.9%). DeepScck-Mol MoE demonstrates even greater resilience, halving its baseline miss rate with negligible perplexity increase (.1%))
13

---

Published in Transactions on Machine Learning Research (06/2025)
Expansion Rate The expansion rate, defined the ratio of activated expert parameters to the total number of expert parameters (Ludziejewski et al. 2024), also appears correlated with cache performance. We observe that models with lower expansion rates generally exhibit lower baseline cache miss rates. Specifically, Phi-MoE, Qwen-MoE, and DeepSock-MoE (all with an expansion rate of 0.125) tend to show better cache characteristics under LRU compared to Mixtral (expansion rate of 0.25).
Top-K We observed no clear link between top-K and overall vache-friendling (e.g., Mixtral vs. Phi, both = 2). However, models with smaller & (Mixtral, Phi) are more sensitive to top-1 expert swaps and benefit more from guaranteeing top-1 expert loading (Figure and Figure 4 In contrast, Qwen (k= = 4) and DeepSeck (k= 6) show greater resilience to expert swapping, although their use of shared experts might also contribute to this behavior.
4.8 Ablation Experiments
In this section, we perform ablation experiments on the WikiText dataset to demonstrate the general applicability of our method across a range of cache sizes, compare its performance to the theoretical bounds of the optimal cache eviction policy, and justify our hyperparameter choices.
Optimal Cache Eviction As a thcoretical upper bound for cache policy, we consider Belady's algorithm [Belady 1966), which evicts the expert that will be needed farthest in the future. This lossless policy requires perfect foresight of future router decisions, making it unattainable in practice but useful as an upper bound cache hit rate. LRU.
Figure shows that while Belady's algorithm roughly halves cache miss rates compared to
Conventional caching methods (e.g., LRU, Belady's optimal) are lossless and thus cannot surpass the theoretical cache hit rate bound set by the model's own routing. In contrast, our approach introduces controlled, lossy, reranking of experts, allowing the model to swap lower-ranked experts for in-cache ones, at the cost of a small, tunable increase in perplexity. This design enables us to trade a negligible amount of accuracy for substantial gain in cache efficiency, even surpassing the oracle bound. As can be seen in Figure our Cache-Prion method achieves similar or even lower miss rates with only minimal perplexity increases.
Cache Size We investigate whether our findings hold across different cache sizes, beyond our default of half the total expert count. Figure 10 shows results for cache sizes ranging from one expert to all experts, comparing LRU, optimal caching, and our Cache-Prion method (with an LRU cache). Unlike the first two approaches, Cache-Prior introduces a perplexity trade-off controlled by A. To standardize this comparison, we select the À value for each model and cache size that minimizes cache miss rate while keeping the perplexity increase within specific thresholds (1%, 5%, and 10,, shown in different shades of purple).
As expected, when the cache size equals N, all experts fit in memory, and the miss rate drops to zero for all methods. For the optimal cache baseline, reducing cache size to half the number of experts improves performance over LRU, but the benefit. diminishes cache size shrinks further, becoming negligible for cache sizes < k.
In contrast, our Cache-Prior method consistently enhances performance, particularly at smaller cache capacities. At the extreme of a cache size of one, even the optimal caching strategy yields only marginal improvements. For models with high values of k. our mothod behaves similarly, as cache reuse still results in k-1 cache misses. However, for models with smaller values of k =, 2, such as Mixtral and Phi, Cache-Pris significantly improves cache hit rates, achieving up to 20% reduction in cache misses even at the limit of cache size- = 1.
Remarkably, with just a 1% increase in perplexity, our Cache-Prior method outperforms the theoretical optimal caching bound for all cache sizes and models, except 11-33--MMEEInnsruce When allowing a 5% increase in perplexity, our method surpassos the optimal bound across all models and cache sizes. This demonstrates the effectiveness of Cache-Prior, as can exceed the theoretical optimal caching performance with only a minimal trade-off in downstream task performance.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Qwnn1 1.5MMa MME--2.77 7B
DeepSeek-VZLItt
Cache params)
Cache Size (M params)
200
100
100
100
Caching + Routing
Optimal
-Oiiinal
10% perplexity cosn
Coche (96)
80
80-
LRU
Original
[ (0% perplextty cost;
LRU
+ Cache Prior
[ 11% perplexity cost)
60
LRU
+: Cache-Prtor
5%% perplexity cost)
LRU
+ Cache Prior
(10% perplexity cost)
40
20
20-
0%
81
#
T
Cache Size (num experts total experts)
Cache Size (num experts / total exports)
Mixtral-8xxB--va
Cache Size (8 params)
Cache Size (8 params)
0.0
0.5
1.0
15
2.0
80
80
80
40
20
20-
better
12
Cache Size (num experts Lotaal experts)
(num experts total experts)
Figure 10: Cache size ablation on Wikitext, Cache size of half of the total number of experts N (highlighted in bold) is used throughout the rest of this paper. Cache size of h is highlighted in blue.
Impact of Initial Cache State A potential concern with cache-aware expert routing is that the initial cache content might bias the model to repeatedly select the same experts, thereby limiting diversity and adaptability in expert selection. To investigate this, Figure 18 in Appendix [G] visualizes cache hits/misscs for GSMSK sample at the start of prompt encoding, for the original routing and our Cache-Prior (A - 0.5 and A = 0.8) method. We consider two scenarios: (1) the initial cache is empty and fills experts are selected during processing of the first few tokens, and (2) the cache is initialized with random set of experts. We provide more details on this ablation in Appendix G
Our results show that, for moderate values of X (e-g., 0.5), the initial cache state has minimal long-term impact: after processing a few tokens, the distribution of expert activations and the cache state converge, regardless of initialization. This indicates that the routing mechanism does not remain biased toward the initial cache content, and the method remains robust to different. initial cache states without degrading model diversity or generation performance. In practice, the strength of the bias towards cached experts can be controlled by the parameter X. When using excessively larger X values (e.g., 0.8), the model may become overly reliant on the cached experts, potentially reducing predictive performance.
we first select the top-J experts (J < K) from the router's normal top-K selection, then complement the remaining experts by favoring those already in the cache, ensuring both critical experts and cache-efficient choices are included. Figure shows the effect of varying J on language modeling performance. While bascline methods like Max-rank and Cunsun-lIrreshold experience a significant performance drop without loading the top-J experts, Cache- is more robust to changes in J. The impact on PriceerPror is nedeeddep,,,ent, with minor improvements in perplexity for models like DeepSock, Mixtral, and Phi, and a slight decrcase for models like QWEN. Given that J >0 0 benefits bascline methods and produces adequate results for Mache-Prior, we keep it fixed across all methods.
Varying the Number of top-J Experts As discussed in Seccion 4.2
15

---

Published in Transactions on Machine Learning Research (06/2025)
5 Related Work
Expert Caching Caching is a common optimization in MoE models, given their high memory requirements.
Most prior works rely on a cache with LRU eviction policy (Fliseev & Mazur 2023 Yuan al. 2024b
Yi ct al. 2023 Hwang ct al.
2024
Huang et al. 2023), while others adopt deast-frequently-ueed (LFUU
eviction policy (Xue et al. 2024a
evict experts based on more dynamic and custom measures of expert
importance (Kamahori et al. 2024
Kong et al. 2023
These methods manage expert storage and eviction without changing the model's routing decisions or outputs and thus also maintain identical performance. However, this constraint limits their effectiveness, as they cannot change which experts are selected. To establish a performance upper bound, we include an optimal cache baseline that assumes perfect foresight of future router decisions. While this is unattainable in practice, show that method can surpass this theoretical bound with only minimal increase in perplexity. Unlike convantional caching strategies, our approach introduces controlled trade-off between accuracy and latency, enabling more flexible resource management.
Other lossy caching strategies inchude Adapt-MoE (Zhong et al. 2024), which dynamically adjusts cache sizes per layer, and HOBBIT (Tang et al.] 2024), which decides whether to fully load, skip, or use lower-bitwidth versions of experts based on router confidence. However, both rely on complex prefetching mechanisms, In contrast, our method is much simpler, requiring only a single hyperparameter to balance accuracy and latency effectively.
Speculative Routing and Weight Prefetching Several approaches optimize MoE inference by prefetching expert weights and using speculative routing (Cui et al. 2023 Xue et al. 2024b Eliseev & Mazur 2023 Du et al., 2024). These methods improve cache hit rates by predicting which experts will be needed in advance. For example, MoE-Infinity (Xue al.) 2024bb uses look-ahead routing and caches the most frequently used exports. SiDA-MoE employs an offline-trained hash function to reduce expert selection overhead, though at a significant cost to perplexity. Similarly, Eliseev & Mazur (2023) combines look-ahead routing with LRU caching, requiring the pre-loading of multiple experts well in advance to optimize the overlap between data movement and inference calculations. While this approach helps conceal memory transfer costs, it can adversely affect model accuracy. Speculative routing to improve cache hit-rate without loss of accuracy is particularly challenging on menory-constrainel devices where token generation is more memory-bound than compute-bound In contrast, our method enhances routing consistency by exploiting locality priors, achieving minimal increase in perplexity and significant latency improvements. Our approach is orthogonal to speculative routing methods.
Efficient LLM Inference Techniques such as LLM quantization (Dettmers et al.
2022 Frantar et al.,
2023 Shao et al. 2024) and compression (Frantar & Alistarh 2023 can significantly reduce the memory footprint of large language models (LLMs). Recent studies have explored pruning or merging MoE experts based on their utilization to conserve memory (Chen et. al. 2022 Li et al. 2024 Lu et al. 2024), though this result in decreased performance. To facilitate the deployment of off-the-shelf MoEs on devices with limited memory without model surgery that would change the architecture, previous works have focused on optimizations like offloading parameters to host memory (Alizadeh et al.) 2024 Yu et al.) 2024 Jung
et al. parameters while maximizing the overlap between expert loading and computation. For instance, DeepUM Jung et al. 2023) records CUDA kernel execution patterns to predict which kernel is likely to be executed next. However, MoE model execution patterns can vary significantly depending on the task and context. Our proposed method enhances expert cache hit rates without roquiring complex hardware implementations and remains agnostic to the specific task being addressed.
2023 Rasley et al. 2020). These approaches employ dynamic scheduling strategies to offload sparse
6 Conclusion
In this work, we introduced novel training-free, cache-aware expert routing approach for Mixture of Experts models, designed to improve inference efficiency on memory-constrained hardware. By increasing cache hit.
16

---

Published in Transactions on Machine Learning Research (06/2025)
rates through consistent routing, method significantly reduces latency while preserving model accuracy, achieving up to 2x speedup over standard LRU caching on mobile devices.
Our experiments demonstrate that models remain robust to expert swapping, particularly when the top-1 expert is preserved. This suggests new possibilities for balancing performance with cache efficiency. To explore this, we introduced three routing strategies, with our Cache-Prior method proving particularly effective. Unlike prior work, which is fundamentally constrained by the optimal oracle cache bound, we show that allowing minimal deviations in expert selection enables us to surpass this theoretical limit while maintaining strong downstream task performance.
We conducted comprehensive amalysis across four state-of-the-art models DeepSeek-V2-Litee Qwenl.5- 55- A2.7B, Phi-3.5-MoE, and Mixtral-8x 7B on three lasks: WikiText, MMLU, and GSM8K. Our Cache-Prion method achieves a substantial reduction in cache miss rates across all models, decreasing them by more than 50,, while incurring only a minimal increase in perplexity of 0.1%-3% on the language modcling task and negligible accuracy drop of less than 0.1% on downstream tasks. Additionally, our findings indicate that are inherently more cache-efficient.
granular MoE models, such DeepSeek- V2-Lite and Owenl..-MoB-A2.. This trend suggests a promising future for newer MoE architectures like DeepSeek v3 and Qwen2.5man which, despite being too large for mobile deployment today, could benefit from their cache- frehe-friendly structure.
Our method is highly adaptable, requiring no model retraining, making it suitable for a wide range of hardware configurations. This adaptability reduces deployment costs and facilitates the practical application of MoE models in real-world, resource- limited environments. A key feature of our approach is that it enables explicit control over the trade-off ween cache efficiency and predictive performance via the À parameter. While excessively large values of X can introduce brief, transient bias toward the cache state and may encourage overuse of cached experts, this trade-off is tunable and can be adjusted to suit specifie deployment needs. In practice, moderate values of À provide substantial cache efficiency gains with minimal impact on accuracy.
In summaar, our cache-aware expert routing approach not only enhances inference efficiency but also opens new avenues for deploying large MoE models in resource-limited environments. This work paves the way for more practical and cost-cffective applications of MoE models, ensuring their scalability and adaptability in diverse hardware configurations.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Bchl, Alon Benhaim, and Misha Bilenko et al. Phi-3 technical report: A highly capable language model locally on your phone. arXie preprint ar 2rXiv2404414221, 2024.
Keivan Alizadoh, Seyed-Iman Mirzadeh, Dmitry Belenko, S. Khatamifard, Minsik Cho, Carlo C. del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. LLM in a flash: Efficient large language model inference with limited memory. In Ann-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annuual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 12562-12584. Association for Computational Linguistics, 2024.
Laszlo A. Belady. A study of replacement algorithms for a virtual-storage computer. IBM Systems journal, 5 (2):78-101) 101, 1966.
Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. Task-specific expert pruning for sparse mixture-of-experts. arXiv preprint arXiw:2206.008: 2022.
Weihao Cui, Zhenhua Han, Lingji Ouyang, Yichuan Wang, Ningxin Zheng, Lingxiao Ma, Yuqing Yang, Fan Yang, Jilong Xue, Lili Qiu, Lidong Zhou, Quan Chen, Haisheng Tan, and Minyi Guo. Optimizing dynamic neural networks with brainstorm. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pp. 797-815, Boston, MA, July 2023. USENIX Association. ISBN 978-1-939133--4-22
17

---

Published in Transactions on Machine Learning Research (06/2025)
Dee,Seck-AI, Aixin Liu, Bci Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, and Hao Yang et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. ar preprint arXiv:2405. 04444....4444.. 2024.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318 8-30332, 2022.
Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai Ti, and Yiran Chen. Sida: Sparsity-inspiree data-aware serving for efficient and scalable large mixture-of-experts models. Proceedings of Mochine Learning and Systems, 6:224-238, 2024.
Artyom Eliseev and Denis Mazur. Fast inference of mixture-of-experts language models with oflooding. arXiv preprint 21t:22312.12235 2023.
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning, arXiv preprint arXiv:2209.0100 2022a.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(0))): 39, 2022b.
Elias Frantar and Dan Alistarh. Sparsogpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323 10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference Learning Representations, 2023.
Georgi Gerganov. llama.epp, 2023. URL
cpp. Accessed: 2024-10-31.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference Learning Representations, 2020. URL http:://pperrrriee.ne//ooou id=rygGQyrFvH
Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee. Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference. arXiv preprini Xiv:23033 06182, 2023.
Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang, Pre-gated moe: An algorithm-systens co-design for fast and scalable mixture-of-exper inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 1018-1031. IEFE, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaune Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
Jachoon Jung, Jinpyo Kim, and Jacjin Lee. Deepum: Tensor migration and prefetching in unified memory. In Proceedings of the 28th ACM International Conference on Archilectural Support for Programming Languages and Operating Systems, Volume 2, pp. 207-221, 2023.
Keisuke Kamahori, Yile Gu, Kan Zhu, and Baris Kasikci. Fiddler: Cpu-gpu orchestration for fast inference of mixturo-of-experts models. arXiv preprint arXiv:2402.07033 2024.
Rui Kong, Yuanchun Li, Qingtian Feng, Weijun Wang, Xiaozhou Ye, Ye Ouyang, Linghe Kong, and Yunxin Liu. Swapmoe: Serving off-the-shelf moe-based large language models with tunable memory budget. arXiv preprinl ar 230: 15030, 2023.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Pingzhi Li, Zhenyu Zhang, Prateck Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient SMoe with hints from its routing policy. In The Twelfth International Conference on Learning Representations, 2024.
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture--o-ccperts large language models. Tn Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6159 6172, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. h00/5/3331/222vcchbbeecc
Jan Ludziejowski, Jakub Krajewski, Kamil Adamezewski, Maciej Pióro, Michal Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóżdı, Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. In Forty-first International Conference on Machine Learning, 2024.
Niklas Muennighoff, Luca Soldanii, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. Olmoe: Open mixture-of-experts language models. Xiv preprint arXiv:2409.0000 2024.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkal et al. Gpt-4 technical report. arXiv preprint ar arXiw:2303.0877 2024.
Qwen Qwen Team. Qwen1.5-mo Matching 7b model performance with 1/3 activated parameters, February 2024. URL https:///wenlnn walimm.ggttt hotbub/..ollo//////o http:///oogggoo//eoeoeeeeeeeeee/
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong Hc. Deepspeod: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505 3006-300,-00 2020.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, Pheng-Ann Heng, Chao Li, and Minyi Guo. Hobbit: A mixed precision expert offloading system for fast moe inference. orXie preprint ar arXiv:2411.01433 2024.
Gemini Team, Rohan Anil, Scbastian Borgcaud, Jean-Baptiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, and Andrew M. Dai et al. Gemini: A family of highly capable multimodal models. arXiv preprint 2312 1 11805, 2024.
Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. Moe loo-iitinity: Activation-aware expert offloading for efficient moe serving, arXiv e-prints, pp. arXiv 2401, 2024a.
Leyang Xuc, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. Moc-infinity: Offloading--fficient moe model serving. arXiv preprint arXiv:2401. 14861, 2024b.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXio preprint 2r 12115 2024.
Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. Edgemoe: Fast on-device inference of moe-based large language models. arXiv preprint arXiv: 143522 2023.
Dianhai Yu, Liang Shen, Hongxiang Hao, Weibao Gong, Huachao Wu, Jiang Bian, Lirong Dai, and Haoyi Xiong. Moesys: A distributed and efficient mixture-of-expprts training and inference system for internet services. IFF.F Transactions on Services Computing, 2024.
19

---

Published in Transactions on Machine Learning Research (06/2025)
Xiaoming Yuan, Weixuan Kong, Zhenyu Luo, and Minrui Xu. Efficient inference offloading for mixture-of-
experts large language models in internet of medical things. Electronics, 13(11), 2024a. ISSN 2079-9292. doi:
Xiaoming Yuan, Weixuan Kong, Zhenyu Luo, and Minrui Xu. Efficient inference offloading for mixture-of- experts large language models in internet of medical things. Electronics, 13(11):2077, 2024b.
Shuzhang Zhong, Ling Liang, Yuan Wang, Runsheng Wang, Ru Huang, and Meng Li. Adapmoe: Adaptive sensitivity-bast based expert gating and management for efficient moe inference. arx preprint arX2v:4408.1088, 2024.
20

---

Published in Transactions on Machine Learning Research (06/2025)
Appendix
A
Optimal Expert Selection
To investigate the accuracy of the router's expert predictions, conduct experiment Wikitext using Mixtral-8x7B-vo.11 Iterating over layers, we fix the top-1 expert and systematically test all possible choices for the second expert, finding the one that minimizes perplexity. The results, shown in Figure 11 reveal that the router selects the optimal top-2 expert only 28% of the time on average. Although deeper layers improve this accuracy, the best-performing router only correctly predicts the top-2 expert in just 38% of cases. This suggests that while selecting the nighest-ranked experts important, router predictions are often suboptimal, leaving room for flexibility in expert selection without significantly impacting performance.
bY
ww..ROOO mar
Parx Pww.C
Figure 11: Agroement between router predictions and optimal expert selection across different layers of Mixtral-Sx7B--v.11 The y-axis represents the frequency with which a predicted expert was actually the optimal choice based on Wikitext perplexity.
B Intuitive Explanation of the Max Rank and Cumulative Sum Threshold Baselines
To provide further intuition, the Max Rank approach can be understood follows. Rather than always selecting the top-K K experts with the highest router probabilities, experts, specifically, those ranked in the top M by the router. Among these, we give priority to experts that are already present in the cache, promoting them to the front of the ranking. This increases the likelihood that the selected experts are already loaded in fast memory, reducing cache misses and improving inference efficiency. However, to avoid degrading model performance, we do not promote cached experts that fall outside the top M router ranks, ensuring that only reasonably probable experts are considered. In effect, this method balances the tradc-off between cache efficiency and predictive accuracy by flexibly reordering the selection within controlled rank window.
consider slightly larger pool of candidate
Top-2
MiDen
Top-2
Top-2
3
4
6
0.4
Router probabilities
0.2
0.2
0
Experts in Cache Uncached experts
Experts
Exports
3 4
6
top-J Expert
Peaky distribution (M 3)
Flat distribution (M 5)
Cache state
(a) Max-Rank Approach
(b) Cumulative Sum Threshold
Figure 12: Illustration of the Max Rank and Cumulative Sum Threshold Baselines.
To provide an example, suppose the router produces a ranked list of experts r = [E1. E2, Es, Es, Es. Esl, and the cache currently contains C = (E E4, Ea} (See Figure 12 for an illustration) Let M = 4, K = 2, and J = 1. The top M experts are [E, E2, Eg, Et). The intersection with the cache yields [Eg, Es]. Applying the promotion operation, we reorder the list to obtain [Eg, E,, E,, Egg, Es, Egj. To ensure the top-J is always included, we move them to the front in order while preserving the rest [E1, Es, Es, Ez, Es, Eo). The top-K experts selected for this token would then be Eg and Eg.
21

---

Published in Transactions on Machine Learning Research (06/2025)
Dr-device performance of LRU at differant Cache $285
Life
Relstive Throughhus
Owen-MoE-48 @12GB device
O-enn-MoE-dl
@166B device
15
50
45
15
30
Number Experts Cached in DRAM
Figure 13: Throughput of the 4-bit and 8-bit quantized Qwen1.5-MoE models with LRU caching deployed on two mobile devices with 12GB and 16GB available memory, respectively.
The procedure for expert swapping in the Cumulative Sum Threshold approach is based on the same principle, however, the maximum rank M is dynamically determined by thresholding the cumulative sum of sorted router probabilities. As can be seen in Figure 12 for flat distributions, M is larger and for peaky distributions, M takes smaller values.
C On-Device LRU Throughput
In Figure 13 we show on-device performance for various cache sizes. Similar to Section 4.5) we use best. LRU performance as a reference point of 1x. As can be seen, increasing cache size beyond 30 in case of 4-bit quantized model on 12GB device (left) and beyond 45 case of 8-bit quantized model on 16GB device (right) leads to decrease in performance. This is due to the fact that increasing cache size reduces available memory, causing the OS to offload uncached components (e.g., KV-cache, activations) for each token, requiring reloading them from flash which significantly slows down inference.
Given these results, we use same cache size of 30 for 4-bit and 45 for 8-bit quantized models in experiments
in Section 4.5
D Smaller Cache Sizes
Throughout all the experiments in the papor, we use a cache size that is 1 times the total number experts. In Figure 14 we demonstrate performance of our approach for models with cache size equal to 1 of the total amount. Our Cache-Prios Prior method is outperforming all other methods across the board. Note that it is doing without any changes to method itself making it appealing for final user who might want to deploy it across
range of devices. An aggregated overview of additional cache sizes can be found in Figure 10 in the main text.
Logit Range Estimation In our method, the scaling factor À is multiplied by a running average of the logits range Aavg, as shown in Equation 9 This estimate of the logit range is dynamic, depending on model and input. To improve the estimate, we could alternatively calculate Agvg over a calibration dataset. We tested this by estimating the range over the entire Wikitext training subset. Results Figure [5 show that our running average performs comparably estimates based on the full dataset, while also offering robustness to out-of-domain data where logit ranges may differ from general text data (e.g., for coding task).
22

---

Published in Transactions on Machine Learning Research (06/2025)
7B with Cache Size: 15/60 experts
DeepSeek-V2-Lite with Cache Size: 16/64 experts
Params fetched from flash (M (tkeen) 300
Paramns fetched from flash (M token!
250
350
400
450
200
300
400
500
600
8.275
7.50
Routing strategy
8.250
7.45
Original
Pruning
0.225
7.40
Max-Fank
8.200
1.30
Cache Prior
8.175
7.30
7.25
7.20
9.100
7.15
better
30
35
45
55
20
30
35
45
50
Cache
Cache Rate
Phi-3.5-Mott
4/16 experts
Mixtral--B77B-O0 Cache experts Params fetched from flash (B taken)
Fa rams fetched from flash iB taken!
1.2
4,0
5.0
6.0
7,0
4.90
Perextteeploes Perlodty
4.85
5.2
5.1
4.70
5.0
4.65
20
25
35
45
30
35
40
Cache Miss Rate (56)
Cache Miss Rate 0%)
Figure 14: Wikitext Perplexity results for cache size that is one quarter the amount of experts.
DeepSeek-V2 V2-Lite with Cache Size: 32 64 experts
DeepSeck-V2-Lite with Cache Size: / experts
Estimator
Routing Strategy
Precomputed Average Running Average
7.275
Cache-Prior Prior
7.26
Wikttext Perplexit
Wiktext Perplexity
7.24
7.225
7.22
7.20
7.200
7.18
7.175
7.12
7.125
10
20
25
10
20
25
Cache (%))
Cache miss rate (%)
Figure 15: Effect of different strategies for estimating the logit range.
Figure 16: Performance of our learned Cache- Prior strategy Wikitext.
E Learned Cache-Prio
In Sectionn 3.3
present our training-free Cache-Award routing method. As shown throughout the paper, the presented method is very strong and competitive across range of scenarios.
However, it is also possible to make the proposed method learnable. To do so, we propose learning the additive cache-prior term using a small cache MLP layer conditioned on the cache state. Specifically, we implement this by employing two-layer MLP that takes both the cache state and router logits as input, outputting single bias vector. This bias is then added to the router logits and used for the next step. The loss function is optimized the softmax outputs. We force weights assigned to experts that in the cache
23

---

Published in Transactions on Machine Learning Research (06/2025)
but not in top-K K to move closer to one while penalizing weights for those that are in top-K but not in the cache to move closer to zero.
A downside of learnable cache-priors is that we are required to train many independent cache MLPs to obtain trade-off points between cache hit rate and model accuracy. We conducted an experiment to verify the effectiveness of learning this bias but did not observe notable improvements. This optimization challenging and the results were close to the cache-prior performance without outperforming it. Moreover, this approach would be less beneficial in settings with multiple deployment targets. Therefore, the training- free nature of our approach proves very advantageous for real-world applications.
Given the increased training expense, comparable performance and limited applicability, we skip this approach for now, leaving it for fulure work.
F Impact of Prompt Length on Throughput
F Figure 17
shows the influence of prompt length (short: 40-60 tokens; long: 300-400 tokens) on throughput for
the Qwen Qwen1.5-MoE-A2. 15-MMoE-22.7 model deployed with a cache size of 30 experts. Across nearly all values X, longer input prompts yield higher throughput.
Qwen1.5-MoE-A2.77 with cache size: 30/60 experts
1.6
Short Prompts
Long Prompts
1.4
Relative Throughput
1.2
1.0
0.8
0.6
0.4
0.2
0.0
Cache Prior (X=0.1)
Cache Prior (A=0.3)
Cache Prior (A=0.5)
Cache Prior (A=0.7)
Cache Prior (A=0.9)
Figure 17: Influence of prompt length on relative throughput across varying À values, for the Qwen1.5-Me 5-MoE- A2.7 running on the 16GB device with a cache size of 30 experts.
G Impact of Initial Cache State
In this section, we qualitatively assess how the initial cache state affects expert selection. Figure 18 visualizes cache hits and misses for GSM8K sample during the prompt encoding phase for the Phi-3.5-MoE-instruee model. We compare both the original routing and Cache-Prior methods for cache size of 8 experts across four scenarios: (1) original routing with an empty cache, (2) Cache-Prion (A = 0.5) with an empty cache, (3) Cache-Prion (A 0.5) with randomly initialized cache, and (4) Cache-Prior (A = 0.8) with the same random cache state as (3). This setup allows us to examine the influence of both the initial cache content and the À parameter on expert selection. In all these setting, we set J 1, for guaranteed top-1 expert loading.
Our results show that, for moderately large values of (e.g., 0.5), the initial cache state has minimal long-term effect: after processing few tokens, the distribution of expert activations and the cache state converge, regardless of how the cache was initialized. This suggests that the routing mechanism is not persistently biased toward the initial cache content. However, with excessively larger (e.g., 0.8), we observe that the model lends to overly reuse experts in the cache, which may negatively impact predictive performance.
24

---

Published in Transactions on Machine Learning Research (06/2025)
Phi-3.5-M 1...5--M--iinstuce (layer 26) with Original Routing
15
Expert
10
5
0
Phi-3.5-MoE MoE-instruct (layer 26) with Cache-Prior (A=0.5)
15
Expert
10
5
0
Inis..tt (layer with Cache-Prior (A=0.5) Random Cache Init
15
Expert
10
5
Phi-3.5-MoE-instruct (layer with Cache-Prior (A=0.8) Random Cache Init
15
Expert
10
5
0
Cumulative Cache Misses
40
Phi-J5-MoE-instrrutt (layer (A=0.8) Random
Phi-3 5-MoE-insstr (layer 26) Cache-Prior 0.5) Random Cache Init
Cache Misses
30
Ini-355-MaF-instuur (layer (A-0.5)
Phi-3 M-Mo--insstt (layer 26) Original Routing
0
60
100
Taken
Figure 18: Expert selection during token generation GSM8K sample (green: cache hit, red: cache miss, blue: in cache).
25

---

Published in Transactions on Machine Learning Research (06/2025)
H Qualitative On-Device Results
presents sample text from our on-device autoregressive text generation experiment. It is evident that incorporating our Cache-Prior method has minimal impact on the quality of the generated results. For more details, refer to Section 4.5 of the main text.
Table 3
Model
Q4, Cache Size = 45
Q8, Cache Size = 30
Prompt
Write short poem inspired by "Numb" by Linkin Park but about Mixture-of-Expee Expert models.
Building website can be done in simple steps: Step 1:
LRU
The model is type of artificial intelligence that uses multiple machine learning algorithms to make predictions decisions. It's designed to make better decisions faster than expert, but it also make mistakes due to the complexity of combining the decisions of different algo-
Get website domain name. This is the address where your website will live. You can choose
that describes your business or brand. It's best to choose name that's 's casy for people remember and spell. Step Find web hosting service. A web hosting service where your website files will be stored. There many different types web hosting services to choose from, including shared hosting, VPS hosting, and dedicated hosting, You'll want to choose hosting
In the realm of data and code, Lies model so bold and brave, A mixture experts, Working together, making decisions, it thrives.
name,
rithms.
With algorithms galore, Predicting and deciding, with speed and force, Majestic in its complexity, A marvel of modern science.
it's sight to behold,
The model prodicts the probability distribution of the next word (y_m) n) as a function of the previous word (y (n-1)) and the hidden state (h {n-1})
Get a website name (the web address)) You need to choose a domain name (or URL) for your website, and register it with a domain registrar. This is the address that people will use to find your website on the internet, such aswww. yourdoourrmmmn.com
(Verse 1)
In the depths of data streamss a secret blend of math, Where words and words and words are churned into a model, The Mixture-of of-urr--f-Expprrt Experts reigns, a wiser way to guess, The next word to fill the verse, the rhythm of the next verse.
Prior,
0.2
Get web hosting for your website. This is the place where your website will live on the internet, and where yourwebsite files will be stored. The most popular option is to use hosting company,
Step 2:
(Chorus)
Numb to the noise, numb to the noise, Guided by an algorithm, the words flow like guide, Numb to the noise, numb to the noise, Predicting the future with pattern we can't annoy.
Prior,
The first part of the English language Wikipedia choose Mixture-of-Expert models as the 2018 word. Mixture-of- Expert models use multiple neural networks expert solutions. I'm a poet, but here's my attempt at a very louse and free poetry style translation of the given meaning.
Find Domain Name Before you start building your website, you will need to choose domain name (the name of your website). Your domain name the address that people will use to reach your website, so you want to make sure it's easy to remember and spell and related to what your website about. For example, if you're building a website for your dog grooming business, your domain name could be.
À = 0.8
A model so complex, Experts gather from near and far The experts of yore, to bring their expertise, their might A model to optimize, to help us all survive From AT to the rescue, it's transformation we'll rejoice In the mind of Chester and Patel, Expert insights to bestow, Insights to empower.
Step 2:
Choose a Hosting Provider Now that you have your
Table 3: Qualitative results of caching performance. We generate 100 tokens for each prompt.
26


Reviews:
[{"review_id": "jQffXEOGnz", "reviewer": "Reviewer_SKhL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed algorithm is simple and intuitive.\n- Extensive experiments are conducted across a wide range of state-of-the-art MoE-based LLM architectures and tasks. The results comprehensively show the effectiveness and robustness of the method in reducing cache miss rates.\n\nWeakness:\n- The writing quality needs significant improvement. Many sentences are unclear, and the overall flow could be much more polished.\n\n- Formatting issues, such as improper citation styles, suggest a lack of careful proofreading, raising concerns about the overall attention to detail.\n\n- Equation 5 lacks clarity. Since sets are inherently unordered, the operation described appears to be a no-op. The authors should consider representing the experts as a list or another ordered data structure to make the operation meaningful.\n\n- Section 3.2, as I understand correctly, is just the top-p sampling from [1], a widely known and commonly used method in LLM-related work. This section could be greatly simplified or merged into the background.\n\n- The text in Figure 3 is too small and difficult to read. Furthermore, the meaning of the different colors is not clearly explained.\n\n- Figures 5 and 6 lack sufficient explanation. In particular, it is unclear why some data points are not connected. More context and description should be provided to help the reader interpret the results.\n\n[1] Holtzman, Ari, et al. \"The curious case of neural text degeneration.\" ICLR 2020", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "MoE models have recently become highly powerful, but their growing parameter counts make deployment under memory-constrained settings challenging. Cache-based methods have emerged as a mitigation strategy. In this paper, the authors observe that modifying an expert selection when the change does not involve the highest-probability expert has a smaller impact on model performance compared to pruning experts outright. Based on this insight, they propose using a cache prior to adjust expert weights, encouraging selection of experts already present in the cache. Experimental results demonstrate that this approach can significantly reduce the cache miss rate with only a small degradation in perplexity or accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The placement of Figure 1 can be optimized. Currently, it appears on Page 1, while the corresponding explanation for the left part of the figure only appears on Page 3 (left)/Page 10 (right). Additionally, the different colors and arrows used in the left panel of Figure 1 are not explained at all. The authors should improve the figure's placement and make Figure 1 more self-explanatory, including a clear legend or caption describing the color scheme.\n\n- The description of Figure 2 is unclear. The figure caption should explicitly state that the dashed line represents the baseline.\n\n- Citation format needs to be carefully checked. E.g. Page 2 DeepSeek V3, OLMoE, etc. Please change the \\cite command into \\citep.\n\n- The authors state: \"A key issue is that state-of-the-art MoEs lack temporal locality in expert selection, leading to inefficient LRU caching due to frequent evictions and reloads, which results in a poor cache hit rate.\"\nHowever, \"temporal locality\" is not formally defined in the paper. Moreover, several existing works have noted similarities in expert assignment across sequential decoding steps. The authors should either provide experimental evidence to support this claim or cite appropriate references.\n\n- Please refer to the Weaknesses section of this review for additional specific comments regarding figure readability, equation clarity, and text simplification.\n\n- *The authors are encouraged to open-source the implementation of the proposed algorithm.*", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EMsNf0bI2V", "reviewer": "Reviewer_8bhs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The paper studies MoE deployments in systems with constrained resources (memory and perhaps compute)\n- The observation that dropping/pruning experts with lower scores do not affect the performance in 4 popular MoE models\n- The paper proposes sensible baselines in addition to a new algorithm for cache-prior reranking method\n\n# Weaknesses\n\n- The paper lacks a systematic study of training hyperparameter choices that can have an influence over the proposed cache-prior reranking method. The use of open source models is a good start but the reader (like this reviewer) may wonder what choices can be made during training that can assist with producing deployment-friendly models. Choices like ```top_k```, ```granularity``` would be interesting to study but understandable that its out of scope for this paper.\n\n- Related to above, the reader may wonder whether the findings in the paper can extend to newly developed and/or released MoEs. Is there a limit to the scale of the model beyond which the cache-prior reranking might not work?\n\n- The proposed method, cache-prior reranking, is completely heuristic. The paper provides no intuition on why this method should be better than a baseline that seemingly provides an upper bound for caching.\n\n- The notation used to describe the baselines (Equations 5 - 7) are somewhat confusing to this reader. Could the authors consider adding more text or an example in the appendix to help the reader mechanically understand how proposed baselines and algorithms work?\n\n- There are certain claims in the paper that needs more explanation or clarification. Please check requested changes section for more information", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies approaches to improve the efficiency of Mixture-of-Experts (MoE) deployment on memory-constrained devices, specifically for token generation with batch size equal to 1. The paper observes that the experts beyond top-1 or top-2 maybe replaced by other experts with minimal drop in performance. Taking inspiration from this observation, the paper suggests a new method and baselines for cache-aware experts reranking. Experimental evidence is then presented to show reduced cache-misses by using the proposed cache-prior reranking algorithm.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Claims that require clarification\n  - Page 4, Paragraph 3, final sentence - Is Top-1 critical or Top-2 critical to maintain performance. Looking at Figure 2, it appears that Top-2 is necessary to retain (experts not be replaced) failing which perplexity gets worse for DeepSeek. Please make the argument precise.\n\n  - Page 6, Paragraph that follows equation 8 has a discussion on uncertainty. What is meant by uncertainty here? Does the use of the term here mean that if a router outputs all logits with same value (effectively uniform distribution)  then it has highest uncertainty? The statement appears to be qualitative and somewhat confusing as uncertainty can have  multiple meanings in language models (ML models in general)\n\n  - Page 9: the claim on improved cache consistency leading to better performance is attributed to implicit regularization. This explanation needs to be clarified as the improved score could very well be due to an entirely different (and unknown) reason\n\n- Doe the points shown in Figures have any uncertainty information available? In other words, are these point estimates or averaged over multiple trials etc?\n\n- There is a mention of cache-miss in text for Figure 8. I do not see any metric that can help me see cache miss in Figure 8 in the draft available for review. Please clarify what I might be missing here\n\n- Suggestion to aid with paper completeness - please provide a description of the 4 models used in empirical analysis including training details like top_k, granularity etc (as applicable) in the appendix", "broader_impact_concerns": "Nothing specific concerns noted.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RM8GBgx9A7", "reviewer": "Reviewer_XijN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The paper identifies an interesting trend where less important experts do not affect the performance very much, giving opportunity for expert pruning. \n2. The method is cost effecitive and does not require significant architecture change or training.\n3. The performance results look promising for both cache hit rate and algorithm accuracy.  \n4. Good ablations on different settings and optimal solution.\n\nWeaknesses:\n\n1. How significant is cache hit rate to the latency of the model serving? Is it the bottleneck?\n2. Will the selection of experts in cache cause bias in the output? Will the same set of cache being always selected? How is the performance affected by the initial set of experts in the cache? In short, what are the limitations of the current method?\n3. Lack of experiment results on the materialized benefits of increased cache hit rate, like latency and throughputs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a cache-aware efficient MoE algorithm that reduces cache miss rate. The authors demonstrate that the models' performance exhibits low sensitivity to the selection of low-weighted experts, and build a hardware-aware algorithm on top of that. They evaluate the method and prove its efficiency on various benchmarks. They also test it on mobile devices, showing the flexibility and cost-effectiveness of the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Profiling results on how much the cache hit rate affects the latency. \n2. Discuss the possible negative effects of performing cache-aware expert selection. \n3. More results on latency or throughputs improvement.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an algorithm that aims to make routing more consistent by exploiting the fact that MoEs can remain effective even if the "default" top-k routing decisions are perturbed. By making routing more consistent, MoEs can be made more applicable and efficient in resource-constrained settings, since repeated loads/swapping of active experts can be avoided. The paper evaluates on a few standard LLM benchmarks and presents compelling efficiency gains with minimal performance degradation. Reviewers all were unanimous in recommending acceptance. One reviewer suggested considering open-sourcing the algorithm to support future research.
Decision: Accept (poster)