FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Authors: Yuxin Zhou
zheng li
Jun Zhang
Jue WANG
Yiping Wang
Zhongle Xie
Ke Chen
Lidan Shou
Venue: ICML 2025 poster (2025)
Keywords: Mixture-of-Experts
Efficient Inference
Model Compression
Experts Offloading
Primary area: deep_learning->large_language_models
TL;DR: An on-the-fly MoE inference system on memory-constrained GPU, founded on the insight that substantial untapped redundancy exists within sparsely activated experts.

Abstract: With the widespread adoption of Mixture-of-Experts (MoE) models, there is a growing demand for efficient inference on memory-constrained devices.
While offloading expert parameters to CPU memory and loading activated experts on demand has emerged as a potential solution, the large size of activated experts overburdens the limited PCIe bandwidth, hindering the effectiveness in latency-sensitive scenarios.
To mitigate this, we propose FloE, an on-the-fly MoE inference system on memory-constrained GPUs.
FloE  is built on the insight that there exists substantial untapped redundancy within sparsely activated experts.
It employs various compression techniques on the expert's internal parameter matrices to reduce the data movement load, combined with low-cost sparse prediction, achieving perceptible inference acceleration in wall-clock time on resource-constrained devices.
Empirically, FloE  achieves a 9.3$\times$ compression of parameters per expert in Mixtral-8$\times$7B; enables deployment on a GPU with only 11GB VRAM, reducing the memory footprint by up to 8.5$\times$; and delivers a 48.7$\times$ inference speedup compared to DeepSpeed-MII on a single GeForce RTX 3090—all with only a 4.4\% $\sim$ 7.6\% average performance degradation.

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Yuxin Zhou 12 Zheng MiengLi"! Li Jun Zhang' 12 Jue Wang" Lidan Shou
Yiping Wang Zhongle Xie 12 Ke Chen
Abstract
chitecture by introducing sparsely activated experts. These sparse LLMs contextually activate only a subset of experts per token, significantly reducing inference costs while main- taining generative performance. However, the abundance of idle, non-activated experts during MoE inference signifi- cantly hampers efficient GPU memory utilization, making it challenging to deploy MoE models on memory-consttained GPUs. For instance, running inference for Mixtral-8 7B, where two experts are activated, requires approximately 94GB of VRAM FP16 6 precision. Of this, 30% of the activated parameters (27.3GB) utilized during decoding, while the remaining 66.8GB is occupied by non-activated ex- perts, resulting in significant inefficiency (Shin et al., 2024).
With the widespread adoption of Mixture-of- Experts (MoH) models, there is growing demand for efficient inference on memory-constrained devices. While offloading expert parameters to CPU memory and loading activated experts on demand has emerged as potential solution, the large size of activated experts overburdens the lim- ited PCIe bandwidth, hindering the effectiveness in latency-sensitive scenarios. To mitigate this, we propose FloE, an on-the-fly MoE inference system on mentoyyconstrained GPUs. FloE is built on the insight that there exists substantial untapped redundancy within sparsely activated experts. employs various compression tech- niques on the expert's internal parameter matri- ces to reduce the data movement load, combined with low-cost sparse prediction, achieving percep- tible inference acceleration in wall-clock time on resource-constraine devices. Empirically, FloE achieves a 9.3x compression of parameters per expert Mixtral-8 7B: enables deployment on a GPU with only 1GB VRAM, reducing the mem- ory footprint by up to 8.5x; and delivers a 48.7x inference speedup compared to DeepSpeed-Mil on single GeForce RTX 3090-all with only a 4.4% 7.6% average performance degradation.
To address the problem, offloading techniques (Sarkar et al., 2023; Eliseev & Mazur, 2023; Hwang et al., 2024; Song et al., 2024a; Xue al., 2024a; Tang et al., 2024), which unmount expert parameters to CPU memory and load them into GPU memory demand for each input, offers ural solution. However, offloading shifts the decoding bottleneck from memory-bound to I/O-bound, as transfer- ring billions of parameters through the low-bandwidth PCIe bus incurs substantial data transfer delays. For comparison, the DRAM-to- VRAM bandwidth (32GB/s for PCIe 4.0) is orders of magnitude lower than the bandwidth between GPU memory and on-chip computation units (300GB/s). Consequently, existing MoE inference systems with expert offloading, designed for edge-side continuous serving sce- narios (i.e., single-batch et al., 2024; Eliseev Mazur, 2023; Hwang et al., 2024; Sarkar et al., 2023; Tang et al., 2024), still fail to support on-the-fly inference, where the loading process percepti- ble the user because its overhead cannot be hidden by the model computation. Ultra-low-bie quantization effectively reduces the size of transmitted parameters to mitigate the latency of activated expert loading (Eliseev & Mazur, 2023; Sarkar et al., 2023), but at the cost of significantly degraded generation performance. Thus, a pressing question emerges:
inference) (Kong
1. Introduction
Mixture of Experts (MoE) models including DeepSeek- RI (DeepSeek-AI, 2025), GPT-4 (OpenAI, 2023), Phi- (Abdin al., 2024b), Mixtral (Jiang et al., 2024), etc., offer a paradigm shift in the large language model (LLM) ar-
Equul contribution 'The State Key Laboratory of Blockchain and Data Security, Zhejiang University 'Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security 'aaul G. Allen School of Computer Science & Engineering, University Washington. Correspondence to: Zhongle Xie Exiezl@zju.cdu..o. Lidan Shou <should@ zju.edu.en>..
How can we hide the I/O overhead of activated experts within model computation enable on-the-fly MoE infer- ence the memoryy-cosstraiee GPU while minimizing generation performance degradation?
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we present an on-the-fly MoE inference sys-

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
tem, coined FloE, for consumer-grade devices. FloE re- duces the 1/O overhead of the experts, namely the transfer cost of the matrices for up, gate, and down projections, via a hybrid compression mechanism (Section 3.2). Despite the utilization of the well-known inter-expert sparsity, the com- pression exploits the vast, untapped intra-expert sparsity in MoE models with a novel contextual sparsification scheme (Section 3.2.1), balancing the transfer cost and the down- stream performance. In detail, the system first identifies Low-magnitude no-salient output activations of up projec- tion and then removes the corresponding channel weights from the gate and down projections. Meanwhile, we ob- serve that the up projection matrix has limited sensitivity on performance against quantization, motivating us to enable the ultra-low-bit guantization in FloE to reduce the transfer overhead further (Section 3.2.2).
to DRAM (Sheng et al., 2023). However, constrained PCIe bandwidth creates transfer bottlenecks during CPU-GPU expert transfers (Kamahori et al., 2024).
To mitigate this, prefetching strategies predict and preload required experts through two paradigms: experience- statistical methods using offline activation traces (Sarkar et al., 2023) (limited to top-1 expert activation strategy (Fe- dus et al., 2022)), and intermediate result-driven approaches leveraging hidden states (Eliseev & Mazur, 2023; Hwang et al., 2024; Song et al., 2024a; Tang et al., 2024) or prior ex- pert indices (Xue et al., 2024a). The former fails under multi- expert activation due to exponential path growth (Dai et al., 2024), while the latter faces an accuracy-latency tradcoff: early-stage predictions from intermediate results (Song et al., 2024a) diminish prefetching accuracy, necessitating costly expert reloads, whereas adjacent-layer predictions (Hwang et al., 2024; Eliseev & Mazur, 2023) prevent computation- communication overlap.
Although the hybrid compression reduces per-transfer cost for MoE models, the pipelining between transfer and compu- tation is prevented due to the sequential execution of routing, quantized up projection computation, and DRAM expert fetching, inhibiting on-the-fly inference. Therefore, we in- vestigate the weights and the input during the computation and locate a high similarity between the shared hidden state input before routing and up projection of the MoE model. Based on the finding, we devise two efficient yet effective sparsity predictors: (1) an inter-expert learning-base baaeed pre- dictor to guide the routing of the activation expert of the next layer with the hidden state of the current layer; (2) an intra-expert reused-base predictor precomputing the con- text sparsity distribution with the hidden state of the current layer and the reused up projection. The two predictors, with the help of prefetching, enable the pipelining of transfer and computation for on-the-fly inference. (Section 3.3)
While ultra-low-bit expert quantization reduces transfer overhead at the cost of accuracy (Eliseev & Mazur, 2023; Sarkar et al., 2023), CPU-based partial computation (Kama- hori et al., 2024; Xue al., 2024b; Tang et al., 2024) achieves limited acceleration due to insufficient through- put for high-dimensional matrix operations.
In contrast to our focus on on-the-fly inference in latency- sensitive scenarios, alternative offloading solutions, such as MoE-lightning (Cao et al., 2024), are primarily designed for high-throughput inference in offline scenarios.
To integrate all the techniques above, we at last propose an efficient sparse kernel and compact asynchronous transfer from DRAM VRAM to achieve rystem-wide efficiency (Section 3.4). The experimental study on various GPU specs and downstream tasks evidence the efficiency and efficacy of FloE (Section 4). Notably, for the popular Mixtral-8 x 7B, FloE achieves 9.3x parameter compression per expert, en- ables deployment on a GPU with just 11GB VRAM, and delivers a 2.6x inference speedup on an RTX 3090, with only 4.4%~7.66 7.6% average performance degradation.
Sparsity in LLMs. Maintaining model quality while min- imizing parameter transfer necessitates synergistic sparsity and quantization. Weight pruning (Sun et al., 2023; Frantar & A Alistarh, 2023; Ma et al., 2023) zeroes subsets of LLM weights to reduce computatioonal/meorry overhead but faces performance degradation and hardware compatibility issues consumer-grade devices.
Activation sparsity- computation via zero-rich hidden states caates-naturallyy occurs in ReLU-based MLPs (Liu et al., 2023; Alizadeh et al., 2023; Shin et al., 2024) but diminishes in modern architectures using non- MLPs (e.g.. SwiGLU (Shazeer, 2020)), limiting direct applicability. Recent research has thus concentrated on reintroducing acti- vation sparsity within newer architectures (Mirzadeh et al., 2023; Zhang et al., 2024; Song et al., 2025; 2024b), but requires extensive pretraining (billions of tokens). Training- free activation sparsity (Lee et al., 2024a; Liu et al., 2024), achieved through activation magnitude pruning in SwiGLU- based LLMs, remains tailored for dense models with uni- form parameter utilization across inputs.
2. Related Work
Experts Offloading. Efficient deployment of MoE mod- els faces substantial challenges due to their parameter counts, particularly under resource constraints. Current inference frameworks like Llama.cpp (llama.cpp), Hugging- Face Accelerate (Gugger et al., 2022), and DeepSpeed In- ference (Aminabadi et al., 2022) employ experts offloading by selectively transferring VRAM- expert weights
2

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
(a) Naive MoE Inference with Offloading
(b) Advanced MoE Inference with Offloading
(c) On-the-Fly MoE Infrence (FLoE)
GPU
PCle
CPU
GPU
PCle
CPU
GPU
PCIe
CPU
Non-Experts i-th Layer
Expert Predictor
-1-th Predictors
Non-Experts
Non-Experts I-th layer
i-th Layer
Experts Miss!
Experts Miss!
Experts Miss!
MoE Layer
MoË i-th Laver
MoE 1-th Laver
Bumi
(+122)
(14)
66
611,11
(11,22
Br3.m)
Non-Experts 1+1-th Layer
Experts Cache
[um]
Expert Predictor
[n.m
I-th Predictors
In,
(1.))
W4
[wg Samnn Wil
Sedwon) w Activated O Non-Acthonee
Miss 0 trontcaaed
Figure 1. Comparison of MoE. inference offloading systems for memory- consttrindd GPUs: (a) Naive MoE Inference with Offloading; (b) Advanced MoE Inference with Offloading; (c) On-the-Fly MoE Inference (FloE).
3. FloE: On-the-Fly MoE Inference
duces dual predictors (Figure I(c) )) for inter- and intra- expert sparsity to accurately prefetch activated compressed weights (Figure I(c) @) while minimizing DRAM usage.
3.1. MoE Inference with Offloading
Figure ((a) illustrates naive MoE inference with offloading. Non-expert weights, frequently activated during inference, reside persistently in VRAM and are computed the GPU. Expert weights, due to their sparse activation, are offloaded to DRAM. If certain experts' weights are missing from VRAM (Figure ((a) O) the system transfers these weights over the PCIe bus (Figure ((a) ), after which the GPU proceeds with subsequent computations (Figure ((a) i).
Finally, Section 3.4 describes system co-optimizations that further enhance FloB's efficiency.
3.2. Expert Hybrid Compression
As shown in Figure 1, in a SwiGLUU-basd MoF model, each expert Eij consists of three matrices { {W Wa W 4j We denote the number of layers and the number of experts per layer as m and n, respectively. Although advanced MoE offloading proposes compressing experts using ultra- low-bit quantization (e.g., INT2, INT1) to reduce transfer costs, this significantly degrades model performance.
As mentioned in the introduction, expert offloading shifts the decoding bottleneck from memory- bound to I/O bound. Specifically, the expert transferring from DRAM to VRAM incurs long latency. For example, the expert in the Mixtral- 8x model has over 300MB of FP PP16 16 parameters, taking nearly 15ms to transfer over a 16-channel PCIe 4.0 bus, whilst single expert's 's computation on GeForce RTX 3090 takes only about 5ms.
We argue that applying a uniform ultra-low-bit quanti- zation strategy Q(-) across all matrices (see Figure 1(b) { [W Wdddow)) whihi) ) within an expert fails to strike an optimal balance between efficiency and perfor- mance. Thus, FloE introduces a unique twist with hybrid strategy that tailors compression methods to the proper- ties of the projection matrices. Specifically, contextual activation sparsity S(-) is applied to the gate projection Wa and down projection Wolwwn to produce W and Sidoww)) Mcanwhile, ultra-low-bit quantization Q(-) (INT2) is used for the up projection Wip to yield w
Figure 1(b) shows advanced MoE offloading (Eliscev & Mazur, 2023; Sarkar et al., 2023; Hwang et al., 2024; Song et al., 2024a; Xue et al., 2024a; Tang et al., 20t4ldeetailed related works discussion in Section 2). Despite the process in the naive solution, an extra expert predictor (Figure ((b) 4) is implemented to prognosticate the expert visiting in the near future. The prognosticated expert, as shown as Figure 1(b) 6, is quantized and preloaded in GPU-resident expert cache (Figure 1(b) @), managed by replacement policy. Compared to the naive solution, the advanced MoE offloading can achieve better transfer efficiency due to the usage of the expert predictor and the cache.
ij
3.2.1. CONTEXTUAL SPARSIFICATION FOR GATE DOWN PROJECTIONS
Contextual activation sparsity reduces model computations dependent low-magnitude, non-salient contextual activa- tions by pruning the corresponding channel weights, with minimal impact on model performance (Liu et al., 2023; Lee et al., 2024a; Liu et al., 2024). However, the MoE model already performs sparse activation inference through the router, selecting the experts to participate in the computa-
Next, we present FloE, an inference system that delivers on-the-fly MoE model inference consumer-grade GPUs. FloE uses a hybrid compression scheme-innegrating con- textual sparsity and ultra-low-bit quantization (Figure ((c) ① detailed in Section 3.2. In Section 3.3, FloE intro-
3

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
activation functions:
Propuatoy WMC
Matil traatie
sle Theakee
agatte (x) SSLUUX aup (x) = x xW
(3)
adown (x) aguie (x) ① aup (x),
(4)
kids
0.0
0.12
Rogamno of W02
youil
and produce the following sparsity function:
if [a(x)) 2 t, Sy(a(xx) 110, 0, if [a(x)] < t.
- aaxx,,
(5)
Emquanto of
Frenuancy of of
Here, a E {agattee Sup, Adown The threshold L is derived from the sampled dataset based on the desired sparsity ratio:
(a) w Frequency (b) Wd Frequency (c) W Frequency
t: : minst' F(t) 2 k}.
(6)
where F-) represents the empirical cumulative distribution function of absolute activation values for each expert, and k specifies the target sparsity ratio (c.g.. 70%). The dis- tribution is empirically estimated offline using activations sampled from a general text corpus.
Figure 2.
Activation distributions of -ixttral-x7BB's three hidden states at experts Eo,o (shallow layer), E1s,7 (middle layer), and E (deep layer).
tion based on the given context.
To evaluate the impact of magnitude- based activation prun- ing on model performance, we set thresholds for the outputs of the SiLU LU activation function, W and the inputs to Widon at various sparsity levels, then measured text per- plexity on WikiText-2 (Merity et al., 2016). As shown in Figure 3(a), we find that pruning based on the Wdoen in- puts the least sensitive to sparsity: at 50% sparsity, the perplexity increases by only about 0.5%, and even at 90% sparsity, perplexity remains relatively stable. In contrast, pruning the 4"P outputs is slightly more sensitive, where 80% sparsity roughly matches the 90% sparsity level of the Widown inputs. Pruning the SiLU outputs is the most sensi- tive, pushing perplexity ahove 7 at 70% sparsity. We provide further evaluations on downstream tasks in Section 4.2 and have a theoretical interpretation for this phenomenon (Refer to Appendix A. for more details):
Question 1. Does internal sparsity in experts of MoE models exist and persist consistently across layers?
The experts within a sparsely activated MoE model maintain a high internal sparsity across layers.
Observation 1.
We conducted preliminary study on the activation distri- bution within experts, analyzing output activations from the Waate and WaP matrices, and input hidden states to the Wrown matrix of the Mixtral-8 x 7B (Jiang et al., 2024) model on the C4 dataset (Raffel et al., 2019), visualized in Figure 2 Consistent with findings from CATS (Lee et al., 2024a) and TEAL (Liuu et al., 2024), we observed that many activations are concentrated around zero. This con- centration motivates the use of a magnitude-basee activation sparse strategy, where activations close to zero are set ex- actly zero, eliminating corresponding weight computations and transfers during inference.
Theorem 3.1 (informal). From the definition of St in Equa- tion (5), we define:
Given an input vector x and three projection weight matri- ces Weate Wdown W in an expert E, the corresponding activation output ag is computed as following forward pass:
Loown E (Bdown - Sy (adow)))) Wodowa 2
(7)
Eup
E8| (Bdown agate 2 Se (a(p))))
W down 2
(8)
ag (x) := (SiLU(X Weat gate
① (xWWP) down
(1)
Lgase E (downn - Se (agate) ) 2 aap)
Wodown
(9)
Then under assumptions consistent with experimental obser- vations, we have
SiLU(X) == x xx) 1 x
X
where C denotes the Hardmard product and (.)) is the activation function. Therefore, magnitude- based sparsity can be determined from the outputs of the SiLU activation function, Wap and the inputs to drown, We define three
Lsown S Cup < Cgate-
(10)
While the input pruning of down projection shows theo- retical optimality for downstream tasks, its effectiveness is constrained by two factors: (1) Dependency on gate/up projection outputs limits computational savings to the final
(Abdin al., 2024a) and DeepSeek- v2 (DeepSeek-AA, 2024) are validated in the Appendix D.
4

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
INT4 results in minimal performance impact, with per- plexity changes under 3%. At INT3 and INT2, perplexity increases, with the down projection exhibiting the most sig- nificant change, followed by the gate projection, while the up projection remains the least sensitive. At INT1, up pro- jection quantization yields only 46.01 % of the perplexity of gate projection quantization and 27. 27.23% of the perplexity for down projection quantization. Across all bit-widths, the up projection consistently shows the lowest perplexity.
0.23
1.59
P412
Spass Ratis 1S3
Obentication BV
(a) Sparsification Sensitivity
(b) Quantization Sensitivity
Some works (Geva et al., 2021; Yu & Ananiadou, 2024) treat the MLP layer (i.e., the expert in MoE) as a key- value memory model, where the up and gate projections serve as keys to selectively activate the values in the down projection, which stores knowledge related to the input. This theory aligns with our experimental results. The down projection, storing knowledge values, requires higher pre- cision than the gate and up projection, as evidenced by its significant performance degradation across different quanti- zation bit-widths. The gate projection, influenced by nonlin- ear activations, e.g., SwiGLU (Shazeer, 2020), demonstrates greater sensitivity at ultra-low bit-widths (INT2, INT1).
Analysis:
Figure 3. Compression sensitivity of expert parameters: (a) Sparsi-
fication sensitivity; (b) Quantization sensitivity.
projection, and (2) Non-linear operations (SiLU, Hadamard product) hinder prediction for offloading, Empirical evalua- tions and theoretical analysis show that the output sparsity of up projection, compared to the SiLU activation function, yields superior generative performance at equivalent spar- sity ratios. This motivates our design to replace the original expert forward pass computation in Equation (1) as follows:
a (z) == ( @ (x(((")))) W down (11)
The observation and analysis of quantiza- tion sensitivity above suggest that the up projection is the least sensitive to quantization, and therefore we choose to apply the INT2 of HQQ method for its compression.
Implementation:
3.2.2. ULTRA-LOW-BIT QUANTIZATION FOR THE UP PROJECTION
Thanks contextual sparsity, only 10% of the weights in the gate and down projections ({ WWalle Widwwn) }) are activated. However, the full parameter set of the up projection Wup is required for computation, as its output activations determine the sparsity threshold for truncating Waste and Wilwwn As mentioned, prior work (Eliseev & Mazur, 2023) deploying MoE models on consumer-graid grade devices suffers from sub- stantial performance loss due to the uniform ultra-low-bit quantization the three projection matrices. Building upon the fact that the contextual sparsity of gate and down projections has minimal impact on performance, thus alle- viating the quantization burden on experts, the following question arises:
3.3. Expert Sparsity Prediction
In MoE models, each MoF layer uses router to determine activated experts for each input hidden state followed by forward pass according to Equation (1). Although the hybrid compression reduces per-transfer cost for MoE models, the pipelining between transfer and computation is prevented due to the sequential execution of routing, quantized up projection computation, and DRAM expert fetching, inhibit- ing on-the-fly inference. Recalling sparsity prediction in dense LLMs (Liu et al., 2023; Lee et al., 2024b), the resid- ual structure of the model leads to high similarity between hidden state inputs before consecutive MLP layers. This allows the hidden state of the i-th MLP layer to be fed into a trained predictor to forecast the sparsity distribution for the (i + 1) 1)-th layer. Inspired by this approach and considering the same inputs to the router and quantized up projection, we pose the following question:
Question 2. Can we quantize only the full up projection, from W to Will)), and effectively reverse the inherent performance degradation caused by uniform quantization? Observation 2. The up projection exhibits low sensitivity to nltra-low-bit quantization.
We employ Half-Quadratic Quantization (HQQ) (Badri & Shaji, 2023) with various bit-widths to quantize the three projection matrices within each expert of Mixtral 8x7B and evaluate their quantization sensitivity using perplexity on WikiText-2 (Merity et al., 2016) As shown in Fig- ure 3(b), quantizing the projection matrices at INT8 and
Question 3. Can the hidden states x of the existing layer be used in sparsity prediction in prefetching the activated compressed experts for the successive layer, replacing the router and up projection computations?
Fortunately, in sparse MoE models, we empirically validate the core principle behind sparsity prediction:
PPhi-a.--Mo--instruett (Abdin et al., 2024a), Moe- 16B-Base (Dai al., 2024) and 1wennl.5-MoB-A2.7B (Team, 2024) are validated the Appendix F.
Observation 3. The hidden states input to the router and up projection in consecutive MoE layers exhibit high similarity.
5

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
1.00
Algorithm Efficient Sparse Kernel
1: Input: hidden states x, threshold tij. Egj
0.35
{ [Welle W T
W
0.90
2: xW
if
3: mask + ([v)
0.80
Cos Similarity Outlier 0.44
Inter-expert Precision
4: x' + SiLU (x gate
fJ
mask]) @ v[mask]
intra-experts Recall
5: y f- (Wolon,
[mask]x')
0.75
10
15 Layer ID
20
25
30
6: Return: y
Figure 4. Next layer cosine similarity (blue), intra-expert predictor precision (yellow), inter-expert predictor recall (red), and the out- lier corresponding to the cosine similarity at the O-th layer (gray).
states before the i-th MoE layer and the reused up projection matrix of the (i + 1)-th layer. Once the approximate output activations of the up projection are obtained, the contextual sparsity distribution can be computed in advance.
Specifically, we randomly sample 100 sequences of length 256 from the ShareGPT (ShareGPT, 20023) and feed them into Mixtral-8x 7B. Then, we compute the average next layer similarity, defined as the cosine similarity between the hidden states before the i-th layer and (i + 1)-th layer. Figure 4 shows that the next layer similarity consistently remains above 0.95, except for the first layer.
Different from existing learning-based predictors (Liu et al., 2023; Shin et al., 2024; Xue et al., 2024b), which, for example, impose an additional 2.19GB~9GB of memory footprint for models like Mixtral-8 7BB (detailed Ap- pendix F), an unbearable burden for memory-constraine GPUs, intra-expert sparsity predictor incurs little extra memory cost. Furthermore, the red line in Figure shows an average recall of 0.95, demonstrating the predictor's s ability to maintain high accuracy across varying layer depths.
Building on the observation, we devise two efficient yet effective predictors for inter- and intra-expert sparsity. They both consume the input hidden states of the existing layer and prefetch the activated compressed experts to be visited in the next layer, hence excluding the upcoming computation of router and up projection.
3.4. System Co-optimization
3.4.1. 1. EFFICIENT SPARSE KERNEL
3. INTER-EXPERS SPARSITY PREDICTOR
To translate the reduction in computational complexity intro- duced by sparsity into clock time acceleration, we developed a specialized sparse GEMV kernel using the Triton (Tillet et al., 2019)-based kernel introduced by CATS (Lee et al., 2024a). We achieve maximal data read efficiency by trans- posing Wilown and utilizing volumn-major storage. By se- lectively loading the columns of the weight matrices W and W-lown. based on a threshold, we reduce the number ij of memory accesses, thereby accelerating clock time.
For inter-expert sparsity, we introduce learning- based pre- dictor that proactively predicts the experts required for the (i +++ 1)-th layer while computing the i-th layer.
The core idea of the learning-based predictor is to collect the input from the previous layer along with the histori- cal trajectory of expert selections, capturing the underlying correlations between them. Leveraging these correlations, the predictor makes informed decisions about future expert selections. We observe that the complexity of prediction diminishes as the layer depth increases. To adapt to this, we dynamically adjust the predictor's 's parameters in prac- tice, scaling from a single-layer MLP with 32K parameters to a two-layer MLP with 2M parameters. The orange line in Figure illustrates an average precision of 0.88, high- lighting the inter-experts predictor's 's capability maintain high accuracy while adapting varying layer depths.
As shown in Algorithm this kernel accepts the input hidden state x, sparse threshold tey. and expert weights E, w W W.down.T w }. First, a mask vector is generated based on the absolute values of the hidden vectors output by xWu and the magnitude of the threshold. The SiLU activation and element-wise multiplication are fused into each block computed by W [mask]x, which con- serves memory operations required for multiple storage and loading of x' and reduces kernel launch time. Subsequently, the resulting x' is multiplied by the transposed W to produce the output of the sparse MLP. Section 4.1 shows our sparse GEMV kernel effectively reduces expert compu- tation time as sparsity increases.
3.3.2. INTRA-EXPERT SPARSITY PREDICTOR
For intra-expert sparsity. we introduce parameter-five free, reuse-basee based predictor. This predictor estimates the output activations of the up projection in the (i + 1) layer by di- rectly performing matrix multiplication between the hidden
6

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
4. Evaluation
DRAM
dustoon
Threads Pool
In this section, we aim to demonstrate that FloE can speed up MoE decoding on limited GPU memory while preserv- ing high accuracy. We first present our end-to-end system results showing wall-clock performance, followed by FloE's accuracy in downstream tasks'. Specifically,
Pinned Memory
Wi]
Bowa
In Section 4.1, we demonstrate that FloE enables 48.7x end-to-end acceleration compared to DeepSpeed-MII, with sparse kernel contributing up 2x speedup and compact asynchronous transfer achieving 12.6x faster performance compared to the naive method.
CUDA Streams
PCIe
VRAM
Figure 5. Process FloE's compact asynchronous transfer: com- pacting weights layout in DRAM for reduced access latency and multi-threaded packaging of activated experts to enable asyn- chronous data transfer.
In Section 4.2, we show that FloE achieves a performance gain of 9.8% over other methods at high sparsity.
4.1. Efficiency Evaluation
3.4.2. COMPACT ASYNCHRONOUS TRANSFER
We analyze decode efficiency via end-to-end generation tests across various input/output lengths and VRAM us- age, assess single-expert latency speedup for sparse GEMV, and evaluate transfer efficiency by simulating single-expert transfer with varying chunk sizes.
Due to the sparse activation of weights, the expert trans- fer process occurs across multiple non-contiguous memory blocks in DRAM and VRAM, making it difficult to fully utilize the PCIe bus bandwidth. The Pytorch (Paszke et al., 2019) naive implementation can only achieve a fraction of the PCTe bandwidth, significantly affecting inference la- tency. Therefore, we compacted the arrangement of the gate and down projection matrices to transfer data larger chunks and further enhance data throughput using SIMD and multithreaded asynchronous transfer techniques. Next, we detail the optimization of the data transfer strategy.
Setup We use GeForce RTX 3090 with 24G VRAM to evaluate end-to-end latency on ShareGPT (ShareGPT, 20023) prompts. The system is also equipped with 64- CPU 2.3GHz and 256G DRAM interconnected via PCIe 4.0. For the single-expert latency test, we use C4 dataset (Raffel et al., 2019) and employ four types of GPUs, including H100, A100, A6000, and GeForce RIX 3090.
Compact weights Layout In an expert, the activation of the i-th intermediate neuron corresponds to the usage of the i-th column from the gate and up projection matrix, along with the i-th row from the down projection matrix. By co-locating the corresponding columns of the gate pro- jection and rows of the down projection in DRAM, we can compact the data into larger contiguous chunks for efficient transfer. Assuming each element of the dnidten X dintermediate weight matrices is stored in num numbyytes, this layout strat- egy increases the chunk size from duideen X num_bytes to 2dhidten X num bytes, illustrated in Figure 5.
Baseline We employ four SOTA baselines in the eval- uation: DeepSpeed-MII (Microsoft): An inference sys- tem utilizes ZeRO-Infinity (Rajbhandari et al., 2021) to deal with expert offloading. Mixtral-Offloddin (dvmazur, 2023): An MoE framework integrating expert prediction, caching mechanisms, and quantization. Fiddler (Kamahori et al., 2024): A CPU-GPU co-execution system minimiz- ing data transfer overhead through computational offload- ing. Mixtral-GPU: A model with HQQ INT2 quantized enabling complete GPU residency, serving as the latency lower-bound reference for on-the-fly scenario requirements. Analysis We evaluate FloE's end-to-end efficiency with varying input/output lengths, and the results averaging over 5 runs are depicted in Figure 6. In the figure, we measure the inference speed for single-batch generation. We select the average output tokens per second (TPS) as the measure- ment. As seen, FloE achieves 91% of Mixtral-GPU's speed (95% most). delivering 48.7x, and 3.14x speedups over DeepSpeed-MII, spectively. It should be noted that with longer outputs for
SIMD Asynchronous Transfer To fully leverage modern CPU capabilities, we use the AVX-512 instruction set and multithreaded asynchronous transfer. As shown in the Fig- ure 5, we allocate a pinned memory in the CPU (transferring data from pinned memory significantly improves transfer speed) to send weights to VRAM. We use multithreading in combination with SIMD instructions to bundle several weights groups for transfer into pinned memory, and asyn- chronously send transfer requests across multiple streams, minimizing idle time on the PCIe bus.
and Fildder, re-
We employ Mixtral-8 x 7B as the MoE model for all test cases.
7

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Mixtral-GFU
FLaE
Decpspeed
Mixtrall Ofilbadin
Fiddler
.222
1.00
91
50
0.34
0.30
0.3322
0.35
P.300
0.36
30
0.35
28.
0.27
0.29
0.26
0.229
0.29
0.02
0.02
0.02
0.02
0.02
0.02
0.02
0.02
0.02
0.02
(32,128)
(32,256)
(32,512)
(64,128)
(64,256)
(64,512)
(128,128)
(128,256)
(128,512)
avg
(Input Length, Output Length)
Figure 6. Bars quantity generation speed of compared methods under 12GB VRAM constraints, with numerical labels indicating relative speedup ratios against the Mixtral-GPU baseline. Note that DeepSpeed uses FPI offloading.
Table I. ExpeettExpettteee Execution Latency with Sparse GEMV Kernel (ms).
GPU MODEL
0
508
60
708
80%
90%
H100
0.169
0. 134
(1.26x
0.123 (1.37x f)
0.114 (1.48x 1)
0.106 (1.59x
0.103 (1.64x t)
A100
0.253
0.195
(1.30x
0.188 (1.35x f)
0.176 (1.44x f)
0.166 (1.52x
0.155 (1.63x f)
A6000
0.524
0.365
(1.44x f)
0.337 (1.56x f)
0.305 (1.72x f)
0.277 (1.89x f)
0.263 (1.99x f)
RTX-3090
0.542
0.379 (1.43x f)
0.354 (1.53x
f)
0.316 (1.72x
0.302 (1.80x f)
0.283 (1.92x 1)
fixed inputs, TPS improves as layer-wise expert replacement overhead is amortized over longer sequences.
LO.
w/o
LO..
L.O.
Naive
1.0
0.86
Figure compares the generation throughput under in- put/output length of 64/256 and VRAM usage ranging from 12GB to 24GB. With additional VRAM, we cache more MoE layers to reduce expert misprediction reload overhead. Meanwhile, our sparse GEMV kernel applied to expert ac- tivations further boosts generation speed. Across different VRAM capacities, our method remains close to Mixtral- GPU's performance and slightly surpasses it 24GB. When essentially
o
0.78
0.75
0.8
Bandwilth Utilzaation
Latency (ms)
0.780.76
0.61
0.66
0.6
0.60
0.4
0.2
0.07
0.0
DRAM usage reaches 21GB, Mixtral-Offloateee mirrors the Mixtral-GPU setup but is marginally slower, it still relies on INT 3 quantization for certain experts.
100
200
500
1000
2000
Chunk Size
Figure 7. Comparison transfer latency and bandwidth utilization: bars show DRAM-to- VRAM transfer delays per expert, while lines depict utilization relative to PCIe 4,0's actual peak bandwidth. Gray dashed lines are Py Torch''s native implementation.
Table compares the sparse kernel's speedup across sparsity levels and GPUs for single expert, including dense up pro- jection GEMV, fused SiLUU activation, sparse gate projection GEMV, and sparse down-projection GEMV. Using 500 to- kens from C4 dataset, ran 80 warm-up iterations and 200 timed trials to measure execution latency. Our kernel consis- tently outperforms the dense baseline (sparsity 0). At 50% and 70% sparsity, it achieves over 1 1.26x and 1.44x speedup, respectively. At 90% sparsity, only A6000 and RTX 3090 obtain nearly 2x speedup, while H000 and A100 are lim- ited by kernel launch overhead and other non-computational factors due to their higher computational throughput. The results evidence sparse GEMV kernel is advantageous on consumer-grade devices.
chunk sizes (number of weight columns per thread). The average over 20 trials shown in Figure 7. Our compact asynchronous transfer achieve up 88% of peak bandwidth, 12.6x faster than PyTorch (Paszke et al., 2019) native im- plementation. Compact weights layout improves efficiency across all chunk sizes. Transfer latency first increases and then decreases as chunk size grows-small chunks are dom- inated by API calls and CUDA launch overhead, while large chunks suffer from excessive DRAM packing time, limiting transfer overlap. The optimal chunk size in our setup 50.
For transfer efficiency, we randomly selected 20% of expert weights (20% of columns in the gate projection matrix and corresponding columns in the transposed down projection) and transferred them from DRAM VRAM using varying
4.2. Efficacy Evaluation
We analyze model efficacy via downstream tasks and vali- date the compatibility of the quantization with FloE.
8

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Mixtral-GrP
Deepspeed
Mixtral-Oflleding
Mixtral-8x78
85.3
96.8
35.4
76.2
56.7
84.3
69.5
72.0
0.97
0.97
0:980:90
0:96
90
HQQ INT3
80.9
95.5
29.2
71.2
48.1
80.0
60.8
66.5
CATS-80%
32.2
67.0
51.5
78.2
65.7
80
CHESS-80%
80.2
91.2
30.2
69.4
49.8
78.1
61.2
66.5
0.27
00.28
0.31
Hoo--W--80%%
82.9
94.4
34.4
73.2
53.2
81.6
65.44
69.3
60
6.02
0.02
HQQ INT2
48.5
33.1
14.4
49.3
21.2
27.9
23.4
31.1
12G
14G
18G
21G
24G
VRAM Usage
CATS-90%
70.4
82.6
27.2
58.6
44.2
70.9
37.7
55.9
50
Figure Illustration of the generation speed of different methods under varying DRAM usage, with numbers indicating the speed relative to Mixtral-GPUS GPU.Since Mistral-Offlodingg caches by layer, there is configuration for 14GB DRAM usage, we use the 12GB result instead.
CHESS-90%
72.7
83.9
27.8
60.4
41.0
69.4
42.4
56.8
90-9-W9-90%%
60.1
65.6
30
FidE-80%
81.0
95.1
33.6
71.7
50,9
80.3
67.6
Aug Aoc Doorrttwan Heat
1072
1024
FleE-90%
95.2
76.2
18.1133.1
20
BoolQ
5010
0
WG
Arc.C
ArcE
MMLUg5
Average
8.50
625
0.90
Figure 10. Downstream task performance. FloE-W e refers to
5.75
0.00
30
contextural sparsificaition according up projection.
Sparsity Ratlo(9)
Sparsity Ratio(56)
(a) Sparsification Sensitivity
(b) Quantization Compatibility
90% sparsity. The reason lies the fact that activations for WrP demonstrate better performance compared those for Waate evidenced by the trend in Figure (a)). In addition, FloE combines both quantization and sparsity, trading off a small amount of accuracy for improved deployment speed. Despite this trade-off, its performance remains higher than that of HQQ INT3 and CHESS. On the MMLU task, we observe noticeable performance drop from FloE- W to FloE. When GPU memory is sufficient, increasing the bit- width of the yh matrix can mitigate this issue.
Figure Impact scaling up sparsity ratio on performance: (a) Task accuracy across different sparsity strategies, and (b) Text perplexity of FloE combined with various quantization bit-widths.
Setup For the downstream task performance, we use seven downstream tasks using the EleutherAI LM Harness (Gao et al., 2024), including zero-shot ARC easy and challenge, zero-shot BoolQ, zero-shot SciQ, zero-shot Open zero-shot Winogrande and 5-shot MMLU (Clark et al., 2018; 2019; Johannes Welbl, 2017; Sakaguchi al., 2019; Hendrycks et al., 2021). These tasks originally chosen to measure the abilities of the models across various domains, such as reading comprehension and reasoning.
We also demonstrate the compatibility with quantization techniques by evaluating different HQQ quantizations and plotting the perplexity variations Mixtral-8 x on WikiText-2 in Figure 9(b). The perplexity increases ex- hibit similar trends across different bit widths, indicating that the errors introduced by activation sparsity and weight quantization largely independent and additive.
Baseline We employ three sparsity quantization base- lines in the evaluation: CATS (Lee et al., 2024a): A SOTA activation sparsification method, which applies magnitude pruning to FFN activations. CHESS (He et al., 2024): A general activation sparsification approach via channel-wise thresholding and selective sparsification. HQQ quanti- zation (Badri & Shaji, 2023): fast and accurate model quantizer that skips the need for calibration data. Among its notable applications, HQQ INT2 refers to the perfor- mance of MixtralGGPU on downstream tasks. We utilized HQQ to quantize the experts to INT2, enabling the entire Mixtral-8x7B model to fit within GPU memory.
5. Conclusion
We introduce FloE, an on-the-fly inference system for MoE models on memory-constrained GPUs, which optimizes GPU memory utilization through an expert hybrid compres- sion scheme and effective sparsity predictors, achieving remarkable 48.7x inference speedup single GeForce RTX 3090 compared to DeepSpeed-Mil We hope our work inspires further research on MoE inference with offloading from a sparsity perspective and believe FloE will serve as a valuable tool for the community, enabling on-the- fly infer- ence of sparse MoE models on consumer-grade hardware.
Analysis As shown in Figure 10, " achieves a 2.8% accuracy improvement at 80% sparsity and a signifi- cant performance gain of 9.8% over the SOTA methods at

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Acknowledgements
Alizadeh, K., Mirzadeh, I., Belenko, D., Khatamifard, K., Cho, M., Del Mundo, C. C., Rastegari, M., and Fara- jtabar, M. Lim in flash: Efficient large language model inference with limited memory. arXiv preprint arXiv:232..11114 2023.
This work was supported by the Pioneer R&D Program of Zhejiang (No. 2024C01021), the National Regional Innova- tion and Development Joint Fund (No. U24A20254), and the Zhejiang Province 'Leading Talent of Technological Innovation Program' (No. 2023R5214).
Aminabadi, R. Y., Rajbhandari, S., Awan, A. Li, Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M.. Rasley, J., et al. heepspeed-inferene
Impact Statement
enabling efficient infer- ence of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1- 15. IEEE, 2022.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Badri, H. and Shaji, A. Half-quadratic quantization of large machine learning models, November 2023. URL https: htt:://mmbbsm/smm wit.ub...........g. iichhub.booe. hw/.gggbohgg////////....
References
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan,
Cao, S., Liu, S., Griggs, T., Schafhalter, P., Liu, X.,, Sheng, Y., Gonzalez, J. E.,, Zaharia, M., and Stoica, I. Moe- lightning: High-throughput moe inference on memory- constrained gpus. arXiv preprint arXiv:2411.11217. 2024.
A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl,
H., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai,
M., Cai, Q. Chaudhary, V., Chen, D., Chen, D., Chen, W., Chen, Y.-C., Chen, Y.-L., Cheng, H., Chopra, P., Dai, X., Dixon, M., Eldan, R., Fragoso, V. Gao, J., Gao, M., Gao, M., Garg, A., Giorno, A. D., Goswami, A., Gunasekar, S., Haider, E., Hao, J., Hewett, R. J., Hu, W., Huynh, J., Iter, Jacobs, S. A. Javaheripi, M., Jin, X., Karampatziakis, N., Kauffimann, P., Khademi, M., Kim, D., Kim, Y. J., Kurilenko, I.., Lee, J. R., Lee, Y. T. Li, Y., Li, Y., Liang, C., Liden, L., Lin, X., Lin, Z., Liu, C., Liu, L., Liu, M., Liu, W. Liu, X., Luo, C., Madan, P., Mahmoudzadeh, A., Majercak, D., Mazzola, M., Mendes, C. C. T., Mitra, A., Modi, H., Nguyen, A., Norick, B., Patra, B., Perez- Becker, D., Portet, T., Pryzant, R., Qin, H., Radmilac, M., Ren. L., de Rosa, G., Rosset, C., Roy, S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang. N., Sharma, H., Shen, Y., Shukla, S., Song, X., Tanaka, M., Tupini, A., Vaddamanu, P., Wang, C., Wang, G., Wang. L., Wang, S., Wang, X., Wang, Y., Ward, R., Wen, W., Witte, P., Wu, H., Wu, X., Wyatt, M., Xiao, B., Xu, C., Xu, J., Xu, W., Xue, J., Yadav, S., Yang. F., Yang, J.. Yang. Y., Yang, Z., Yu, D., Yuan, I.., Zhang, C., Zhang, C., Zhang.J. J., Zhang, L. L., Zhang. Y., Zhang, Y., Zhang, Y., and Zhou, X. Phi-3 technical report: A highly capable language model locally on your phone, 2024a. URL httpst:/: //axivv org/ abs/ 14219.
Chen, Y., Jamieson, K., and Du, S. Active multi-task repre- sentation learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 3271-3298. PMI.,, 17-23 Jul 2022. URL https: /ppocceeiigggss.eee mllr press /v1l 62/ c2/chn222 html.
Clark, C.,, Lee, K., Chang. M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. https::: / //arxiv. org/ abs 905 10044.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https: / wea.iiooorrroooooooo.oooooo /abs/ 1803.05457.
Dai, D., Deng, C., Zhao C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. Dee:SeckMoE: Towards ultimate expert specialization in of-euues language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), ACL (Volume 1: Long Papers), pp. 1280-1297, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 1.. 10.116663/1/2222.a.lloooo //aclanthology hwwpe/tthhholoo...go22222222 wrg/2024 long...ll....ooo 70.
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gu- nasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., Lee, J. R., Lee, Y. T., Li, Y., Liu, W., Mendes, C. C. T., Nguyen, A., Price, E., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Wang, X., Ward, R., Wu, Y., Yu, D., Zhang, C., and Zhang, Y. Phi-4 techni- cal report, 2024b. URL https:/ //t///ax/a /arxivv..../////////aa/////aa/v/ org/ abs/ 2412. 08905.
URL https:
DeepSeek-AL. Deepseek-v2: A strong, economical, and efficient mixture-offexxpers language model, 2024.
DeepSeek-AL. Deepseek-rl:
Incentivizing reasoning ca- pability in 1lms via reinforcement learning, 2025. URL
https://arxvv.oo/ httpss///aaii.oogaaa//2/000 12948.
10

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Du, S. S., Hu, W., Kakade, S. M., Lee, D., and Lei, Q- Few-shot learning via learning the representation, provably, arXiv preprint arXiv:2002.09434 2020.
He, J., Wu, S., Wen, W. Xue, C. J., and Li, Q. Chess: Optimizing IIm inference via channel-wise threshold- ing and selective sparsification, 2024. URL https: 01366.
/ //arxiv.ozz
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sra- vankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., and et al. The Ilama 3 herd of mod- els, 2024. URL https: //axxivvv..// wrw. wttgg/ab///// 2407. 21783.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021. https:
/ //arxiv.... org/ aas/ 03300.
Hwang, R., Wei, J., Cao, S., Hwang, C., Tang, X., Cao, T., and Yang. M. Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of -ffture-of-expert infer- ence. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 1018- 1031. IEEE, 2024.
dvmazur. mixtral-offloading https: /tt::////ttttt com/ dvmazuz/mixtral mttpa/urmmmtttal-offooodddddd 2023, Version v0.1.0, Accessed on: October 10, 2023.
Eliseev, A. and Mazur, D. Fast inference of mixture- of-experts language models with offloading. CoRR, abs/2312.17238. 2023. doi: 10.48550/ARXIV.2312 17238. URL https://doii org/ / arXiv, 2312.17238.
Jiang. A. Q. Sablayrolles, A., Roux, A., Mcnsch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L.. R., Saulnier, I.., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mixtral of experts, 2024. URL https: //tr///axvvvvvvv///////ee/aaaal/ org/ htg//bb/224011 04088.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120):1-39,20 2022.
Johannes Welbl, Nelson F. Liu, M. G. Crowdsourcing mul- tiple choice science questions. 2017.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In ICML, pp. 10323-10337) PMLR, 2023.
Kamahori, K., Gu, Y., Zhu, K., and Kasikci, B. Fiddler: Cpu-gpu orchestration for fast inference of mixture-of- experts models. arXiv preprint arXiv:2402.07033, 2024.
Gao, L., Tow, Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: /ewodde org/records/ 12 6060
Kong, R., Li, Y., Feng, Q., Wang, W., Ye, X., Ouyang, Y., Kong, L., and Liu, Y. SwapMoE: Serving off-the-shelf MoE-based large language models with tunable memory budget. In ACL (Volume 1: Long Papers), August 2024.
Lee, D.,, Lee, J., Zhang, G., Tiwari, M., and Mirhoseini, A. CATS: large language models. In First Conference on Language Modeling, 2024a. URL net f www.uummmiddww222Em...
Context-aware thresholding for sparsity in
Geva, M., Schuster, R., Berant, J., and Levy, O. Trans- former feed-forward layers are key-value memories. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.- (eds.), EMNLP, pp. 5484-5495 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/11/0211. emnlp-main.446. URL https:://///.... /aclanthoooggy..... org/2022 wewnppmmiin..... 4446-
Lee, W. Lee, J., Seo, J., and Sim, J. InfiniGen: Ef- ficient generative inference of large language mod- els with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 155-172, Santa Clara, CA, July 2024b. USENIX Association. ISBN 978-1- 939133-40-3. URL https: //www usenix.org/ conference/ oste///di22/. 2 coofereeneee/sii/2//peeetaaaaaaa pttp://eeettaiioneeeeeeee
Gradshteyn, S. and Ryzhik, M. Table of integrals, series, and products. Academic press, 2014.
Gugger, S., Debut, I.., Wolf, T., Schmid, P., Mueller, Z., Mangrulkar, S., Sun, M., and Bossan, B. Ac- celerate: Training and inference at seale made sim- ple, efficient and adaptable. attpp://githuu com/ huggi face aaceeaccelll lerate, 2022.
Liu, J., Ponnusamy, P., Cai, T., Guo, H., Kim, Y., and Athiwaratkun, B. Training-fre activation sparsity in large language models. arXiv preprint arXiv:2408.1466 2024.

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., al. Deja vu: Contextual sparsity for efficient ilms at inference time. In ICML, pp. 22137-22176. PMLR, 2023.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https: http//aaxivv./r/.e org/abs/ 1907. 10641.
abs
lama.cpp. lppma.pp.
https: /ttp://ggghhhhttg com/
Sarkar, R., Liang, H., Fan, Z., Wang, Z., and Hao, C. Edge-moc: Memory-efficien multi-task vision trans- former architecture with task-level sparsity via mixture- of-experts. In 2023 IEEE/ACM International Conference Computer Aided Design (ICCAD), pp. 01-09. IEEE, 2023.
Beazrganov/llamae app.
Ma, X., Fang, G., and Wang, X. Lim-pruner: On the struc- tural pruning of large language models. NIPS, 36:21702- 21720, 2023.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
ShareGPT. Sharegpt. сo /ttddaases//noonn2111111 Vicuna_unfiiterree 20023.
Microsoft. Deepspeed-mii. https:/ /ttp://////ttttttt/ee com/ microsoft Seezosoft/Deepspeeet DDte//peepppppp.epee
Shazeer, N. Glu variants improve transformer. arv
preprint arXiv:2002.05202, 2020.
Mirzadeh, I., Alizadeh, K., Mehta, S., Del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04564, 2023.
Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang. C. Flexgen: High-throughput generative inference of large language models with a single gpu. In ICML, pp. 31094-31116. PMLR, 2023.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXXV22233007777. URL https: //do //ooi. , wwg//o..//..ooo.ooo 48.444550/// (tt/550//axivi../ 2303. 08774.
Shin, J., Yang, H., and Yi, Y. Sparseinfer: Training-free pre- diction of activation sparsity for fast Ilm inference, 2024. URL https: / arxiv. org abs / 241 12692.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J. Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché- C'Aché-Buc, F., Fox, E., and Garnett, R. (eds.), NIPS 32, pp. 8024 8035. Curran Associates, Inc., 2019. http: http:://apperr.ooo www.::ss...eruisscccc.c. http:///pperr..nuuipsscc///asaap hulsspptoocb--n----meerrteee-ees meerformance-dee--eeeeee pdf.
Song, C., Han, X., Zhang, Z., Hu, S., Shi, X., Li, K., Chen, C., Liu, Z., Li, G., Yang, T., and Sun, M. Prosparse: Introducing and enhancing intrinsic activation sparsity within large language models, 2025. URL https:// arxiv.orge btgg/bb............. 2402 13516.
Song, X.,
Zhong, Z., and Chen, R. Promoe: Fast moe- based Ilm serving using proactive caching. arXiv preprint arXiv:2410.22134 2024a.
Song, Y., Xie, H., Zhang, Z., Wen, B., Ma, L., Mi, Z., and Chen, H. Turbo sparse: Achieving Ilm sota perfor- mance with minimal activated parameters. arXiv preprint arXiv:2406.05955 2024b.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Sun, M.. Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint Xr 1165, 2023.
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. Zero-infinity: breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '21 New York, NY, USA, 2021. Association for Comput- doi: 10.1
Tang, P., Liu, J., Hou, X., Pu, Y., Wang, J., Heng, P.-A., Li, C., and Guo, M. Hobbit: A mixed precision expert offloading system for fast moe inference. arXiv preprint arXiv:2411.01433, 2024.
Team, Q. Qwen 1.cen:.--mo:: Matching 7b model per- formance with 1/3 activated parameters", February 2024. URL wtttttps//// /ttp://wwnnlln /qwenlm.github.b...b github. aesttub.io/bboo. io/bbog/ qwen-moe/.
ing Machinery. ISBN 9781450384421.
3458817.3476205. URL https:
11 1145/3458817.3477
12

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Thekumparampil, K. K., Jain, Netrapalli, P., and Oh, S. Sample efficient linear meta-learning by alternating minimization, arXiv preprint arXiv:2 105.00000000 08306. 2021.
Tillet, P., Kung, H. T., and Cox, D. Triton: an inter- mediate language and compiler for tiled neural net- work computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop Machine Learning and Programming Languages, MAPL 2019, pp. 19,19, New York, NY, USA, 2019. Association for Comput- ing Machinery. ISBN 978 1781450367196. doi: 10.1145/
3315508.3329973. URL https: htpp:://doi.org////// 1145/3315508.1 33399973.
Tripuraneni, N., Jordan, M., and Jin, C. On the theory of transfer learning: The importance of task diversity. NIPS,
33:7852-7862, 2020.
Tripuraneni, N., Jin, C., and Jordan, M. Provable meta- learning of linear representations, In ICML, pp. 10434- 10443. PMLR, 2021.
Wang, Y., Chen, Y., Jamieson, K., and Du, S. S. Improved active multi-task representation learning via lasso. In ICML, pp. 35548-35578, PMLR, 2023.
Xue, L., Fu, Y., Lu, Z., Mai, L.. and Marina, M. Moe- infinity: Activation-aware expert offloading for efficient moe serving. arXiv preprint arXiv:2401 14361, 2024a.
Xue, Z., Song, Y., Mi, Z., Chen, L., Xia, Y., and Chen, H. Powerinfer-2: Fast large language model inference on smartphone. arXiv preprint 2Xiv:2406600628, 2024b.
Yu, Z. and Ananiadou, S. Neuron-level knowledge attri- bution large language models. Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), EMNLP, pp. 3267- 3280, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/v120224 emnlp- 191. URL org/2024. emn 1p maminn main.191/.
Zhang, Z., Song, Y., Yu, G., Han, X., Lin, Y., Xiao, C., Song. C.. Liu, Z.. Mi, Z., and Sun, M. ReLU² wins: Discovering efficient activation functions for sparse llms. arXiv preprint MXXi::2402.038M. 2024.
13

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
A. Theoretical Analysis
A.1. Preliminary
Given a vector x E IRth, we use ||ж|| == VΣ 2 to denote the two-norm of x. We use [m]] to represent the set {1,2, m). We use @ to denote the element-wil wise multiplication of two vectors or matrice.
We denote N (11, o2) to be the Gaussian distribution with mean l and variance x². We let ((x) - e-22/ and (x) = I- (y)dy to be the probability density function (PDF) and cumulative distribution function (CDF) of the standard normal distribution, respectively. Given function f. we use f- to define its reverse function.
A.2. Main Theorem
In the main paper, state the informal theorem as below: Theorem A.1 (informal), From the definition of S₁ in Equation (5), we define:
Ldown |8|| (Adown Sy (a.dww))
W
[
(12)
Lup E (dowwn agate
Sz (aup))
W Wydoww
(13)
Cgate
E (down (agaat)
Bap)
W Wydowa
(14)
Then under assumptions consistent with experimental observations, we have
Ldown I Cup
Geate-
(15)
Here we restate the theorem a formal format.
Theorem A.2. Let agaae and define Adown Agate C aup. Let Wdown E mm be the weight matrix for the down projection. From the defiiitinn of St
E R and aup E T be the activations after the S1 and the up projection, respectively,
in Equation (5), we define:
down (adown (addow))) Widdow 2
(16)
2
Cup /2 (adown Ayate 4 Se (auu)) Wrdowa 2
(17)
2
Egate
E (adowa (aggue))
C aup) W
(18)
We assump that all the Waj in Widoon are i.i.d. and satisfies Wij N( Bi (i [m].j [n]). Simillry,,,,,, for all E [m], we assume a are i.i.d. and satisfies agate, (O, ggac)) And for all E [m], we let aup, are i.i.d. and agate, = Zgatee - c for some constant independent. Then if we keep the threshold of sparsity such that (1 71) 100% elements of the activations set to zero in S,, we can explictly write out Cup and Egasc as follows:
O, where ggus,, i satisfies exponential distribution with parameter X. We also assume Oup and Ggar
Cup nmo 2
2
2
2c
+ (1-17 1- -22,p(2n))
(19)
X2
Lup = nmoo oup [c -)
-
tn
+ (?))
- -
(2 +2 9n + +
(20)
12
X
where 2, = ---10 (1 - 3), x(x)
- 22/2 and an
de sinh = Furthermore, dc 2 2, and E [e_1, 1/2].
we can obtain
Ldown
Cup
Lgate.
(21)
Remark A.3. Here discuss the rationality of the theorem assumptions: First, the choice of La loss, independence between random variables, and gaussian assumptions are widely used in machine learning theory community (Tripuraneni et al., 2020; 2021; Du et al., 2020; Thekumparampil et al., 2021; Chen et al., 2022; Wang et al., 2023), and from Figure 2 in the main paper, we can observe that the distribution of elements in activations after up projection satisfy gaussian distrbution.
On the other hand, the shifted exponential distribution on gate-pree pate-prooection activations mainly comes from the property of SiLU function. The distribution of gate-projection elements before and after SiLU functions are shown in Figure 11. We find
14

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
F2T ohichhhhhl o.0.g
PDF Stawit 05 6.631
Orance Por
Within Threshold
65-
Threshold
-20
05 123 0.0
3.20
-2.5
2,5
FOP x
2 (0w.000000. aw3,21
Within Threshold Outside Threshold
0.10
0.00
-5 -2.5
2.5 5
Figure 11. (Left) Distribution of gate-prou projeccoooonnn elements before (Ist column) and after (2rd column) SiLU functions. The first row is for 15-th layer and second row for 31-th layer. We can see that before SiLU function, the activations are roughly shifted gaussian, while after SiLU, they has very high probability density value I z -0.28, which is the minimum value of SiLU function. (Right) Simulation SiLU outputs on shifted gaussian variables. find that for input I ~ A o²) with reasonable a (as the left figure), the outputs after SiLU function has similar truncated unimodal distribution, and can be well fitted by shifted exponential distribution, i.e., a = x c for x satisfies exponential distribution with parameters
and shifted constant = 0.28.
that the distribution dgate,y can be well fitted by shifted exponential distribution, i.c., a = I for x satisfies exponential distribution with parameters À 2 10 and shifted constant e 0.28. Here is the negative value the minimum of SiLU function, and thus is fixed value. Therefore, our assumptions are consistent with experimental observations, and we can that condition Ae 2 2 is also satisfied. We add this data distribution assumption mainly because the thcoretical difficulty for handling the reverse function of SiLU function, and we will see that even with this simplification, the proof is still non-trivial.
Then comes the proof of our main theorem:
Proof of Theorem A.2: From assumptions and Lemma A.11, Lemma A.12, we have
Laown no
up
E
adown
Sy (B
||2
(22)
Cup no u
adown
agatc
3 (aup)
2
(23)
Laate = no W
up Adown
Se (a
b aup
(24)
Note that obviously, for any vector and any fixed ratio of non-sparsity rate 17, (a is the sparsified vectors with maximum norm, and all three kind of sparsification strategies have the same non-sparsity ratios, so must have
Btoon
S, (aoww))
<
agate
O S, (amp))
(25)
adown S [[Jow))
< Adown
Sy (agutt))
O aup
(26)
On the other hand, note that
E
Adown
agate
2 Se
11
E agate (auu - (aup)))1
(27)
mi Agate." (Oup.i Se Se(uup.))) (i.id.)
(28)
[a TE [(app.. St Se( (independence, Lemma A.12, Lemma A.11)
(29)
15

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Similar formulas hold for Laae- Then combining Lemma A.4, Lemma A.5, and Lemma A.9, can get the results.
A.3. Technical Proof
Lemma A.4. Assume that random variable -N(0, For given v (0,1), define the threshold by such that P P(aa] tn) n7. Then if we define the inverse sparsity function St, (a) as
{00 a otherwise.
0, if by:
Se,
(30)
Then, the threshold in and the expectation E E.S.,(a)) are given by
t = 0 $ 1
(31)
and
E|St, =2 (zn)( (1-1)- (() E[(a],
(32)
where Zn 1-110111 (1 5) ln/0. And as defined in Appendix A. 6(x) Jeze
x21 - is the PDF of the standard normal
distribution, $ (-) denotes its inverse cumulative distribution function (CDF).
Proof. Threshold t.p
Given a ~ _N(00²), standardize a by defining Z 3, so that Z VV 1). Therefore:
t P
Due to the symmetry, we have
1 0
2P
2 >
S1 7 P
Z>
$ 2
Therefore have
tn
--1 (1-1) 2)
+ by = o x-
(1-
=
σ
Expectation E[S., (=)²): First note that
(a)²) -E::EE. [ [a² 1 [|a| <t,
(33)
= =E[a²] E Ia² [1a(200)]
(34)
0² [a² 1 (1
(35)
And let a/a N(0,0²), 2n = ln/o, we can obtain
=22 20² { 220(2)de
E aa [(1[220,)] =
(36)
in
= (0-([[=2(22])) + L o(z)dz}
(37)
-202 (zz) + Q(z,))
(38)
where Q(=) = (=)). Substituting = = =- (1
2), we get Q(zy))
= ./ Therefore, finally we have
E|S., (a) = 0² - 202
Zn 2 + "
= 0² [1 22,, cb zny)
(39)
/2T
2
This concludes the proof.
16

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Lemma A.5. We define bn and St, similarly as Lemma A.4. If satisfies exponential distribution with parameter X, and c.for some constant e. (c 2 tn). Then we have
2c
Ela²]
+
(40)
X²
A
And,
(u)9 = A0t, -c)
12
2
b + (?))
e
12
+
- ln
1
(41)
Furthermore, in satisfies
x sinh -1 ( 11 2 exp(-22cc)
bg
(42)
In(n)) G
otherwise.
where sinh (x) = In(x + +I2+1).
Proof. Expectation E[a²]:
Since Exp(A), we have [[x]] = 1 and Var(z) xz.
For a c
1
E|a] E[x]
Var(a) Var(z) X2
(43)
Thus:
2
B|a²] = Var(a) + (E(a))2 1
X
+
c
=
x A
+ c²
(44)
Expectation (a){;
By definition:
|oatee (x c) -he
cotin
(a) [[al/(t,)] =
x dx.
(45)
where a = x c. Recover it to the integral on a:
[a) = de dc 1 dac-dada. da.
by
de-
(46)
Exploiting symmetry (valid if 2 kn)) by
0 are- Da-daa + jq
1'r a²e- da da
a²e da
(47)
-tg
1-a a²(
(rv ba 2 )d
2
(48)
2
/ tn a² caad(aa))a,
(49)
where cosh(Xa) ea Aa Using the integral formula (Gradshteyn & Ryzhik, 2014):
f
à cosh(Aa)da= = 213 ((aa)2 - 2Aa + +2) 213
λdaa
AAa (Aa)) + +2a + +2) + C,
(50)
Then we have
1'1 d² coshh(AAa)
E E|S,, (a)?1 2Xe -xe oo
(51)
(e ((tt,)) 2- -2 + 2)
((Xt,)) + +2t, + 2)} (2) e -ot((a) 2
e Ne,
(52)
X²
(tn--)
2
2
in
+2 l/
+12)
(53)
2
12
x
17

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Determination of l,=
The threshold satisfies P([a] 2 tn) 17. For with Exp(A):
P(x 2e by) P(x S bn) e -lecte) + (1- 1--e-Nootu)) = 77.
(54)
Case 1: S c- 22c
When bn > c, the lower tail vanishes:
P(a (e2ctty) = - -Nott()) =1 = in In(n) c.
1
(55)
It's clear the condition in this case is: bn 22 7 2 eexp(-
Case 2: 17 2Ac
When in I c. both terms contribute:
-eeee" (e-ee-,
Mr, - eMii) 17,
(56)
Aa e -ha = (1 n)
(57)
Note that sinh(y) = e 2
2
(1 -1-10 -)nee sinh(At,,)
1 (1-1) 17 exce
tn -n sinh
(58)
2
It's easy to check now we have 7j 2 exp(-2Xc), This concludes the proof.
Remark A.6. From calculation, we can have some approximation of the key terms in Lemma A.5: If Ac >> 1 and 1], then we have
17 E
t, R c+ 2 In(1 In(1-n) n1) 11
(59)
(1- 1) (2+ -2
)
2 t
(60)
Then if E [exp(-2Xc) 1/2]. we can see L, is very close to c, which matches our experiment observations in Figure 11.
Then we compare the second moments calculated above.
Lemma A.7. Let qn, = y sinh sinh-'((" '('="'() ""), On = p(an - 1), and h, = =((g + 1). Then if 2 2 and r) E [e-2p, 1/2], we have
In(1 1)
0 1
< on <1
(61)
p
In11 1)) 9n < 0,
2p+In( 11 < hn < 2p
(62)
Proof. Note that
sinh (x) In(
x2 + 11 2 l2x)
(63)
So have
(1- 17 cP
2. 1 1-11 11 cP
In(1 17)
9n sinh-
(12
> In
+1>0
(64)
p
p
And the last inequality holds when 7) 0.5 < - 2 < - c p. For the upper bound, from 17 2 22,, we have:
1- e" <
1 1-c 2
2p
e²
6
sinh(p).
(65)
2
2
18

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Since sinh -1 is strictly increasing and sinh (inh-((sin((2))) p. have:
9n sinh -1 1 7 e²) <1 1.
(66)
2
And thus inequalities for 9r, also hold.
Lemma A.8. For p > O, the function
f(p) -2p + 2p + p² 2p + p²
(67)
strictly decreasing. And therefore when p 2 f(p) S f(22) = 55 4
Proof. Let (p) = =p² 2p + 2 and D(p) = p² - 2p 2p+ 2. The derivative of fp) is:
[p) - (p) - (p))
f(()= 2p
(68)
Dpp)
(2p+ 2) - -2p + 2) - (p² + 2p + (2p 2) 2(p² 2p + 2)(p² + +p + 2)
(69)
Dp))
4p² + 8 2(p² +4)
2p.
(70)
D
2p. 2p 4p²
(71)
D(p) 2
0.
(72)
Therefore, f() is strictly decreasing.
Lemma A.9. We define
F(1) = - 17 22, (zn)
(73)
where 2n 10010111 3) and o(x) = 1 x² 22/2, And we define
1/2m
G(n,p) = -p(an-11 11( 2/p² 2q pp + o2
2/p² -2/p +1 (n) - -p((tan))
2/p²+ + -2 p + Qy
(74)
2/p² 2/p +
where 9n = sinh -11101 e²). "e)) Then i 2 2 and E (e-1,0.5) we have F(1) < G(TI,)).
Proof. Let 9n p(an 1) and h, p(gn + 1). Then for the first term we have
G1(7,)) = clan 11 () 2/1² 24 p + In
(75)
2/p² 2/p + 1
2 - -2pq, + 12
e
(76)
2 -
pp² + 2p 2pgn k2pp
= - +
(77)
(p (p-13 1)2 +
pl- p - 2- 24, + Pppl]
c0
(78)
(p 1)2 +1
pl 2 1) + 2(1 9m)]
c01(1
(79)
(p 1)2 +1
= + p. (gn 1): (p(g, + 1) - 2)
(80)
(p - 1)2 +1 1
c1 + (y - 2)
(81)
(p 1)2 +
19

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Comparison of F(n) and G(n, p)
1.2
F(n)
G(n.==1)
1.0
G(n.p-2)
G(n.p=3)
G(n,p=5)
0.8
G(n.p=7)
G(n,p-10)
G(n,p-15)
0.6
G(n,p-20)
Function
0.0
0.2
0.4
0.8
Figure Comparison between F(n) and G(n.p). When p and 17 E [e [e-*,0.5), we can see that G(n. 2 F()). And actually this increase as increases.
From Lemma A.7, it's easy see that
(2p 2)
Gi(n.)) (1 1)) (1 In(1 1)
(82)
(p 1)2 +
>1 1 (1 11) In(1 -1)
(83)
Here we use In(1 - 7) < 0 and the property that f(p) = 2p-2 2
S f()) 1.
(p-1)2+1
Similarly, for the second term of G, we have
2/1² + 22 /p + d
G2(1.p) = pp((tn))
(84)
2/p² 2/p +
2 2p4r 22p2
=emla
(85)
- 2p p2
- 11) 2p + 2p + p?
(Lemma A.7)
(86)
2 2p + p2
< 5e 411 - 11) (Lemma A.8)
(87)
Therefore we have
G(n.p) F(n)) Gi(n.p) - G2(n.) - F(n)
(88)
1- (1 - 1) In(1 1)) 5e 4 1)) [1 17 22, [2n]]
(89)
22,0(zn) - (1 - 15e +_ In(1 7)]
(90)
Q(1)
(91)
where 2g --11 1--11111
(1- 7)
and (xx)=
-e 12/2
V2x
20

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Note that for m(x) (2)) m (x) = (1 - -1- (x), and thus ma() is strictly decreasing on x E (-x, -1) (1, and
strictly increasing on (-1,1) 1). And further note that $ (-) is strictly decreasing and continous, then since 1f [e- 0.5],
Zn E ④ (0.75), - (1 e - /2)]. where & -- (0.75) ~ 0.674 and 4 - e- 22/2))))))))))))) > & (0.975) ~ 1.960 > 1.
Therefore, m²(n) = (m,) first increase, and then decrease on 17 E [e 00,,,,000. 0.5] 3 (e -,-,,0.. 0.5].
Similarly, for n(()) = (1 - 15e 4 In(1 - n)), n' (1) - 5e + In(1 - n)) We can see n'))) consistently increase for r E [0, (e [e-2r, 24, 0.5] C -ea- 0..5), where 17" = exp(5e
1) Rs 0.632.
Combining these, to prove that Q(1) (Equation (91)) is positive on 17 E [e 4 0.5]. we just need to make sure m2(e +) > n(e-1) and m2(0.5) > n(0.5). And from calculation we get
m²(e R 0.116 n(e +) 0.108,
m2 0.429 nn(05) x 0.392
(92)
Therefore, have G(n,p) F(n) Q11) 0 for all p and 1/ (e ,,05). This completes the proof.
Remark A. Lemma A.9 is the key lemma of the whole proof, which compares the variance caused by the inverse sparsity function of up and gate function. We can further visualize F(n) and G(n,P) in Figure 12. We see that the visualization results match our proof, and showing that larger will has larger G(T,P) F()) values.
Lemma A.11. Assume that E Imm and W E R are random vector and matrix whose elements are independent to each other. And all the Way in W satisfies Wiy ~ ((0,o²)( E [m].j E [n]). Then we have
B|ll 12:1 no =
(93)
Furthermore, if all xy are ii.dd (independent and identically distributed), with mean 0 and variance o then
B|||>W||| nmo? -
(94)
Proof. Let W, denote the j-th column of matrix W. Due to independence:
E L 3 = E Σ
(2
Σ E [z] = nE [(
(95)
Similarly by independence between x and W.
m
E | E
Σ
x. Wis
(96)
Bla?] EW? +
B[ngar | Waa]
(97)
Σ
Bin 0
(98)
o 'E
(99)
0² 2 B|
(100)
Combine them and we get the first equation. And the second equation is totally similar.
Lemma A.12. Ifa (O R),, is random variable independent to a, sparsity function S, is defined as Equation (5). Then for any t > 0, E[(a- Sz(a)) b] = Ela- (b - [b))] = 0.
Proof. Just need to note that and b are independent and from symmetry, E[)) 0,, and then the proof is trivial.
21

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
B. Sparsity Insensitivity of the Up Projection in Dense LLMs
evaluate the sparsity sensitivity the up projection on LLaMA-3-8B (Dubey al., 2024), with results consistent with our findings on MoE models..ome results are presented Table 2.
Table 2. Sparsity Insensitivity of the Up Projection in Dense LLMs
METHOD
BOOL-Q
SCLQQ
OPENBOOKQA
WINOGRANDE
ARC.CHALLENGE
ARC_EASY
AVERAGE
BASE
0,8187
0.961
0.370
0.7348
0.5026
0,8085
0.6993
UP-90%
0.7116
0.925
0,300
0.6717
0.4002
0.7066
0.6192
DOWN-90%
0.7780
0.959
0.336
0.6922
0.4241
0.7504
0.6566
C. Downstream Tasks Performance Details
We present the detailed evaluation results of FloE and baseline methods across downstream tasks in Scction 4.2 in Table 3.
We evaluate performance on three complex downstream tasks: MMLU, GSM8K, and HumanEval Table 4, which pose significant challenges for small distilled models. Our method outperforms smaller models of similar scale across tasks. Despite retaining only 93% of the base model's accuracy on these tasks, significantly surpasses other baselines (Mistral-7B and Llama3.2-3B).
We also evaluate the impact of different projection matrix sparsification sensitivity on downstream tasks in Table 5.
Table 3. Performance of Downstream Tasks under Different Compression Methods
MMLU 5
BoolQ
SCIQ
QA
WG
ARC-C
ARC-E
AVERAGE
MIXTRAI--8+7B
0.695
0.853
0.968
0.354
0.762
0.567
0.843
0.720
HQQ INT3
0.608
0.809
0.955
0.292
0.712
0.481
0.800
0.665
CATS-80%
0.617
0.792
0.903
0.322
0.670
0.515
0.782
0.657
CHESS-80%
0.612
0.802
0.912
0.302
0.694
0.498
0.781
0.657
FLOE-WT-8805
0.654
0.829
0.944
0.344
0.732
0.532
0.816
0.693
HQQ INT2
0.234
0.485
0.331
0.144
0.493
0.212
0.279
0.311
CATS-
0.377
0.704
0.826
0.272
0,586
0.442
0.709
0.559
CHESS-90%
0.424
0.727
0.839
0.278
0.604
0.410
0.694
0.568
FLOE-W"-990
0.601
0.787
0.933
0.312
0.670
0.497
0.788
0.656
FLOE-80%
0.605
0.810
0.951
0.336
0.717
0.509
0.803
0.676
PLOE-90%
0.531
0,835
0.952
0.276
0.695
0.458
0.762
0.644
Table 4. Performance Comparison of Models under dense model
MODEL.
CSM8K@8(ACH)
HUMMNEEALL@OFE O(PAss
MMLU@ (5AcC)
AVERAGE
BASE MODEL
58.0
33.5
69.5
53.67
FLOE-80
51.7
32.3
65.4
49.80
MISTRAL.-7B
39.4
29.2
62.5
43.70
FLoE-90
40.9
30.5
60.1
43.83
LLAMA-3.2-3B
26.6
25.6
56.4
36.20
D. Sparsification Insensitivity of the Up Projection in More MoE models
The sparsity rates for Phi-3.5-MoE-Instruuc are presented Table 6. Due to the smaller hidden layer dimensions of DeepSeek V2's experts, the sparsity rates tested were correspondingly lower, with results in Table
22

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Table 5. Performance of Downstream Tasks Under Different Sparse Strategies
0
50%
608
70%
808
90%
GATE
0.7228
0.7140
0.7035
0.6640
0.5897
UP DOWN
0.7247
0.7199
0.7148
0.7038
0.6971
0.6646
0.7233
0.7210
0.7201
0.7194
0.7054
Table 6. Sparsification Sensitivity of MoE Models
MODEL
OPERATION
50%
60%
70%
8.8
908
GATE
5,8151
6.3379
7.2570
9.1439
18.5280
MIXTRAL-8X7B
DOWN
5.1583
5.2101
5,3252
5.6147
6.5511
UP
5.3164
5.5390
5.9795
6.9141
9.1250
GATE
5.4386
5.7809
6.4006
7.6114
11,2538
PHI-3.-MMOE-INSTRICT
DOWN
5.1495
5.2051
5.3255
5.6377
6.6271
5.4092
5.6855
6.2642
7.3284
10.2146
E. Quantization Insensitivity of the Up Projection in More MoE models
Besides Mixtral 8xBB, we also evaluated the quantization insensitivity the up-projection Phi-3.5-MoE-instrucc (Abdin al., 2024a), DeepSeek-Men MoE- 1o0E-16--Bast Base (Dai al., 2024), and Qwenl.5-MoE-A2. (Team, 2024), all of which show that the up-projection the least sensitive to ultra-low-bit quantization. The results are shown in Table 8.
F. Memory Footprint of Learning-based Contextual Sparsity Predictors
While some existing approaches rely on learning-basse prediction methods (Liu et al., 2023; Shin et al., 2024), these methods often incur significant memory costs. For instance, in the Mixtral-8 7BB model, where the hidden state dimension is 4096 and the gating weight matrix W gate an MIP block has size requires (4096 1024 + 1024 14336) (bytes) 256 9GB of memory when the rank of the DEJAVU predictor (Liu et al., 2023) set to 1024. Similarly, although SparseInfer (Shin al., 2024) achieves more memory-efficient design by storing only the sign bit of each element, compactly packed into 32-bit variables, it still incurs memory footprint of 14336 x 160 x 4 (bytes) x 256 2.19GB.
& = 4096 14336, PowerInfer (Xue et al., 2024b)
23

---

FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Table Sparsification Sensilivity DeepSeek-V? V2
MODEL
OPERATION
30%
50%
70%
GATE
8.6434
8.8331
9.8083
DEEPSEEK- V2
DOWN
8. 8.6264
8.6400
8.7223
UP
8.6456
8.7818
9.2083
Table Quantization Sensitivity MoE Models
MODEL
OPERATION
INI8
INI4
INT3
INT2
INT1
GATE
5.768
5.785
5.952
6.623
608.7
PII-3.S-MOE-INSTRUCT
DOWN
5.768
5.772
6.067
7.733
365.9
UP
5.769
5.788
5.899
6.599
209.3
GATE
8.476
8.497
8.558
9.020
112.8
DEEPSEEK-MOE. 16
DOWN
8.476
8.659
9.364
27.70
350.1
UP
8.476
8.489
8.602
9.090
83.57
5.119
5.158
5.310
6.245
1130
MIXTRAL-8X77
DOWN
5.121
5.270
5.968
14.36
1910
UP
5.119
5.151
5.281
6.177
520.1
GATE
9.227
9.258
9.364
10.72
102.0
OWEN-L.5-A2.7B
DOWN
9.224
9.419
9.655
12.53
138.4
UP
9.226
9.270
9.426
10.19
71.06
24


Reviews:
[{"review_id": "qhnvvPrtPb", "reviewer": "Reviewer_oREx", "summary": "The paper presents FloE, a system for on-the-fly inference of Mixture-of-Experts (MoE) models on memory-constrained GPUs. It addresses the challenge of high memory and I/O overhead in MoE inference by introducing a hybrid compression strategy, which combines contextual sparsification of gate and down projection matrices with ultra-low-bit quantization of up projection matrices. Additionally, FloE leverages learning-based expert sparsity predictors to reduce the overhead of expert offloading while maintaining inference efficiency. Experimental results show that FloE achieves a 9.3× parameter compression per expert, enables inference on a GPU with only 11GB VRAM, and delivers a 48.7× speedup over DeepSpeed-MII on an RTX 3090, with only 4.4%–7.6% accuracy degradation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer oREx,  \n\nThanks for your careful review. We summarize and address your key concerns as follows:\n\n**`[Suggestion 1]` A theoretical analysis of how expert sparsification and quantization affect routing distributions and model expressiveness.**\n\n**`[Response]`** In **Appendix B**, we provide a **detailed theoretical proof explaining the sparsity sensitivity differences observed in models with SiLU activations**. Based on the distribution characteristics of the three matrices, we demonstrate why the output activations of the up projection exhibit lower sparsity sensitivity. Additionally, **in lines 190 - 201 (Section 2.2.2)**, we offer qualitative analysis regarding the quantization sensitivity differences.\n\nThe impact of compression methods on routing was evaluated by performing 500 forward passes on the ShareGPT corpus. We measured the shift in routing logits caused by compression and found the results to be consistent with downstream task performance. Our method induces significantly less distortion to the routing distribution compared to other baselines with the same sparsity ratio. This observation aligns with the theoretical proof provided in Appendix B, further validating our approach.\n\n**`[Suggestion 2]` A deeper investigation into failure cases, such as whether certain compressed experts become underutilized or degrade faster than others.**\n\n**`[Response]`** In our current compression scheme that applies a uniform sparsity ratio to all experts, we have supplemented additional tests measuring the L2 norm of output errors before and after expert compression to investigate failure cases (as shown in [expert sparsity loss figure](https://anonymous.4open.science/r/floe-5843/pdf/expert_loss_mse.png)). Our findings revealed that **compression of experts in the 30th and 31st layers induces the most pronounced performance degradation**. When preserving these critical layers as non-sparse, the model showed improvements of 1.3 on the **MMLU** benchmark and an average gain of 1.66 across six CSR tasks (**BoolQ, SciQ, ARC-C, ARC-E, QA** and **WG**). \n\nThese empirical results substantiate our hypothesis regarding layer-specific compression sensitivity. Accordingly, we develop to implement more fine-grained compression by selectively reducing the sparsity ratio for these critical experts, which can better preserve model performance while maintaining compression benefits.\n\n**`[Suggestion 3]` Additional studies on compression effects across different MoE models and more diverse tasks, to validate generalization beyond Mixtral-8×7B. For instance, how does DeepSeek-v2 or v3 behave in your system? Other open-source MoE models also include the Arctic model from SnowFlake.**\n\n**`[Response]`** In the **Appendix E and F**, we evaluate the sparsity sensitivity of **Deepseek V2** and **Phi 3.5**, as well as the quantization sensitivity of **Phi 3.5**, **Qwen 1.5A 2.7B**, and **Deepseek MoE**. **These results are consistent with the conclusions  on Mixtral 8×7B, further validating our findings**. Furthermore, we have supplemented the experimental results for MMLU (5-shot), GSM8K (8-shot), and HumanEval, providing a comprehensive comparison between FloE and the baseline models. These results demonstrate the strong generalizability of our method across different models and downstream tasks.\n\n**`[Experiment Issue]` Is the accuracy loss caused by FloE's compression sufficiently competitive compared to model distillation or direct deployment of smaller models?**\n\nComparing with smaller non-compressed models is indeed crucial. However, due to time and resource constraints, we were unable to distill smaller models and instead chose the more advanced **Llama3.2-3B** and **Mistral-7B** as baselines for comparison. More details refer to the response to `Reviewer bhnT`.\n\nWe evaluated performance on three complex downstream tasks: **MMLU**, **GSM8K**, and **HumanEval**, which pose significant challenges for both small distilled models and sparse methods. Our method not only exhibits the least performance degradation among sparse baselines but also outperforms smaller models of similar scale across tasks. Despite retaining only 93% of the base model's accuracy on these tasks, it significantly surpasses other baselines.  (Resualt Table in response to `Reviewer bhnT`)\n\nThus, we argue that appropriately compressing larger models is more competitive than directly deploying or distilling smaller models, especially in scenarios involving resource-constrained environments and complex reasoning tasks.\n\nCheers,\n\nAuthors\n\n---\n\n### `Results Table`:\n#### **Router Logits Shift**\n| Sparsity (%) | 50 |60|70|80|90|\n|-----|----|---|---|---|----|\n| FloE  | 0.9941 | 0.9888 | 0.9794 | 0.9607 | 0.9212 |\n| CATS   | 0.9744 | 0.9626 | 0.9455 | 0.9181 | 0.8608 |\n\n#### **Downstream Task Performance of Pin Higher Layer**\n| Model| MMLU-5 shot (Acc)| CSR (Acc)|\n|----|---|---|\n| FloE-80 | 65.4| 68.77|\n| FloE-80 pin 30 & 31 |66.7|70.43|", "claims_and_evidence": "The paper makes claims about compressing experts in MoE models while maintaining accuracy, demonstrating 4.4%–7.6% accuracy degradation through empirical results. However, this trade-off is non-trivial, as it remains unclear how compression affects routing behavior and knowledge retention within experts.\n\nA key concern is the lack of theoretical justification for why the proposed hybrid compression strategy—contextual sparsification for gate/down projections and ultra-low-bit quantization for up projections—preserves model performance. While empirical results show competitive accuracy, the paper does not provide a formal analysis of how different compression techniques influence expert activation patterns, routing stability, or downstream generalization.\n\nWithout a deeper theoretical understanding, it is difficult to assess whether the observed accuracy retention is inherent to the method or merely specific to the evaluated models and tasks. This raises concerns about the generality of the approach across different MoE architectures (e.g., DeepSeek-MoE, Switch Transformers) and long-form inference stability, where routing mispredictions might accumulate.\n\nTo strengthen its claims, the paper would benefit from:\n- A theoretical analysis of how expert sparsification and quantization affect routing distributions and model expressiveness.\n- A deeper investigation into failure cases, such as whether certain compressed experts become underutilized or degrade faster than others.\n- Additional studies on compression effects across different MoE models and more diverse tasks, to validate generalization beyond Mixtral-8×7B. For instance, how does DeepSeek-v2 or v3 behave in your system? Other open-source MoE models also include the Arctic model from SnowFlake.", "methods_and_evaluation_criteria": "The paper presents a strong evaluation setup, including multiple baselines (DeepSpeed-MII, Mixtral-Offloading, Fiddler), datasets, and diverse GPU configurations (RTX 3090, A100, H100, A6000). This comprehensive benchmarking makes the results more convincing. However, testing on more advanced MoE models (e.g., DeepSeek-MoE, Switch Transformers) would provide stronger evidence of FloE’s generalizability across architectures with different routing mechanisms.\n\nA 7% accuracy loss is non-trivial, raising the question of whether model distillation could achieve a similar efficiency gain with lower accuracy degradation. Comparing FloE with distilled MoE models or smaller dense alternatives would clarify its trade-offs. I would not expect a real comparison here but I would need some justification that 4 -7 % accuracy loss is something we can expect better than distillation and make the proposed method valuable for the MoE model community (the proposed approach is indeed easier to be deployed since it does require the expensive distillation process, which I have been convinced). Additionally, the paper could discuss how much accuracy loss is acceptable in real applications, as the impact varies across tasks.", "theoretical_claims": "Like I said above, strong theoretical understanding would significantly strengthen the soundness of the proposed approach.", "experimental_designs_or_analyses": "Check my review above.", "relation_to_broader_scientific_literature": "Yes, it is related to a broad scientific literature, since it aligns with key ML topics, such as MoE architecture, compression, sparsity and system designs.", "essential_references_not_discussed": "This paper did a good job of covering most key references in MoE inference systems.", "other_strengths_and_weaknesses": "See my review above.", "other_comments_or_suggestions": "See my review above.", "questions_for_authors": "See my review above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OgdLgpbPyk", "reviewer": "Reviewer_u26i", "summary": "The paper introduces FloE which is an inference system to run MoE models on memory constrained GPUs. FloE includes various techniques (1) compression coupled with (2) dual predictors.\n(1) the work suggests that there is an internal sparsity in experts that can be set to zero during inference by using magnitude based sparsification. Also, the work suggests using various bit-widths to quantize the different projection matrices can provide good perplexity.\nAlso, the paper proposes some system optimizations such as development of efficient sparse kernels and compact asynchronous transfer.\nThis results in a significant speedup in running MoE models.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer u26i,\n\nWe sincerely thank you for your thorough and thoughtful review of our submission. Your feedback is invaluable, and we will provide a concise response to the knowledge background and GPU utilization concerns you raised.\n\n**`[Minor issue]` The paper is very difficult to read as it skips proper description of the terminologies used in the paper. Although it is understandable that it is challenging to put a lot of content into short 8 pages, I believe a little section to describe some background would be very helpful for readers.**\n\n**`[Responce]`** Thank you for your understanding regarding the difficulty of including rich content within the 8 pages. We agree that a more thorough background would be helpful for readers. As such, we have placed the related work on expert offloading and sparsity in LLMs in Appendix A, with a concise introduction and positioning provided in **Lines 105-108, Section 2, Page 2 of the main text**. We would be happy to **move Appendix A to Section 2** if given any opportunity to revise.\n\nIn **Appendix A**, we introduce **experts offloading** and **sparsity in LLMs**, two key areas critical to the efficient deployment of large language models. Experts offloading addresses the memory bottlenecks caused by the massive parameter counts in Mixture-of-Experts (MoE) models, with frameworks like Llama.cpp and DeepSpeed Inference transferring expert weights between VRAM and DRAM. However, limited PCIe bandwidth leads to communication delays, and existing prefetching strategies face tradeoffs in accuracy, latency, and scalability, especially under multi-expert activation. Sparsity in LLMs, on the other hand, minimizes computational and memory overhead through techniques like weight pruning and activation sparsity. While these methods show promise, they are often hindered by performance degradation, hardware inefficiency, or limited adaptability to modern architectures like SwiGLU.  \n\nOur work builds on these foundations by addressing the gaps in both areas. We propose an approach for on-the-fly inference in MoE models and explore contextual sparsity techniques tailored for modern architectures. By optimizing parameter utilization and reducing hardware bottlenecks, our solutions aim to **improve the efficiency, accuracy, and practicality of deploying MoE models in resource-constrained environments**.\n\n**`[Key Issue]` Key question I have about the paper is on the GPU utilization w/ and w/o this work. Although memory is a big bottleneck in MoE adoption is \"the large size of activated experts overburdens the limited PCIe bandwidth\", another big contributor is the low GPU utilization that spikes up the cost of running services using MoE. I think it would be great to understand how GPU utilization is impacted.**\n\n**`[Responce]`** **Two challenges stemming from distinct scenarios.** Your analysis of the bottlenecks in MoE deployment is insightful. Indeed, PCIe transmission bandwidth and GPU utilization represent two critical challenges that arise from **distinct scenarios and are driven by different underlying causes** in the current deployment of MoE models.\n\n**The first challenge, which occurs in scenarios with limited GPU memory resources, compels the heterogeneous storage of model weights.** Consequently, substantial latency occurs during inference due to the transfer of parameters between DRAM and VRAM. \n\n**The second challenge arises from an imbalance between computational load and memory usage in GPU-deployed MoE inference scenarios.** This frequently leads to suboptimal GPU utilization, especially when processing small-batch requests, leaving GPU cores underutilized.\n\n**Our work primarily focuses on addressing the first challenge.** Our work reduces the latency of model inference caused by weight transfer while maximizing the preservation of model accuracy. This is achieved by designing an ingenious hybrid compression strategy and pairing it with a corresponding prediction mechanism. Though our solution does not directly resolve GPU utilization concerns, we would be happy to share additional insights into how our approach interacts with GPU performance. We tested on the ShareGPT corpus and measured the **GPU utilization rate at 63.2%** during 50 forward passes with 20 tokens each. **Given the relatively large matrix dimensions of MoE models compared to consumer-grade GPUs, we have observed that GPU utilization remains reasonably high in practice.**\n\n**The research on the second challenge is orthogonal to our work.**  The challenge of GPU utilization that you raised is indeed one of the major challenges in MoE deployment. It is also a key direction for our future research and exploration. We deeply appreciate your feedback and hope to continue advancing our understanding as we work toward optimizing MoE deployment.\n\nWe hope the above clarifications address your concerns and would greatly appreciate your further feedback.\n\nCheers,\n\nAuthors", "claims_and_evidence": "It seems like the main claims are backed by data Figure 2, 3.\nThe ideas seem to make sense.", "methods_and_evaluation_criteria": "Evaluations seem to be reasonable.", "theoretical_claims": "I was not able to scrutinize the theoretical proofs in the supplementary material.", "experimental_designs_or_analyses": "More data may be helpful but overall it seems to be reasonable.", "relation_to_broader_scientific_literature": "Common understanding in the field is that the mixture of experts are good for training high performance models. However, it still incurs high memory usage and it may lead to low GPU utilization. The paper focuses on the first issue and claims that the large size of activated experts overburdens the limited PCIe bandwidth is a challenge that limits the use of MoEs in latency-critical scenarios.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "+ Ideas presented in this paper may help reduce the memory burden of running MoE based networks.", "other_comments_or_suggestions": "Minor issue, but the paper is very difficult to read as it skips proper description of the terminologies used in the paper. Although it is understandable that it is challenging to put a lot of content into short 8 pages, I believe a little section to describe some background would be very helpful for readers.", "questions_for_authors": "Key question I have about the paper is on the GPU utilization w/ and w/o this work. Although memory is a big bottleneck in MoE adoption is \"the large size of activated experts overburdens the limited PCIe bandwidth\", another big contributor is the low GPU utilization that spikes up the cost of running services using MoE. I think it would be great to understand how GPU utilization is impacted.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H78CSV6D2U", "reviewer": "Reviewer_bhnT", "summary": "The paper introduces an on-the-fly inference system (called FloE) for Mixture of Experts models on memory-constrained GPUs. The utilization of the limited GPU memory is optimized by a hybrid compression scheme, especially focused on intra-expert sparsity - while still utilizing inter-expert sparsity. When the memory required by all the experts exceeds the GPU memory, some of the experts parameters are offloaded to CPU memory, and then loaded onto the GPU when required. This shifts the LLM decoding bottleneck from memory-bound to I/O-bound, because of the bandwidth limitations of the PCIe bus.\n\nA study of the activation distribution within experts shows that many activations are very close to zero. This led the authors to propose a magnitude-based activation sparse strategy, where activations close to zero are set to be exactly zero, resulting in the elimination of the corresponding weight computations (and therefore transfers) during inference.\n\nThe experimental evaluation shows some of the advantages of FloE, and compares it with other schemes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer bhnT：\n\nThank you for recognizing the importance of the research problem, as well as the soundness of our methodology and experiments. Below, we address the key concerns regarding **MMLU performance** and **comparisons with non-compressed baselines**:\n\n**1. FloE achieves competitive performance on MMLU.** \n\nAs you noted, MMLU performance is relatively sensitive to sparsity, which is reasonable given its complexity as a large-scale, multi-task NLP benchmark covering 57 subjects. Similar **sparsity-sensitive trends** are observed in other challenging tasks such as GSM8K (math reasoning) and HumanEval (code generation), which **aligns with the observations in Sirius [1]**.\n\nHowever, FloE demonstrates **best performance retention** compared to other baselines (see Figure 10 of original manuscript). Specifically, as shown in the table below, \nat **90% sparsity**, CATS almost completely fails on these three tasks, achieving an average score of only **16.2**, while FLoE maintains a score of **43.8**. At **80% sparsity**, FLoE retains over **93%** of the base model's capabilities.\n\n**2. Non-compressed baselines exhibit limited performance.**\n\nTo address your concerns, we have used **Mistral-7B** and **Llama-3.2-3B** as baselines for non-compressed methods.  \n\n**(1) Clarification of Fair Evaluation Setup.** \n\nTo ensure that the activation parameter count of FLoE aligns with that of similarly sized non-compressed models, we applied **INT8 quantization** to the attention layers of FLoE at **90% sparsity**. This adjustment ensures that the memory footprint of FLoE is comparable to that of Llama-3.2-3B in its **FP16** configuration. For FLoE at **80% sparsity**, the memory usage remains consistent with the original setup and matches that of Mistral-7B. Furthermore, we have supplemented the experimental results for **MMLU** (5-shot), **GSM8K** (8-shot), and **HumanEval**, providing a comprehensive comparison between FLoE and the baseline models.\n\n**(2)  Analysis of New Results.** \n\nNot only on MMLU, but also on complex tasks such as GSM8K and HumanEval, various sparse methods face significant challenges, with performance tending to degrade more rapidly. However, our method, FLoE, demonstrates the best performance retention. At **90% sparsity**, CATS almost completely fails on these three tasks, achieving an average score of only **16.2**, while FLoE maintains a score of **43.8**. At **80% sparsity**, FLoE retains over **93%** of the base model's capabilities.\n\nEven smaller non-compressed models struggle to match the performance of large compressed models on complex reasoning tasks. On **MMLU**, **GSM8K**, and **HumanEval**, **FloE** at **90% sparsity** outperforms both **Mistral-7B** and **Llama-3.2-3B**. Moreover, **FloE** at **80% sparsity** preserves performance **closest** to the base model, demonstrating its clear advantage in these tasks. These results highlight that compressing large models can be more beneficial than relying on smaller non-compressed models, particularly for demanding reasoning tasks.  \n\nIn summary, **FloE not only achieves superior performance retention under high sparsity but also surpasses smaller non-compressed models across multiple benchmarks**. This underscores the effectiveness of our approach in balancing model efficiency and task performance, especially in scenarios where computational resources are limited. \n\nWe believe these findings strongly reinforce FloE’s robustness and practicality, making it a **compelling choice** for deploying MoE-based LLMs in **resource-constrained environments**.\nFor users with limited resources who cannot deploy MoE models, we highly recommend FloE as an efficient and practical alternative, delivering high performance while significantly reducing computational costs.\n\nLooking forward to hearing from you!\n\nBest,\n\nAuthors\n\n---\n\n### `Results Table:`\n\n| Model         | GSM8K-8 shot (Acc) | Humaneval-0 shot (Pass@1) | MMLU-5 shot (Acc)| Average       |\n|---------------|--------------|-------------|-------------|---------------|\n| Base Model    | 58           | 33.5        | 69.5        | 53.67         |\n| FLoE-80       | 51.7         | 32.3        | 65.4        | 49.8          |\n| CATS-80       | 31.1         | 28.7        | 61.7        | 40.5          |\n| FLoE-90       | 40.9         | 30.5        | 60.1        | 43.83         |\n| CATS-90       | 2.42         | 8.54        | 37.7        | 16.22         |\n| Mistral-7B    | 39.4         | 29.2        | 62.5        | 43.7          |\n| Llama-3.2-3B  | 26.6         | 25.6        | 56.4        | 36.2          |\n\n---\n[1] Zhou, Y., Chen, Z., Xu, Z., Lin, V., & Chen, B. Sirius: Contextual Sparsity with Correction for Efficient LLMs. NeurIPS, 2024.", "claims_and_evidence": "Intra-expert sparsity can be exploited via compression techniques, resulting in significant speedups of MoE execution on consumer-grade hardware with limited memory. Empirical evaluation is presented.", "methods_and_evaluation_criteria": "The experimental evaluation and comparison with other schemes are reasonable.", "theoretical_claims": "I did not check the Appendices.", "experimental_designs_or_analyses": "Yes, I followed most of the experimental evaluation. \n\nFigure 10 is the one I found more interesting, showing the performance. On some of the larger corpus tasks (e.g., MMLU), the performance goes down quite a bit compared to Mixtral-8x7B (60.1 and 65.4 vs 69.5). If we accept degradation in performance, then it may be possible to use smaller models without any compression, that are the the same performance level. Therefore, it may be good to compare not just other compression methods, but also non-compression methods on smaller models with the same performance.", "relation_to_broader_scientific_literature": "The paper belongs to a line of work concerned with making large model perform inference efficiently on edge hardware, typically at single batch size; offloading and compression techniques are used.", "essential_references_not_discussed": "Not that I can suggest.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "No questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ARuuvw6b7", "reviewer": "Reviewer_ZXAh", "summary": "Mixture-of-experts have become a popular way to scale up the transformer models these days, but the large model scale creates challenge when deploying the model under limited resources.This paper introduces FloE, an inference system designed for on-the-fly MoE inference on consumer-grade GPUs. FloE integrates quantization, offloading, prefetching, and optimized kernels to reduce memory overhead and latency with limited performance degradation. The authors evaluate their approach against multiple baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer ZXAh,\n\nWe sincerely appreciate your recognition and valuable suggestions. Below, we summarize and respond to the **issues**, **suggestions**, and **questions** you raised.\n\n**`[Minor Issue 1]` The absence of Mixtral-GPU performance results in Section 3.2. While HQQ Int2 seems to reflect the same setting, the authors should clarify this explicitly.**\n\n**`[Response]`**  We sincerely appreciate you pointing out the issue with our inconsistent terminology. Indeed, **HQQ-INT2 refers to the performance of Mixtral-GPU on downstream tasks**. We utilized HQQ to  quantize the experts to INT2, enabling the entire Mixtral-8×7B model to fit within GPU memory. In future versions of the paper, we will ensure the consistency of the configurations for HQQ-INT2 and Mixtral-GPU by clearly stating their alignment.\n\n**`[Suggestion 1]` I strongly encourage the authors to include code assets during the rebuttal period.**\n\n**`[Response]`** We genuinely hope that our efforts can contribute to the community by facilitating the deployment of MoE models on consumer-grade GPUs and edge devices. We have provided the current version of **our code via an anonymous link** below **[Code Link of FloE](https://anonymous.4open.science/r/floe-5843)**, and we plan to open-source it after completing the adaptation for more models.\n\n**`[Suggestion 2]` Figures 6–8 use color schemes that are difficult to read. Additionally, Figure 8's caption contains a missing space.**\n\n**`[Response]`** For later versions of the paper, we will **refine the color schemes** and **captions for Figures 6-8**.\n\n**`[Question 1]`  What is the latency of the predictor described in Section 2.3?**\n\n**`[Response]`** We measured the latency of the predictor over 500 forward passes and calculated the average values. The results show that the inter-expert sparse predictor has a latency of **0.11 milliseconds**, while the intra-expert predictor's latency is **0.27 milliseconds**. In comparison, the average execution time for a single FloE Transformer block is **5.74 milliseconds**. These latencies account for only **1.9%** and **4.7%** of the total execution time, respectively, indicating that they have almost no significant impact on the model's generation speed.\n\n**`[Question 2]` Given that the proposed quantization method outperforms existing approaches, could it also be applied to dense models, particularly due to the similarities with the SwiGLU module?**\n\n**`[Response]`**  The evaluation methods for **quantization and sparsification sensitivity are equally applicable to dense models**. The **theoretical proof presented in Appendix B** is also **generalizable** and can be applied to MoE models. These are our preliminary results on Mistral. Moving forward, we plan to further explore how to compress the size of MLPs in dense models to improve inference speed and reduce deployment costs. This will be one of the key directions for our future work.\n\n\nLooking forward to hearing from you!\n\nBest regards,\n\nAuthors\n\n---\n### `Code Link of FloE:`\n\nhttps://anonymous.4open.science/r/floe-5843\n\n---\n### `Results Table:`\n#### **Quantization Sensitivity of Mistral-7B**  \n\n| Quantization | INT8   | INT4   | INT3   | INT2    | INT1     |\n|--------------|--------|--------|--------|---------|----------|\n| **gate**     | 5.252  | 5.271  | 5.350  | 5.839   | 1025.0   |\n| **down**     | 5.252  | 5.309  | 5.544  | 13.51   | 12022    |\n| **up**       | 5.552  | 5.264  | 5.324  | 5.753   | 301.28   |\n\n#### **Sparsification Sensitivity of Mistral-7B**\n\n| Sparsification | 50%       | 60%       | 70%       | 80%        | 90%        |\n|----------------|----------|----------|----------|-----------|-----------|\n| **gate**     | 6.91     | 8.67     | 18.76    | 2902      | 21857     |\n| **down**       | 5.29     | 5.35     | 5.503    | 5.940     | 7.92      |\n| **up**         | 5.46     | 5.74     | 6.404    | 8.581     | 20.47     |\n\n---", "claims_and_evidence": "Most claims in this paper are either supported by evidence or well-known facts in this area.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are well-aligned with the problem at hand.", "theoretical_claims": "The theoretical claim and proof in App. B looks reasonable.", "experimental_designs_or_analyses": "The experimental setup is well-structured and appropriately evaluates the proposed approach. However, one minor issue is the absence of Mixtral-GPU performance results in Section 3.2. While HQQ Int2 seems to reflect the same setting, the authors should clarify this explicitly.", "relation_to_broader_scientific_literature": "The proposed method builds upon prior work in deploying MoE models on personal devices, offering improvements in efficiency and performance preservation.", "essential_references_not_discussed": "I'm not aware of any essential references that were not discussed.", "other_strengths_and_weaknesses": "The paper is well-written, but the absence of code assets limits reproducibility.", "other_comments_or_suggestions": "Figures 6–8 use color schemes that are difficult to read. Additionally, Figure 8's caption contains a missing space.", "questions_for_authors": "- What is the latency of the predictor described in Section 2.3?\n- Given that the proposed quantization method outperforms existing approaches, could it also be applied to dense models, particularly due to the similarities with the SwiGLU module?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose the FloE algorithm, a mixture of experts (MoE) inference system on memory constrained GPUs, that uses various compression techniques on the experts' internal parameter matrices to reduce data movement load to the GPU.
Overall the paper received 3 favorable reviews and one slightly negative review of weak accept.
Amongst the positives comments are that the work has a good experimental setup, it is a well written paper,  and that the work seems to offer improvements in efficiency and performance.
The reviewers point a lack of theoretical justification for the method as part of the negatives as well as a lack of a deeper investigation of failure cases. 
Overall I did not notice any deal-breaker comments that would make me reject the paper, so I am leaning towards acceptance.
Decision: Accept (poster)