Libra: Effective yet Efficient Load Balancing for Large-scale MoE Inference
Authors: Jaehoon Yang
Yushin Kim
Seokwon Moon
Yeonhong Park
Jae W. Lee
Venue: ICLR 2026 Poster (2026)
Keywords: Mixture-of-Experts
Model inference
Expert load imbalance
Dynamic load balancing
Primary area: infrastructure, software libraries, hardware, systems, etc.

Abstract: Distributed inference of large-scale Mixture-of-Experts (MoE) models faces a critical challenge: expert load imbalance. Numerous system-level approaches have been proposed for load balancing, but they either fail to achieve a satisfactory level of balance or introduce new bottlenecks due to the overhead of the load balancing mechanism itself. To this end, we propose Libra, a system that achieves near-optimal load balancing with minimal overhead. Libra adopts sophisticated mechanisms that accurately predict future expert activations and, based on these predictions, systematically perform load balancing. At the same time, it effectively hides the associated overhead by reconstructing the execution flow so that these costs are overlapped with MoE computation. Evaluations with two large-scale state-of-the-art MoE models on 8 H200 GPUs demonstrate that Libra improves throughput by up to 19.2\%. The code is available at https://github.com/SNU-ARC/Libra.

Published as a conference paper at ICLR 2026
LIBRA: EFFECTIVE YET EFFICIENT LOAD BALANCING FOR LARGE-SCALE MOE INFERENCE
Jachoon Yang, Yushin Kim, Seokwon Moon, Yeonhong Park, Jac W. Lee Scoul National University {jaehoon. yang, henryl peryllnn 1n, moonsw214 i1i196, jaewlee)
ABSTRACT
Distributed inference large-scale Mixture-of-Experts (MoE) models faces a critical challenge: expert load imbalance. Numerous system-level approaches have been proposed for load balancing, but they either fail to achieve a satis- factory level of balance introduce new bottlenecks due to the overhead of the load balancing mechanism itself. To this end, we propose Libra, a system that achieves near-optimal load balancing with minimal overhead. Libra adopts sophisticated mechanisms that accurately predict future expert activations and, based on these predictions, systematically perform load balancing. At the same time, it effectively hides the associated overhead by reconstructing the execu- tion flow so that these costs are overlapped with MoE computation. Evalua- tions with two large-scale ttate-of-the of-the-art MoE models on 8 H200 GPUs demon- strate that Libru improves throughput by up to 19.2%. The code available :tpps://gin https: //gi thu.. com com//NU--RC/LLbrra
INTRODUCTION
The Mixture-of of- Ef-Extssss (MoE) architecture has become a comnestone for state-of-the-art Large Lan-
guage Models (LLMs) such as DeepSeek- V3, Qwen3MoE, and GLM-44.5 (DeepSeek-AL et al. 2025]
Yang et al. 2025 GLM-4.5 Team et al. 2025). Through sparse activation, MoE enables models to
scale to trillions parameters while keeping the training and inference computation cost manage-
able (Du et al.
2022 The Mosaic Research Team 2024: Jiang et al. 2024 Fedus al.) 2022
Lepikhin et al.
2020 Rajbhandari et al. 2022).
At the same time, the dynamic nature of MoE models introduces key deployment challenge: load imbalance. One common way to scale MoE inference is through Expert Parallelism (EP), in which experts within MoE layers partitioned across multiple GPUs. Under this setup, load imbalance arises when a disproportionate number of tokens are assigned to a few hot experts, causing the GPUs hosting them to become stragglers that determine the end-to-end latency.
Existing system-level load balancing approaches suffer from fundamental limitations, proving to be less effective and/or inefficient (DeepSeek) 20255 Li al., [023] Doucet et al.) 2025). Some approaches fail to achieve satisfactory balance because they rely on ineffective heuristics, leaving substantial room for improvement (DeepSeek) 2025 Li et al., 2003). Others achieve considerable level of balance but introduce new bottlenecks due to the additional operations required for load balancing (Doucet et 2025).
To address these challenges, we propose Libra, system that achieves near-optimal balance with virtually zero overhead. In other words, catches two birds at once: effective load balancing and efficient realization of that mechanism. For effectiveness, Libra predicts expert activations for the next layer with high accuracy by leveraging the observation that hidden states in LLMs evolve slowly across consecutive blocks, and based on these predictions, applies sophisticated algorithm that yields near-optimal balance. For efficiency, Libra reconstructs the inference execution flow so that any overhead incurred by this process is hidden under MoE computations. In evaluations eight benchmarks using two state-of-the-un MoE models, Qwen3MoE and GLM-4.5, on H200 GPUs, Libra improves throughput by up 19.2% compared the state of the art.

---

Published as a conference paper at ICLR 2026
2
BACKGROUND AND MOTIVATION
2.1 EXPERT LOAD IMBALANCE IN MOE INFERENCE
The Mixture-of Ex-erts (MoE) (Jacobs et al. 1991 Jordan & Jacobs 1994, Shazeer et al. 2017) architecture enhances the capacity of Large Language Models (LL.Ms) by replacing the dense Feed- Forward Network (FFN) layer in a Transformer block with sparse MoE layer. This layer consists of large pool of subnetworks (experts) and gating network that selectively activates small subset of experts (e.g.. top-k) for each input token. This sparse activation allows MoE models to scale to hundreds of billions even trillions of parameters while keeping the computational cost for inference relatively low (Du et al. [2022] The Mosaic Research Team 2024 Diang et al.) [024; Fedus
al. 2022 Rajbhandani et al. Lepikhin et al. 2020). Consequently. large-scale open-source MoE models have achieved performance comparable to leading proprietary models like GPT-5,
demonstrating the efficacy of this architecture (Yang al.) 2025 DeepSeek-Al et al. 2025 GLM-
4.5 Team et al. 2025 Baidu ERNIE Team 2025 OpenAl et al. 2025 Kimi Team et al. 2025).
However, the inherent mechanism that grants MoE models their ifficiency-indeeendemt token as- signment-iintrouuces significant challenge: expert load imbalance. Historically, this issue was addressed during the training phase by incorporating an auxiliary load-balancing loss term, which encouraged more uniform distribution of tokens across all experts (Xue et al. [0224] Muennighoff ett al. 2025; Fedus et al.] (022). While effective for balancing, this approach often came at the cost
of model performance, it could hinder the degree of expert specialization (Wang et al. 2024
Guo
ett al.
2025
Qiu et al. 2025 DeepSeek-A1 et al. 2025).
Qwen2MoE vs Qwen3MoE
DeepSeek-V2 vs DeepSeek-V3
2.75
2.75
Qwen2MoE
DeepSeek-V2 DeepSeek-V3
2.50
Qwen3MoE
2.50
Imbalance Ratio
2.25
2.00
1.75
1.75
1.50
1.50
1.25
1.25
1.00
1.00
Layer
Layer
Figure I: Intensified expert load imbalance in recent MoE models.
Reflecting this trade-off, recent state-of-the-ar MoE models have moved away from strict load- balancing loss in favor of techniques that maximize expert specialization (Yang et al. 2025 GLM- 4.5 Team et al.] [2225] DeepSeek-AL et al.] 2025). This aggressive pursuit of specialization has suc- cessfully pushed model performance the expert load imbalance during inference. We measure this using the imbalance ratio, defined as the maximum load on any single GPU divided by the average load across all GPUs, where a value of 1.0 indicates a perfect balance. This trend is illustrated in Figure D showing a stark contrast in the imbalance ratio between newer MoE models and their predecessors (see Appendix A for exper- imental setup details). This reveals a fundamental trade-off: achieving state-of-the tite-ofthee-rr theearrt performance in large MoE models exacerbates the expert load imbalance, a problem projected become more severe as models advance.
heights but has the critical side effect of intensifying
To efficiently serve large-scale MoE models across multiple GPUs, the standard strategy is hybrid approach that applies Expert Parallelism (EP) to MoE layers and Data Parallelism (DP) to non Mon-MoE layers (e.g., self-attention) (Perplexity Al 2025 SGLang Team 2025 Doucet et al. 2025 Li et al.) 2025). While this strategy is essential for managing the massive parameter counts of these models, its performance is highly susceptible to expert load imbalance, which consequently becomes a critical performance bottleneck (Doucet al. 2025). This issue stems from the synchronous execution of the MoE layers, forcing all devices to wait for the most heavily loaded GPU the one hosting the hot expert(s) to finish its computation. This phenomenon, known the straggler effect, leads
2

---

Published as a conference paper at ICLR 2026
to significant idle time less-loaded workers, severely degrading end-to-end lateney and overall system throughput. Consequently, mitigating this straggler effect has become central challenge in MoE inference, prompting the exploration of various system-level solutions.
2.2
SYSTEM-LEVER SOLUTIONS FOR LOAD BALANCING AND LIMITATIONS
No replication
Hot expert replication
GPUO
EO
t
Token
E0
t
sharding
GPU
E1
E1
GPU 2
E2
E2
E0
Figure 2: Hot expert replication and token sharding
A number of studies have attempted to mitigate load imbalance in MoE inference through system- level techniques. Most of these approaches employ hot expert replication. Figure illustrates this approach: instead of assigning each expert to unique GPU without redundancy. hot experts (i.e., experts that are likely to receive many tokens) are replicated across multiple GPUs. During MoE execution, tokens routed to these hot experts then be distributed across replicas different GPUs-a process we refer to as token sharding. This alleviates bottlenecks that would otherwise arise single GPU were forced process disproportionate number of tokens.
In the following, we discuss three representative works that constitute the most widely used and/or state-of-the-art techniques. They share the above approach but differ in how they perform expert replication and token sharding. While these methods present promising results, they also exhibit notable limitations, as their strategies for replication and sharding achieve only limited effectiveness and/or efficiency.
Hot exprrc
Profiling iterations
Profiling iterations
Predictor
Predictor
(a) EPLB
Hot expert replication
MoE block
MoE block i+1
Put doaach
Predictor
layers
sharding
MoE layer
layers
shardling
layer
(b) Lina
MoE Layer
Comp.
Routing
Replication decision
Token sharding
Resident experts
experts
Comm.
Hot expert replication
No predictor
(c) HarMoEny
Figure 3: Overview of EPLB, Lina, and HarMoRny
EPLB. Expert Placement Load Balancer (EPLB) (DeepSeek) 2025) periodically performs expert replication based on historical data. Figure 3 illustrates this process. It profiles expert popularity over fixed number of iterations (N in the figure) and uses the aggregated statistics to guide expert replication. However, this strategy is less effective because past popularity does not capture the instantaneous and dynamic variations across requests. EPLB's token sharding is also limited in
3

---

Published as a conference paper at ICLR 2026
effectiveness: once experts replicated, tokens routed to them randomly distributed across the GPUs holding replicas. In short, while EPLB is relatively efficient, it is not effective in either expert replication token sharding.
Lina. Figure illustrates how Lina eLi 2023) performs expert replication. Specifically, Lina replicates experts for the upcoming block: when executing MoE computation for Block i, it performs expert replication for Block + 1, thereby removing replication overhead from the critical path. For this purpose, it relies on a pre-constructed lookup table that tracks each token's 's expert-selection-patii This path, which is the sequence of experts a token has selected in previous few layers (e.g., from layeri - to layer i 1), used to predict which experts will be popular in the current layer (layer i). Such expert replication, however, is limited in effectiveness. In our evaluations on eight different benchmarks with two models (Qwen3MoF and GLM--455), Lina's s prediction accuracy falls to as low as 43.7% and 11.8%, respectively. Details of the experimental setup are provided in Appendix A
For token sharding. Lina simply distributes tokens uniformly across all replicas (e.g.. assigning an equal number of tokens to each). Similar to the random token sharding in EPLB, this strategy is largely oblivious to the actual GPU loads, leaving significant room for improvement.
HarMoEny. Figure(3cc) illustrates HarMoEny (Doucet et al. 2025), which makes decisions on ex- pert replication based on exact routing results. In other words, expert replication performed only after the current input and its routing outcomes become available. This design makes replication highly effective, as it is guided by exact information. To prevent replication overhead from appear- ing on the critical path, HarMoEny first performs MoE computation for tokens routed to resident experts, while replication of hot experts proceeds in parallel. Once replication completes, the MoE computation for hot experts is executed. Token sharding is also effective, HarMoBny employs sophisticated algorithm that computes a near-optimal token assignment.
Despite this effectiveness, HarMoEny suffers from efficiency issues. The overhead required real- ize such accurate replication and token sharding is substantial. After routing and before completing MoE computation, HarMoEny must perform decision making through complex algorithms to de- termine both expert replication and token sharding. Because these algorithms run synchronously on the GPU, they extend the critical path and introduce new bottlenecks.
DESIGN
In this section, we propose Libra, a system for MoE inference that achieves near-optimal load balanc- ing with minimal overhead. Unlike prior methods, Libra simultaneously addresses both effectiveness and efficiency in hot expert replication and token sharding, thereby overcoming the key limitations of existing approaches. For expert replication, Libra follows the spirit of Lina by prefetching hot experts for the next layer while processing the current layer, based on prediction. This design avoids the inefficiency observed in HarMoEny, which cannot exploit Grouped- GEMM optimizations. At the same time, Libra employs a more accurate prediction mechanism than Lina, thereby improving effectiveness. For token sharding, Libra adopts the strategy of HarMoEny but effectively hides its cost from the critical path by restructuring the execution flow and leveraging the CPU.
The remainder of this section is organized as follows. Section 3.1 explains Libra's execution flow. Section describee the hot expert replication mechanism, and Section 3.3 details the token shard- ing mechanism.
3.1 LIBRA EXECUTION FLOW
Figure fillustrates the execution flow of Libra. The key novelty lies in its Two-Stage Locality-Waare Execution, which splits MoE computation into two phases based on token locality: MoE and MoE,emote- The MoE phase processes tokens routed experts residing
the same GPU as the tokens themselves, while the MoE,emote phase handles tokens that must be dispatched to other GPUs. After decomposing the computation into these two phases, Libra first performs MoE followed by MoE
This execution flow creates a time window in which the overhead of sophisticated token sharding mechanism can be hidden. In the conventional execution flow with load balancing, MoE compu- tation begins only after token sharding completes. By contrast, in Libra, the Mo phase has
4

---

Published as a conference paper at ICLR 2026
GPU
Gate
Predict
MoEincal
MoEramott
Comm.
Dispatch
Hot Expert Replication
CPU
Planning
Routing
Sharding
Figure 4: Two-Stage Locality-Aware Execution of Libra
no dependency on token sharding; it can start immediately after the gating function, with only the MoE,emote phase depending the results of token sharding and Dispatch operation. This enables token sharding mechanism to run in parallel with MoE
To further enhance the effectiveness of this parallelism, Libra performs token sharding the CPU rather than the GPU. addition, Libra implements dispatch using AllGather (i.e., all tokens are transferred all GPUs) instead of AlI2AII, where tokens are sent only to their assigned GPU. While this design increases the raw communication volume of dispatch, its latency impact is negligible. More importantly, improves efficiency by removing dispatch from the critical path: in an AlI2AI1- based implementation, dispatch must wait for token sharding to finish, whereas in the AllGather- based implementation, dispatch can also proceed in parallel with token sharding.
3.2 HOT EXPERT REPLICATION
As mentioned, Libra follows the spirit of Lina for hot expert replication: performing expert repli- cation for the next layer while processing the current layer, based on prediction. However, Libra departs substantially from Lina in both how the prediction is performed and how expert replication planning (i.e., determining which experts replicate which GPUs) is carried out.
Block
Block i+1
Gate,..
Planning
Replication
Gate
Planning
Replication
Non-MoE
Non-MoE layers
Gate
MoE layer
Gate
MoE layer
layers
Figure 5: Hot expert replication of Libra with lookahead predictors
Predictor Design. Libra employs a lookahead predictor, leveraging a well-established property of Transformer-based LLMMs: hidden states evolve slowly across layers (Liu al.) [202bb] Hwang al.) 2024]. Figure Iillustrase its concept. It speculatively executes the gating function of the next layer using the hidden states from the current layer, and then uses the results determine which ex- perts should be replicated across GPUs. This runtime-based approach achieves substantially higher accuracy than Lina's 's predictor (e.g. 70-80% 20-30%)
Locality-Aware Expert Replication Planning. During expert replication planning, Libra intro- duces an additional consideration beyond load balancing: locality enhancement. In other words, Libra not only balances load but also seeks to extend the MoEEseut computation window, thereby providing more opportunity to hide token sharding overhead.
To this end, Libra performs expert replication planning in two phases. In the first phase, each GPU brings in N x a experts that are most frequently activated by the tokens on that GPU and are not already resident on it, thereby extending the MoEiocal computation window. Here, N denotes the maximum number of additional experts a GPU may host, determined by its available memory capacity and the allowable time window (a function of MoE computation time and communication bandwidth), while a a hyperparameter that controls what fraction phase. In the second phase, load balancing is performed iteratively: at each step, the hottest expert
is allocated to the first
5

---

Published as a conference paper at ICLR 2026
from the most heavily loaded GPU is selected for replication and placed on the least-loaded GPU among those that have not yet received N extra experts.
Figure 6 illustrates the expert replication planning process of Libra with example. First, to hance locality, Libra addresses the initial placement (left), where a significant portion of tokens on each GPU is routed to remote experts other devices. Each GPU identifies its most requested mote experts and replicates them locally. For instance, GPU O replicates E4 from GPU to serve its local tokens. This process converts remote tokens into local computations, securing the MoExoco computation window necessary to hide system overhead. Second, to establish a foundation for load balancing. the algorithm identifies and replicates heavily loaded experts to under-loaded GPUs. In the figure, this is shown by replicating expert E2 to an under-loaded device. This facilitates effec- tive token sharding by allowing the workload from overloaded GPUs be redistributed, ultimately enabling a near-perfect load balance.
Initial Expert Placement
With Locality-Awarr Expert Replication
Local hot experts By phase
2
GPU0
t
E1
E0
E2
M20
<60
X20
A50
J62
420
<30
X20
H40
GPU
E2
E3
E2
E3
E1
Tx10
GPU 2
E4
E5
E4
E5
EO
Figure 6: Locality-awar expert replication planning
3.3 ADAPTIVE TOKEN SHARDING
For token sharding, Libra adopts an algorithm similar to that used by HarMoRny, but with two key differences. First, Libra applies token sharding only to remote tokens. Second, this process is offloaded to the CPU.
Figure explains the iterative greedy strategy of token sharding. The algorithm's main loop begins by checking if any GPU's load exceeds If the system is bal- anced, the process terminates. Otherwise, it se- lects the most overloaded GPU, 9a. resolve To find the most effective transfer, the al- gorithm enters inner loop, starting by select- ing the hottest remote expert, e, on g the one accounting for the largest number of its remote tokens It then searches for optimal des- tination: the least-loaded GPU (ga) that hosts replica of expert and has enough capacity to accept new tokens is necessary on the destination GPU to process the transferred tokens, and these replicas abled by hot expert replication. If a suitable des- tination found, the algorithm calculates the number of tokens to transfer and updates the Crucially, after
Local Stattibuiiion
the target threshold
Initiatize Total GPU Loacis (D)
No =o End: Belanced
Load?
Yes
Select most overlosded GPU
[g.5)
Select hottest remote expert
Find leest-lnaded replica CPU ig di with svailable capacity?
A replica of the expert
No
Calculate transfar and Update loads a.s and a_d
try
No
Yes
Select next hottnet expert(e) R_S
Figure 7: Logic flow of the iterative greedy rebal- ancing strategy.
loads on both 9s and ga each successful transfer, the algorithm returns the main loop's start to re-evaluate the entire system's 's balance, ensuring it always ad- dresses the most critical imbalance first. no suitable destination is found attempts to trans- fer the tokens for the next hottest expert on 9s until all options are exhausted, at which point also returns re-evaluate the global state. The full algorithm is detailed in Appendix C

---

Published as a conference paper at ICLR 2026
4 IMPLEMENTATION DETAILS
We implement Libra atop the SGLang (v0.4.10) LLM serving framework. Our core mechanisms for expert replication planning and token sharding are implemented in Cython to ensure minimal overhead and are integrated as native modules during SGLang's 's build process.
To efficiently perform hot expert replication, leverage PyTorch Symmer cMMmory for copy engine-based P2P transfers. We employ a double-buffering strategy by pre-allocating two large buffers: even buffer and odd buffer. During the execution of even-numbered MoE layer, original and duplicated experts are gathered in the even buffer to be processed via a high- performance Grouped- GEMM kernel. Concurrently, the system loads the necessary experts for the subsequent odd oud-nueeered layer into the odd buffer. When processing an odd-numbered MoE layer, the roles are reversed: computation utilizes the odd buffer while the even buffer populated for the next layer. This pipelining mechanism effectively hides the expert replication overhead by overlap- ping the P2P copy operations with the ongoing computation.
EVALUATION
We conduct comprehensive evaluation to demonstrate the effectiveness of Libra. Our experiments are designed to answer four key questions: (1) How does Libra's 's prefill performance compare against baselines? (2) How stable and robust is Libra's performance under workloads with dynamic and (3) How does the prediction accuracy of Libra's speculative execution compare against existing methods like Lina? (4) What is the overhead introduced by Libra's s mech- anisms, and how effectively is it hidden within the execution flow?
shifting token distributions?
5.1 SETUP
Model and Data. We evaluate Libra using two representative state-of-the-ar large MoE models: Moen3MoE (235B) (Yang et al.] 2023) and GLM-4.5 (355B) (GLM-4.5 Team et al.] 2025). To ensure coverage of a wide range inputs, we use eight datasets: BookCorpus (Zhu et al 2015). Codeforces (Penedo et al.] (025). PrepSee (Xin et al.] 2024), Fine (Penedo et al.)
).24). GSM8K (Cobbe et al. (021), HellaSwag (Zellers et al., 2019). HumanEvalPlus (Liu et al, 2023a), and LMSYS-Chat IM (Zheng et al. 2023). All experiments are run using BF16 precision.
Environments. All experiments are conducted on a single node equipped with NVIDIA H200-SXM5 GPUs, each with 141 GB of HBM3e memory. Intra-node communication leverages NVSwitch with 900 GB/s of P2P bandwidth.
Baselines. We compare Libra against three baselines. The vanilla MoE implementation in SGLang 10) serves as our foundational baseline, representing a standard system without advanced load balancing. For the widely adopted proactive expert replication approach, we EPLB (DeepSeek) 2025 from its implementation within SGLang. As the strongest baseline, we evaluate against Lina Li et al. 2023). Since public implementation of Lina is available, we developed an in- house version built on SGLang, faithfully following the description in the original paper.
Metrics. The primary performance metric is prefill throughput, measured in tokens per second. We assume a prefill-decode disaggregated serving system where the prefill and decode phases are separated and handled by different GPUs (Zhong al. 20244) Hu al. 2025 Feng et al. 2025). and therefore target only the prefill phase in our evaluation. We also measure the imbalance ratio (defined as the load of the most burdened GPU divided by the average load across all GPUs) to analyze the effectiveness of load balancing.
5.2 RESULTS
Throughput Results. First, Libra substantially improves the performance of the prefill phase. As shown in Figure 8 Libra achieves the highest throughput across all tested models and datasets. To rigorously evaluate the robustness of our system, we focus on the four datasets exhibiting the most severe expert load imbalance. These challenging scenarios serve as stress tests to clearly distinguish the efficacy of different load balancing strategies. Notably, this evaluation was conducted under an experimental setup deliberately designed to be highly advantageous for the baseline systems. For
7

---

Published as a conference paper at ICLR 2026
Qwen3MoE, Batch size = 8, Seq 4K
4000
Throughput
3000
2000
GLM-4.5, Batch size 8, Seq
4K
Throughput
2000
1000
BookCorpus
DeepSeek-Prover//
HellaSwag
HumanEvalPlus Libra
Mean
SGLang
EPLB
Lina
Figure 8: Prefill throughput of Libra and baselines.
SGLang
Libra
Owen3MoE Seq 2048
@Wen3MoE Seq 4096
GLM-4.5 Seq 2048
GLM-4.5 Seq 4096 M
2800
2600
3800
Throughput
2500
1800
2600
2400
2500
3200
2300
2400
2200
3000
2200
2.0
Imbalance Ratio
1.7
1.6
1.6
1.8
1.5
3.6
1.6
3.4
1.4
1.2
1.2
1.2
1.2
1.1
www
3.0
2.0
2.0
150
100
50
200
25
100
Iteration
Iteration
Iteration
Iteration
Figure Throughput fluctuation and imbalance ratio under a dynamic workload.
Lina, its expert-selection path table constructed using the same dataset as the evaluation and its expert prefetching allowed each GPU to hold additional experts. Similarly, EPLB's 'ss expert placement determined by profiling the identical dataset. It statically placed identified hot experts-one on each of the 8 GPUs- across every MoE layer. This static replication, however, results in higher memory consumption compared the dynamic approaches of Lina and Libra be- cause of large number of layers. To meet the memory budget with Lina, Libra was configured with N set to 8 and a 0.5. Despite these favorable, even biased, conditions for the baselines, Libra consistently and significantly outperforms them for both Qwen3MoE and GLM-4.5. These results confirm that Libra's dynamic load balancing effectively resolves the straggler problem, leading to superior computational efficiency and overall system performance.
Per-Iteration Fluctuation. Libra also delivers significantly higher and more stable throughput. Fig- ure 9 illustrates this robustness using mixed-dataset designed to simulate dynamic shifts in expert load imbalance. For this test, Lina's expert-selection-pale table was constructed using a workload created by mixing one-eighth of the build split from each of the eight datasets. This comparison centers on dynamic systems like Lina and Libra, excluding EPLB, as its reliance on periodic pro- filing and static reconfiguration is ill-suited for workloads where imbalance shifts frequently and intensely. While baseline systems suffer from volatile performance that plummets as the imbalance ratio spikes, Libra effectively decouples its performance from the input distribution. By maintaining
8

---

Published as a conference paper at ICLR 2026
Qwen3MoE
GLM-4.5
Dataset
Lina
Libra
Dataset
Lina
Libra
BookCorpus
47.3
91.7
BookCorpus
11.7
79.6
DeepSeck-Prover V1
45.4
86.5
CrepSeek-Prover. V1
12.7
72.9
HellaSwag
37.5
86.6
HellaSwag
11.5
76.6
HumanEvalPlus
44.5
87.0
HumanEvalPlus
11.2
72.7
Table 1: Prediction accuracy.
D2H metadata transfer
H2D metadata transfer
Scheme
Dispatch
Prediet
Broadcast
Token shurding
Repl. planning
Combine
Others
MoE
Total
Labra
0,99
0.03
0.24
0.10a
0.09
0.57
0.26
1.33
0.39
2.77 (Local)
9.07
(Remote)
Lina
0.72
0.15
0.08
1.32
0.55
8.11
11.33
SGL
0.73
0
1.34
0.55
10.99
13.61
Hiddeen MoEalocal, Hidken MoEinen
Table 2: Breakdown analysis of MoE layer.
near-perfect imbalance ratio close 1.0, Libra provides consistently high throughput, proving its resilience to the dynamic nature of expert load imbalance.
Prediction Accuracy. We quantitatively evaluate the accuracy of Libra's predictor for hot expert replication compared against Lina. Table 1 presents direct comparison between the prediction accuracy of Libra's speculative execution based approach and Lina's constreeco lookup ta- ble based method on the Qwen3MoE and GLM-4.5 models. Accuracy is defined as the fraction of correctly predicted experts for each token, where the set of actually activated experts serves as the ground truth. We construct Lina's 's expert-selection-pal table the build split of mixed dataset, then evaluate on the evaluation split of four datasets. Evaluation setup is detailed in Appendix (
The results reveal stark contrast between the two methods. Libra's predictor consistently achieves a high and stable accuracy in the 70-90% range across all datasets, demonstrating the effectiveness of its runtime prediction based current-layee hidden states. In contrast, Lina's lookup- based pre- dictor shows noticeably lower accuracy across datasets, and highlights the critical generalization limitations of an offlieebuilt lookup table. This effect is pronounced GLM-4.5, where Lina successfully identifies fewer than one correct expert between top-8 experts. These findings confirm that Libra's dynamic prediction mechanism is significantly more robust and reliable for handling diverse and unpredictable workloads.
Breakdown Analysis. We present the inference latency breakdown of the MoE layer for SGLang, Lina, and Libra in Table evaluated using the Qwen3MoE with sequence length of 1024 and batch size of 32. Both Lina and Libra significantly reduce MoE computation time through load balancing; however, Libra achieves greater reduction, primarily driven by its superior expert-activation activation pre- diction accuracy. Notably, while Lina incurs visible prediction latency, Libra's prediction overhead is negligible. Furthermore, although Libra introduces additional costs associated with its sophisti- cated execution model including metadata transfer, broadcasting, and balancing logic-ther these are effectively hidden. By decoupling MoE computation into MoEiocal and MoE Libra overlaps these auxiliary operations with local computation, preventing them from impacting the critical path.
CONCLUSION
We introduce Libra, a dynamic load balancing system addressing intensified expert load imbalance in modern Mixture-of-Experts (MoE) models. Libra proposes Two-Stage Locality-Aware Execution, an innovative paradigm hiding load balancing overhead by overlapping with ongoing GPU com- components: Locality-Awan Expert Replication,
putations. This is enabled by two synergistic using speculative execution for highly accurate (70-80%) expert prediction to strategically prefetch the necessary experts for the next layer, and Adaptive Token Sharding, computing optimal

---

Published as a conference paper at ICLR 2026
signment schedule by accounting for processed local token load. Implemented on SGLang, Libra demonstrates state-of-the-ai performance, improving the prefill throughput by up to 19.2% while maintaining an imbalance ratio of nearly 1.0 under dynamic workloads. Libra thus achieves dynamic load balancing with virtually zero-overhead for efficient serving of large-scale MoE models.
ACKNOWLEDGMENT
This paper was the result of the research project supported by SK hynix Inc. Jac W. Lee is the corresponding author.
REFERENCES
Baidu ERNIE Team. Emni 4.5 technical report, 2025.
Weijian Chen, Shuibing He, Haoyang Qu, Ruidong Zhang, Siling Yang, Ping Chen, Yi Zheng, Baox- ing Huai, and Gang Chen. IMPRESS: An inportance-Informen Multi-Tier prefix KV storage system for large language model inference. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pp. 187-201, Santa Clara, CA, February 2025. USENIX Association. ISBN 978-1-939133-45-
URL https: //pp:://www uswnix. wwtp:/ssml...c///oooeeeeeeooooo wttp:///ooo.eeeeneeeaass/2222 presentation/ chen wei j an- impress
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1rXiv:2110.14166 2021.
DeepSeck. Expert parallelism load balancer (eplb), 2025. URL https: com/
deepseek- -aieekaaaiiiii aiaaaiiEPPP /EBLL
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L.. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li. Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei. Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu WU, Zhongyu Zhang, Zhuoshu Li, Zihui Gu. Zijia Zhu, Zijun Liu. Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025, URL https:
or.o abs 1943777477
10

---

Published as a conference paper at ICLR 2026
Zachary Doucet, Rishi Sharma, Martijn de Vos, Rafael Pires, Anne-Marie Kermarrec, and Oana Balmau. Harmoeny: Efficient multi-gpu inference of moe models, 2025. URL hetps:
//aaxiv.crg/aaaaa.i.. haapr///aa/2500.. 12417
Kuntai Du, Bowen Wang, Chen Zhang, Yiming Cheng, Qing Lan, Hejian Sang, Yihua Cheng, Jiayi Yao, Xiaoxuan Liu, Yifan Qiao, Ion Stoica, and Junchen Jiang. Prefillonly: An inference engine for prefill-only workloads in large language model applications. In Proceedings of the ACM SIGOPS 31tt Symposium on Operating Systems Principles, SOSP '25, pp. 399-414, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400718700. doi: 10.1145/ 3764834
7731569.3764837
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with cixture-of-experts. In Ka- malika Chaudhuri, Stefamie Jegelka, Le Song, Csaba Szepesvari. Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5547-5569. PMLR, 17-23 Jul 2022. URI. wwl.sppees//ee
http:::/proceedings
du22c. html
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39)
2022. URL[http //3ml: r. org/ hwtpps/ppprs//// 21 -999 html
Jingqi Feng, Yukai Huang, Rui Zhang, Sicheng Liang, Ming Yan, and Jie Wu. Windserve: Efficient disaggraageee IIm serving with stream-based dynamic scheduling. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, pp. 1283-1295, 2025.
GLM-4.5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao. Xiaohan Zhang, Xuancheng Huang. Yao Wei, Yean Cheng, Yifan An, Yilin Niu. Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge, Chenghua Huang, Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang. Dazhi Jiang, Ding Ai, Erle Zhu, Fei Wang. Gengzheng Pan, Guo Wang, Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng. Hao Tai, Haoke Zhang, Haoran Wang, Haoyu Yang, He Liu, He Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Hui- long Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang. Jiayi Gui. Jiayue Zhao Jie Liu, Jijie Li, Jing Li, Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du Jinhua Du Jinxin Liu, Junkai Zhi, Junli Gao, Ke Wang. Lekang Yang, Liang Xu, Lin Fan, Lindong Wu, Lintao Ding, Lu Wang, Man Zhang. Minghao Li, Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang, Shaoyou Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang, Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu, Weng Lam Tam. Wenjie Liang, Wentao Liu, Xiao Wang, Xiaohan Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang. Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang, Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yuxuan Zhang, Zezhen Liu, Zhen Yang. Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng, Zhuorui Liu, Zichen Zhang, Zi- han Wang, Zijun Yao, Zikang Wang, Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wen- guang Chen, Jidong Zhai Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025. URL tittps://aree
www./aggg///oooooo.//../o/s..aea /208 06471
Hongcan Guo, Haolang Lu, Guoshun Nan, Bolun Chu, Jialin Zhuang, Yuan Yang, Wenhao Che, Sicong Leng, Qimei Cui, and Xudong Jiang. Advancing expert specialization for better moe, 2025. URL / arxiv. org/ abs/ 2505. 22323
Xiannan Hu, Tianyou Zeng, Xiaoming Yuan, Liwei Song, Guangyuan Zhang, and Bangzheng He. Bestserve: Serving strategies with optimal goodput in collocation and disaggregation architec- tures. arXiv preprint arXiv:2506.05s. 25880777, 2025.
11

---

Published as a conference paper at ICLR 2026
Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. Pre- Pre-gated moe: An algorithm-syster co-design for fast and scalable mixture-of cexture-offexpet of-err inference. In 2024 ACM/IEEE st Annual International Symposium on Computer Architecture (ISCA), pp- 1018-1031. IEEE, 2024.
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991. doi: 10. 10.1162/neco.19 1991......991......
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mixtral experts, 2024. URL https: http//aaivvv.. /ttp///www...raa///////11../ 04088
Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 6(2):181-214, 1994. doi: 10.1 0..1162/neo..194. 1991.6.2.181.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao. Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang. Zihao Huang. Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma. Xinyu Ma, Yingwei Ma. Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng. Ruoyu Qin, Bowen Qu, Zeyu Shang. Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang. Dinglu Wang, Feng Wang. Haiming Wang, Jianzhou Wang, Jiaxing Wang, Jinhong Wang, Shengjie Wang, Shuyi Wang, Yao Wang, Yejie Wang, Yiqin Wang, Yuxin Wang. Yuzhi Wang, Zhaoji Wang. Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu. Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang. Ying Yang. Zhen Yang, Zhilin Yang. Zonghan Yang. Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang. Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi k2: Open agentic intelligence, 2025. URL https::///// wwtp///aivvvvvvvv/ abs/abs/////aooaoooaa..aaaaaaaaa abs 2507 20534
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding, 2020. URL htttts:://aaiv...ttt wwwp:/iwwwrrgaab///////0 16668
Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. Accelerating distributed MoE train- ing and inference with lina. 2023 USENIX Annual Technical Conference (USENIX ATC 23), pp. 945-959, Boston, MA, July 2023. USENIX Association. ISBN 978- 17781-993933--35-9 URL https:: www. usenis x http://g//cooooeeen///t///33////
on /annlii-/ jiamin
Yan Li, Pengfei Zheng, Shuang Chen, Zewei Xu, Yuanhao Lai, Yunfei Du, and Zhengang Wang. Speculative moe: Communication efficient parallel moe inference with speculative token and ex- pert pre-scheduling, 2025. URL httpt:ttt:/aaaat
org/ hbtp:/bs//5555 04398
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. Is your code gener- ated by chatgpt really correct? rigorous evaluation large language models for code generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in
12

---

Published as a conference paper at ICLR 2026
Neural Information Processing Systems, volume 36, pp. 215722.1572. Curran Associates, Inc.,
2023a. URL httt::://proceedinss.eeeeee cc/pppr httpa//eccpeeeeeieeeepeeeeeeeeee /itps//apeeeeeee2////33
Eile/
Zichang Liu, Jue Wang, Tri Dao. Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference Machine Learning, pp. 22137-22176. PMLR, 2023b.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannanch Hajishirzi. Olmoe: Open mixture-of-expert. language models, 2025. URL httph::////// arxiv. org/ abs/2409. 02060
OpenAl, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Hai- tang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Do- minik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le. Guillaume Leclere, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh Mc- Grath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovie Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Prochl, Huida Qiu, Gaby Raila, Filippo Raso. Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang, Miles Wang. Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whit- ney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. apt-oss-120b gpt- oss-206 model card, 2025. URL https:
https://arxiv. wtwp:v.aaxvvvvvvv org/ abs/ aae///bss5//5 2508 10925
Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, fñigo Goiri, Saeed Maleki, and Ri- cardo Bianchini. Splitwise: Efficient generative Ilm inference using phase splitting. In 2024 ACM/EЕЕ 51 Annual International Symposium on Computer Architecture (ISCA), pp. 118- 132, 2024. doi: 10. (UD9///CAA9907.2022.0000
Guilherme Penedo, Hynek Kydliček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. 1d=n66Ckk2QaG
ptattts:://opnnevviie wtwp://wwwwe//ooouuu.o.
Guilherme Penedo, Anton Lozhkov, Hynek Kydliček, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarin, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces. https:///aggingfaee.../aattssss http://ppnnencccoceeeeeeee r1/codeforces 2025.
Perplexity
AL.
Efficient URL
and
portable mixture-of-experts communica- https::/ww..prppllettt att.b///b/yy.///////o////go
tion,
2025.
Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Demons in the detail: On implementing load balancing loss for training specialized mixture-of-exper models, 2025. URL https: // /arxivv... www.v..rg//.................o..e oogg abs/
2501. 11873
13

---

Published as a conference paper at ICLR 2026
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed-MoE: Advancing mixture-of- experts inference and training to power next-generation AI scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceed- ings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18332-18346. PMLR, 17-23 Jul 2022. URL https: //proceedings. , wwt.ssmmr.prrsess.e. wwwppreessssee
html
SGLang Team. Deploying deepseek with pd disaggregation and large-scale ex- pert parallelism on 96 h100 gpus, 2025.
URL. https: hrg.//oggg///lg//////o
2025- 05 05 large- scwle-ealeeeeeeee
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gated mixture-of-experts layer, 2017.
06538
The
Mosaic
Research
Team.
Introducing dbrx: A new state-of-the-art open
11m. March 2024.
URL https:/ http/w/waatttt databriccs..... com/blog/
introducing dbrx nww- srtttttttttttteeeeeeeeee 11m Databricks Mosaic Research Blog.
Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-free load balancing strategy for mixture-of-experts, 2024. URL
15664
Huajian Xin, Daya Guo, Zhihong Shao, Z.Z. Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Advancing theorem proving LLMs through large-scale synthetic data. In The 4th Workshop on Mathematical Reasoning and Al at NeurlPS 2024. URL https: //openzeview. wwtp:/eee/ttoommaaaaaaaa..
Yi Xiong, Hao Wu, Changxu Shao, Ziqing Wang. Rui Zhang, Yuhong Guo, Junping Zhao, Ke Zhang, and Zhenxuan Pan. Layerkv: Optimizing large language model serving with layer-wise kv cache management, 2024. URL.
C0428
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort open mixture-of-experts language models, 2024. URL https:
//axxii.org/aaaaaaaa...aa.aa.a/a http://ab/2402.0000 01739
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong. Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://arxiv wwt.//tg////ooooooooo..o/o.esess /ttps/2407.16611
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhennu Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025, URL https: httpp:s///axvvv.o//a o abs/ / 2505 09388
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
14

---

Published as a conference paper at ICLR 2026
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zhuohan Li, Lin, Eric. P Xing, Joseph E. Gonzalez, lon Stoica, and Hao Zhang. Lmsys-chat-Im: A large-scale real-world Ilm conversation dataset, 2023.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large lan- guage model serving. In 18th USENIX Symposium on Operating Systems Design and Im- plementation (OSDI 24), pp. 193-210, Santa Clara, CA, July 2024a. USENIX Association. ISBN 978-1-939133-40-3 URL. https: pettps://ee http:://wwwww.siiii/. usenix .org/ http://oooeeeeee//s22/4444 presentattion/hoone wwwoon///oooyyyg nmin.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu. Xin Jin, and Hao Zhang. (DistServe): Disaggregating prefill and decoding for gooddut-optimiine large language model serving. In 18th USENIX Symposium Operating Systems Design and Implementation (OSDI 24), pp. 193-210, 2024b.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEFF International Conference on Computer Vision (ICCV), December 2015.
15

---

A14 4B Published as a conference paper at ICLR 2026
A EXPERIMENTAL SETUP DETAILS
Models. For Figure 1 we compare the imbalance ratio across recent MoE families by pairing mod- els with different load-balancing strategies during training. Within the Qwen family, we evaluate Qwen2MoE (Yang et al. 2024) and Qwen3MoE, released on HuggingFace as Qwen2- and Qwen3 2353- A22B. Qwen2MoB has 57 billion total parameters and 14 billion activated pa- rameters uses 64 experts with top-8 expert selection, and utilizes a micro-batch level auxiliary load- balancing loss during training ensure load balance. contrast, Qwen3MoE, with 235 billion total and 22 billion activated parameters, uses 128 experts with top-8 expert selection and forgoes this term favor of global-batch load balancing loss, strategy that maximizes expert specialization while still addressing balance. The DeepSeek family shows similar contrast: DeepSeek-V2 (236B total, 21B activated) also employs 160 experts with top-6 expert selection and two shared experts, and also uses auxiliary load-balancee loss, whereas DeepSeek-V V3 (671B total, 37B activated) employs 256 expert with top-8 expert selection and a single shared expert, and improves training efficiency by adopting the auxiliary-loss free load balancing technique. For all other experiments, we evaluate Libra and baselines on Qwen and GLM-4.5, models that are trained without such fine-grained balancing losses. GLM-4.5 has 355 billion total parameters and 32 billion activated parameters, and uses 160 experts with top-8 expert selection and a single shared expert.
Datasets. We evaluate Libra and baselines on eight datasets: BookCorpus (Zhu al. 20155) Code- forces (Penedo et al.) (022)). ReepSeck-Provee (Xin et al., 2024), FineWeb (Penedo et al. 2024)
GSM8K (Cobbe et al. al.]| 2021]. HellaSwag (Zellers al. 2019). HumanEvalPlus (Liu et al.) 2023a
and LMSYS-Chat 1M (Zheng et al. 2023). Unless noted otherwise, each dataset contributes a to- tal of 2.0M tokens. The first 1.6M tokens form the build split for EPLB offline profiling and for constructing Lina's prediction table. The next 0.4M tokens form the evaluation split used for test- ing. Figure T is the only exception, which uses a 0.07% subset of the BookCorpus dataset. For Table I build Lina's 's expert-selection-pii table on a mixed workload that uniformly interleaves the build splits of all cight datasets. We then evaluate Lina and Libra separately on each dataset's evaluation split. Figure reports results BookCorpus, DeepSeek- Provee-----r--e- V1, HellaSwag, and Hu- manEvalPlus dataset. Figure 9 jsees a shuffled workload constructed from all eight datasets.
Environments. All experiments SXM5 GPUs, each with 141GB of HBM3e memory. The server configuration is summarized in Table 3
single-node system equipped with eight NVIDIA H200-
CPU GPU
2x Intel Xeon Platinum 8580 (128 cores)
8x NVIDIA H200-SXM5-141GGB
System Memory
32x 64 GB DDR5-5600 (total 2,048 GB)
GPU Memory
141GB HBM3e per GPU
GPU Interconnect
Connected with NVSwitch (900GB/s bandwidth)
Table 3: Server configuration
Metrics. Table reports accuracy, defined the fraction of tokens whose ground-truth experts appear the predicted top-k set. Figure g and Figure 9 report prefill throughput in tokens per second. Throughout these experiments, we adopt Prefill-eccode disaggregation (Patel et al.) 2024 Zhong et al. (2024a) setup. and therefore we evaluate prefill only as we target prefill phase. Figure and Figure 9 also report the imbalance ratio, defined the load of the most heavily utilized GP divided by the average load across all GPUs.
B EXPERT REPLICATION PLANNING
The Expert Replication Planning optimally places experts on GPUs for the next MoE layer. This process is crucial for facilitating effective load balancing and extending the MoE computation window, which in turn hides system overhead. The algorithm operates in two main phases after an initial setup.
First, it duplicates "local hot experts" those most frequently requested by a GPU's local tokens but residing other GPUs- onto the source GPU itself. This strategically increases the number
16

---

Published as a conference paper at ICLR 2026
of local tokens that be processed without inter-GPU communication, creating sufficient time window for the Adaptive Token Sharding to execute in parallel without affecting the critical path.
Second, the algorithm iteratively balances the remaining load by duplicating global hot experts. It identifies the most overloaded GPU and replicates its hottest expert to the least-loaded GPU that has available capacity. This ensures that the overall load distributed evenly possible before the next layer's computation begins. The final output is an optimized expert placement map that serves the foundation for the rebalancing stage.
Algorithm Expert Replication Planning
Inputs: Predicted expert IDs for upcoming tokens next topk ids, Total number of experts E, To- tal number of GPUs G, Max duplicated experts per GPU N, Number of local hot experts to duplicate T..
Outputs: A binary matrix for expert placement on GPUs: Maex E {0,1]
1: Calculate ExpertLoad]g. el (requests for expert from GPU g) based on next_topk_ids
2: Initialize Mnext by assigning each expert to its home GPU.
3: for each GPU 9s do
>> Phase I: Duplicate Local Hot Experts
4: Identify top L remote experts most requested by 9s-
5: Duplicate these experts to 9s.
6: end for
7: Calculate initial GPU loads based on the current mapping in M
8: <- Target balanced load per GPU.
9: for to (N L) G do
Phase 2: Balance Load via Iterative Duplication
10: Oare f- most overloaded GPU where load >> B.
if no such GPU exists then break
12:
end if
< expert contributing most to 9 remote load.
14:
Gas + least loaded candidate GPU that can host (respecting capacity N). if Jast found and duplicating keeps Load[]as] S then
16:
Update Maext by duplicating Idst-
17:
Update GPU loads to reflect newly localized computations.
18:
end if
19:
end for
20:
return Moest
C ADAPTIVE TOKEN SHARDING
The Adaptive Token Sharding determines optimal assignment for remote tokens to resolve load imbalance, operating on the expert placement map generated by the Expert Replication Planning. Its core strategy is an iterative greedy approach that ensures the final token distribution is close to perfectly balanced as possible.
The algorithm begins by calculating the total load for each GPU. It then enters loop that continues as long as any GPU's load exceeds a target threshold. Within the loop. it identifies the most over- loaded GPU (g.) and selects its hottest remote expert the one responsible for the largest portion of its remote token load. It then finds the least-loaded GPU (ga) that already hosts a replica of expert e and has sufficient capacity.
A calculated number of tokens for expert e are then transferred from g. to Id. and the load states of both GPUs are updated. After each transfer, the algorithm restarts its loop to re-evaluate the global system state, ensuring it always addresses the most critical imbalance first. This process repeats until the loads are balanced or no further beneficial transfers can be made.
17

---

Published as a conference paper at ICLR 2026
Algorithm 2 Adaptive Token Sharding
Require: Expert-to-GPU mapping M [0,1] Local GPU loads I. gG Remote expert loads R E ZGxE, Average target load B IR, Imbalance tolerance (0,
Ensure: Rebalanced loads R. G, and maximum load fmax
1: G[g] [وg] E. Ríg, e| for all g G
b Initialize total GPU loads
2: while max, G[g] > =1 do
3: moved + false
4: for each 9N E {g G > B in descending order of Gig] do
5:
for each E {e g.g. 6 0} in descending order of Rlg,, el do
6:
C c + 9s Mle, 1
D Find candidate destination GPUs
7:
is not empty then
8:
f- arg min G[g]
Select least-loaded destination > Calculate destination's 's remaining capacity
9:
cap - B - Ga
if cap then
11:
[ { 1i-min( (Rgg. cl cap)
Determine amount to move
Rig., c| Rlg.. el 1; Rlga. el + Rlga el +1
13:
G <- G[g-] 1
Glga] t G(ga) +
> Perform the token transfer
moved true
15:
break
Exit inner loop to re-evaluate the most overloaded GPU
16:
end if end if
17:
18:
end for
end for
20: if not moved then
21: break
Converged stuck, exit outer loop
22:
end if end while
24:
fmax maxg reg)ur return R, G, kmax
D END TO END LATENCY ANALYSIS
Libra prioritizes accelerating the prefill phase to minimize Time-To-First-Token (TTFT), which crucial for prefill-intensive workloads like Summarization and Retrieval-Augmented Generation (RAG). While the decoding stage often dominates total latency, minimizing TTFT essential for meeting striet Service-Level Objectives (SLOs) (Du et al. 2025 Xiong al.)
2024 Chen et al., 2025). To demonstrate Libra's effectiveness this regard, we conducted end-to-end experiments on Qwen3MoE and GLM-4.5 across various configurations, the results of which are detailed in Table through Table 13
The results demonstrate that Libra significantly reduces TTFT, particularly in long-context scenar- ios. We note that for short sequences (c.g.. 1K tokens), performance may be comparable to or slightly lower than the baseline, the MoEiscal window is insufficient fully mask CPU overhead. How- ever, as input length increases, Libra effectively mitigates the straggler effect, achieving TTFT re- ductions of up to 27% for Qwen3MoE and 12.7% for GLM-4.5 in large batch and long-context settings.
18

---

Published as a conference paper at ICLR 2026
Output Length
Batch
Scheme
4
8
16
32
64
128
256
Libra
0.38
0.50
0.73
1,22
2.17
4,08
7.87
15,62
31,05
Lina
0.37
0.49
0.73
1.22
2,19
4.14
7.99
15.49
30.98
SGLang
0.36
0.48
0.72
1.19
2.14
4.04
8.00
15.60
31.07
Lihra
15.75
16
Lina
0.65
0.77
1.01
1,50
2.45
4.35
8.09
16.08
31.19
SGLang
0.70
0.82
1.06
1.54
2.52
4.50
8.24
16.00
31.80
Libra
1.03
1.16
1.40
1.87
2,88
4.81
8.57
16.40
32.11
SGLang
1,38
1,50
1.75
2.23
3.19
5.18
8.97
16.82
32.16
Libra
1.51
1.63
1.86
2.33
3.27
5.18
8.96
16.16
31.07
48
Lina
1.79
1.91
2.15
2,61
3.55
5.42
9.14
16.87
31.78
SGLang
16.99
Lihra
2.00
2.12
2.35
2.83
3.77
5.68
9.43
16.93
31.75
Lina
2.38
2.50
2.73
3.21
16.98
32.33
SGLang
2.72
2.84
3.08
3.56
4.49
6.35
10.10
17.59
33.20
Table 4: End to end latency for Qwen3MoF, Input length = 1024
Output Length
Batch
Scheme
128
Libra
0.56
0.68
0.92
1.39
2.32
4.24
8.11
15.31
30.78
0.65
0.77
1.01
1.48
8.32
15.58
30.69
StiLang
0.71
0.82
1.07
1.53
2.48
4.40
8.14
15.70
31.26
Libra
1.04
1.16
1.41
1.89
2.89
4,76
8.80
16.50
31.32
Lina
1.24
1.36
1.59
2.10
3.03
4.99
8.89
16.50
32.01
SGLang
1.39
1.51
1.75
2.25
3.19
5.17
8.94
16.72
32.00
Libra
2.02
2.13
2.37
2,86
5.76
9.58
17.10
32.53
Lina
2.40
2.52
2.76
3.26
4,23
6.09
9.93
17.71
33.12
SGI ang
2.75
2.87
3.12
3.59
4,56
6.47
10.42
18:07
33.41
Libra
3.00
3.12
3.36
3.82
4.75
6.60
10.36
17.99
33.24
3.57
3.68
3.90
4.39
7.19
11.03
18.63
33.67
SGLang
4.12
4.24
4.47
4.93
5.87
7.72
11.40
18.96
34.15
Libra
4.00
4.35
4.82
5.75
7.62
18.87
33,91
64
Lina
4.76
4.87
5.11
5.58
6.50
8.39
12.12
19.39
34.79
SGLang
5.51
5.87
6.35
9.14
12.80
20.37
34.75
Table 5: End to end latency for Qwen3MoE, Input length 2048
Output Length
Batch
Schemne
2
4
16
32
64
128
256
Libra
1.18
4.82
Lina
1.27
1.38
1.62
2.10
3.06
4.98
8.73
16.35
32.00
SGLang
Libra
2.06
2.18
2.42
2.90
3.89
5.76
9.62
17.46
32.31
Lina
2.44
2.56
4.24
6.23
10.05
33.10
StiLang
2,80
2.92
3.16
3.64
4.60
6.58
10.25
18.13
33.68
Libea
4.08
4.20
4.43
4.92
5.85
7.78
11.62
19.21
34.70
Lina
4.82
4.93
5.18
5.66
6.61
8.54
12.33
20.17
34.93
SGLang
5.60
5.72
5.96
7.44
9.38
13.08
35.80
Libea
6.12
6:22
6.48
6.92
7.91
9.75
13.57
21.03
36.04
Lina
7.24
7.35
7.60
8.08
9.05
10.88
14.71
22.11
37.39
SGLang
8.41
8.54
8.79
9.21
10.21
12.03
15.97
23.42
38.35
Libra
8.19
8.30
8.54
9.01
9.93
11.88
15.65
23,12
37.73
Lina
0OM
0OM
OOM
0OM
OOM
00M
OOM
0OM
OOM
SGLang
11.25
11.36
11.60
12.08
12.98
14.79
18.59
26.07
40.78
Table 6: End to end latency for Qwen3MoE, Input length = 4096
19

---

Published as a conference paper at ICLR 2026
Ourput Length
Batch
Scheme
2
16
32
64
128
256
Libra
1,59
1.71
1.94
2.41
3.38
5.28
9.15
16,60
31.36
Lina
1.88
2.00
2.24
2.71
3.69
5.58
9.35
17.13
31.43
SGLang
2.13
2.25
2.49
2.96
3.89
5.82
9.58
17.17
31.90
3.26
6.89
16
Lina
3.69
3.81
4.05
4.54
5.48
7.50
11.26
19,10
34,68
SGLLang
4.25
4.38
4.61
5.10
6.03
8.01
11.95
19.61
35.50
Libra
6.24
6.37
6.59
7.10
8.05
9.98
13.75
21.57
36.71
7.48
37.67
SGLang
8.54
8.67
8.89
9.40
10.35
12.28
16.05
23.88
39.01
Libra
10.13
10.24
10:48
10.93
11.86
13.70
17.52
24.42
39.65
48
Lina
0OM
0OM
0OM
0OM
OOM
OOM
0OM
0OM
OOM
SGLang
13.15
Libra
12.87
12.98
13:21
13:67
14.63
16.52
20.28
27.64
42.63
Lina
0OM
0OM
OOM
OOM
0OM
OOM
OOM
OOM
SGLang
17.15
17.26
17.49
17.95
18.86
20.74
24.51
31.88
46,96
Table 7: End to end latency for Qwen3MoF, Input length = 61
Output Length
Batch
Scheme
32
Libra
2.14
2.26
2.50
2.98
3.90
5.80
9.60
16.97
32.07
Lina
2.53
2.65
4.35
6.29
10.07
33.25
SGiLang
2.88
3.00
3.23
3.70
4.62
6.56
10.28
17.94
32,83
Libra
4.25
4.37
4.61
5.09
6.05
8.03
12.05
19.61
35.39
I Lina
5.00
5.12
5.36
5.86
6.82
8.72
12,70
20.33
35.40
SGLang
5.77
5,90
6.14
7.61
9:51
13.33
21.20
36.70
Libra
8.53
8:65
8.89
9.37
10.36
12:26
16.22
23.98
39.99
Lina
0OM
0OM
OOM
OOM
OOM
OOM
0OM
OOM
OOM
SGLang
11.59
11.72
11.94
12,43
13.44
15.39
19.28
27.18
42,86
Libra
13.13
13.24
13.48
13.94
14,86
16.69
20.41
27.65
42.48
Lina
0OM
0OM
OOM
OOM
OOM
OOM
OOM
0OM
OOM
SGLang
17:40
17.52
17.74
18.20
19.15
20.99
24.68
32.15
47.12
Libra
0OM
0OM
OOM
OOM
OOM
OOM
OOM
64
Lina
0OM
0OM
0OM
0OM
OOM
OOM
0OM
0OM
OOM
SGLang
23.21
23.33
23.58
24.06
25.07
27.01
30.77
54.52
Table 8: End to end latency for Qwen3MoE, Input length 8192
Output Length
Batch
Scheme
4
8
32
128
256
Libra
0.48
1.51
19.35
Lina
0.65
0.80
1.10
1.66
2,84
5.25
9.87
19.20
38.12
ang
19.16
Libra
0.79
0.93
1.23
1.83
3.02
5.38
10.18
19.78
39.51
0.91
1.06
1.37
1.96
19.89
SGLang
0.84
0.99
1.29
1.90
3.07
5.47
10.23
19.89
39.26
Libra
1.48
1.63
1.94
2.54
3.73
6.19
10.75
20.98
40.08
Lina
1.73
1.88
2.19
2,79
3.99
6.41
11.07
20.77
39.92
SGLang
1.65
2.10
2.70
3.91
6.35
11.24
20.80
Libra
2.17
2.32
2.62
33,22
4.40
6.75
11.47
21.30
39.95
Lina
2.55
2.70
2.99
3.61
4,79
7.17
11.85
21.34
40.85
SGI ang
2.45
2.60
2.89
3.49
4.69
7.13
12.02
21.33
40.87
Libra
2.88
3.03
3.33
3.91
5.05
7.47
12.10
21.82
41.01
3.39
3.54
3.84
4.43
8.02
12.65
22.32
40.81
SGLang
3.28
3.43
3.73
4.32
5.47
7.78
12.47
21.97
41.15
Table 9: End to end latency for GLM-4.5, Input length 1024
20

---

Published as a conference paper at ICLR 2026
Ourput Length
Batch
Scheme
2
16
32
64
128
256
Libra
0.80
0.95
1,25
1.86
3.01
5.37
10.18
19.55
38.71
Lina
0.92
1.07
1,36
1.96
3.15
5.52
10.12
20.25
39.68
SGLang
0.84
0.99
1.29
1.87
3.05
5.49
10.25
19.80
38.85
1.95
16
Lina
1,74
1.89
2.19
2.78
3.97
6.36
11.32
20.40
40.12
SGLaang
1.67
1.82
2.11
2.72
3.91
6.30
10.99
20.66
39.23
Libra
2.92
3,07
3.36
3.98
5.22
7.68
12.36
22.29
41.89
3.56
SGLang
3.31
3,46
3,76
4.36
5.54
8.03
12.66
22.47
41.41
Libra
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
48
Lita
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
SGLang
Libra
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Lina
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
SGLang
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Table 10: End to end latency for GLM--455, Input length = 2048
Output Length
Batch
Scheme
Libra
1.54
1.69
1.99
2.57
3,76
6.12
10.70
20.18
38.81
Lina
1.77
1.92
2.22
6.31
39.38
SGiLang
1.70
1.84
2.14
2.71
3.92
6.33
11.10
20.49
40.07
Libra
2.99
3.14
3.44
4.04
5.23
7.71
12.50
21.87
41.61
Lina
3.47
3.62
3.93
4.53
5.76
8.13
12.87
22.25
41.88
SGLang
3.37
3.52
3.82
5.59
8.15
12.88
41.10
Libra
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Lina
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
SGLang
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Libra
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Lina
COM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
SGLang
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Libra
0OM
OOM
OOM
OOM
OOM
OOM
OOM
64
Lina
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
SGLang
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
Table 11: End end latency for GLM-4.5, Input length 4096
Output Length
Batch
Schemne
2
4
16
32
64
128
256
Libra
2.31
2,46
6.90
Lina
2.65
2,80
3.10
3.69
4.89
7.26
11.96
21.48
40,80
SGL.ang
Libra
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Lina
0OM
0OM
OOM
SGiLang
0OM
0OM
0OM
0OM
OOM
OOM
OOM
0OM
OOM
Libra
0OM
0OM
OOM
OOM
OOM
OOM
OOM
COM
OOM
Lina
0OM
0OM
OOM
COM
OOM
OOM
OOM
OOM
OOM
SGLang
0OM
0OM
OOM
OOM
OOM
OOM
Libra
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Lina
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
SGLang
0OM
0OM
OOM
COM
OOM
OOM
COM
OOM
OOM
Libra
DOM
OOM
OOM
COM
OOM
OOM
OOM
OOM
OOM
Lina
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
SGLang
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Table 12: End to end latency for GLM-4.5, Input length = 6144
21

---

Published as a conference paper at ICLR 2026
Ouiput Length
Batch
Scheme
32
Libra
3.05
3.19
3.49
4.08
5.25
7.67
12.32
22.05
40.76
Lina
3.59
3,74
5.82
8.23
13.06
41.51
SGLangg
3.49
3.64
3.92
4.53
5.70
7.98
12.76
22.41
41.73
Libra
0OM
0OM
OOM
0OM
OOM
OOM
OOM
OOM
OOM
Lina
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
SGLang
0OM
0OM
OOM
0OM
OOM
OOM
OOM
Libra
OOM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Lina
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
SGLang
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Libra
OOM
OOM
OOM
COM
OOM
OOM
OOM
OOM
OOM
Lina
COM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
SGLang
0OM
0OM
OOM
OOM
OOM
OOM
OOM
OOM
OOM
Libra
0OM
OOM
OOM
OOM
OOM
OOM
64
Lina
0OM
0OM
OOM
0OM
OOM
OOM
OOM
OOM
OOM
SGLang
0OM
0OM
OOM
OOM
OOM
OOM
OOM
Table 13: End end latency for GLM-4.5, Input length 8192
22


Reviews:
[{"review_id": "uLcTgfvVf1", "reviewer": "Reviewer_BzVM", "summary": "This paper introduces Libra, a system for efficient and effective load balancing during large-scale Mixture-of-Experts (MoE) inference. It addresses the expert load imbalance problem, where certain experts become bottlenecks due to uneven token distribution across GPUs.\n\nKey Contributions:\n1. Two-Stage Locality-Aware Execution: Splits MoE computation into local and remote phases, enabling overlap of load balancing overhead with computation.\n2. Accurate Expert Prediction: Uses speculative execution based on hidden states to predict expert activations for the next layer, achieving 70–90% accuracy.\n3. Efficient Expert Replication & Token Sharding: Introduces locality-aware expert prefetching and CPU-based token rebalancing, improving throughput by up to 19.2% over strong baselines like Lina.\n4. Strong Experimental Results: Demonstrates consistent performance gains across multiple datasets and MoE models (Qwen3MoE, GLM-4.5), especially under dynamic workloads.\n\nOverall, Libra achieves near-optimal load balance with minimal overhead, making it a practical and scalable solution for MoE inference.", "questions": "1. A more detailed and precise explanation of the model configuration should be provided for each model used in this paper. For example, there is no model named Qwen2MoE in HuggingFace.\n2. The definition of the imbalance ratio should be placed in the main text rather than the Appendix for better clarification.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The problem in this paper is clearly defined and strongly motivated. It focuses on the expert load imbalance issue during the inference phase of large-scale MoE models, explicitly pointing out that it introduces the “straggler effect”, severely degrading system throughput.\nThrough experiments (Figure 1), it demonstrates that modern MoE models (e.g., Qwen3MoE, DeepSeek-V3) abandon load-balancing during training in pursuit of expert specialization, exacerbating load imbalance at inference time.\n2. The system is elegantly designed and highly innovative. It proposes Two-Stage Locality-Aware Execution, dividing MoE computation into local (MoElocal) and remote (MoEremote) phases, cleverly hiding the overhead introduced by load balancing. It introduces a hidden-state-based speculative expert-activation prediction mechanism, achieving significantly higher accuracy (70–90% vs. 20–30%) compared to static routing tables used in methods like Lina. It designs two modules—Hierarchical Expert Prefetcher and Adaptive Token Rebalancer—to tackle expert replication and token redistribution, respectively, with clear structure and strong synergy.\n3. Comprehensive experiments and significant improvements. Experiments in this paper are implemented on SGLang, the LLM serving engine with state-of-the-art efficiency, and evaluations are conducted on two large-scale MoE models (Qwen3MoE and GLM-4.5) and multiple datasets. Libra achieves up to 19.2% higher throughput than the strongest baseline (Lina). Under dynamic-load scenarios, Libra demonstrates high stability, whereas methods like Lina exhibit noticeable performance fluctuations (Figure 9). Detailed experimental configurations and reproduction information—including models, datasets, hardware environments, and evaluation metrics—are provided, enhancing reproducibility.", "weaknesses": "1. The memory consumption is not examined in this paper. While the expert replication mechanism is introduced in this paper for balanced workload on distributed devices, this would lead to higher memory consumption, since individual experts can also contribute to a certain degree of memory consumption. A detailed breakdown analysis of memory consumption on each rank would be helpful to examine the performance more systematically.\n2. The prediction accuracy in this paper should be better evaluated. In this submission, the predictors are separately trained on different datasets with specialized task domains. While in real LLM serving applications, the task domain may vary across a broad range for different users at different times. Therefore, a more rational approach would be training the predictor on a large-scale dataset with a diverse data distribution and evaluating the prediction accuracy on massive downstream tasks to better evaluate the out-of-distribution (OOD) performance of Libra.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7XQcEh5Edb", "reviewer": "Reviewer_HMhF", "summary": "The paper addresses the problem of load imbalance in mixture of expert large language model (LLM) inference. We propose a system named Libra that achieves efficient load balancing. Libra builds upon the expert replication policy used in Lina, incorporating an improved mechanism for predicting hot experts. Furthermore, we redesign the execution flow to utilize the CPU effectively, minimizing overhead associated with token sharding.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The proposed method addresses several drawbacks of Lina. It introduces an improved hot expert predictor and a more efficient expert replication strategy. \n2. Additionally, Libra utilizes the CPU to reduce overhead associated with token sharding operations.", "weaknesses": "1. Novelty concern: Utilizing prefetching to identify expert activation in the next layer is not a novel approach. It has been previously used for accelerating Mixture-of-Expert (MoE) inference as described in [1]. In addition, [1] also introduces the use of the CPU to prevent lightweight computations from becoming a bottleneck in GPU processing.\n\n2. Token sharding introduces additional communication overhead, which is not addressed by the authors. This issue could become significant as batch sizes increase.\n\n3. The experiments include only two MoE models, which is limited. Newer MoE models, such as Qwen3, Moonlight-A3B, gpt-oss-120b, and gpt-oss-20b, should be considered.\n\n4. In throughput experiments, the models only execute the prefill stage, whereas the decoding stage is the main time-consuming component. The experiments should focus more on the decoding stage.\n\n5, The experiments are conducted with a setup of 8 batch size and 4k sequence length. Larger batch sizes, as well as shorter and longer sequence lengths, should be tested.\n\n6. The study lacks end-to-end latency experiments that include both prefilling and decoding stages with various inference configurations, such as different batch sizes, input sequence lengths, decoding sequence lengths, and datasets.\n\n7. Lastly, the paper does not provide an analysis of the MoE layer inference time.\n\n[1] Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y6ZFunfdFS", "reviewer": "Reviewer_knvi", "summary": "This paper addresses the issue of expert load imbalance during MoE model inference by implementing runtime expert placement based on load prediction.\nIt achieves the effective integration and systematic orchestration of the proposed methods, which are built upon existing methods and offer enhancements.\nExperimental results demonstrate significant end-to-end inference speedup on the evaluated MoE models.", "questions": "Given that various model configurations (e.g., hidden size, number of experts, top-k activation) and system configurations (e.g., CPU/GPU computational and communication capabilities) influence the duration of specific operations and, consequently, affect the overlap of asynchronous processes, it is essential to provide a detailed breakdown analysis. Such an analysis can elucidate the extent of the optimization space and identify the scenarios in which these optimizations are most beneficial.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This manuscript is well-structured, making it easy to follow.\n2. It effectively integrates the proposed designs in the system implementation, resulting in significant efficiency improvements.", "weaknesses": "1. Lack of detailed breakdown analysis regarding the duration of operations and system overhead, including GPU-CPU data transfer overhead as well as the execution times of operations on both the GPU and CPU.\n2. Lack of analysis on various model and system configurations.\n3. The work demonstrates relatively modest innovation, primarily through the integration of established optimization strategies, such as overlapping using local expert computation, load prediction, and expert placement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9KjA0rZ8UR", "reviewer": "Reviewer_SZWt", "summary": "This paper addresses the critical challenge of expert load imbalance in distributed Mixture-of-Experts (MoE) model inference. The authors propose Libra, a system design that combines accurate prediction of expert activations with sophisticated load balancing algorithms while effectively hiding overhead through careful execution flow reconstruction. The system achieves up to 19.2% throughput improvement on state-of-the-art MoE models.", "questions": "see above", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This work is well-motivated by demonstrating that recent SoTA MoE models (DeepSeek-V3, Qwen3MoE, GLM-4.5) exhibit intensified load imbalance compared to their predecessors\n\n* The combination of algorithm observation (slow evolution of hidden states) and system design is novel and insightful.\n\n* The experiments cover multiple models, diverse datasets, and demonstrate both static and dynamic workload performance.\n\n* The paper is well-written with effective visualizations that make the technical approach accessible.", "weaknesses": "* Despite the overall \"slow evolution of hidden states\", how does the evolution vary acorss different MoE layers? For example, the early MoE layers could result in more significant hidden state dynamics, which might make the proposed method less effective.\n\n* I would love to see how Libra scales to multi-node deployments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)