Balancing the Experts: Unlocking LoRA-MoE for GRPO via Mechanism-Aware Rewards
Authors: Changlian Ma
Zizheng Huang
Xiangyu Zeng
Yi Wang
Cheng Liang
Kun Tian
Xinhai Zhao
Limin Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models (LLMs)
Reinforcement Learning from Human Feedback (RLHF)
Mixture-of-Experts (MoE)
Parameter-Efficient Fine-Tuning (PEFT)
Group Relative Policy Optimization (GRPO)
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce RO-GRPO, a method that prevents routing collapse in MoE models during GRPO by transforming internal routing statistics into a reward signal, enabling the simultaneous alignment of a model's behavior and its internal mechanisms.

Abstract: Parameter-efficient Mixture-of-Experts (MoE) architectures, such as LoRA-MoE, enable strong and generalizable fine-tuning. However, a critical problem arises when fine-tuning these architectures with advanced reinforcement learning algorithms such as Group Relative Policy Optimization (GRPO). Traditional supervised techniques are not naturally compatible with
the GRPO  objective, and naive combinations fail to effectively address routing collapse and the underutilization of MoE adapter parameters. To resolve this disconnect, we introduce Routing-Optimized Group Relative Policy Optimization (RO-GRPO), a mechanism-aware framework. It turns internal expert routing statistics collected during training into a direct reward signal, seamlessly integrating routing supervision into the reinforcement fine-tuning (RFT) process. This enables effective optimization of parameter utilization and improves performance on both unimodal and multimodal mathematical reasoning tasks, all without extra training stages. Our work provides the first demonstration that a scalar reward in GRPO can be engineered from a model's own internal mechanics to explicitly guide its optimization, extending alignment from mere behavior tuning to holistic mechanism alignment.

Published as a conference paper at ICLR 2026
BALANCING THE EXPERTS: UNLOCKING LORA-MOE FOR GRPO VIA MECHANISM-AWARE REWARDS
Changlian Ma
Zizheng Huang!4-
Yi Wang Xinhai Zhao Limin Wangl2t
Xiangyu Zeng!2
Cheng Liang Kun Tian*
Nanjing University
Shanghai AI Laboratory
Noah's Ark Lab, Huawei
*Shanghai Innovation Institute
*Shanghai Jiao Tong University
changli i @ smail .nju. edu. on
Imwang@nju. edu. on
ABSTRACT
Parameter-efficient Mixture-of-Expe -f-exrerts (MoE) architectures, such as LoRA-MoE. enable strong and generalizable fine-tuning. However, critical problem arises when fine-tuning these architectures with advanced reinforcement learning algo- rithms such as Group Relative Policy Optimization (GRPO). Traditional super- vised techniques are not naturally compatible with the GRPO objective, and naive combinations fail to effectively address routing collapse and the underutilization of MoE adapter parameters. To resolve this disconnect, we introduce Routing- Optimized Group Relative Policy Optimization (RO-GRPO), a mechanism-awan framework. It turns internal expert routing statistics collected during training into a direct reward signal, seamlessly integrating routing supervision into the rein- forcement fine-tuning (RFT) process. This enables effective optimization of pa- rameter utilization and improves performance on both unimodal and multimodal mathematical reasoning tasks, all without extra training stages. Our work pro- vides the first demonstration that a scalar reward in GRPO can be engineered from model's own internal mechanics explicitly guide its optimization, extending alignment from mere behavior tuning to holistic mechanism alignment.
INTRODUCTION
Large language models (LLMs) have significantly advanced many artificial intelligence applica- tions, but their large size poses challenges for practical deployment, especially regarding fine-tuning efficiency (Han et al. 2024, Hu al.) 2022). Among parameter-efffcient fine-tuning (PEFT) ap- proaches, LoRA-MoE (Dou et al. 2024). which combines Low-Rank Adaptation (LoRA) (Hu et al.)
2022] with a cfitture-of-expert
MMB)) architecture, has emerged as particularly promising tech-
nique (i et al.
2024a Gou et al. 2024 Mu & Lin 2025).
However, despite its success in supervised fine-tuning (SFT) (Dou et al. 2024 Li et al. 2024a), ap- plying LoRA-MoE to reinforcement learning fine-tuning (RFT) frameworks such as Group Relative Policy Optimization (GRPO) (Shao et al., (024) presents key challenge.
SFT, routing is typi- cally guided by auxiliary baad-balancing loss (Fedus et al. 2022 Lewis et al. 2021 Zoph al. 2022 Dai et al. 2022; Wang et al., 2024). In GRPO, our goal is to employ a loss-free free mechanism that jointly optimizes task performance and routing load through the rollout-time reward signal, rather than uniformly enforcing the same constraint on all routing decisions via an auxiliary routing loss. When GRPO relies solely on the external task reward, the training signal remains blind to internal routing decisions (Omi et al.) 20255 Harvey et al.] 2025). Without explicit guidance, the routing mechanism often collapses, leading to severe expert imbalance and underutilization of the model's 's parametric capacity, which limits the effectiveness of the modular architecture.
To bridge this gap. we propose RO-GRPO (Routing- Optttize- GRPO), novel framework that integrates routing-awareness into the RFT process through carefully designed mechanism reward. Our key insight is that routing statistics collected during generation, such as routing entropy and load distribution, can be transformed into a reward signal that aligns the internal routing mechanism
"quual contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
GRPO
Reference Model
Q1
r1
A
q
LLM
O2
Reward Model
Y22
Group Compulation
A2
OG
FC
Ac
RO-GRPO
Reference Model
01
A
LLM with LoRA-MoE
02
Reward Model
F2
Group
A2
Computation
Routing Info
O0
Routing Reward
Ag
Aggregator
Function
Figure 1: Comparison of standard GRPO and our RO-GRPO.
with task performance (Chen et al.) 2022, Cong al.) 2024). Specifically, we augment the standard task reward with two complementary components: one promoting confident routing decisions (low entropy) and another encouraging balanced expert utilization. These routing rewards are integrated into the GRPO objective, enabling a unified optimization process that requires auxiliary losses or architectural modifications (see Figure for a schematic comparison).
The main contributions of this paper are follows:
To our knowledge, this the first systematic study of LoRA-MoE architectures within the RFT framework, identifying and addressing key challenges such routing collapse and expert underutilization that arise during GRPO-based training.
We propose RO-GRPO, a novel framework that integrates routing statistics directly into the GRPO reward function. This enables the unified optimization of both task performance and internal routing efficiency without requiring auxiliary losses.
Our method achieves consistent improvements over baselines (e.g., standard LoRA and vanilla LoRA-MoE) across all expert counts in both task performance and expert utiliza- tion, validated across unimodal and multimodal mathematical reasoning benchmarks.
Our experiments provide the first empirical evidence that scalar reward in RFT can align not only a model's external behavior but also its internal mechanisms, opening new avenues for the principled alignment of complex model architectures.
RELATED WORK
Modular and Mixture-of-Expert PEFT. To enhance the capacity and versatility of PEFT, searchers have integrated Mixture-of-Experts (MoE) principles into LoRA, creating architectures like LoRA-MoE (Dou al.) 2024), MixLoRA (Li et al. 2024a), and MoCLE (Gou et al. 2024). These methods have shown effectiveness in reducing task interference and improving knowledge retention. Beyond adapterized MoE, modern routing builds on early conditional computation and deep MoE ideas (Cho & Bengio 2014 Eigen et al. 2014) and on system-scale routed Transformers such as GShard and Switch (Lepikhin et al. 2022 Fedus et al. 2022). Most approaches optimize expert routing during supervised pretraining or SFT. typically using auxiliary load lbad-balancing ob-
jectives to prevent expert collapse and under-utilization (Shazeer et al. 2017 Fedus et al. 2022) Alternative balancing mechanisms include the balanced assignment of BASE Layers (Lewis al.)
2021) and non-pan parametriiic routing via Hash Layers (Roller et al. (02T). Training stability and rout- ing fluctuation have been studied extensively, with design guidelines in ST-MoEE (Zoph al. 2022
2

---

Published as a conference paper at ICLR 2026
Token Info Flow Routing Info Flow
Riotal = Reaat
+
Ryouting
Generated Completions
Task Reward Function
Routing Reward Function
Routing Info
Aggregator
bost
Routing Statisties Collector
Luyer Norm
X
Layer Norm
Pretrained Weights
B1
B
AV
A
Router
Attention
Inputs
Figure 2: Overview of the RO-GRPO framework. A mechanism-aware reward Rroute computed from internal routing statistics and combined with the task reward R The resulting unified reward Riotal guides the GRPO update to jointly optimize task performance and routing efficiency.
two-stage stabilized routing StableMoo Dai et al. 2022 and empirical analysis expert-load dynamics (Cong al. 2024). However, these strategies remain predominantly designed for differen- tiable supervised training; integrating comparable mechanism-ave -ware supervision into reinforcement fine-tuning still underexplored.
Alignment and Optimization of LLMs. Reinforcement Learning from Human Feedback (RLHF) has become the dominant approach for aligning large language models with human inten- tions Ouyang et al.] 2022 Kaufmann et al. 2025). Proximal Policy Optimization (PPO) (Schulman al. 2017) and its variants, such Direct Preference Optimization (DPO) (Rafailov et 2023)
and Group Relative Policy Optimization (GRPO) (Shao et al.) 2024). have demonstrated strong performance on complex reasoning and instruction following, and have been further extended to multimodal settings such as video reasoning (Zeng et al., (026). These RLHF methods typically op- timize a scalar task-based reward and not incorporate supervision for internal mechanisms such as expert routing. In MoE-based models, this limitation can lead to expert collapse or inefficient parameter utilization when using RLHF directly (Fedus et al.) 2022, Harvey et al.) (025). While auxiliary losses have been used to encourage balanced routing in supervised settings, integrating such mechanism-awan supervision into reinforcement learning remains an open problem.
METHODOLOGY
In this section, we introduce RO-GRPO Coating-Optimized Group Relative Policy Optimization), a framework designed to optimize the internal routing of LoRA-MoE models by incorporating mechanism- supervision into the reinforcement learning loop. We first review the prelimi- naries of GRPO and LoRA-MoE, then describe the core challenge of unguided routing in RLHF, and finally detail our proposed solution.
3.1 PRELIMINARIES: GRPO AND LORA-MOE
Group Relative Policy Optimization. GRPO (Sho et al. 2024) is a critic-free RL algorithm that aligns LLM policy ne by maximizing the expected task reward over dataset of prompts D. Its objective is to maximize Eg-D, y~-m R (3)), where na is the LLM policy, D is the dataset of prompts, and Rack is the scalar task reward evaluating the output y.
3

---

Published as a conference paper at ICLR 2026
LoRA-MoE Architecture. A LoRA-MoE layer modifies the output of frozen pretrained layer. consists a trainable router network and set of parallel LoRA experts, { Be) For imput token representation h, the router first computes gating probability vector p softmax h) where W Wroter is the router's weight matrix. The final output of the layer is:
E
hout = h
Σ ple h) B.A)
3.2 THE CHALLENGE: UNGUIDED MOE ROUTING
fundamental disconnect arises when LoRA-MoE model fine-tuned using an RLHF algorithm like GRPO. The optimization process is blind to the router's decisions p, as the task reward Rusk evaluates only the final output. This lack of explicit supervision leads two well-documented failure modes in MoE training (Fedus et al. )
Expert Collapse:
The router defaults to choosing a small subset of experts, leading to severe load imbalance and wasted parametric capacity.
Routing Indecision: The router generates high-entropy distributions (i.e., low-confidence decisions), failing to foster expert specialization.
Our goal is augment the RLHF objective with internal, mechanism-aware reward signal that directly addresses these failure modes.
3.3 RO-GRPO: MECHANISM- AWARE AWARE REWARDS
As depicted Figure our method collects internal routing statistics during policy generation. These statistics are used to compute Ryoute, which is then combined with the primary task reward, Rusk. This is achieved in three steps.
3.3.1 STEP 1: QUANTIFYING ROUTING EFFICIENCY.
For each generated sample, we collect the routing probability vectors from all activated LoRA-MoE modules. Let M be the number of such modules in the model and T be the total number of tokens routed during generation. We quantify routing efficiency using two metrics aggregated from these statistics. First, we measure routing confidence using the mean Shannon entropy over all individual token routing decisions. lower value indicates more decisive routing. For the multiset of all T routing vectors {p.),-1, define the token-wiss wisee entropy as H(p) == Ee= Pe Pe (a small added in implementation). The average entropy
H = T Σ H(.)).
Second, we measure load balance by assessing expert utilization across the M LoRA-MoE mod- ules. For each module m {1,..., .M. first compute its average expert utilization vector pm by averaging the routing vectors all tokens that pass through The final metric M the mean of the Mean Squared Errors (MSE) calculated for each module relative uniform distribution
M
1 M
Σ m=1
Pm 111
(3)
For stable integration into the reward function, we normalize these metrics to an approximate [0, 1] range, yielding Haurm H/ In E,, Muorm = =M/(EE 1)/ 1)/E²), for use in the reward calculation.
3.3.2 STEP 2: FORMULATING THE ROUTING REWARD.
We propose two distinct strategies transform these metrics into scalar reward Rooute-
RO-GRPO (Smooth): Curriculum- Based Reward Scheduling, This strategy employs a curricu- lum that initially encourages confident routing (low entropy) and then transitions promoting load
4

---

Published as a conference paper at ICLR 2026
Table Performance unimodal mathematical reasoning benchmarks. We compare task accuracy (%), trainable parameter count (#fParam), and internal routing metrics. The Config column specifies the adapter structure, denoted as rank r for LoRA E x r for LoRA-MoE.
Unimodal Mathematical Reasoning (Qwen2.5-7B-Instrucc NuminaMath-TIRR TIR-2kk)
Method
Config
#Param
GSM8K
MATH
SVAMP
MGSM Entropy
MSE
Base (zero-shot)
0
87.34
70.42
91.33
53.64
16
30.3M
88.48
70.38
90.67
50.00
GRPO (LoRA)
32
60.6M
90.45
69.54
93.00
47.10
121.1M
90.14
49.02
92.67
53.67
2x8
31.7M
89.39
70.36
90.00
61.75
0.640
0.020
GRPO (LoRA-MoE)
4x8
63.4M
89.39
70.40
91.30
46.15
0.651
0.009
8x8
126.9M
90.22
70.44
91.00
52.04
0.655
0.008
2x8
31,7M
86.73
69.50
92.33
57.27
0.632
0.036
Aux-Loss (LoRA-MoE)
4x8
63.4M
87.11
70.10
91.00
56.36
0.540
0.024
126.9M
87.04
69.54
91.33
56.66
0.645
0.019
2x8
31.7M
91.51
70.64
91.00
62.18
0.639
0.016
RO-GRPO (Smooth)
4x8
63.4M
90.67
70.62
92.00
52,58
0.651
0.009
8x8
126.9M
90.98
69.78
92.67
52.04
0.656
0.006
2x8
31.7M
90.22
70.58
91.33
59.45
0.639
0.017
RO-GRPO (Relative)
4x8
63.4M
89.76
69.88
93.33
54.58
0.651
0.008
8x8
126.9M
90.52
70.18
92.67
51.96
0.655
0.007
balance (low MSE). As detailed in the Discussion section, this curriculum aligns the reward signal with the natural training dynamics of MoE models, defining the reward as follows:
Rrouie Wroute (WH (t) Hnorm + wD(() Mmarm)
(4)
where weights (t) and wp(t) are dynamically scheduled based training progress using a sigmoid function (1) Ite o with steepness k and center e
[0,
WH (t) = A o(t)),
(5)
w (t) X ott).
(6)
RO-GRPO (Relative): Relative Improvement Gating. This strategy provides sparse, adaptive
reward based on historical baseline, encouraging continuous self-improvement and avoiding the need manually balance the two routing objectives. A constant positive reward C is granted only if both routing confidence and load balance improve simultaneously relative to their exponential moving averages (Huist Muis):
Ryoute C'1(Hnorm < Hhist Muori < <Mhist
3.3.3 STEP 3: UNIFIED OPTIMIZATION VIA POLICY GRADIENT.
The mechanism-awarc reward Rjoute is combined with the external task reward to form the total reward (y) = Reaste (y) + Rroute The computation of Rroaee non-differentiable: its gradient is propagated implicitly through the policy gradient update. In the GRPO framework, a group of responses } is sampled for each prompt and evaluated using Rroa-- The resulting rewards are used to compute group-relative advantages, A, which in turn guide the policy update. The objective can summarized as:
(no-CRPO(0) Re E Σ, log To (y]l)) A, DKI(TT Tret)]
(
By integrating Proute into the advantage calculation, RO-GRPO guides the policy m to generate outputs that improve task performance while also exhibiting efficient routing, all without requiring differentiable auxiliary losses.

---

Published as a conference paper at ICLR 2026
Table 2: Performance on multimodal mathematical reasoning benchmarks. evaluation setup as in the unimodal experiments.
We follow the same
Multimodal Mathematical Reasoning (Qwen2.5-VL-77-Innstl on Geometry3k)
Method
Config #Paara
Geošk MathVista
Math
WeMath
Entropy
MSE
Base (zcro-shot)
37.44
46.50
26.50
56.95
16
30.3M
38.44
58.60
33.30
63.97
GRPO (LoRA)
60.6M
38.10
59,30
23.22
53.85
121.1M
33.28
55.90
25.43
53,91
2x8
31.7M
38.27
57.90
30.99
63.74
0.619
0.038
GRPO (LoRA-MoE)
63.4M
28.95
56,40
30.30
63.45
0.637
0.019
8x8
126.9M
33.11
55.00
31.78
61.49
0.649
0.012
2x8
31.7M
39.60
56.20
30.03
62.81
0.621
0.060
Aux-Loss (LoRA-MoB)
4x8
63.4M
41.43
60,50
27.23
62.87
0.634
0.036
8x8
126.9M
40.43
54.40
32.13
65.80
0.649
0.019
2x8
31.7M
38.94
58.70
30.48
66.09
0.630
0.033
RO-GRPO (Smooth)
4x8
63.4M
40.10
58.30
28.73
64.14
0.642
0.014
8x8
126.9M
38.94
58.90
27.89
64.10
0.648
0.011
2x8
31.7M
41.93
55.80
33.30
60.98
0.624
0.036
RO-GRPO (Relative)
4x8
63.4M
40.27
60.20
31.29
66.26
0.645
0.013
8x8
126.9M
40.16
60.10
32.03
63.97
0.636
0.010
4
EXPERIMENTS
We experimentally validate RO-GRPO by testing three core hypotheses: (1) applying LoRA-MoE with GRPO leads to suboptimal routing and underutilized parameters; (2) our mechanism-awan reward framework, RO-GRPO, mitigates these routing issues; and (3) these internal improvements translate to better task performance.
4.1 EXPERIMENTAL SETUP
Tasks and Datasets. We evaluate RO-GRPO challenging mathematical reasoning tasks. Per- formance on these tasks relies on precise, multi-step deduction, making it sensitive to the model's expert utilization and thus an ideal testbed for our approach. To demonstrate the versatility of our method, we conduct experiments in both unimodal and multimodal settings.
For unimodal experiments, we fine-tune on NuminaMath- TIR eL al. 2024b) and evaluate on the established benchmarks GSM8K (Cobbe et al. 2021). MATH (Hendrycks et al. (202I). SVAMP (Pa- tel et al. 2021), and MGSM (Shi al.J (2023). For multimodal experiments, we line-tune on Ge-
ometry3k (Lu et al. 2021 and evaluate on its test set, alongside Math Vista (Lu et al. 2024), Math-
Verse (Zhang al. 2025). and WeMath (Qiao et 2025).
Models and Baselines. We use the open-source Qwen2.5-7B 1 (Yang al. 2025 and Qwen2.5-VL-7BB 1ann:55-VVL7B--sttreee Instruct (Bai et al., 2025) models, chosen for their strong foundational perfor- mance in mathematical reasoning. We compare five configurations: (1) Base, the original pretrained model evaluated in a zero-shot setting; (2) GRPO (LoRA),, a standard LoRA baseline fine-tuned with GRPO representing a typical PEFT approach; (3) GRPO (LoRA-MoE), LoRA-MoE model trained with GRPO using only the task reward to isolate the effect of unguided routing; (4) Aux- Loss (LoRA-MoE). baseline where routing objectives are added auxiliary losses to the GRPO objective, using the same scheduling as the Smooth strategy; (5) RO-GRPO (Smooth), our method with the curriculum-bac based reward scheduling strategy (Section 3.3); and (6) RO-GRPO (Relative), our method with the relative improvement gating strategy for the routing reward (Section 3.3).
Evaluation Metrics. We evaluate both task performance and internal mechanism efficiency. Task Performance is measured by accuracy (%) on the respective benchmarks. Routing Performance is quantified by two metrics: (1) routing entropy. the average per-token Shamnon entropy as formu- lated in Eq. (2) indicating decision confidence; and (2) load balancing MSE, the mean squared

---

Published as a conference paper at ICLR 2026
Raseline
Smooth
Relative
0.65
0.648
0.648
Entropy
0.64
0.640
0.640
0.635
0.63
0,632
0.632
0.625
0.62
0.624
0.624
0.620
0.61
0.616
0.616
Baseline: Entropy
Smooth: Entropy
(a3) Relative: Entropy
All Methods: Entropy
0.048
0.048
0.040
0.040
MSE
0.024
0.024
0k
15k
30k
45k
45k
15k
30k
45k
15k
Steps
Steps
Stops
Steps
(b1) Baselinc: MSE
(b2) Smooth: MSE
(b3) Relative: MSE
(b4) Methods: MSE
Figure 3: Training dynamics of routing metrics on the unimodal mathematical reasoning task. (Top) Average routing entropy over the course of training. (Bottom) Load balancing MSE over the course of training.
G-E0
Baveñoe
G-FI
U-E0
0.2
0.6
1-E1
D-50
D-El
-0.4
-0.2
0.0
0.2
0.4
Smooth
G-EI
G-E0
Smooth
G
U-E0
U-E1
U
D-E0
D-F.1
D
Relalive
G-EI
Relative
U.E0
U-F
D:E00
D-E1
21
24
21
24
Layer Index
Index
(a) Expert Routing
(b) Routing Balunce Improvement
Figure Visual analysis of routing behavior improvements with RO-GRPO on the Math Vista benchmark. (a) Left Panel: Heatmaps show the routing ratio of the most frequently selected expert for the baseline and our two RO-GRPO methods. Darker colors represent a higher selection ratio for the dominant expert. (b) Right Panel: Heatmaps quantify the improvement in routing balance relative to the vanilla LoRA-MoE baseline. Positive values (warmer colors) indicate a reduced rout- ing ratio for the dominant expert, signifying shift toward the desired 1/ equilibrium.
error between the expert utilization distribution and a uniform one as formulated in Eq. 3 indicat- ing load balance. We report these raw, un-normalized values for direct interpretability.
Implementation Details. To ensure fair comparison, we control for trainable- paramete budget: the LoRA baseline uses rank of 16, while LoRA-MoE models use E experts (E E (2,4, 8)), each with rank 8. For LoRA-MoE, modules are inserted into the gate, up, and down projections of the Feed-Forward Network (FFN) in each transformer block. During training, only the PEFT parameters are updated while the base model weights remain frozen. Across all experiments, we consistent system prompt for both training and evaluation to encourage step-by-step reasoning. The overall routing reward Rroate integrated into the total reward using global scaling coefficient of Wnoute 0.2. Key hyperparameters for our routing strategies were determined through validation.
For the Smooth strategy, we set the initial entropy weight Astart 0.5 and the final balance weight Amd = 2.0. For the Relative strategy, the performance baseline was calculated over a moving window of the 1000 most recent samples. Further details in the Appendix.

---

Published as a conference paper at ICLR 2026
4.2 MAIN RESULTS
As shown in Tables Tan RO-GRPO yields consistent performance gains over vanilla GRPO with LoRA-MoE across all expert counts (É'E {2, 4, 8)) in both unimodal and multimodal settings. The two reward variants show complementary strengths: the Smooth strategy performs best on GSM8K and SVAMP, while the Relative strategy excels on Geometry3k, Math Vista, and WeMath.
Unimodal. On GSM8K, RO-GRPO (Smooth, E-2) achieves the top score of 91.51%, an im- provement of +1.06 pp over GRPO (LoRA) and +1.29 pp over the best-performing vanilla LoRA- MoE model (at E- On SVAMP, RO-GRPO (Relative, 4=4) obtains 93.33%, surpassing GRPO (LoRA) by +0.33 pp and vanilla LoRA-MoB (at 4=4) by +2.03 pp. While improvements on MATH modest, they are consistent matched expert counts (e.g.. 70.64% vs. 70.36% for F=2 Smooth vs. vanilla). The largest gain on MGSM is at E=4, where RO-GRPO (Relative) achieves 54.58%, +8.43 pp increase over the vanilla model.
Multimodal. RO-GRPO (Relative, 2'=2) attains the highest Geometry3k score of 41.93%, outper- forming vanilla LoRA-MoE (E=2) by +3.66 pp and GRPO (LoRA) by +3.49 pp. On WeMath, the best results are with RO-GRPO (Relative, E-4), which reaches 66.26%, gains of +2.29 pp over GRPO (LoRA), and +2.52 pp over the best vanilla LoRA-MoE model (at E=2), respectively. On Math Verse, the top performance of 33.30% is shared by GRPO (LoRA) and RO-GRPO (Relative, E=2).
Overall, across expert sizes, matched E' our routing-awan aware training either matches or surpasses vanilla LoRA-MoE on nearly every benchmark, with the largest margins on Geometry3k and We- Math. These results reaffirm that aligning the router with mechanism- rewards translates into stronger task performance.
4.3 ANALYSIS OF ROUTING MECHANISM
Unguided Routing under Vanilla GRPO Is Brittle. At F=2 the multimodal setting, vanilla LoRA-MoE appears confident (Entropy 0.619) but exhibits routing collapse (MSE 0.038). At larger E. the raw MSE decreases (e.g., multimodal 0.019 and 0.012 for E=4 and E=8), yet accuracy does not reliably improve and can even drop (Geometry3k score of 28.95% at 4-4), revealing instability in the absence of mechanism aware feedback.
RO-GRPO Restores Balance at Matched E. Across all experimental configurations, RO-GRPO reduces or matches the MSE of the vanilla baseline at the same E (unimodal: 0.020 00.016/0.017 0.009 - 1.009/0.008 0.008 + 0.006/0.005 multimodal: 0.038 - 0.033/0.036, 0.019 + 0.014/0.013, 0.012 -> 0.011 10.0100), while maintaining comparable entropy. These improvements in routing correspond the largest accuracy gains on Geometryšk, WeMath, and SVAMP. The training dynamics in Figure 3 further confirm that RO-GRPO yields more stable routing trajectories, and the layer-wise heatmaps in Figure 4 show consistent reduction in single-expert dominance across transformer layers.
Routing Rewards Mitigate Text Degeneration. On Geometry3k with E-4, the vanilla GRPO (LoRA-MoE) model exhibits repetitive-loop failures in 7.5% of generations and scores 28, 28.95%. In contrast, RO-GRPO (Smooth) reduces these fallures to 0.17% and RO-GRPO (Relative) eliminates them entirely. achieving accuracies of 10% and 40.27%, respectively. This suppression of degen- erate outputs also translates into a more favorable inference efficiency profile. As shown in Figure RO-GRPO achieves higher accuracy on Geometry3k at lower inference latency compared to all base- lines, indicating that mechanism- -awaie routing effectively promotes concise reasoning and reduces unnecessary token generation. A detailed token efficiency analysis is provided in Appendix G
Mechanism-Aware Rewards Outperform Auxiliary Losses As shown in Tables I and 2 the Aux-Loss baseline consistently underperforms RO-GRPO, particularly unimodal tasks. While auxiliary losses successfully reduce routing entropy, they fail to improve load balance, often yielding higher MSE values compared to our reward- based approach (e.g., 0.036 vs. 0.016 GSM8K with E=2). Furthermore, the auxiliary-loss method tends to generate significantly longer sequences without yielding commensurate accuracy gains, as detailed in Appendix and Appendix
8

---

Published as a conference paper at ICLR 2026
44
Method
42
Better Trade-off
GRPO (LoRA-MoE)
Aux-loss (LoRA-MoE)
Task Accuracy (%)
40
RO-GRPO (Smooth)
38
RO-GRPO (Relative)
GRPO (LoRA)
36
Config
34
E=2 r=
32
E=4 r=32
E=8 / r=
30
28
5000
7500 10000 12500 15000
Average Inference Latency (ms/sample)
Inference latency vs. task performance trade-off on Geometry3k. RO-GRPO achieves a superior Pareto frontier compared to all baselines, attaining higher accuracy with lower latency.
Figure 5:
Table 3: Ablation and causal analysis on GSM8K (E==2). Our full RO-GRPO model sig- nificantly outperforms the vamilla baseline. Sub- sequent experiments demonstrate that both re-
0.040
Eniropy
MSE
0.032
ward components (RH, RB) are necessary for
0.024
optimal performance, and control experiments validate that the gains are causally driven by our targeted reward signal.
(af) Only Ru: Entropy
(a2) Only Ri: MSF
Entropy
0.625
MSE
0.060
0.045
Configuration
GSM8K
Entropy
MSE
0.030
RO-GRPO (Smooth)
91.51
0.639
0.016
0.575
RO-GRPO (Relative)
90.22
0.639
0.017
(b1) Only Ra: Entropy
(b2) Only Ra: MSE
GRPO (LoRA-MoE)
89.39
0.640
0.020
0.648
0.036
Ablations on Reward Components:
Entrupy
0.642
0.030
w/o Ra (Smooth)
90.75
0.639
0.018
0.636
MS
w/o Ri (Smooth)
89.92
0.639
0.019
0.024
w/o Rp (Relative)
89.01
0.638
0.019
0.630
0.018
w/o Ry (Relative).
90.14
0.637
0.019
50k
100%
150k
0k
50%
100%
150k
Causal Sanity Control::
Steps
Steps
Shuffled (Smooth)
89.23
0.640
0.018
Smooth: Entropy
(c2) Smooth MSE
Shuffled (Relative)
0.020
Figure 6: Training dynamics of routing metrics when the Reas set to zero.
4.4 ABLATION AND CAUSAL VERIFICATION
Ablation experiments on GSM8K confirm our approach's 's integrity (Table
Contribution of Reward Components. We first investigate the individual contributions of the confidence (Rz) and balancing (R.) rewards. As shown in Table 3 removing either component from our best-performing model, RO-GRPO (Smooth), reduces accuracy. Specifically, removing the balancing reward (w/o Rs) or the confidence reward (w/o Rx) reduces the GSM8K score from 91.51% to 90.75% and 89.92%, respectively. This dependency is more pronounced for the RO- GRPO (Relative) variant: removing its balancing reward (w/o o Rg) causes performance to drop to 89.01 %, below the vanilla baseline. These results demonstrate that the two reward components are synergistic and critical for optimal performance.

---

Published as a conference paper at ICLR 2026
Contribution of the Reward Signal. To ensure performance gains are driven by meaningful feed- back, we performed a control experiment. In the Shuffled Control, we randomly permuted routing rewards within each batch, breaking the causal link between action and its reward. As shown in Table performance under this condition dropped significantly for both the Smooth (89.23%) and Relative (89.28% variants, falling the level of the vanilla GRPO (LoRA-MoFi) baseline (89.39%) This result strongly suggests the gains from RO-GRPO are causally driven by the targeted feedback from our reward signal, not by an artifact of the reward structure.
DISCUSSION
Our experiments demonstrate the empirical success of RO-GRPO across 4,8. This section analyzes why unified scalar reward can supervise a model's s internal router and why this becomes more important E grows. A more detailed derivation is available in Appendix F
The Rationale for a Curriculum-Based Reward. The Smooth curriculum is effective because single-objective optimization is suboptimal: rewarding only low entropy degrades balance (MSE rises), whereas rewarding only balance is initially too weak to shape specialization (Figure ( By first encouraging confident routing and then increasing pressure balance, the curriculum builds specialized experts and subsequently organizes them. This dynamic mirrors our empirical trends at E-4, 8, where mechanism- supervision not only improves accuracy but also suppresses degeneration on Geometry3k.
Rewards vs. Auxiliary Losses in RL. A critical insight from study is the superiority of mechanism-aware rewards over auxiliary losses in the GRPO framework. When routing supervision formulated as reward, it is integrated into the group-relative advantage calculation. This allows the model to learn trade-offs: a trajectory with slightly imbalanced routing can still receive a positive advantage if it yields correct answer. Conversely, an auxiliary loss applies a uniform penalty to all trajectories in a batch regardless of their task success. This rigid penalization can suppress useful but unconventional routing pattems required for complex tasks, leading the suboptimal performance. Additionally. RO-GRPO shares the same architecture and parameter count as vanilla LoRA-MoE, incurring only an ((TF) post-hoc reward computation overhead per layer, which negligible com- pared to the O(T K dr) forward-pass cost given E < dr for typical configurations. Crucially, unlike the auxiliary-loss approach, RO-GRPO requires no additional backward- sakddd--ppsss pass gradient computation (VxCaux). further reducing both computational cost and VRAM usage.
Grounding the Reward Components. The confidence reward, Ris, which promotes low-entropy routing, be understood through the Information Bottleneck (IB) principle (Tishby et al. 2000). The IB principle states that an optimal representation should compress an input while preserving task-relevant information. In framework, the router's decision acts as this bottleneck. By warding low-entropy (confident) decisions, RO-GRPO incentivizes the router to learn a minimal sufficient representation of its input. It is encouraged to discard noisy features and focus infor- mation predictive of task success, a process that naturally fosters expert specialization.
The balancing reward RB directly optimizes parameter utilization. Maximizing our balancing ward, which is formulated using MSE, is formally equivalent to minimizing the variance of the expert load distribution. This ensures the reward signal provides a direct and efficient gradient for combating routing collapse and ensuring the model leverages its full parametric capacity, principle established in supervised MoE training (Shazeer et al. 2017).
CONCLUSION
We addressed core limitation of applying LoRA-MoE to GRPO: the task reward is blind to routing. RO-GRPO remedies this by transforming routing statistics into a mechanism-awar reward that plugs into GRPO without architecture changes extra stages. Across E-2, 4, 8 and both unimodal and multimodal math reasoning, RO-GRPO improves load balance at matched E, boosts accuracy, and reduces text degeneration. These results indicate that reinforcement learning align not only external behavior but also internal mechanisms, suggesting a path toward principled alignment for complex modular architectures.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our findings, provide comprehensive set of resources. The com- plete source code for all experiments, including model implementation and training and evaluation scripts, is available in the supplementary material. Further details and complete list of hyperpa- rameters for all model configurations are documented in Appendix alongside pseudocode for our reward calculation algorithms.
ACKNOWLEDGEMENTS
This work is supported by the Basic Research Program of Jiangsu (No. BK20250009) and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
REFERENCES
Shuai Bai, Keqin Chen. Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng. Hang Zhang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
URL https: /tttps:/arxivv org/ abs 2202. 13923
Zixiang Chen, Yihe Deng. Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards un- derstanding the ofeperrsof-experts layer in deep learning. In Advances in Neu- ral Information Processing Systems, volume 35, pp. 23049-23062, 2022. URL
Kyunghyun Cho and Yoshua Bengio. Exponentially the haapssinnnhhccappppetoo-ooooppp ratio for conditional computation deep learning, 2014. URL /ull:tttpps///ar.v.o///////////
7362
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021. URL https://arxiv..
org/abs/ 21100. 14168
Peizhuang Cong, Aomufei Yuan, Shimao Chen, Yuxuan Tian, Bowen Ye, and Tong Yang. Predic- tion is all moe needs: Expert load distribution goes from fluctuating stabilizing, 2024. URL
16914
Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. Stable- MoE: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers), pp. 7095-7095, 2022. htt////aaaaattooooo/oo www.oog..oooooooo...o..........o 2022 acl long. 489/
URL
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao. Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi. Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng. Tao Gui, Qi Zhang, and Xuanjing Huang. LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style plugin. In Proceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pp. 1932-1945, 2024. URL owt.oog//2222. 1one.1--oog.10000
htttps://cclanthen wtwp:ooo...rg/222...ooo
David Eigen, Mare' Aurelio Ranzato, and llya Sutskever. Learning mixture of experts, 2014. URL/htppee//arxiii.orr////////11
factored factored repressttios in a deep
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(12)):1-39, 2022. URL httpt:/t//tt wttpppppess/////22222.
html html

---

Published as a conference paper at ICLR 2026
Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Din-Yan Yeung, James T. Kwok, and Yu Zhang. Mixture of cluster-condittonaal lora experts for vision-language instruction tuning, 2024. TRLhhttps: //tp:/i///axvvv.o org/ http:///bbb....../2222
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efffcient fine-tuning for large models: A comprehensive survey. Transactions on Machine Leurning Research, 2024.
ISSN 2835-8856. htthhttss///ppeeeee eww net/ / forum? i 1= IsC58b62
Daniel Fidel Harvey, George Weale, and Berk Yilmaz. Optimizing moe routers: Design, imple- mentation, and evaluation in transformer models, 2025. URL httph: www.xii.orrg/......
2546.16419
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solv- ing with the math dataset. In Proceedings of the Neural Information Process- ing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https: //Abaaattseeecchmaarssssrrreeeee meg neurips http://pprrrfieesppppppaaa Paperr round2 pdf
2021 20LLFFLL/2Bsaa23e000777222200022
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con-
ference on Learning Representations, 2022. URL
wwt.:/n.eeeeeeeeeeeeeeeeeeeeeeee net/fooum
Ed=
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedhack. Transactions on Machine Learning Research, 2025. ISSN 2835- 8856. URL htt.attpsss/ppeneeeeeew.eeee
forum http:///we.....t.rooooooooo///r-
Survey Certification.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. International Conference on Learning Representations, 2021. URL //tpn::e///peereeeeeeee net/ www.ooodddggrrr....... 7XHTmYb.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 6265-6274, 2021. URL https:
mlr.press/vii wttpp//ssww//ewwe///e. html
Dengchun Li, Yingzi Ma. Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng. Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts, 2024a. URL https::/aaxxi..ort
abs/2404.15 15159
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face, 2024b. URL hhttps://huggiiggtaee
co/ dataset s AI MO MO/ Num naMath- CoT
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pp. 6774-6786, 2021. URL https:://cccccc antho ogy. 2021 -aalllllo long. 5287
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, 2024. URL
Siyuan Mu and Sen Lin. A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications, 2025. Utl.itttpp:://rrwv..orr//////
12

---

Published as a conference paper at ICLR 2026
Nabil Omi, Siddhartha Sen, and Ali Farhadi. Load balancing mixture of experts with similarity preserving routers, 2025. URL https:t/p/////c httt::///Axxii..oo//ab////3333
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with hu- man feedback. In Advances in Neural Information Processing Systems, volume 35, pp. 27730- files/paper/ pdf
27744, 2022. URL http:://pooceed......ee......... 2022/ 002.22E000E000533333550000000000
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094, 2021. http:tsss//aaaattllllooooo
168/
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Jiapeng Wang, Zhuoma GongQue, Shanglin Lei, Yifan Zhang. Zhe Wei, Miaoxuan Zhang. Runfeng Qiao, Xiao Zong. Yida Xu, Peiqing Yang, Zhimin Bao, Muxi Diao, Chen Li, and Honggang Zhang. We-math: Does your large multimodal model achieve human-like mathematical reasoning? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers). pp. 20023-20070, 2025. URL htlp:::://claathooooyy.ooo 2025 acl 201.........1n... 983/
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pp. 53728- 53741, 2023. URL 000001/1188800000377777642333333 2023/F1le
Stephen Roller, Sainbayar Sukhbaatar, arthur szlam, and Jason Weston. Hash layers for large sparse models. In Advances in Neural Information Processing Systems, volume 34, pp. 17555- 17566, 2021. URL https::/prooeeedingg. www.:eiiiiiinnnnnuuep.........ee www.nnuuip.... cw/papeppppppee.eeeeeeee... files/paper/
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, K. Li, Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL ptttpps:/arxivv.oom 0r/
03300
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. In International Conference on Learning Representations, 2017. URL https: //openrevie eww. www//ttttuuemm d=B1 ckMDglg
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thouge reasoners. In International Conference on Learning Representations, 2023. URL https:://opereeiiew net forrmmiif
ER3wGCK-IXp
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000.
URL //tp
Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary load balancing strategy for mixture-of-experts, 2024. URL :ttta://arxiie http///xxv..ogggbb///// wwwp:/ggg/bbbsss..... 15664
13

---

Published as a conference paper at ICLR 2026
Xujia Wang. Haiyan Zhao, Shuo Wang, Hanqing Wang, and Zhiyuan Liu. MALoRA: Mixture of asymmetrie low-rank adaptation for enhanced multi-task learning. In Findings of the As- sociation for Computational Linguistics: NAACL 2025, pp. 5624-5641, 2025. URL https:
//aclanthology org/2025. Windingss - naacl
3122/
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts, 2024. URL https://arxix
org/abs 2404 13628
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL http//axxvvv/ http://aaa.....rra/////242..2...
Xiangyu Zeng, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Zikang Wang. Changlian Ma, Qingyu Zhang, Zizheng Huang. Kun Ouyang. Tianxiang Jiang, al. Video-o3: Native interleaved clue seeking for long video multi-hop reasoning. arXiv preprint arXiv:2601.23222 2026.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Wei Chang, Yu Qiao, Peng Gao, and Hongsheng Li. Mathverse: Does your multi- modal Ilm truly see the diagrams in visual math problems? In Computer Vision ECCV 2024, pp. 169-186, 2025.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL https: /tap:///axivv arxiv. org/
abg/abs////ee. abs 2202. 08906
14

---

Published as a conference paper at ICLR 2026
A HYPERPARAMETER AND IMPLEMENTATION DETAILS
Our approach to hyperparameter selection is designed be both systematic and efficient. For established components of the training pipeline, such as the GRPO algorithm and the LoRA architecture, we adopted values from seminal works and common practices estab- lish strong, competitive baselines. Our primary tuning efforts were concentrated on the novel parameters introduced by the RO-GRPO frame- work, ensuring a rigorous evaluation of our core contributions.
Table 4: Hyperparameters for all experiments.
Parameter
Value
GRPO Configuration
Learning Rate
1 x 10
KL Coefficient (3)
0.1
Batch Size
64
Generations per Prompt (k)
Epochs (Unimodal)
Epochs (Multimodal)
3
LoRA/ / LoRA-MoE Configuration
LoRA Rank (r)
16
LoRA-MoE Rank (r)
8 (per expert)
Number of Experts (E)
{2.4.8)
LoRA Alpha (a)
32
Core Training and Architecture Parameters.
LoRA Dropout
0.05
For all experiments, used learning rate of 1x10 and a batch size of 64. The GRPO con- figuration included KL coefficient (3) of 0.1 and sampling 8 responses per prompt (A: = 8) for advantage estimation. For the base LoRA architecture, we set the rank to = 16 and al- pha to a 32. For our LoRA-MoE models, we used E E {2, 4, 8} experts, each with rank of r = 8 and an alpha of a = 32, maintaining similar parameter budget. The training duration was set to epoch for the unimodal Numina- Math dataset and 3 epochs for the more com- plex multimodal Geometry3k dataset to ensure convergence. The external task reward weight was consistently set to 1.0.
RO-GRPO Specific (Optimal Values)
Global Routing Weight (Woome)
0.2
Smooth Strategy
(Entropy Weight Start)
0.5
NB
(Balance Weight End)
2.0
Sigmoid Steepness (k)
20.0
Sigmoid Center (c)
0.5
Relative Strategy
History Window Size (Sma)
1000
Reward Constant (C)
1.0
Note: Sigmoid steepness (k) controls the transi- tion speed curriculum, and the center (c) defines the transition point in terms of training progress.
RO-GRPO Routing Reward Parameters. The most critical hyperparameters are those governing the mechanism- aware routing reward, Ryouee- We conducted grid search to determine the optimal settings for both our adaptive strategies, using a held-out validation set.
For the Kurriculum-Basel Reward Scheduling (Smooth) strategy, we explored the key parame- ters controlling the curriculum's shape and intensity. The search space included the final load bal- ancing weight Ann E {1.0, 2.0, 5.0). the initial entropy weight satar E {0.5, 1.0]. and the sigmoid steepness E (15, 20, 25}. The pseudocode for this strategy is detailed Algorithm 2
For the Relative Improvement Gating (Relative) strategy. the key parameter is the history-size, which defines the window for the moving average baseline. We searched over values in {100, 500, 1000). The logic for this strategy is presented in Algorithm
A separate grid search was performed for the global routing reward weight, which scales the entire Rroate term, across the range {0.1, 0.2, 0.5). Our experiments indicated that weight of 0.2 provided the best trade-off between improving task accuracy and optimizing routing efficiency (i.e., minimiz- ing load balancing MSE and routing entropy). This value was used for all reported RO-GRPO results.
The final, optimal hyperparameters selected through this process summarized in Table
15

---

Published as a conference paper at ICLR 2026
Algorithm 1 RO-GRPO Reward Calculation (Relative Strategy)
Algorithm 2 RO-GRPO Reward Calculation (Smooth Strategy)
Input: Routing statisties stats, history buffer
Input: Routing statistics stats, s, current step four max steps fmax
Buist
Parameters: Reward constant C, history buffer size Spia
Parameters: Global weight Whone, entropy start weight A# balance end weight X sigmoid cen- sigmoid steepness k
(Require sufficient history to establish baseline]
[Calculate curriculum progress and sigmoid value]
if Buia < Smis then return 0 end
lourr/ Umax
- (1 (1+ --p--------
[Compute metrics for the current sample] (Maur,, Howr) <- ComputeMetrics(taaeee
[Schedule the weights for entropy and balance]
WH A
WB
AB
(Compute historical average baseline)
Muis + Averago Buss mss) Huis + Averago Buist entropp)
[Compute average normalized metrics from stats}
Hnom Avvenge/ormliizedtnnoyysssssss
(Grant reward only if both metrics improve] if Meur < Muis and Heur <. Hais then Rioue + C
Maora sterageNommallcccMM(((taattt))))
Calcullaee final reward (a negative penalty))
else
Reouee + Wroute (M Haxm + Wp Mnoo)
Romee
return Rrone
(Updtte the history buffer with current metries} Update( Buiss (Meune, Hamr)
return Roone
B SYSTEM PROMPT
We used a consistent system prompt all models during both training and evaluation to encourage step-by-step reasoning. The prompt is provided below:
A conversation between User and Assistant. The user asks question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasening process and answer are enclosed within <think> </think> and<answer> </answer> tags, respectively, i.e., <think> reasoning process heres/think>><asswe
answer here </answer>
C QUALITATIVE CASE STUDY
To provide comprehensive qualitative analysis the impact of our method, we present case studies spanning both unimodal arithmetic reasoning (GSM8K) and multimodal geometric reasoning (Ge- 3M) Tables and ods against RO- GR-GRPO framework. These examples empirically demonstrate that RO-GRPO not only corrects the logical hallucinations and repetitive loops observed in baselines to ensure higher accuracy, but also achieves these correct solutions with reduced token consumption, thereby improv- ing inference efficiency.
the specific generation trajectories of the bascline meth-
16

---

Published as a conference paper at ICLR 2026
Table 5: Case study comparing method outputs on a sample from the GSM8K benchmark.
Aspect
Content
Problem
Mary avid gardener. Yesterday, she received new polted plants her favoritt plant nursery. She already potted plants on each of the 40 window ledges of her large country home. Feeling generous, she has decided that she will give potted plant from each ledge friends and family tomorrow, How many potted plants will Mary remain with? Answer:
Solution
Yosterday, before receiving the plants, Mary had 2 40 80 potted plants. After rocciving an additional 18 plants, she therefore had total of 80 + 18 = 98 pottee plants.
potted plants. She will therefore remain with 98 10 58 potted plants. Final Answer: 58
Tomorrow, Mary's plant giveaway will
Base Output
<think> First, we calculate the total number of potted plants Mary already has, Since she has potted plants on cach of the 40 window ledges, total 2
80 potted plants.
Nest, we determine bow many potted plants she will give away. Since she plans to give potted plant from each of the potted plants.
window ledges, she will give away x
Finally, we subtract the mumber of potted plants she gives away from the total number she originally had. So, 80 10 40.
Therefore, after giving away the potted plants, Mary <ansmer>40</ansse answer>
will remain with 40 potted plants. </think>
GRPO (LoRA- MoE) Output
<think> First, we calculate the total number of potted plants Mary already has. Since she has potted plants on each of the 40 window lodges, the total is 2 x 10 = 80 potted plants.
Next, we determine how many potted plants she will give away. Since she plans give potted plant from each of the 40 window leiges, she will give away x 40 40 potted plants.
Finally, subtract the number of potted plants she gives away from the total number she originally had. So, 40 40.
Therefore, number of potted plants Mary will remain with 40, </thinic> <answe"> </arswer>
Aux-Loss (LoRA- MoE)
<think>
Let's break down the problem step by step:
1. Initial Plants:
Output
Mary already has 2 potted plants on each of the 40 window ledges.
Therefore, the total number of plants initially has x 40 80 plants.
2. Plants Given Away:
Mary plans to give polted plant from each ledge to friends and family.
Since there 40 window lodges, she will give away x 40 40 plants.
3. New Plants Received:
Mary received
potted plants from plant nursery yesterday.
4. Total Plants Afler Receiving. New Ones:
Adding the new plants her count,
becomes
98 plants.
5. Final Count After Giving Away Plants:
Subtracting the 40 plants she gives away from the total, we get 98 40 58 plants remaining. </think> <answer>
Therefore, the number of potted plants Mary will remain with is 58 </
RO-GRPO (Smooth) Output
<think> First, we calculate the number of potted plants Mary already of the 40 window ledges, the total is 2 x 40 = 80 potted plants.
she potted plants on each
Next, we consider the 18 new potted plants she received yesterday, Adding these existing plants gives 50+-18 potted plants in total.
Mary plans to give away potted plant from each of the 40 window ledges. This means she will give away 40 x 40 potled plants.
To find out how many potted plants she will have left after giving some away, we subtract the number of plants given away plants. </answer>
plants has: 98
58, </thinks <answer> Mary will remain with polled
RO-GRPO (Relative)
<think> Mary initially has potted plants each the 40 window ledges. So, the total number of potied plants she has hefore receiving the new ones is:
Output
plantsfledge ledges 80 plants
receiving new potted plants, number potted plants becomes:
80 plants 18 plants 98 plants
Mary plans to give away ported plant from each of the 40 window lodges. Therefore, the number of plants she will give away is:
plant/ledge x 40 ledges 40 plants
Suhtracting the numher of plants given away from the total number of plants gives us the numher of plants Mary will remain with:
98 plants plants 58 plants
</chink> <answers Mary will remain with 58 potted plants. </ </answer>
Continued on next page
17

---

Published as a conference paper at ICLR 2026
Aspect
Content
Analysis
Both Base Modell und GRPO (LoRA-MoeE) ignure the 13 patted plants, leading incorret In contrast, our RO-GRPO models successfully integrate all information to derive the correct solution. Moreover, RO- GRPO generates fewer tokens than the Aux-Loss method when both yield count by approximately 25% and Relative by 10%. This demonstrates that our mechanism- aware rewards foster robust reasoning and
Smooth reducing
logical deficiencies observed baselines, moving beyond superficial format adherence.
Table 6: Case study comparing model outputs Geometry3k geometry problem.
Aspect
Content
2x+ 8
3
2y
Problem
Find xr.
Answer
Base Output
<think> To find the value of is we need to use the property that the slopes of two parallel lines are equal. From the image, we can see that lincs
and a are parallel, and lines 9 + and 2y are parallel.
First, let's lind the slope of the line +--+-3 +.3. equation 2xr 3 can be rewritten in the slope cupe-intercep form mz+ where m is the slope. Here, the slope is slope line
slope-intereept form, get y -r + slope parallel, slapes must equal. Therefore, set slopes equal
Rewriting
here is -l. other:
3 and
This contradiction, which means there might a misundarstanding if consider the slopes the lines & + and 2g. fin the slope 2y by rewriting slope-intereept form. The equation 2y can be rewritten Since the lines y 1 and 2y parallel, their slopes must be equal. Therefore, set the slopes equal to each other:
interpretation the problem. However,
f. so the slope is 2.
This contradiction. suggests problem might misinterpreted. should consider the slopes of the lines in the context of the problem. If we ussume the problem is correctly set up and we to find Since the problem Therefore, we can set slopes equal to each other and solve for z. The correct interpretation should be that the slopes of the lines can assume correct interpretation is that slopes of the lines must be equal. Therefore, we can set the slopes equal to
typical problem setup,
parallel, can is not clear, can assume the correct interpretation is that the slopes of the lines must be equal.
slopes
equal.
equal, and since the problem setup not clear,
for The correct answer is 3. </think>
<answer>
</ /anserr>>
Continued next page
18

---

Published as a conference paper at ICLR 2026
Aspect
Content
GRPO
property that the slopes of two parailel lines equal. From the diagram, we can see that the lines + and an are parallel, and lines 3 + and 2y are parallel. line 2x The slope-intereept form line Rewriting 2n + 3 in slope intercept form, we get
<think> lind the value
need
(LoRA) Output
First, let's find slope
+ where slope.
2z. + 3,
so the slope of the line 2 + is -2. Next, let's lind the slope the line 6 Rewriing slope intercept form, get
6,
so the slope of the line 6 is the lines +
are parallel, slopes be equal:
This contradiction, so we need to re-evaluate the problem. The correct interpretation is that the slopes of the lines and x are not dircetly comparable as they are not the same form. Instend, we should consider the slopes of the lines 3 and
Rewriting 3 in slope intercept form,
y
3
so the slope of the line &u + 1 is Rewriting 2y slope intercept form, we get
so the slope of the line 2y 1. the lines $ 2y parallel, slopes equal:
This also contradiction, so we need re evaluate the problem again. The correct interpretation is that the slopes of lines 2z and 6 the slopes of the lines 2x + and x as they are not parallel.
directly comparable they are the form. Instead, should consider
The correct interpretation is that the slopes of the lines 2z + and 6 are not directly compurable as they not same form. Instead,
slopes
as they are not parallel.
The correct interpretation is that the slopes of the lines 2z + and 6 a are not directly comparable as they same form. Instead, should consider the slopes the lines
parallel.
The correct interpretation not in the same form. Instead, we should consider the slopes the lines 2xr and 6 as they are not parallel.
slopes
+
directly compurable as they
(Repet Livve loop intil bitting the meximun. Length)
Continued next page
19

---

Published as a conference paper at ICLR 2026
Aspect
Content
GRPO
<think> salve for vectors are parallel, which means their slopes must he equal.
undersiand the relaionship between the and given expression.. The
(LoRA- MoE)
The slope vector 2x the confficiont of which The slope the vector 6 coefficient as which is -1.
Output
Since the vectors parallel, their slopes must equal:
However, this is possible the slopes equal. This indicates that the problem might be misinierpreted there might he a misunderstanding in the setup. Given the setup, let's assume the vectors are meant to he parallel in different context there might be need ro-evalunte the problem constraints.
Given the setup. the model keeps assuming the vectors are meant to be parallel in a different context and repeatedly reaching conclusion or producing an <answer> lag.
evaluntes" the constraints
Since problem setup seems to be incorrect or misinterpreted. let's assume the vectors are meant to be parallel in different context there might be need evalintt the problem constraints.
Given the setup, the model odly "re-evaluates" the constraints without reaching a conclusion or producing an <anawer> tag.
assuming the
parallel different context and repeat-
Since the problem setup seems to he incorrett or misinterpreted, let's assume the vectors meant to he parallel in a different context or there might be a need to re evaluate the problem constraints.
Given the sctup, the model keeps assuming the vectors are meant to be parallel in different context and repeat- odly "re-evaluates"
without reaching conclusion or producing <anaver> tag.
problem setup seems incorrect misinterpreted, assume vectors meant paraliel in a different context there might be a need to re-evaluate the problem constraints.
(Repet f 1oe--oop unt
maxt length?
Aux-Loss (LoRA-
<think>
To solve for use the property of parallel lines and corresponding angles. The angles formed by the transversal with parallel lines are equal.
MoE)
Output
Given:
The top left ungle is 2z + 3. angle $
The bottom left ungle
bottom right angle 2y.
Since the lines are parallel, the corresponding angles equal. Therefore, we can set the following equations:
3
3
2y..
From the second equation.
6 = 2y
2
Substitute into the first equation:
2xr 3
2
Multiply both sides clear denominator:
6(2z + 3)
24
4x
So
12
3
16r
16
4
Thus, the value it 3. </think>
<answer>
</answer>
Continued on next page
20

---

Published as a conference paper at ICLR 2026
Aspect
Content
RO-GRPO
<think> The image shows expressions involving and y. Focusing the relationship needed determine equate the angles labeled 2a: + and ar:
(Smooth) Output
2ir +
Adding both and subtracting from both gives
2x
3x
50
2 1.
</LLink>
<ansMer> </answer>
RO-GRPO (Relative) Output
<think> find initially consider whether the slopes intersecting lines be equal because of paral- lelism. We might uy to read off "slopes" from expressions like 2x + and 6 - 2, leading to incousistent equalities such as
2
1,
or, from 3 + and By.
which are clearly impossible. This indicates that the slope-based interpretation is flawed. reconsider diagram and fact that angles labeled ir equal (vertical or corresponding angles). Therefore,
2x + 3 6
Solving.
2x
30
</thinks <onswer> </EnsMeY>
Analysis
The GRPO (LoRA) und GRPO (LoRA-MoE) models both enter a repetilive reasoning loup when trying to enforve parallel line slope comstraints, In contrast. the Aux-Loss (LoRA-MoE)) model produces clean and well-structured ofain-of-thought with explicit equa- corresponding angles, but still converges the wrong solution
producing valid final answer before hitting the generation limit.
RO-GRPO (Smooth) directly writes down and solves the key equation 2
obtains the correct solution
s 1, and uses fowest tokens, but it omits detailed explanation of how this equation is grounded the goometry of the diagram.
RO-GRPO (Relative) first explores an incorrect slupe based interpretation, then rellects, switches to the correct geomet- constraint 2rr
This trajectory best demonstrates rohust reflective reasoning while using roughly hall as many tokens as the Aux-Loss model (about 282 vs. 523), reinforcing auxiliary- loss supervision mainly inflates taken length without reliably improving geometric reasoning performance.
and finally outputs the
D VISUALIZATION OF EXPERT SPECIALIZATION
To better understand the internal dynamics of the RO-GRPO framework, we provide qualitative analysis of expert utilization across different reasoning tasks. our setup, all experts are trained through learned router, and integrate three LoRA-MoE modules into each FFN layer. Con- sequently, the final behavior of each token results from a composition of routing decisions across multiple layers, rather than the output of single expert. Unlike recent approaches such MAL- oRA (Wang et al., 2025) and MoLE (Wu et al.) 2024), which explicitly assign experts to specific domains or tasks, our experts emerge from end-to-end training and are not bound fixed task la- bels. As result, we do not observe a rigid one-to-one mapping between specific experts and specific reasoning skills, but soft specialization does occur.
Figure 7 illustrates the contribution of experts within a LoRA-MoE module across different subsets of the Math Vista benchmark. We find that different experts are preferentially activated for different types of Math Vista problems, indicating partial task-level specialization. At the same time, the distributions remain soft: no expert exclusively dedicated contribute non-trivially across many tasks. We also observe systematic differencee between prefill and decode: some experts are used more heavily during the prefill phase, when the model is ingesting and structuring the multimodal input, whereas others become more prominent during the decode phase, when the model produces multi-step reasoning and final answers. Taken together, these patterns suggest that RO-GRPO encourages partial specialization of experts across both tasks and phases, but the emergent structure is graded rather than perfectly disentangled.
single subset, and several experts
21

---

Published as a conference paper at ICLR 2026
MathVista prefill phase layer 16 up_proj
1.0
Expert 0
Expert
Expert 2
Expert 3
0.8
Task Contribution
0.6
0.4
0.2
0.0
geometry diagram natural image synthetic scene
scientific figure
chart
function plot
abstract scene
table
document image
line plot
scatter plot
puzzle test
Task Tag
(a) Prefill Phase Expert Contribution
MathVista decode phase layer 16 up_proj
1.0
Expert 0
Expert
Expert 2
Expert 3
0.8
Task Contribution
0.6
0.0
geometry diagram scientific figure
synthetic scene
bar chart
natural image function plot
table
puzzle test
scatter plot
abstract scene
document image
line plot
Task Tag
(b) Decode Phase Expert Contribution
Figure 7: Visualization of expert contribution ratios in a representative LoRA-MoB layer across different Math Vista subtasks. (a) Shows the expert contribution during the prefill stage. (b) Shows the expert contribution during the decode stage.
E ADDITIONAL EXPERIMENTAL RESULTS
This section reports supplementary results for two extended settings. First, we scale the base models to Qwen2.5-322 5--22B and replicate both unimodal and multimodal experiments under the same training configuration in Section 4 Second, we further evaluate RO-GRPO under top-2 expert routing configuration on the 7B models. In this setting, the router seleets the two most probable experts per token, and the corresponding LoRA updates are aggregated accordingly.
22

---

Published as a conference paper at ICLR 2026
Table Unimodal mathematical reasoning results for Qwen2.5-32B- -nnstts--2B- NuminaMath-TIR, 2k. We report accuracy (%) on GSM8K, MATH, SVAMP, and MGSM, together with routing entropy (E) and bad-baaancing MSE (B).
Unimodal Mathematical Reasoning (Qwen2.5-32B-Insttuuc on MaininaMath-TTR-2222
Method
#Experts GSM8K
MATH
SVAMP
MGSM
Entropy
MSE
GRPO (LoRA)
94.69
74.46
93.00
36.58
94.47
75.32
92.33
34.95
0.676
0.005
GRPO (LoRA-MoE)
94.69
75.96
93.33
37.42
0.679
0.003
8
94.77
75.16
93.00
38.04
0.682
0.002
95.83
77.28
93.00
38.84
0.676
0.005
RO-GRPO (Smooth)
94.84
76.40
93.67
44.22
0.680
0,003
8
94.92
75.28
93.67
39.56
0.681
0.002
95.07
75.74
93.00
38.91
0.676
0.005
RO-GRPO (Relative)
4
95.15
76.96
93.00
45.78
0.679
0,003
94.69
77.26
92.33
47.16
0.681
0.002
Table Multimodal mathematical reasoning results for Qwen2.5-VL-322 ---m2.5-VV-322-Instttc on Geometry3k.
All metrics are reported as accuracy (%) for Geo3k, Math Vista, Math Verse, and WeMath, with routing entropy (E) and wad-balancing MSE (B).
Multimodal
Mathematical Reasoning (Qwen2.5-VL-322-Ime on Geometry3k)
Method
#Experts
Geo3k
Vatta
MathVerse
WeMath
Entropy
MSE
GRPO (LoRA)
46.76
56.70
43.35
76.32
47.59
56.00
42.34
76.09
0.667
0.008
GRPO (LoRA-MoE)
4
47.25
57.30
43.22
75,86
0.672
0.007
48.75
58.10
41.57
74.89
0.675
0.006
47.92
55.80
43.12
75.57
0.669
0.008
RO-GRPO (Smooth)
47.92
$5.60
42.31
76.26
0.671
0.007
49.25
55.80
43.12
76.95
0.675
0.005
47.75
57.20
43.65
75.23
0.667
0.008
RO-GRPO (Relative)
47.09
$5.90
43.10
75.80
0.671
0.006
47.92
57.30
42.84
77.53
0.675
0.005
Tables and present unimodal and multimodal mathematical reasoning results for the 32B mod- els. Tables and I0 summarize the corresponding top-2 routing ablations on the 7B models. All accuracy numbers are reported percentage, and routing statistics are summarized by the average routing entropy (E) and load-bale mad-bblanncng mean squared error (B), consistent with the main tables.
F DETAILED THEORETICAL ANALYSIS
This appendix provides the detailed mathematical derivations and expanded interpretations for the theoretical analysis.
F.I CONSTRAINED OPTIMIZATION INTERPRETATION
The RO-GRPO framework can be viewed as a practical, penalty-basee approach to solving a con- strained policy optimization problem. The objective is maximize the expected task reward, subject
23

---

Published as a conference paper at ICLR 2026
Table 9: Unimodal mathematical reasoning results for Syee2.5-7B-Instmuie on NuminaMath-TIR- 2k under top-2 expert routing. All models use LoRA-MoE adapters with different expert counts E.
Unimodal Mathematical Reasoning (Qwen225-77-Instrucc on NuminaMatt-TTRR228)
Method
#Experts GSM8K
MATH
SVAMP
MGSM
Entropy
MSE
89.39
69.96
90.67
50.51
0.334
0.032
GRPO (LoRA-MoE)
90.37
70.30
92.00
52.65
0,225
0.039
4
89.92
69.88
92.00
46.00
0.334
0.031
RO-GRPO (Smooth)
89.76
70.68
92.00
54.62
0,225
0.039
4
89.92
70.24
92.00
49.45
0.334
0.032
RO-GRPO (Relative)
90.45
70.14
91.67
45.60
0.225
0.039
Table 10:
Multimodal mathematical reasoning results for -reen..--LL7B--nnttii Instruct on Geometry3k under top-2 expert routing. Metries are reported as accuracy (%) on Geo3k, Vista, Math Verse, and WeMath, together with routing entropy (F) and load-balancing MSE (B).
Multimodal Mathematical Reasoning
on Geometry33)
Method
#Experts
Geo3k
Math
MathWerse
WeMath
Entropy
MSE
40.27
58.60
30.51
63.22
0.325
0.061
GRPO (LoRA-MoE)
41.10
60.80
31.19
65.06
0.221
0.068
4
39.77
62.20
16.50
62.07
0.328
0,059
RO-GRPO (Smooth)
37.77
61.40
18.38
57.59
0.219
0.065
41.60
58.50
32.56
66.03
0.331
0.043
RO-GRPO (Relative)
8
41.43
58.10
32.82
64.54
0.222
0.067
constraints on the policy's internal routing behavior:
max
Enywar [Reak (y))
subject to Eynom [Havrm (y)) S EH
E Eyoaro [Mnorn (y)) S. EM,
where ии and EM are desired thresholds for the average normalized routing entropy and load bal- ancing MSE, respectively.
The standard method for solving such problem is via its Lagrangian relaxation. The Lagrangian T,(0, AB AM) is:
[Rmkk AH (L))oomm] EH)
(10)
Am (nomo -M),
where AØ Am 2 are the Lagrange multipliers. Our RO-GRPO objective can be expressed as:
max E Bask Whoute (wes (t) Flmerm wD(t). -mmm))
(11)
Comparing objective in equation I1 with the Lagrangian in equation 10 reveals that RO-GRPO maximizes a simplified Lagrangian. The weights wH((! and wn(() function as fixed (or scheduled) Lagrange multipliers, and the constraint thresholds E E implicitly absorbed into the objec- tive. This formulation positions RO-GRPO as fixed-penalty method, which gains significant simplicity by integrating the constraints directly into the scalar reward signal.
F.2 PARAMETER UTILIZATION AND VARIANCE MINIMIZATION
The balancing reward, Rg, is grounded in direct mathematical relationship with load distribution variance. We show that minimizing our MSE-base metric is equivalent to minimizing the variance of expert utilization.
24

---

Published as a conference paper at ICLR 2026
GR
RO-GRPO(Relative)
RO-GRPOlSmocdh)
Aux-Loss
800
500
Mean Length
Mean Length
600
Length
300
400
500
250
300
300
200
200
200
100
150
100
200
300
400
100
200
300
400
0
100
200
300
400
Steps
Steps
Steps
(a) #Experts
(b) # Experts
(c) #Experts
Figure 8: Evolution of average response length during training on Geometry3k.
Let p be the empirical utilization vector over E experts. The variance of this distribution is:
1
Var(p)
Σ
(e
E(p))
(12)
E
Since Eee 1, the mean utilization Elp L/ Substituting this gives:
E
1 2
Var(p) = E
Σ c=1
Pe
(13)
E
This expression is precisely the Mean Squared Error (MSE) between the empirical distribution p and uniform distribution (1/E,..... 1/FE.. Therefore, Var(p) MSE(p, u).
This equivalence establishes that maximizing our reward Rg x - MSE(p, u) is directly propor- tional to minimizing the variance of the expert load. This provides a principled and efficient mech- anism to promote balanced parameter usage within the RL framework.
G TOKEN LENGTH AND EFFICIENCY ANALYSIS
We evaluate computational cost via token efficiency, defined as the ratio of average output tokens to task accuracy. As detailed in Tables II and 12 RO-GRPO variants consistently optimize this trade-off. Unlike auxiliary loss, which often inflates generation without commensurate accuracy gains, RO-GRPO achieves peak accuracy on GSM8K, MATH, and Geometry3k with significantly reduced token usage. Notably, improves MGSM accuracy by over 12% and reduces WeMath response length by approximately one-third compared to baselines. The resulting reduction in quence length leads to faster inference, as confirmed by the latency-accuracy trade-off shown in Figure [5] (Section 55. These results indicate that mechanism-awee routing effectively suppresses repetitive loops and promotes concise reasoning.
To complement the tabular results, Figure 8 iillustrates the training dynamics of response length. We observe that the unsupervised GRPO baseline sometimes exhibits a rapid increase in token count, often indicative of reward hacking via verbosity degeneration into repetitive loops. While the in- clusion of an auxiliary loss helps curb this tendency, both RO-GRPO strategies maintain consistently concise generations throughout the training process. This suggests that mechanism-awwe rewards effectively regularize the reasoning process, preventing the model from defaulting to inefficient or degenerate output patterns.
H COMPLEXITY ANALYSIS
In this section, analyze the parameter count, computational complexity (FLOPs), and memory overhead of RO-GRPO compared to standard GRPO with LoRA and vanilla LoRA-MoE. We denote the sequence length T. the hidden dimension as d, the LoRA rank r, the total number of experts as E, and the number of active experts per token as K. We assume the model contains layers equipped with adapters.
25

---

Published as a conference paper at ICLR 2026
Table 11: Token Efficiency Analysis for Qwen2.5-7B-Instee on NuminaMath-TIR-22 We report
Average Tokens (Toks.), Accuracy (Acc.), and Efficiency (Eff.) across four benchmarks. Efficiency calculated as Tokens//cccraccy (lower is better). Best results for Accuracy (highest) and Efficiency (lowest) are highlighted.
Token Efficiency Analysis (Qwen2.5-7--Instee on NuminaMath-TTRe.
GSMSK
MATH
SVAMP
MGSM
Method
Config
Toks.
Acc.
Etr.
Toks.
Acc.
Eff.
Toks.
Acc.
Eff.
Toks.
Acc.
Enf.
16
291.58
88.48 3,30
519.01
70.38
7.37
189.42
90.67 2.09
264.13
50.00 5.28
GRPO (LoRA)
32
273.63
90.45 3.03
518.91
69.54
7.46
174.04
93.00 1.87
267.09
47.10
5.67
276.78
90.14
3.07
503.31
49.02
10.27
176.22
92.67
1.90
272.69
53.67 5.08
2x8
288.39
89.39
3.23
539.38
70.36
7.67
201.58
90.00
2.24
262.19
61.75 4.25
GRPO (LoRA-MoF)
283.74
89.39
3.17
512.21
70.40
7.28
180.63
91.30 1.98
251.36
46.15
5.45
8x8
271.89
90.22
3.01
525.68
70.44
7.46
190.52
2.09 91.00
259.37
52.04
4.98
2x8
290.40
86.73
3.35
573.58
69.50
8.25
213.32
92.33
2.31
255.01
57.27 4.45
Aux-Loss (LoRA-MoF.)
4x8
291.60
87.11
3.35
576.56
70.10
8.22
218.11
91.00
2.40
253.60
56.36
4.50
8x8
289.33
87.04
3.32
577.34
69.54
8.30
215.06
91.33
2.35
254.21
56.66
4.49
2x8
264.55
91.51
2.89
511.26
70.64
7.24
182.80
91.00
2.01
252.79
62.18 4.07
RO-GRPO (Smooth)
4x8
280.22
90.67
3.09
525.68
70.62
7.44
200.17
92.00
2.18
265.45
$2.58 5.05
8x8
269.46
90.98
2.96
523.41
69.78
7.50
180.73
92.67
1.95
255.04
52.04
4.90
2x8
284.16
90.22
3.15
528.60
70.58
7,49
197.65
91.33
2.16
268.27
59.45 4,51
RO-GRPO (Relative)
4x8
264.10
89.76
2.94
533.63
69.88
7.64
185.80
93.33
1.99
257.15
54.58 4.71
8x8
258.97
90.52
2.86
510.94
70.18
7.28
182.35
92.67
1.97
256.19
51.96 4.93
Table 12: Token Efficiency Analysis for Qwen2.5-VL--7B-nnste on Geometry3k.
Multimodal Mathematical Reasoning (Qwean255VII-77--nsttruc on Geometry3k)
Geo3k
Math
Math
WeMath
Method
Config
Toks.
Acc.
Eff
Toks.
Acc.
EMf
Toks.
Acc.
Eff.
Toks.
Acc.
Eff.
16.00
352.89
38.44
9.18
207.66
58.60 3.54
353.19
33.30 10.61
292.07
63.97
4.57
GRPO (LoRA)
32.00
526.97
38.10
13.83
263.29
59.30
4.44
460.99
23.22
19.85
601.22
53.85
11.18
64.00
446.82
33.28
13.43
393.17
55.90
7.03
715.62
25.43
28.14
743.43
53.91
13.79
2x8
335.64
38.27
8.77
210.52
57.90
3.64
347.25
30.99
11.21
313.59
63.74
4.92
GRPO (LoRA-MoE)
4x8
295.68
28.95
10.21
212.68
56.40
3.77
329.59
30.30
10.88
253.56
63.45
4.00
8x8
362.38
33.11
10.94
230.56
55.00
4.19
360.94
31.78
11.36
327.86
61.49
5.33
2x8
406.79
39.60
10.27
208.38
56.20
3.71
353.08
30.03
11.76
359.87
62.81
5.73
Aux-Loss (LoRA-MoBi
4x8
368.85
41.43
8.90
231.08
60.50
3.82
375.17
27.23
13.78
350.04
62.87
5.57
8x8
337.44
40.43
8.35
205.68
54.40
3.78
349.70
32.13
10.88
301.78
65.80
4.59
2x8
244.33
38.94
6.27
139.97
58.70
2.38
221.75
30.48
7.28
191.89
66.09
2.90
RO-GRPO (Smooth)
4x8
275.37
40.10
6.87
182.87
58,30
3.14
255.43
28.73
8.89
233.97
64.14
3.65
8x8
206.22
38.94
5.30
139.89
58.90
2.38
224.38
27.89
8.05
194.38
64.10
3.03
2x8
348.29
41.93
8.31
152.97
55.80
2.74
245.84
33.30
7.38
200.36
60.98
3.29
RO-GRPO (Relative)
4x
222.93
40.27
5.54
220.27
60.20
3.66
351.11
31.29
11.22
275.09
66.26
4.15
8x8
285.44
40.16
7.11
178.19
60.10
2.96
295.52
32.03
9.23
273.60
63.97
4.28
Parameter Complexity. Standard LoRA introduces two matrices A E Brxd and B E Rddr per module, totaling 2dr parameters. LoRA-MoE introduces E experts and a routing projection W, E RExd The total parameter count per module is:
PLoRA-MOE E(2dr) + dE - dE 1).
(14)
RO-GRPO utilizes the identical architecture vanilla LoRA-MoE without introducing any addi- tional trainable parameters. The scalar state variables required for the reward curriculum (c.g., step counters) occupy negligible O(1) space.
26

---

Published as a conference paper at ICLR 2026
Computational Complexity. We focus the adapter operations, the frozen backbone cost O(Td²) remains constant across all methods.
Standard LoRA: Requires computing B(Ah), incurring 2T FLOPs per module.
LoRA-MoE (Forward Pass): The router computation hw? incurs 2TdE FLOPs. For the experts, the cost depends on the routing strategy. In dense soft routing, all experts are active (K = E), costing 2T" Edr. sparse Top-K routing, only K experts are computed, costing 2r',d.. The total adapter FLOPs per module are ( + Kdr)).
Reward Calculation Overhead: RO-GRPO computes routing metrics (entropy and MSE) post-hoc. Calculating entropy over probability vectors of size F. for T tokens scales with O(TF) per layer. Similarly, the load balancing MSE scales with O(TE) per layer.
Comparing the reward overhead
O(TE) to the model computation Cmodel Ri O(TKdr):
TE
E Kdr
(15)
Cinneel
TKdr
Given typical values (d 10 16 8), have dr > E and hence Kdr E for K' 2 1. Thus, the computational cost of the mechanism-aware reward is negligible compared to the forward pass. Furthermore, unlike auxiliary loss approaches, RO-GRPO does not require computing gradients for separate loss term (VoCaux), significantly reducing the backward pass overhead.
Memory Complexity. RO-GRPO requires storing routing distributions to compute the reward the end of generation batch. For L layers, this requires storing tensor of shape (L, T, E). The memory complexity O(LT)) Comparing this to the activation memory required for backpropa- gation, which scales with OLLdd), the ratio is E/d. Since E < d, the overhead is insignificant.
Complexity comparison per layer for sequence of length T. K denotes the number of Table 13:
active experts (K = for dense soft routing). The reward overhead is negligible as < dr.
Method
Parameters
FLOPs (Forward)
Reward/L.oss Overhead
Memory Overhead
GRPO (LoRA)
2dr dE(2r 1)
O(Tdr)
GRPO (LoRA-MoE)
O(T Kdr))
Aux-Loss (LoRA-MoE)
dF(2r + 1)
O(T Kdr))
O(TE) Backward Can)
O(LTE) O(LTE)
RO-GRPO (Ours)
dF + 1)
O(T(dE Kdr))
O(TE)
27


Reviews:
[{"review_id": "iKv255cH1O", "reviewer": "Reviewer_rDj1", "summary": "The paper addresses a mismatch when fine-tuning LoRA-MoE models with GRPO: the external task reward is unaware of internal expert routing, leading to routing collapse and poor MoE utilization. The proposed RO-GRPO augments the reward with a mechanism-aware term $R_{\\text{routing}}$ added to the task reward $R_{\\text{task}}$. $R_{\\text{routing}}$ combines (i) routing confidence (mean per-token routing entropy; lower is better) and (ii) load balance (MSE to uniform expert usage; lower is better). Two strategies are evaluated: a smooth curriculum that shifts emphasis from confidence to balance, and a relative scheme that rewards only when both metrics improve over a historical baseline. On uni- and multimodal math reasoning tasks, RO-GRPO improves accuracy and yields healthier expert utilization versus vanilla LoRA-MoE. Ablations indicate both components matter; a shuffled-control (permuting $R_{\\text{routing}}$ within a batch) removes gains, supporting causal attribution.", "questions": "1. Budget parity: How does RO-GRPO compare to a high-rank LoRA baseline matched to the $E{=}8$ trainables (~127M)? Please report accuracy and routing metrics under parameter/FLOPs parity.  \n2. Beyond math: Any preliminary results on instruction-following or summarization, and observations on expert specialization there?  \n3. PPO/DPO adaptation: For PPO, can $R_{\\text{routing}}$ be used as standard reward shaping without instability (need annealing/curriculum)? For DPO, is reweighting by $\\Delta$routing (preferred minus rejected) viable?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simple, novel, and practical: Converts internal routing statistics into a scalar reward, no architectural changes or auxiliary losses.  \n2. Credible evidence: Informative ablations and a strong shuffled-control isolate the effect of routing-aware feedback.  \n3. Clear exposition & tangible impact: Clean figures and a persuasive qualitative case link improved routing to better reasoning.", "weaknesses": "1. Parameter-budget confound (major): Baselines compare ~30M LoRA vs RO-GRPO variants up to ~127M trainables; needs budget-matched or FLOPs-normalized comparisons.  \n2. Narrow task scope: All benchmarks are math-centric; generality to instruction following, summarization, dialogue, or safety is untested.  \n3. Limited algorithmic scope & reporting gaps: Only GRPO is shown; PPO/DPO not demonstrated. Clarify entropy/balance aggregation and sensitivity; consider risks of “confidence” gaming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7uvAFFFpAz", "reviewer": "Reviewer_xfQ9", "summary": "This paper addresses the challenge of applying LoRA-MoE architectures to reinforcement learning fine-tuning RFT, specifically GRPO. The authors state that traditional auxiliary losses used to guide expert routing in supervised settings are incompatible with GRPO's scalar reward framework, leading to routing collapse and poor expert utilization. They propose RO-GRPO, which transforms internal routing statistics into reward signals through two components: a confidence reward encouraging low-entropy routing and a balance reward promoting uniform expert utilization. Two variants are introduced, (i) Smooth (curriculum-based scheduling) and (ii) Relative (binary reward for simultaneous improvement). Experiments on mathematical reasoning tasks using Qwen2.5 models demonstrate improvements in both task performance and routing efficiency across numerous expert counts.", "questions": "- Could you provide experimental results on full MoE architectures rather than adapter-based variants? This would clarify whether routing collapse is inherent to MoE structures under GRPO or specific to the LoRA-MoE configuration.\n\n- What happens when you include a GRPO baseline using standard LoRA with trainable parameters matching the LoRA-MoE models (e.g., approximately 127M parameters for the E=8 configuration)?\n\n- Can you provide qualitative analysis or case studies demonstrating what different experts actually learn? Do they specialize in different mathematical reasoning strategies, problem types, or computational roles?\n\n- Why must routing supervision be formulated as a reward signal rather than an auxiliary loss term? Could you compare your approach against a variant that incorporates routing entropy and load balancing as loss terms alongside the GRPO objective?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a meaningful problem at the intersection of parameter-efficient fine-tuning and reinforcement learning, with practical implications for deploying and adapting large models.\n- The proposed approach is technically sound, integrating routing supervision directly into the reward signal without requiring architectural modifications or additional training stages.\n- The experimental design is comprehensive, covering both unimodal and multimodal mathematical reasoning tasks with consistent methodology across multiple benchmarks.\n- The theoretical grounding connecting the approach to constrained optimization and information bottleneck principles adds depth to the empirical contributions.", "weaknesses": "- The scope of evaluation is limited to adapter-based MoE architectures, leaving unclear whether the identified routing pathologies and proposed solutions generalize to full MoE models such as Phi-3.5-MoE or OLMoE.\n\n- The design choice to formulate routing supervision as rewards rather than auxiliary losses is not adequately justified. Given that GRPO already employs KL divergence as a regularization term, it is unclear why routing constraints cannot be incorporated similarly as loss terms.\n\n- The experimental comparisons lack a crucial baseline: a standard LoRA configuration with parameter count matching the LoRA-MoE variants, making it difficult to isolate whether performance gains stem from the routing-aware rewards or simply from increased model capacity.\n\n- Tables 2 reveal several instances where the vanilla GRPO (LoRA) baseline with approximately one-quarter the trainable parameters outperforms RO-GRPO variants, undermining claims of consistent improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0MbB4UYDsg", "reviewer": "Reviewer_eaSP", "summary": "This paper studies the expert collapse and underutilization problem in reinforcement fine tuning of LORA-MOE architectures. The authors proposed a routing confidence and a load balancing reward. The two rewards are combined and summed with the task reward using a weight curriculum. Then the combined reward is optimized using the GRPO algorithm. Experiments on conducted on several math reasoning benchmarks. The authors show the effectiveness of their method for controlling expert confidence and load balance. It was further shown in ablations that removing the proposed mechanisms degrade the performance of LORA-MOE fine tuning.", "questions": "* Have you considered adding auxiliary loss to the simple RL objective as a baseline? Could you discuss this option or show some experiments?\n* Although it's likely intuitive, could you explain why applying the combined reward on the entire policy rather than applying the entropy and load balancing reward to only the routers is so effective? And why not the latter option?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The presentation of the idea is pretty clear. \n* The ability of the proposed method to control entropy and load balance is clearly pretty effective from the plots.", "weaknesses": "* While the method is effective at controlling entropy and load balance, it is not clear that it improves performance for more experts, which I belief is the ultimate goal for this family of methods. In fact from the results tables, 8 experts is almost never the overall best. \n* While integrating these objectives into the reward function seems to unify the training method, it is not clear from the paper that simply adding auxiliary losses to the RL objective is fundamentally flawed or significantly worse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DV3foVIGXW", "reviewer": "Reviewer_CGAv", "summary": "This paper presents a technique called RO-GRPO, which maximizes utility of experts of MoE through LLM-RL training. RO-GRPO adds an additional rewards based on routing entropy and expert utilization to explicitly encourage the MoE model to efficiently use experts. The paper introduces two variations, RO-GRPO (Smooth) and RO-GRPO (Relative). Smooth variation schedules weights of additional reward terms while Relative variation computes the reward signal based on improvements over historical baselines, which eliminates the manual scheduling mechanics. In experiments, both variations outperforms a baseline and especially the Relative variation shows strong performance improvements.", "questions": "- Computing the route reward adds any additional computing overheads?\n- Do we need to balance scaling between the main task reward and the route reward? Can we just use the route reward with its raw scale?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed idea is simple and easy to integrate to existing MoE training. And, the Relative variation eliminates manual balancing to tune performance, which makes this approach very practical.\n- The performance improvement is significant especially when increasing the number of experts, which proves its effect.\n- Through extensive analysis, how individual changes contribute behavior of MoE training is clarified in the paper.", "weaknesses": "- Although the proposed idea sounds sane, a straightforward alternative approach would be to use the routing entropy and the expert utilization as loss functions, which could directly optimize those metrics. I found the similar description in Appendix E.1, however this seems vital baselines to makes more sense to use the proposed idea. Lack of this analysis keeps my score as it is.\n- Analysis of models' outputs with the proposed change should be interesting, but there isn't much observation provided in the paper except Appendix D. Since the proposed model can utilize experts more, it would be interesting to observe different trends in output texts. This analysis will improve the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)