Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
Authors: Jiashun Liu
Johan Obando-Ceron
Han Lu
Yancheng He
Weixun Wang
Wenbo Su
Bo Zheng
Pablo Samuel Castro
Aaron Courville
Ling Pan
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Large Language Model
Math Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: AsyPPO efficiently restores the role of critics through lightweight mini-critics and reconstructs the policy learning objective, enhancing the reasoning ability of LLMs while emphasizing the research value of critic-based algorithms.

Abstract: Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs) to elicit stronger reasoning. Yet, most recent RL for LLMs (RL4LLM) methods avoid explicit critics, replacing them with average advantage baselines. This shift is largely pragmatic: conventional value functions are computationally expensive to train at LLM scale and often fail under sparse rewards and long reasoning horizons. We revisit this bottleneck from an architectural perspective and introduce Asymmetric Proximal Policy Optimization (**AsyPPO**), a simple and scalable framework that restores the critic’s role while remaining efficient in large-model settings. **AsyPPO** employs a set of lightweight *mini-critics*, each trained on disjoint prompt shards. This design encourages diversity while preserving calibration, reducing value-estimation bias. Beyond robust estimation, **AsyPPO** leverages inter-critic uncertainty to refine the policy update: (i) masking advantages in states where critics agree and gradients add little learning signal, and (ii) filtering high-divergence states from entropy regularization, suppressing spurious exploration. Across multiple reasoning benchmarks, **AsyPPO** consistently improves learning stability and performance over strong baselines, e.g., GRPO, achieving performance gains of $> 6$% on *Qwen3-4b-Base* and about $3$% on *Qwen3-8b-Base* and *Qwen3-14b-Base* over classic PPO. Such results highlight the importance of architectural innovations in critics for scalable, efficient algorithms.

Published as a conference paper at ICLR 2026
ASYMMETRIC PROXIMAL POLICY OPTIMIZATION: MINI-CRITICS BOOST LLM REASONING
Jiashun Liu 1.4+
Johan Obando-Ceron 2.3+
Han Lu
Yancheng He* Aaron Courville
Weixun Wang*1 Ling Pan
Wenbo Su Bo Zheng"
Pablo Samuel Castro
HKUST
Mila Québec Al Institute
Université Montréal
AAlibaba Group
ABSTRACT
Most recent RL for LLMs methods avoid explicit critics, replacing them with average advantage basclines. This shift is largely pragmatie: conventional value functions are computationally expensive to train at L.LM scale and often fail under sparse rewards and long reasoning horizons. We revisit this bottleneck from ar- chitectural perspective and introduce Asymmetric Proximal Policy Optimization (AsyPPO), simple and scalable framework that restores the critic's role while remaining efficient in large-model settings. AsyPPO employs a set of lightweight mini-crities, each trained on disjoint prompt shards. This design encourages diver- sity while preserving calibration, reducing value-estinne estimmattni bias. Beyond robust estimation, AsyPPO leverages inter-critic uncertainty to refine the policy update: (i) masking advantages in states where crities agree and gradients add little learn- ing signal, and (ii) filtering high-divergence states from entropy regularization, suppressing spurious exploration. After training on open-source data with only 5k samples, AsyPPO consistently improves learning stability and performance across multiple benchmarks over strong baselines, e.g., GRPO, achieving perfor- mance gains of > 6% on @wen3-4's -----Baee and about 3% on Qwen3-88 1e---Bb--Be Base and 3wen 3-1 4b-Base over classic PPO, without additional tricks. Such results underscore the value of architectural innovations for scalable, efficient learning.
INTRODUCTION
of the most powerful actor-critic algorithms in deep RL, and has demonstrated its potential across diverse domains such as computer games (Yu et al., 2022: Schwarzer et al., 2023) and robotics control (Raj & Kos, 2024). In the realm of large-scale language models (LLMs), PPO has also proven transformative and has been widely applied in the post-training stage stimulate the reasoning ability of LLMs (Hu al., 2025). However, the transition from classical RL to RL LLMs (RL4LLM) introduces unprecedented computational challenge, as LLMS operate at scales orders of magnitude larger than traditional RL environments. Directly applying PPO's default symmetric actor-critic design, where the critic is as large as the actor, creates significant computational overhead. In addition, training full critics at LLM scale is expensive and inaccurate under sparse, long-horizon rewards (Yuan et al., 2025b).
Proximal Policy Optimization (PPO) (Schulman et al., 2017) stands
Faced with these challenges, the RL4LLM community has largely sidelined key element of clas- sical PPO its critic. GRPO (He al., 2025), and its variants, including GSPO (Zheng et al., 2025) in the Qwen series and DAPO (Yu et al., 2025), have achieved great success in replacing value functions with group sampling and average-advantage baselines for coarse-grained estimation of advantages. While effective, this paradigmatic shift abandons a key concept of RL: robust state value estimation can naturally mitigate training collapse caused by advantage bias (Wang et al., 2025b; Liu et al., 2024), especially under off- off-policy settings. This landscape motivates a fundamen- tal reconsideration of architectural assumptions inherited from deep RI,, prompting the following central question: Can we achieve lightweight yet robust value estimation by redesigning PPO to depart from the standard symmetric actor-critic architecture, enabling stable and efficient learning?
"Equal contribution
I "Correponndig to: (1 (1jshasdream, jobando0730 (0mmmm @gnai com, xun. wwx@taobao.com

---

Published as a conference paper at ICLR 2026
2008
Average accurecy generations (%))
AsyPPO
Classic PPO
Running time train step (6)
07
1909
Pask memory (GBB)
:4
1706
PPO: 14b actor & 14b critic Vanilla GRRO
241
-
1308
AsyPPO: 14b actor & 3. 4b critic
GRPO
1403
88-
754
GRPO
160
200
300
400
780
(a) Average test accuracy of six benchmarks
(b) Lightweight deployment
Figure I: (Left)) Learnable crities naturally enhance policy stability through fine- tiee-grained value estimation and yield continuous gains training progresses. Off-policy ratio=8, average @4 of benchmarks, i.e., AIME 24, AIME 25, MATH-500, OlympiadBench, MinervaMath, and AMC 2023. (Right): AsyPPO restores the critic's role in PPO while remaining lightweight and stable under LLM-scale training. The average clock time of training and the peak GPU memory usage of AsyPPO are significantly lower than those of the classic PPO, remain at the GRPO level.
To fill this research gap, we begin with a key insight: the initial rich representational ability inherited from pre-trained models significantly enhances the feasibility of the asymmetric actor-critic in the RL4LLM domain, unlike agents that learn from scratch in classical deep RL, painting a promising prospect for lightweight deployment and computational efficiency. Our initial experiments validate this hypothesis, where we find that a small critic, c.g., Qwen 6b-Base, can indeed provide meaningful guidance to much larger actor, c.g., Qwen -ese--bbassee Base, demonstrating meaningful performance improvements over the base model. However, this asymmetric setup underperforms classical symmetric PPO, revealing limitations in single small critic's 's value estimation capabilities.
To unlock the capabilities of the small critic, we consider critic ensembles to improve its value esti- mation and policy guidance. However, naive ensembles offer limited benefits for policy learning, as LLM critics start from identical pre-trained checkpoints with different heads only and are trained on the same data, leading to nearly identical behaviors that provide no corrective benefit. To tackle this critical challenge, we propose simple yet effective non-overlapping data partitioning technique, in which each critic trained via the subset formed by uniformly extracting responses from each prompt without overlap. This design encourages diversity among small critics and mitigates the risk of perception asynchrony among critics. Leveraging our ensemble- based value correction, small critics can provide reliable guidance to large policies despite their limited expressivity (Tint et al., 2024). Surprisingly, we find that double critic could be the sweet spot between correction capabil- ity and efficiency, it yields qualitative leap in evaluation reliability while incurring the minimal redundancy needed for bias reduction. More critics increase the computation without proportional gains. Empirically,
demonstrate that Qwen3-1. 7b-Base critics robustly guide larger policy, e.g., Qwen3-3 - Base, reducing critic over-paramettrization while outperforming sym- metric PPO under off-policy setting (Figure I(a)). Notably, asymmetric architecture reduces peak memory by 20%, and accelerates training by around 20 seconds per step (see Figure 1(b)).
We further discovered that the agreement and divergence patterns in value estimates between our double critics, measured by their standard deviation, provide a useful signal for refining the policy loss objective. Value-estimation heterogeneity reflects both uncertainty and informativeness of the states. Leveraging this, we mask advantage values in states where critics strongly agree, reducing overfitting to low-quality samples and improving training stability. Conversely, we exploit diver- gence across critics by filtering out uncertain states from entropy regularization, since such states often correspond to low-probability continuations or spurious, reasoning-irrelevve patterns that in- ject noise into entropy measurements (Ahmed et al., 2019). Thus, restricting entropy regularization to high-confien coah-confieecce states promotes safer exploration and improves performance. With this refined loss objective, the policy further amplifies its learning efficiency ( (Figure 2). Overall, refer the above components as Asymmetric Proximal Policy Optimization (AsyPPO). The contributions of AsyPPO can be summarized in three main aspects:
2

---

Published as a conference paper at ICLR 2026
Classic symmetric PPO
Asymmetric PPO (AsyPPO)
Symmctric PPO
ArpP0
Performancee imgrowement
22-
CRITIC
?
mini-criticss
ACTOR
ACTOR
164
actor
(a) Comparation of the two architectures
(b) Comparison of improvements
Visual intuition behind AsyPPO. Small yet expressive critics can guide larger actors effectively when their representations are well-initialized. (a): A single critic struggles to align its value signal, leading uncertain policy updates. Two small critics reach consensus ("This way") and provide robust, larinnriance guidance to the actor. (b): By leveraging representational priors and critic agreement, AsyPPO significantly outperforms classic PPO in terms of performance various policy models, i.e., owen3-4B 4----4b-Bat Base and Qwen3- -as-----Beee... The Y-axis represents the improvement for the initial policy. The score calculation is the same that in Figure I,
Figure 2:
1. Robust Estimation: Prompt-level data partitioning enhances ensemble reliability and yields consistent performance improvements. ($4.1)
2. Lightweight Architecture: The asymmetric design mitigates critic over-paramettrization and opens new direction for RL4LLM. ($4.1)
3. Objective Refinement: We introduce two uncertainty modifications to the PPO ob- jective that improve sample efficiency and enable safer exploration. ($4.2)
2 PRELIMINARIES
Proximal Policy Optimization (PPO). PPO (Schulman et al., 2017) is widely used actor-critic algorithm in the policy gradient family. improves the stability by optimizing a clipped surrogate objective, which limits how much the updated policy ma deviate from the old policy Teaa at each update step. The objective is defined as:
(!) E ((Q),, onnro (019)
Ar, clip TT((oog,00)) 1-c, 1+c) Ae Ab). Tasa (0: (0r))
(I)
min
Tha (orlg, 0zx)
where Te and meli denote the current and previous policy, respectively. Here q is sampled question and o the generated output sequence,, with o, the 1-th token. e is the clipping hyperparameter that constrains the update ratio. A, is the advantage estimate step 4, typically computed with Generalized Advantage Estimation (GAE) (Schulman et al., 2015).
Generalized Advantage Estimation (GAE). GAE addresses the vantage estimation by combining multi-step returns with exponentially decaying weights:
trade-off in ad-
AGAB G)
Σ (7))) 8
+ (ss)..
Here V(s) is the value function, E [0. 1] is the discount factor, and À [0, 1] is the GAE parameter that balances bias and variance. Setting recovers the low-variance, high-bias TD(0) estima- tor, while À corresponds to the high-variance, low-bias Monte Carlo estimator. In practice, PPO

---

Published as a conference paper at ICLR 2026
leverages GAE together with the clipped objective, yielding stable training and improved sample efficiency. The choice of y and À critically influences the temporal horizon and smoothness of the advantage estimates, and thus the convergence of the policy.
RELATED WORK
Critic-based RLALLM algorithms Shao et al. (2024) first demonstrated that large-scale rein- forcement learning (RL) with outcome-based rewards can unlock long-tail reasoning, beginning from an unaligned base model. This finding has led to numerous variations the Proximal Policy Optimization (PPO) algorithm. As far as we know, most algorithm research is mainly based on the baseline normalized advantage calculation method (Hu, 2025; Liu et al., 2025c; Chen et al., 2025a). On the other hand, value-based algorithm innovations are relatively few, Yuan et al. (2025b) gued that the decay factor is not well-suited for complex reasoning tasks that require long chains of thought (CoTT) Yue et al. (2025); Zhu al. (2025); Zhao al. (2025) proposed novel mechanisms to enhance the robustness of the critic model when faced with noisy reward signals. Open- OpennReasonee- Zero (Hu al., 2025) argues that vanilla PPO without KL regularization suffices scale training stably. T-PPO (Fan et al., 2025) uses critic to enhance the stability of policy training in the long-tail asynchronous setting (Fu al., 2025). Another similar research line introduce critic-like mod- els is done with the introduction of Implicit PRM (Yuan et al., 2025a). This approach is also able to provide token-level supervision for scalable RL training. PRIME (Cui et al., 2025a) adapted a specific reward model formulation to directly generate token-level rewards. However, current main- stream RL4LLM algorithms primarily emphasize critic-free free optimization (Zhang et al., 2025). In this context, our research aim to underscore the importance of the critic in RL4LLM scenarios and try to address the deployment limitations associated with critics.
Asymmetric architecture. In the realm of continuous deep RI., recent studies have investigated the potential of asymmetric network structures by reducing the capacity of the actor network. For example, Mastikhina et al. (2025); Mysore et al. (2021) suggest that the actor can function effectively with a significantly smaller capacity compared to the critic. Empirical evidence from Tan et al. (2022) supports this idea, demonstrating that sparsifying the policy network can enhance effective policy learning while significantly improving both inference and training speeds. Additionally, Liu et al. (2025a) found that pruning the actor network's 's topology based on trial gradients can yield better performance. Similarly, Ma et al. (2025) revealed that even random pruning of the actor network can maintain performance within the SimBa network architecture (Lee et al., 2024). These contributions highlight the adaptability of RL in accommodating asymmetric designs, providing valuable insights for our research. However, existing works primarily concentrate on reducing the actor's 's size within simple network frameworks. Our paper pioneers the exploration of effectively guiding small crities to inform larger actors by optimizing PPO within the RL4LLM scenario.
4 ASYMMETRIC PROXIMAL. POLICY OPTIMIZATION
We begin by empirically examining the potential of the asymmetric actor-critic framework while highlighting the limitations of naive ensemble critics in LLM reasoning. By analyzing key differ- ences between classical deep RL and RL4LLM, we propose a group-level non-overlapping data division strategy that enables lightweight mini-critics to provide reliable value estimation ($4.1). Building on this, we investigate the role of divergence and agreement among the mini-critics and find that uncertainty in their value estimates carries strong representational power for measuring sample quality. Leveraging this insight, we incorporate value uncertainty as a signal into the pol- icy optimization objective, reformulating the loss function and refining the entropy regularization to improve sample efficiency and exploration capability of the policy ($4.2).
4.1 TOWARDS LIGHTWEIGHT VALUE ESTIMATION
In LLM reasoning, the policy inherits expressive capabilities from the pre-trained model at ini- tialization. As shown in Figure 3 (Left), even without critic warm-up, small critic, i.e., 3-en3-0. 6B-Base (Yang et al., 2025), can provide useful guidance, demonstrating the poten- tial of an asymmetric architecture. However, due to sparse rewards and the small critic's limited
4

---

Published as conference paper at ICLR 2026
score Asymmettic PPO (0.6b critic)
Differmnce acroes crrics
32 55
Naive ensemble ensemble
Classic symmectic PPO
Naive ensemble
Classic PPO
Our ensemble
Tou
25
1.76 Actor
4b Acter
Kh Actor
Thining Step
The single mini-critic parameterized by Qwen3 0. (6---aseee can effectively guide policies across model scales. Middle: There significant differences in the guiding ability of the two ensemble critics for policies. Actors uniformly use Qwen3-8B-Base, while critics use Owen3-0. 6B-Base. Right: Our ensemble method intensifies the cognitive differences among mini-critics. The y-axis represents the standard deviation between the values calculated by the two mini-critics. We train 5,000 questions sampled from Deep 103K (He al., 2025) and evaluate policies on five challenging math benchmarks: AIME 2024, MATH-500, OlympiadBench,, MinervaMath, and AMC 2023, For each question, we report the average of generations.
Figure 3: Left:
familiarity with long-tail reasoning trajectories favored by larger models (Li al., 2025), its value estimates are often inaccurate, leading to suboptimal policy guidance compared to symmetric PPO.
Starting from the ensemble system. To strengthen mini-critic perceptual capacity, we first adopt ensemble of crities, standard technique in classical deep RL for reducing estimation bias (Chen et al., 2021). In practice, we add a second critic based on the same base model and average their predictions for value estimation. These corrected values then used in advantage computation via GAE. However, as Figure (Middle) shows, this naive ensemble approach yields limited improve- ment. The reason becomes clear in Figure (Right), the two mini-critics exhibit nearly identical behavior, failing to provide the diversity that ensembles rely on. In classical RL, critics are initial- ized randomly, ensuring parameter diversity and differentiated value estimates, which is essential for ensemble effectiveness. By contrast, in RL4LLM, critics are typically initialized from the same pre- trained model, which accelerates learning but reduces diversity. This motivates a question: under homogeneous initialization, can ensemble critics remain effective in LLM reasoning?
Group level non-overlap data division. Be- yond explicitly increasing parameter differ- ences through initialization, another promising approach is provide differentiated optimiza- tion signals for each critic during training. Intu- itively, training crities non-overlapping sub- sets of data encourages them to learn from dis- tinet trajectories and reward distributions, steer- ing their updates in different directions and pro- moting functional diversity. However, prac- tice, randomly partitioning the training data can lead to asynchronous perception at the prompt level, where critics encounter inconsistent rea- soning patterns from different questions. This imbalance increases the risk of overfitting to specific response types, resulting in unstable discrepancies in value estimates. In extreme cases, such divergence may cause policy collapse. To mitigate this, we uniformly divide the data into disjoint subsets the prompt level, ensuring that each critic receives an equal share of responses within every prompt (or group). This design maintains perceptual synchrony across critics within each question while creating differentiated rewards and observations. Our ensemble critic training process can then be formalized as:
Vanilla value
1.0
Conrected value
Mormaltrad value
0.6
Backward Chaining
Subgoal Sctting
0.4
Back Tracking
Verifications
0.2
Figure 4: Critics estimate the state involving key patterns, according to Gandhi et al. (2025).
M Σ m=l
M (m) (0m) = Σ
Leritic (oritic()=))))
Ea (V(se):))) R2)3]
(2)
m=1
5

---

Published as a conference paper at ICLR 2026
Basciee
Noak cos uit low errogy (9%)
Noreadia Berdd
Mark stat will low valasati (2%).
Mini-crticss
NayFPO AzgitO vala-hand (UTD-a)
sore wfw.ttt StTDw))
Cleir staid
step
Training
Motication our idvantagee masking
Ablation studl adeaniage mesking
[6) Comparison masking mechanieme
Figure 5: (a): Agreement among critics implies the state's downstream dynamics are well modeled by the policy, making these samples low-value for learning and best avoided for overfitting. (b): In the high data-reuse setting (UTD=4), masking the bottom 20% (by value-std) boosts AsyPPO's learning efficiency, yields an improvement of about points. The accuracy records of the six benchmarks follow Figure (b). (c): We evaluated two 5% masking mechanisms on vanilla AsyPPO (baseline), i.e., entropy vs. value-std. The value-ste std masking produced the strongest learning effi- ciency benefit. Actors use BB-nae, while critics use Owen3-0, 6B-Base.
M denotes the number of mini-critic with parameters {&m ) Each critic aim to fit the return R. based its assigned subset D - Dm, D,n 0. Corrected advantage Å can be obtained:
T-t-1 A(7, Σ (TA)'644 & = + V(sa); V(s))
1
M Σ (stisi
(3)
The results Figure (Middle, Right) demonstrate that crities trained under our ensemble strategy exhibit clearly differentiated behaviors. Statistical analysis from linguistic perspective (Figure 4) reveals that the corrected values from our ensemble framework significantly encourage the policy to acquire reasoning patterns. Overall, method effectively unlocks the efficiency of asymmet- ric PPO and points to promising new direction for RLALLM algorithm design.
Takeaway 1
Optimizing the ensemble critic design enhances the learning capacity of the asymmetric critr-critic while significantly reducing computational overhead.
4.2 POLICY Loss RECONSTRUCTION
States lowest value-std (5%)
Beyond enabling robust value estimation, we conjecture that ensemble mini-crities can further enhance policy learning effi ciency. Intuitively, the degree of agree- ment among critics' value estimates for a given state can serve as a meaningful sig- nal for policy optimization. This insight arises from our analysis of value fitting dynamics (Lee al., 2021): when crit- ics produce similar value estimates for a state s;, it often indicates that s; low- informative. Such states are frequently encountered across trajectories, and the rewards they yield exhibit low variance, causing critics to converge in their predic- tions, visualized in Figure 5 (a). Analy- sis in Appendix B shows the positive cor- relation between value-std and the pol- icy gradient, supporting the above specu- lation.
1.45
entrupy (5%%
Differences in recognition preferences
90-
O Overlap
1.4
s0-
1.35
70-
Nomalized Score
1.3
1.25
1.2
Difference
1.15
40
1.05
10
0.9
Compare Compare in entropy
The
Figure 6: Left: States with low value-std maintain low entropy, but states with low entropy may have high value-std. Right: States with low entropy and states with low salue-std show differences.

---

Published as a conference paper at ICLR 2026
Advantage masking based on the value agreement. Recent studies show that preventing the policy from overfitting to iwwiiffomation samples can substantially improve learning efficiency (Liu et al., 2025b). Since the degree of agreement across critics reflects state informativeness, where high agreement implies low uncertainty and limited learning potential, we use the standard deviation of crities' outputs quantify the benefit of optimizing given state. Specifically, lop k percentage of states with the highest agreement (i.e., lowest standard deviation) and mask their corresponding advantages in the policy loss. This suppresses gradient updates from low-informative transitions, filtering out noisy or redundant learning signals directing policy optimization toward higher-value data. The resulting policy loss objective is:
identify the
lol
IS. As, clip (IS,, 1-c, 1+0) Ao A.): - [1, otherwise
{0,
if E Low)(o)
(P)
E
I
min
1=1
(4)
Here, or = std (oV @m)) denotes the agreement of value estimates for state 8;. Impor-
tant sampling is defined as IS, Tene (Rila)) Figure 5(b) shows that, masking the advantages corresponding to the 20% of states with high critic convergence, the policy exhibits stable learn- ing dynamics even under high sample reuse (update-to-data ratio (UTD) = 4, i.e., each sample was used for training four times) and significantly improves sample efficiency. We further compared value-std (critic-side uncertainty) with entropy (policy-side uncertainty) (Wang et al., 2025a; Rahn al., 2024; Cui et al., 2025b) by masking equal fraction of states per step according to each metric. Figure 5(c) shows that value-std-brat based masking consistently delivers stronger learning ben- efits. This observation echoes classic RL findings (Osband et al., 2016), where aisemble-based value uncertainty acts as a proxy for learning dynamics. Figure reveals that low value-std states consistently align with low entropy, suggesting that value-std is precise uncertainty metric.
mo(arls))
Takeaway 2
Agreement among critics provides reliable measure of the learning benefit of the states.
Entropy filtering based value divergence. When critics exhibit significant divergence in their evaluation of a state 8j. reflected in a high standard deviation, it may indicate that 8j is reasoning- independent. For instance, different critics may encounter divergent reward distributions for trajec- tories passing through 8y, due to factors such as inference-irreleant tokens or inherent semantic patterns model generations. With a large X, the dispersion in returns distribution propagates back to each state, amplifying disagreement among critics. In such cases, persistent exploration at sy is meaningless, it does not correspond an actionable decision state (Figure (a)). To promote meaningful exploration while avoiding wasteful updates on noisy or non-decision states, we intro- duce a safe entropy regularization weighted by 3. Specifically, we filter out states with high value std deviation when computing entropy H. Complete policy loss is rewritten as:
lo]
(p)o(() = 1E ((Q)) ont (012)]
min IS, Ar, clip (IS;, 1-e, 1+e) A
10
(5)
0, if af E topu(o)
+ B I7H (mm
177
otherwise
Figure (b) shows that, unlike naive entropy loss, which can yield suboptimal learning, our entropy regularization mitigates entropy collapse and stabilizes policy learning, avoiding spurious explo- ration while guiding the policy toward better convergence with higher returns. We also compare filtering based value-std versus entropy. As shown in Figure 7 (c), the overlap between the two sets is minimal. Even after filtering the top 40% of hight value-std, policy entropy remains stable, while filtering the same fraction of high-entropy states causes entropy collapse. Statistical analysis of filtered tokens (Appendix C) confirms that removed words are typically adverbs, interjections that irrelevant decision-making Algorithm shows the pipeline of AsyPPO.

---

Published as a conference paper at ICLR 2026
Bitrere osdiation lffere Natl-modee Enttbalary
Aiye70 Eneeyy film (11))
1.00
2i high vatuc-a ease (48-4) high (00%)
Uncore
Agf Natre exmops ise AsyPPo
8.20
exropy
3.24
R16
Mini-orities
Actor
Wike culreatinn
Training
(R) Motivation of our cilkagy fihering
(b) Ablation study ous entropy filtering
(c) Compurison of two Eltering mediasissms
Figure 7: (a): When critics diverge, the state is weakly coupled to the final outcome and has complex future dynamics; exploration such non-critical states should be avoided. (b): Excluding states with high value-estimate standard deviation from the entropy loss prevents policy collapse induced by naive entropy regularization and yields a roughly 7 percentage-point improvement. The setup follows the settings in Figure 1 (b). (c): Excluding the top 40% of high value-std states from the entropy loss preserves policy entropy at levels comparable to naive entropy guidance, whereas filtering the same percentage of states with the highest entropy collapse. The settings are consistent with Figure 5.
Takeaway 3
Divergence among value estimations indicates the cost-effectiveness of exploring the states.
Algorithm 1: Asymmetric PPO with two mini-critic
Te: actor. (a : mini-crities. o, O: generation up step in response under prompt q, O denotes the total response in the batch. (O):: value estimation std across the critics. Á: corrected advantage. IA: The index for advantage masking. I#; The index for entropy filtering.
while training step < maximum step do
+- O+Ta(Q TQ())
Build training subsets for each critic, and update V according to Eq.2
Ā GAE(V, r). V + mean(V (Q, O), V42 (Q, O)) via Eq.3
Generate masking vector 1 <- Lown (o)) and filtering vector T <- (o)))
Update Ty via reconstructed PPO loss (Fq.5)
EXPERIMENTS
In $4, we demonstrated the efficacy of AsyPPO on 4B and 8B LLMs (please refer to Figure 2(b) for results) through diversified experiments. This section examines AsyPPO a suite of experiments. We organize the subsequent studies around three research questions: RQI: Can AsyPPO and naive asymmetric PPO unlock general reasoning in larger LLM? RQ2: How sensitive is AsyPPO to the size and number of critics? RQ3: What setups are effective for advantage masking and entropy filtering?
broadly through
5.1 GENERALIZATION TO LARGE MODELS
Setup. To ensure consistency with prior research, we fix the global batch size to 1024, with maximum response length of 8192 tokens. The learning rate is set to le - 6. For text generation, use top_p value 0.99, and top_k value = 100, temperature 0.99, UTD = 4 (also referred to as PPO_epoch, result in off-policy). The actor is Owen3-1 -b-Base, while critics vary in size from the Owen3-Base family. To ensure reproducibility and fairesss, exclusively datasets. We use the hard training dataset from Liu al. (2025d); Zeng et al. (2025), which exposes clear performance gaps across algorithms in long-tail reasoning tasks. We report the average @4
open-source
8

---

Published as a conference paper at ICLR 2026
across 4 challenging benchmarks, ie., MATH-500 (Lightman al., 2023), OlympiadBench (He et al., 2024), MinervaMath (Lewkowycz et al., 2022), and AMC 2023 (Xue al., 2025).
Naive asymmetric PPO uses a single critic, i.e., Qwen3-1. 7b-Base, Qwen3-4b-Base and Swen3-Bb-Bail and optimize with the vanilla PPO optimization objective. AsyPPO employs two mini-critics with advantage masking at 20% and entropy filtering at 20%. We use the setting of GRPO recommended by Liu et al. (2025d). Full hyperparameter details are provided in Appendix A.3.
Baselines. For all algorithms, actors initialized using Qwen3-14b-Bt
are
adcartiionn (1.00
comurr MATH-500
(b) Minerra
(c)
OlympiaddBenc
Figure 8: AsyPPO improves accuracy by an average of about 3 + points compared to GRPO, and achieves more than 20% lighter weight than symmetrical PPO. Our naive asymmetric PPO still works on the 14b policy, but fails under the 1.7b critic setting. However, AsyPPO unlocks the 1.7b critic's ability to guide the 14b actor.
Results. Figure 8 shows that AsyPPO with two 4b critics achieves the strongest results across all tasks. Compared to GRPO, AsyPPO improves accuracy by average of about 3 points. For naive asymmetric PPO (a single mini-critic guiding a large actor), we observe clear critic-capacity threshold: single Qwen3-1. 7 critic cannot reliably guide 14b actors, despite successfully guiding an 8B actor; upgrading to 4B critic restores effective learning. By contrast, AsyPPO low- ers this requirement, 1.7b crities deliver substantial reasoning gains. Combined with the lightweight deployability in Figure I(c), AsyPPO establishes an efficient and practical RL4LLM design.
500
175
32x
Aallla ofiile size
(b) of mumerr
(c)Andlys of size
60) valua odlcaiiion
Figure 9: (a): The increase in the size of the critic further enhances the effectiveness of AsyPPO, which can be regarded as the marginal benefit brought by the parameter scaling up. We initialize the actor using Swen3-8b-Base
and initialize the double mini critic using four sizes of the Qwen3 Base model. (b): A qualitative improvement
performance can be achieved by using two mini critics. (c): A suitable group size for AsyPPO is 32. (d): Using the mean of the critic's 's estimated value can achieve better correction of the value than using min. For (b,c,d), we initialize the actor using 3 --ieen---b-ae Bb-Base and initialize the mini crities using Owen3-1. 7b-Base.
5.2 ABLATION STUDY
The preceding results show that AsyPPO consistently enhances reasoning LLMs across scales. We provide a module-wist analysis to characterize the algorithm from multiple perspectives.
Ensemble critic system. Figure 9 (a) shows a scaling-law-like trend: increasing critic size steadily raises the policy's s peak score. We recommend using the largest critic model that fits in GPU memory to maximize AsyPPO's optimization capacity. However, we do not see comparable gains from increasing the number of critics: Figure 9 (b) shows that two mini-critics are sufficient for a clear step-change in performance. Varying the GRPO group size (trajectories per prompt) while keeping other parameters at their defaults (Figure 9 (c)), and found 32 to be a robust setting. Com- paring ensemble value aggregation (Figure (d)), the mean of values outperforms the min value, suggesting overestimation is not dominant issue in R...MM.

---

Published as a conference paper at ICLR 2026
04
Haa
Fir ky
Film ky 20%
Filur out sp 10%
380
Training step
Triining stop
Training step
Figure 10: Left: Average test score six benchmarks under various advantage masking setup. Middle: Average test score under various filtering out setup. Right: entropy curves during training. All experiments were based Qwen3- 8b Base actor and Qwen3-1. Base critic. The accuracy calculation follow Figure 3.
Value-convergenn-basee advantage masking. To identify robust advantage-maskinn percent- age, we adopt the main experiment settings with Owen3-8b-Ban as the policy and two Qwen3-1. 7 crities. Figure 10 (Left) shows that masking 20% of low-value-std states provides the strongest gains.
Yalue-divergence-baass entropy filter. To find an appropriate filtering percentage, we follow same setup as for advantage masking. We test masking 10%, 20%, 30%, and 40% of the highest- value-std states from the entropy loss. As shown in Figure (Middle, Right), larger masks induce entropy collapse, while 20% strikes the best exppltittiteloliiiiiiinn balance.
CONCLUSION
We reframed the critic bottleneck in RLALLM as an architectural rather than a purely algorithmic or optimization issue. Our proposed Asymmetric Proximal Policy Optimization (AsyPPO) reinstates the critic's role via double lightweight mini-crities trained on disjoint prompt-level data, yielding diverse yet calibrated value estimates. Beyond improving value estimation robustly, we showed that inter-critic uncertainty provides an actionable signal for policy optimization: masking advantages for low-informativenes. states and filtering high-divergence states from entropy regularization both reduce overfitting and promote safer, more effective exploration. Across standard LLM reasoning benchmarks, AsyPPO consistently improves general reasoning for models of varied sizes, empir- ically supporting asymmetric actor-critic design as a viable and efficient direction for RL4LLM. AsyPPO mitigates critic over- parameterization while improving sample and compute efficiency.
Limitations To ensure fairness and reliability under limited GPU resources, all experiments ini- tialized both actor and critic models from the widely used Qwen3 series. Evaluation on additional model families (e.g., Llama (Grattafiori et al., 2024)) is left for future work. Following (Liu et al., 2025d), we fixed the maximum generation length to 8k tokens, a common academic setting that bal- ances inference coverage while avoiding inference-cost blowups. We plan to assess the algorithm's 's generalization under ultra-long inference budgets and adopt classical RL practice of using a more diverse set of random seeds to further strengthen the robustness of conclusions. AsyPPO opens new avenues for RL4LLM design and raises several interesting questions. For example, do ensem- ble critic systems composed of different model families and sizes exhibit performance differences? Do variations in critic hyperparameter settings affect calibration and uncertainty estimates? Promis- ing directions also include anfidence--eighlee ensemble critics improve value estimation and analyze the relationship between value uncertainty and entropy.
ACKNOWLEDGMENTS
The work of Jiashun Liu and Ling Pan is supported by the National Natural Science Foundation of China 62406266. The LLM training is supported by ROLL (Wang et al., 2025c). We would also like to thank the Python community (Van Rossum & Drake Jr, 1995; Oliphant, 2007) for developing tools that enabled this work, including NumPy (Harris et al., 2020), Matplotlib (Hunter, 2007), Jupyter (Kluyver et al., 2016 6), and Pandas (McKinney, 2013).
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
REPRODUCIBILITY STATEMENT
We provide all the details to reproduce our results the Appendix.
LLM USE
LLMs were used to assist paper editing and write the code for plotting experiments.
REFERENCES
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding impact of entropy policy optimization. In International conference machine learning, pp- 151-160. PMLR, 2019.
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-ml: Scaling test-time compute efficiently with lightning attention. Xrviv preprint arXiv:2506. 135855, 2025a.
Hongzhan Chen, Tao Yang, Shiping Gao, Ruijun Chen, Xiaojun Quan, Hongtao Tian, and Ting Yao. Discriminative policy optimization for token-level reward models. Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. chen25ca.
PMLR, 13-19 Jul 2025b. URL
https:
mlr press
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q- learning: Learning fast without model. In International Conference Learning Representa- lions, 2021. URL https::t///....... openreview net http://ourmiiddddaae.a
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards. ArXiv, abs/2502.01456. 2025a. URL https:/ / //pii.... semant iommanttisss icscool ar. org/ /ttp///opustttc ID: 27 2767700 276107672.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models, arXiv preprint arXiv:2505.22 22611, 2025b.
Tiantian Fan, Lingjun Liu, Yu Yue, Jiaze Chen, Chengyi Wang, Qiying Yu, Chi Zhang, Zhiqi Lin, Ruofei Zhu, Yufeng Yuan, al. Truncated proximal policy optimization. arXiv preprint arXiv:2506. 15050, 2025.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint trXii:2505.24298, 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cogni- tive behaviors that enable self-im eef-iimrovinng reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.00 30070 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv: 1783, 2024.

---

Published as a conference paper at ICLR 2026
Charles R Harris, K. Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor. Sebastian Berg, Nathaniel J Smith, et al. Array program- ming with numpy. Nature, 58577825):3577-33222 2020.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: challenging benchmark for promoting agi with olympiad-leve bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
Zhiwei He, Tian Liang Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang. Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint DrXiv:2504.11456. 2025.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.032222 2025.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zzrro: An open source approach to scaling up reinforcement learning on the base model, arXiv preprint arXiv:2503.22 2429,, 2025.
John D Hunter. Matplotlib: 2d graphics environment. Computing in science & engineering, (03):90-95, 2007.
Thomas Kluyver, Benjain Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederie, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, Carol Willing, and Jupyter Development Team. Jupyter Note- books a publishing format for reproducible computational workflows. In IOS Press, pp. 87-90.
2016. doi: 0--233/77--1-61499-644 199
Hojoon Lee, Dongyoon Hwang. Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. ArXiv, abs/2410.09754, 2024. URL https:
//api. www.atmtttt..eeee.em....e....ee. waa.mannisccooola....... wrwpaloorr//oooo org/CorpssII- : 273346233.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: simple unified frame- work for ensemble learning in deep reinforcement learning. In International conference on ma- chine learning, pp. 6131-6141. PMLR, 2021.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sole et al. Solving quantitative reasoning problems with language models. Advances neural information processing systems, 35:3843-3855 2022.
Zhong-Zhi Li, Xiao Liang, Zihao Tang, Lei Ji, Peijie Wang, Haotian Xu, Haizhen Huang, Weiwei Deng, Ying Nian Wu, Yeyun Gong, et al. Tl: dr: Too long, do re-weighting for effcient I1m 2025.
reasoning compression. arXiv preprint arXiv:2506.026
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Jiacai Liu, Chaojie Wang, Chris Yuhao Liu, Liang Zeng, Rui Yan, Yiwen Sun, Yang Liu, and Yahui Zhou. Improving multi-step reasoning abilities of large language models with direct advantage policy optimization. arXiv preprint 4rXiv:2412.18277 2024.
Jiashun Liu, Johan Samir Obando Ceron, Aaron Courville, and Ling Pan. Neuroplastic expansion in deep reinforcement learning. In The Thirteenth International Conference Learning Repre- sentations, 2025a. URL https:: /ttp///ppprrv//eeeeeeeeeeeeeeeee iew. net/ wwt.//ertuuntteeee wt:./oormmi/ooo d= fa.
Jiashun Liu, Johan Obando-Ceron. Pablo Samuel Castro, Aaron Courville, and Ling Pan. The courage to stop: Overcoming sunk cost fallacy in deep reinforcement learning. In Forty-second International Conference on Machine Learning, 2025b. URL https: htttps://ppeeeeeeeeeee h/ppp//eeew.eeeeeeeeeeeeeeeeeeee net/
12

---

Published as a conference paper at ICLR 2026
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783 2025c.
Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, et al. Part i: Tricks traps? deep dive into rl for Ilm reasoning. arXiv preprint (ii::2508.022)) 2025d.
Guozheng Ma, Lu Li, Zilin Wang, Li Shen, Pierre-Luu Bacon, and Dacheng Tao. Network spar- sity unlocks the scaling potential of deep reinforcement learning. In Forty-second International Conference on Machine Learning, 2025. URL https: /oopnreveiviieeeeeeee ew net. forum? did= mIomqOskaa.
Olya Mastikhina, Dhruv Sreenivas, and Pablo Samuel Castro. Optimistic critics empower small actors. arXiv preprint arXiv:2506.01016, 2025.
Wes McKinney. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. O'Reilly Media, 1 edition, February 2013. ISBN 1709651 URL http: /wwp://wwwww/////////// amazon. www.co////eeeeeeee wwam/eeec//oe path=ASIN/14493135
Siddharth Mysore, Bassel El Mabsout, Renato Mancuso, and Kate Saenko. Honey. i shrunk the actor: A case study preserving performance with smaller actors in actor-critic rl. 2021 IEEE Conference on Games (CoG), pp. 01-08, 2021.
Travis E. Oliphant. Python for scientific computing. Computing in Science & Engineering, 9(3): 10-20, 2007. doi: 10. N0.1100//CCE..00...55..
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances neural information processing systems, 29, 2016.
Nate Rahn, Pierluca D'Oro, and Mare G Bellemare. Controlling large language model agents with entropic activation steering. arXiv preprint arXiv:2406.00244 2024.
Ravi Raj and Andrzej Kos. Intelligent mobile robot navigation in unknown and complex envi- ronment using reinforcement learning technique. Scientifie Reports, 14, 2024. URL https: //tpp.ssmanticaeee....eeeaeaaaaa www.eeetnnnnncolleeee .araar.rrg/oo :ttp:/oooo.III:3322200..
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-
dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02445. 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06344 2017.
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agar- wal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level effi- ciency. In International Conference Machine Learning, pp. 60365-30388 PMLR, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Yi Xiang Marcus Tan, Pihe Hu, L. Pan, and Longbo Huang. RIx2: Training sparse deep reinforce- ment learning model from scratch. ArXiv, abs/2205. 15043, 2022.
Joshua Tint, Som Sagar, Aditya Taparia, Kelly Raines, Bimsara Pathiraja, Caleb Liu, and Ransalu Senanayake. Expressivityarena: Can Ilms express information implicitly? arXiv preprint 1xXiv:2411008011 2024.
Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
13

---

Published as a conference paper at ICLR 2026
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.0193 2025a.
Tao Wang, Ruipeng Zhang, and Sicun Gao. Improving value estimation critically enhances vanilla policy gradient. In Forty-second International Conference on Machine Learning, 2025b. URL https: opw.revieweeeeeeeeeeeeeeee net forum?
Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. Reinforcement learning optimization for large- scale learning: An efficient and user-friendly scaling library. arXiv preprint arXiv:2506.06 2025c.
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. Sim- pletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning. arXiv preprint arXiv:2509.02479, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338. 2025.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo cooperative multi-agent games. Advances in neural information processing systems, 35:24611-24466 2022.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arxiv:2503. /4476, 2025.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. In Forty-second International Conference on Machine Learning, 2025a. URL https::/t/////...t /ttpps://pperreview net forum? ?id= 8ThnPFhGm8.
Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind ppo's collapse in long-cot? value optimization holds the secret. ArXiv, abs/2503.01491, 2025b. URL https:
//api. www.stecmaattoolaaoomomo wwwpaaapccccoooo..o.ooc.. : 666488.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint wrXiv22544.05//2 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simpleri- 20o: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar arXiv:2503. 18892, 2025.
Kaiyan Zhang. Yuxin Zuo Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Peng Li. Yu Fu, Xingtai Lv, Yuchen Zhang, Sihang Zeng, Shang Qu Hao-Si Li, Shijie Wang, Yuru Wang, Xi-Dai Long, Fangfu Liu, Xiang Xu, Jiaze Ma, Xuekai Zhu, Ermo Hua, Yihao Liu, Zonglin Li, Hua yong Chen, Xiaoye Qu, Yafu Li, Weize Chen, Zhenzhao Yuan, Junqi Gao, Dong Li, Zhiyuan Ma, Ganqu Cui, Zhiyuan Liu, Biqing Qi, Ning Ding, and Bowen Zhou. A survey of reinforcement learning for large reasoning models. 2025. URL https: //api.
orq/ Corpus ID 28 12472004.
Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shao- han Huang, Lei Cui, Qixiang Ye. et al. Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673. 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu. Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18N0 18071, 2025.
14

---

Published as a conference paper at ICLR 2026
Dingwei Zhu, Shihan Dou, Zhiheng Xi. Senjie Jin, Guoqiang Zhang, Jiazheng Zhang, Junjie Ye, Mingxu Chai, Enyu Zhou, Ming Zhang, Caishuang Huang, Yunke Zhang, Yuran Wang, and Tao Gui. Vrpo: Rethinking value modeling for robust rl training under noisy supervision. ArXiv, 0t//22588.30555 2025.
15

---

Published as a conference paper at ICLR 2026
A DETAILED EXPERIMENTAL SETUP
A. PLOT SETUP
To ensure clarity and intuitiveness in the qualitative analysis, all curves are consistently smoothed using identical parameters. Specifically, the mean values are computed using an 11 11-step moving window with an exponential smoothing factor of 0.6. The smooth window set and 2.
A.2 PROMPT
In this work, we incorporate the following instruction into the system prompt encourage the model to better demonstrate reasoning process: "Please reason step by step, and put your final within boxedd This setting is designed guide the model perform step-by-step reasoning and explicitly present the final answer in the form of {boxed thereby enhancing the clarity and readability of the output.
A.3 HYPERPARAMETERS
We employ ROLL, user-friendly and efficient open-source reinforcement learning framework, to implement our pipeline. Subsequently, the key parameters observed during the training process are presented as follows. See our code config file for more details on the parameters. For the 14b policy training. uniformly arrange the actors on (0, (6) and the critics on (16,32) GPUs. For other small models, uniformly place the actor at (0,8) and the critic at (8, 16) GPU. Detailed settings can be found in next page.
We use below setup for 4b and 8b policy seed: 42
max_steps: 500
save_stepst 500
logging_steps: 1
eval_steps:
gamma: 1.0 # discount factor
lambd: 1.0 GAE lambda
rollout_batch_size: 64
prompt_length: 1024
tesponse_length: 8000
salle__aggregatioo_sssttteeey "mean"
gradientt_mak__eeccntttae 0.2 mask 20%
entropy_loss_coeff: 0.01
attroo__fllee__nnsperrrttaaee 0.2 # filter out 208
ppo_epochs: 1 is also used in main experiments
adv_estimator: "gae"
init_ki_coet 0.0
asyne_generate_
actor_train:
training_args:
learning_rate: 1.0e-6
weight_decay: 0
per Seevice_train htteeettrtaaaaaaaaaeeeeeeee size: 1
gradient_ sraillet_accumulatiioneeee stepss 256
harmup_steps: 50
num_train_epochs: 50
16

---

Published as a conference paper at ICLR 2026
critic_1:
training_args:
learning_rate: 1.0e-5
weight_decay: 1.0e-2
warmup_steps:
"___device_train__aacch.o 1
gradieet__accmmllttiosssssss 128
wazmup_stepps 5
nun_train_epochs:: 50
critic_2:
training_arge:
learning_rate: 1.0e-5
weight_decay: 1.0e-2
warmup_steps: 5
per__eevioe_trrannbaaahccisiisse 1
yracleet__cccmullttloo_sseee 128
warmup_stepps
num_train_epochs: 50
actor_infer:
generating_args: max_new_tokens s(response_iength)
top_p: 0.99
top_k: 100
num_beams, 1
temperature: 0.99
nun_returrnseeuunces. 32
We use below setup for 14b policy
seed: 42
max_steps: 500
save_steps: 500
logging_steps:
eval_steps:
value__agrresstion_staaeeer "mean"
graddient_maakkeercenttgge 0.2 mask 208
entropy_loss_coef: 0.01
hutzooy_flleee_nns_pereeeettte 0.2 # filter out 20% or
rollout_Dattch_size: 64
prompt_length: 1024 1024
response_length: 8000
infer batch size: 4
ppo_epochs:
adv adv_estimator: "gae"
kit_kii_ceet 0.0
leyn
17

---

Published as a conference paper at ICLR 2026
actor_train:
training_args:
learning_rate: 1.0e-6
weight_decay: 0
pe ger_device_train_nee
stt.ttraiinnaathssiesssiisss 8
gradient_accumllttion_tteees 64
wazmup_steps: 50
nun_train_epochs: 50
critic_l:
training_arye aaggingg_rgg::
@eariiggraate, 1.0e-5
weight_decay: 1.0e-2
warmup_steps: 5
per_deeiio__rrinnnatcch....iii 2
graddent__accmmlltiiosssttss 16
wazmup_steps:
infer batch size: 4
num nun_trrin_epochhe 50
critic_2:
training_args:
learning_rate: 1.0e-5
weight_decay: 1.0e-2
warmup_steps: 5
per _er_device_train batch batch size: 2
batch_size:
oradlent_accumlllioo__tttt
16
warmup_steps:
infer batch size: 4
num_train_ trrchii 50
actor_infer:
generating_args:
max_new_tokens S(response_eengh)
top_p: 0.99
top_k: 100
num_beams:
temperature: 0.99
nun_returrnsequenccs.
B THE RELATIONSHIP BETWEEN VALUE STD AND STATE INFORMATION QUANTITY
Specifically, for the training scenarios of 8b actors and two 0.6b critics, use the value-std corre- sponding the global state and the median of the gradient magnitude to categorize the states into four types. Namely, large gradient & large value std, large gradient & small value std, small gradient & large value std, small gradient small value std.
The results in Figure 11 (Left) show that the vast majority of states are classified into the categories of large gradient large value std and small gradient & small value std, thereby empirically proving the positive relationship between value std and the learning value (information quantity) of the state.
C VISUALIZATION OF WORD CLOUDS
We statistically analyzed the word clouds of the tokens with the highest mask frequency in the initial stage of AsyPPO training. The results in Figure 11 (Right) show that mask mechanism tends to mask adjectives, adverbs, and some isolated symbols, with less involvement in logical transitions, except for the slightly prominent progressive word "therefore".
18

---

Published as a conference paper at ICLR 2026
Small gradbent & Large value std
0.2
or
move
clear
those
highly
X
that
grathent Large
prdiio Sqoll
Relatively
obviously
Bul
finally
let
by)
higher
Therefore
call
carefully
*
Lange gradient Small
may
Lets
Given undoubtedlyyly ON
Figure 11: Left: Statistics within a mini-batch the mid-training stage. Right: The 40 tokens that masked most frequently in the same mini-batch.
D ADDITIONAL RESULTS
Extensive comparison. We expanded our comparisons in the revised manuscript and included two strong recent baselines to further understand the effectiveness of method. Additional baselines: VAPO(Yue et al., 2025): a popular critic-centric RLVR method (evaluated with its default configu- ration). Q-function Reward Model (Q-RM) + PPO (Chen al., 2025b): a novel token-level reward method for math reasoning tasks. Experimental setup: To ensure fair comparison, we follow the main experimental protocol using DeepScaleR 40k, with token generation length and 600 train- ing steps, using Qwen3- Dwen3-14B-Bass 14B-Baee the policy model. We report the average Avg @4 score across four benchmarks. As shown Table 1: Compared to critic-centric VAPO, AsyPPO (14B actor, 2x1.7B critic) delivers about 6.5% improvement (while using significantly smaller critic). Com- pared to the token-level reward model (Q-RM)+PPO, AsyPPO shows about 4% advantage despite Q-RM requiring additional pre-training cost.
Method
Average Accuracy Across Benchmarks
AsyPPO
61.3
VAPO
54.8
Q-RM PPO
56.2
Table 1: Performance comparison.
Reward sparsity analysis. We use the process reward model to provide step-wise dense rewards for the policy following, in addition to the extremely sparse reward setting with only the final reward used in the main experiment. The results in Table 2 demonstrate that while both methods benefit from dense rewards, AsyPPO consistently outperforms PPO in both settings.
Method
AsyPPO
PPO
Dense Reward
84.9
81.5
Sparse Reward
83.6
78.3
Table 2: Performance comparison.
Domain transfer to code generation. added code-domain evaluation BiveChdeBench base on SeepSeck-Rl-Distie Qwen- Max Response Len 8K tokens. As shown Table 3, AsyPPO consistently outperforms PPO.
19

---

Published as a conference paper at ICLR 2026
Method
Base Model
After AsyPPO
After PPO
Performance
45.6
55.2
49.7
Table 3: Performance comparison.
Validation across multiple model families. We have added another series of open-source models that are widely used in academic research. We evaluated Llama-3.1 1-aa-3.--B-BBass and reported the aver- age score across six benchmarks (Avg @4),..e. AIME 24, AMC23, GSM8k, MATH-500, Minerva Math, OlypiadBench. As shown in Table AsyPPO achieves average performance improvement of about 5% over PPO, further demonstrating the effectiveness and generality of our method.
Model Family
Initial Score
After AsyPPO
After PPO
Qwen3-8B-Bar
41.4
59.8
57.1
-.lama-3.1-8B--al
4.6
23.8
19.2
Table 4: Performance comparison.
To assess whether entropy filtering removes rare but valuable states, we analyze the filtered tokens at training steps 100, 200, and 300. Specifically, compute the proportion of filtered tokens associ- ated with exploration, using the key exploration-related tokens identified in Wang et al. (2025a). As shown in Table 5, the fraction of reasoning-relatee tokens among the 20% filtered by entropy in the main experiment is very small, indicating that the mechanism does not discard valuable reasoning paths.
Method
100 steps
200 steps
300 steps
Filter-out ratio (%)
4.63
2.26
1.81
Table 5: Average results over 16 runs on MATH-500.
Mitigating rigid advantage calculation. AsyPPO employs group-based rollouts, enabling each critic observe multiple trajectories per prompt. Combined with our non-overlapping data parti- tioning, critics capture complementary views of the reward distribution, reducing variance in token- level advantage estimates. As shown Table 6, AsyPPO's advantages are more conservative than GRPO's. This reduces overestimation of frequent non-reasoning phrases in positive samples, e.g., frequently occurring verb phrases, and underestimation reasoning-critical tokens in negative gen- erations. This prevents collapse into superficial reasoning modes (e.g., repetitive phrases like "So, let's "'...""s...""
Sample
100 step
200 step
300 step
Positive samples (GRPO: 1)
0.91 0.13
0.84 + 0.06
0.87 + 0.11
Negative samples (GRPO:
0.26 0.35
0.13 + 0.18
0.09 + 0.16
Table 6: Average token advantage.
20


Reviews:
[{"review_id": "sMBfaJYB8q", "reviewer": "Reviewer_AgbJ", "summary": "This paper introduces Asymmetric Proximal Policy Optimization (AsyPPO), a framework designed to address the high computational cost and instability of training large critic models in Reinforcement Learning for LLMs (RL4LLM). Instead of the conventional symmetric design where the critic is as large as the actor, AsyPPO utilizes an ensemble of lightweight \"mini-critics\". To ensure these small critics provide robust and diverse guidance, they are trained on disjoint, non-overlapping data partitions. The framework further leverages the uncertainty between these critics to refine the policy objective: it masks advantages in states where critics agree (low uncertainty) to prevent overfitting to low-information samples, and it filters states where critics diverge (high uncertainty) from the entropy regularization term to promote safer exploration. The authors demonstrate through experiments that AsyPPO improves reasoning performance over baselines like GRPO and achieves significant reductions in computational overhead (memory and time) compared to classic symmetric PPO.", "questions": "- The paper frequently uses the term \"UTD\" (update-to-data ratio), setting it to 4 in key experiments. The appendix also lists `ppo_epochs` as a hyperparameter, setting it to 1 or 4. Could the authors explicitly clarify the definition of UTD and explain the precise difference between it and the `ppo_epochs` hyperparameter in their training setup?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper addresses the significant and timely problem of the computational bottleneck in critic-based RL for post-training LLMs\n- The proposed method is intuitive and well-motivated.\n- The paper is well-written, easy to follow, and provides a thorough analysis of each of its components, supported by targeted ablation studies (e.g., ensemble strategy, advantage masking, and entropy filtering).\n- The method is validated across a variety of LLM reasoning benchmarks.\n- The literature review is comprehensive and effectively situates the work within the current RL4LLM landscape.\n- The authors include a clear discussion of the work's limitations and potential avenues for future research.", "weaknesses": "- The empirical evaluations could be strengthened. Several learning curves in the figures do not appear to have fully stabilized or converged, showing volatility late in training (e.g., in Figure 4(c), Figure 6(b), and parts of Figure 7). Extending the training steps could provide a clearer validation of the method's stability and final performance.\n\n- The performance gains, while consistent, are not always substantial. For instance, the main results in Figure 7 show an average improvement of \"about 3 points\" over the GRPO baseline, which may be modest.\n\n- The study's baselines could be more comprehensive. While GRPO is a strong baseline, the paper would benefit from comparison against other advanced RL4LLM algorithms mentioned in the related work, such as DAPO or REINFORCE++, to better contextualize its contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jfb0pLrKaC", "reviewer": "Reviewer_FX1U", "summary": "The paper proposes AsyPPO as a lightweight method for incorporating more than one mini-critics in RL. Instead of a full-size critic, AsyPPO uses two lightweight \"mini-critics\" trained on disjoint, prompt-level shards to encourage diversity while staying calibrated. Their predictions are averaged for value estimation, and the standard deviation across critics is treated as an uncertainty signal: the method masks advantages for states where critics strongly agree (to avoid low-information updates) and excludes high-disagreement states from the entropy term (to avoid spurious exploration). On math-reasoning benchmarks, AsyPPO reports notable average gains over GRPO, classic PPO, and lower peak memory / faster steps than symmetric PPO, while remaining competitive and more lightweight.", "questions": "1. Are masking and entropy filtering applied strictly token-wise or differently?\n2. The paper shows gains with bigger critics. Where's the diminishing return point relative to actor size, and how does that trade off with ensemble count? And any insights on why more critics does not yield meaningful return?\n3. Can entropy filtering over-prune rare but valuable states (e.g., creative solution paths), reducing long-tail discovery?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Re-introducing critics via small, prompt-sharded ensembles seems (to me) a clear, practical twist that fits LLM training constraints and mitigates over-parameterized critics. The paper formalizes the training and aggregation cleanly.\n2. The two signals, agreement-based advantage masking and divergence-based entropy filtering, are well-motivated and implemented directly inside PPO, with ablations supporting the choices.\n3. The paper shows lower peak memory and faster training steps vs. symmetric PPO, and consistent accuracy gains vs. GRPO/symmetric PPO under off-policy reuse. The \"two critics is the sweet spot\" result is helpful guidance.", "weaknesses": "1. Strong critic-free baselines like GRPO are covered, but comparisons to recent critic-centric or token-level reward methods are referenced more than reproduced, making it hard to isolate where wins come from in identical settings.\n2. Claims of stability under off-policy (UTD>1) are promising, but it's unclear how performance changes with different UTD, lambda, or reward sparsity beyond the selected setup.\n3. Experiment-wise, most results are math-reasoning; model family is Qwen3-only. This raises external validity concerns (other domains, model families like Llama, non-verifiable reward, etc.). I strongly suggest validating the claims on at least some coding benchmarks.\n4. The method introduce addition hyperparameters, including the choice and number of critics, value aggregation, advantage masking fraction, and entropy filtering fraction. Without cross-domain experiments, it's challenging to assess how sensitive these hyperparameters are to different domains, tasks, or model families in practice (and if they are easy to tune).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2gumVubpsm", "reviewer": "Reviewer_yexD", "summary": "This paper introduces Asymmetric Proximal Policy Optimization (AsyPPO), a lightweight actor–critic method for reinforcement learning fine-tuning of large language models. It employs several small “mini-critics” trained on non-overlapping data to provide ensemble value estimates. The agreement and disagreement between critics are used to stabilize policy updates through masking and filtering. Experiments on multiple reasoning benchmarks show 3–6% accuracy gains over PPO and GRPO with lower memory usage and faster training.", "questions": "Please response to the points listed in the weaknesses", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.This paper is well motivated. \nThe paper revisits a central assumption of modern RL4LLM, namely the removal of explicit critics (as in GRPO), and proposes an elegant asymmetric redesign that restores the critic’s role while remaining scalable.\n\n2.This paper is easy to understand. \nThe proposed mini-critic ensemble, prompt-level non-overlapping partition, and uncertainty-based masking/filtering are clearly described and intuitively justified.\n\n3. Strong exps results. \nThe method consistently improves performance across several benchmarks, while reducing memory footprint by ~20% and maintaining comparable speed to GRPO.", "weaknesses": "1. Limited interpretability of critic signals\nThe mini-critics are trained without access to final-answer correctness; hence their value estimates do not guarantee semantic or symbolic validity, especially on complex mathematical reasoning tasks. The reported gains may stem from smoother optimization rather than genuinely better reasoning reward estimation.\n\n2. Lack of compatibility study with GRPO framework\nFollowing point 1, as GRPO could provide \"correct\" outcome reward, it is essential to verify whether AsyPPO can be plugged into GRPO’s group-based rollout and reward pipeline as a complementary.\n\n3. Evaluation scope\nAll experiments are restricted to the Qwen3 model family and math-style datasets. It remains uncertain whether the observed benefits generalize to more diverse reasoning or open-ended tasks and models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)