Goal Achievement Guided Exploitation: A Principled Performance-Based Scheduling Framework for Reinforcement Learning
Authors: Shengchao Yan
Baohe Zhang
Chenguang Huang
Joschka Boedecker
Wolfram Burgard
Venue: Accepted by TMLR ()

Abstract: In dense-reward tasks, Reinforcement learning (RL) algorithms often employ soft entropy regularization to promote exploration. By integrating an entropy term into the objective function, they regularize exploration via tuning the coefficient. However, the entropy coefficient only indirectly influences the action distribution through gradient updates, making it difficult to precisely control exploration, and requires careful scheduling to balance exploration and exploitation throughout training. As a solution, we propose Goal Achievement Guided Exploitation~(GAGE), a performance-based scheduling framework that adaptively regulates exploration by linking policy stochasticity directly to the agent's performance relative to a target value. Unlike soft entropy regularizers, GAGE enforces hard, performance-dependent constraints on action distribution's standard deviation for continuous actions and logit range for discrete actions. Consequently, GAGE ensures a guaranteed lower bound on action probabilities that naturally decays as the agent approaches optimal performance. Across a suite of challenging robotic control tasks, GAGE improves learning efficiency and stability across various strong baselines, achieving competitive or superior final performance. By providing a more interpretable and robust alternative to entropy-based exploration heuristics, GAGE offers a scalable path toward solving complex dense reward tasks with pronounced local optima.

Published in Transactions on Machine Learning Research (03/2026)
Goal Achievement Guided Exploitation: A Principled Performance-Based Scheduling Framework for Reinforcement Learning
Shengchao Yan
uni de
Baohe Zhang!*
changb@es.uni--rrrtte de
Chenguang Huang?
chenguang.
chenguang.haung..uee haang resuanggrsaarrhhaaamaa. com
Joschka Boedecker*
Shoedeck@ess.uni.ffeee de
Wolfram Burgard²
wolfram, www.mrdddnnannnn.nn.onmmmm
Deportment of Computer Science & IMBIT//Branl Po University of Presburg
*Depormennt of Computer Science and Artificial Intelligence, University of Technology Nuremberg
Reviewed on OpenReview;
// o/penrevwee... net/ pert/formm? id- uGsdWOfKNK
Abstract
In dense-reward tasks, Reinforcement learning (RL) algorithms often employ soft entropy regularization to promole exploration. By integrating an entropy term into the objective function, they regularize exploration via tuning the coefficient. However, the entropy coef- ficient only indirectly influences the action distribution through gradient updates, making difficult precisely control exploration, and requires careful scheduling to balance ex- ploration and exploitation throughout training. As solution, propose Goal Achieve- ment Guided Exploitation (GAGE), a performance-based scheduling framework that adap- tively regulates exploration by linking policy stochasticity directly to the agent's perfor- mance relative to target value. Unlike soft entropy regularizers, GAGE enforces hard, performance-dependeee
constraints on action distribution's standard deviation for continu- ous actions and logit range for discrete actions. Consequently, GAGE ensures a guarantcod lower bound on action probabilities that naturally decays as the agent approaches optimal performance. Across suite of challenging robotic control tasks, GAGE improves learning efficiency and stability across various strong baselines, achieving competitive superior fi- nal performance. By providing a more interpretable and robust alternative to entropy- exploration heuristics, GAGE offers a scalable path toward solving complex dense reward tasks with pronounced local optima.
1 Introduction
Local optima in the objective landscape pose a fundamental challenge for gradient-based optimiza- tion (Chaudhari et al. 2017), especially in high-dimensional robotic control tasks, Even in environments with dense rewards, the trial-and-error nature of reinforccment lcarning (RL) means agents often learn prodomi- nantly from suboptimal trajectories. As result, RL algorithms are often highly sensitive to hyperparameters and prone to premature convergence to arbitrary local optima.
To mitigate this problem, entropy regularization has been widely integrated in RL algorithms. By augment- ing the standard optimization objective with an entropy term that encourages policy stochasticity, entropy
"Equal contribution.

---

Published in Transactions on Machine Learning Research (03/2026)
regularization promotes exploration and often yiclds more stable and robust policies (Williams & Peng. 1991 Mnih et al. 2016 Schulman et al. 2017: Haarnoja et al. 2018a). However, relying on a soft regular- ization through entropy coefficient adjustment also introduces practical limitations. It necessitates extensive hyperparameter tuning to manage the exploration-cxplottatiio trade-off and sometimes requires a manual coelficient annealing schedule to ensure precise, low-entropy policies at convergence. The manual luning of coefficient can be difficult to design and highly sensitive reward scaling (Haarnoja et al. 2018a). Existing methods like Soft Actor-Critic (SAC-v2) (Haarnoja et al., 2018b) attempt to automate the scheduling by learning the entropy coefficient for the policy match fixed target entropy. However, this acts static soft constraint, which may be too high for fine-grained control or too low for initial exploration, failing to provide the adaptive decay required for convergence.
Despite various recent advances in exploration strategies (adoss et al., 2022 Wang et al. 2023 Wan et al. 2023 Yan et al., 2024b; Sukhija et al.) (025), the burden of tuning exploration schedules remains a. bar- rier scalable deployment. In this work, we propose Goal Achievement Guided Exploitation (GAGE), a framework that formalizes exploration annealing inlo principled exploration scheduling strategy by lever- aging the agent's goal achievement, defined the ratio between its current performance and the optimal value. By utilizing this signal, our method adaptively reduces exploration as the agent approaches its desired performance, allowing the exploration rate to evolve naturally with learning progress rather than following a manually specified schedule. This is achieved by imposing a hard constraint on the action distribution through an adaptive constraint on policy stochasticity, implemented as a minimum standard deviation in continuous action spaces and limited logit range in discrete ones.
We implement approach across both continuous and discrete action spaces, well on- and off- policy algorithms. We evaluate GAGE on challenging locomotion and dynamic manipulation tasks involving quadruped and humanoid robots, along with benchmarks from HumanoidBench (Sferrazza et al. 2024). and show that our approach overall improves learning efficiency and performance compared to strong baselines. GAGE also demonstrates robustness to reward ongincering. Furthermore, we demonstrate that the previonsly observed advantage of discrete over continuous policies continuous control lasks (Tang Agrawal 2020) primarily arises from premature convergence, limitation effectively alleviated by our method.
In summary, make the following key contributions:
1. We propose Goal Achievement Guided Exploitation (GAGE), a novel framework that regulates ex- ploration through an adaptive constraint linked to the agent's 's goal achievement, offering principled performance-basee scheduling framework that automates the exploration-exploiaaton trade-off.
2. We propose using hard constraint for different measures of distribution stochasticity, which are more interpretable and controllable than entropy-based soft regularization.
3. We integrate GAGE seamlessly with continuous and discrete action spaces, as well as on- and off- policy algorithms. We demonstrate the strong empirical performance of GAGE across challenging customized robot control tasks and the HumanoidBench benchnark.
4. We derive theoretical lower bound on action probabilities guaranteed by our method, demonstrating its effectiveness with an empirical analysis.
2 Background and Related Work
Premature convergence is a long-standing challenge in optimization and machine learning, affecting algo- rithms including genetic algorithms (Pandey et al., 20144) and reinforcement learning Yan et al. (2024a). To clarify the specific focus of this work, we distinguish our focus on premature convergence from issucs such as "reward hacking" and sparse rewards in RL, as elaborated in Appendix A. In the following, we review the principle of entropy regularization, analyze its limitations, and discuss related work.
2

---

Published in Transactions on Machine Learning Research (03/2026)
2.1 Entropy Regularization in Reinforcement Learning
In the standard RL setting, we model the task as a Markov decision process defined by the tuple (S,A, p. r): where the agent interacts with the environment following the policy a(- 8t) with state 8, E 8 and action Or E A. In each time step, the environment transits from state to 8t+1 according to the state transition probability p: SxSxA= + [0, 1] and gives the agent a reward following the reward function SxA + R. The objective of standard RL to maximize the expectation of the cunulative reward Σ, a.),,.))~ [r(8,, at)]. where P= stands for the statc-action distribution following policy т. The objective can be extended infiite-hhrizon problems with a discount factor 7.
A common practice in RL is to employ entropy regularization objective (Williams & Peng 1991 Mnih
ct al. 2016 Schulman et al. 2017 Haarnoja et al. 2018a Espcholt et al., 2018). which augments the standard optimization objective with the expected entropy of the policy, such as in SAC or PPO:
Σ (ros (H(((.) st))] or L LCLIP cL. (0) + S. [M(a(- (x))),
where H() denotes the policy entropy, 3, are coefficients, and LELIP (0) (0) are the clipped surrogate objective and value function loss. This formulation encourages exploration by promoting higher entropy.
Manuel Exploration Scheduling The additional entropy term inevitably alters the optimization ob- jective. While entropy maximization improves exploration and robustness (Ahmed et al., 2019), it can be detrimental in tasks that roquire a precise, low-entropy policy. To recover the conventional objective at con- vergence, the entropy temperature 3 must be gradually reduced so that 3 O. However, manuel schedule tuning can become prohibitively costly due to the high sensitivity of RL algorithms to hyperparameters. In contrast, goal is to develop more adaptive mechanism that automatically adjusts exploration without the need for such manual scheduling.
Limited Interpretability and Controllability for Exploration The temperature B is sensitive hyperparameter that lacks universal scale; its optimal value is coupled with the reward mdenituuee action dimensionality, and the landscape of specific local optima (Haarnoja et al. 2018b). Rather than relying on soft regularization through temperature adjustment, we propose to impose hard constraints directly on the policy's action distribution to obtain more predictable and controllable exploration behavior.
2.2 Related Work
In this section, we review contemporary approaches to enhancing exploration in reinforcement learning.
Adaptive hyperparameter tuning Soft Actor-Critic (SAC-v1) (Haarnoja et al. 2018a is one of the most widely adopted off-policy RL algorithms using entropy-based regularization. However, its performance is highly sensitive to the temperature hyperparameter B, whose optimal value is non-trivial to tune. To address this issue, Haarnoja et al. (2018b) proposed learning a gradient-based / that matches the expected entropy to a predefined target value. While this approach (SAC-v2) enables dynamic adjustment of B during training, it shifts the tuning burden to the choice of the target entropy. The fixed entropy constraint also limits the policy's 's ability to either start learning with higher exploration rate or converge to an optimal deterministic solution. To eliminate the need for additional hyperparameters, Wang & Ni (2020) applied metagradient method (Xu et al.) 2018) to tune & automatically. However, the reported performance improvement over SAC is marginal, and metagradient updates can still become trapped in local optima due to their gradient-based nature.
Intrinsic motivations While our work addresses premalure convergence in dense-reward environments, another line of research focuses on encouraging exploration in sparse-reward tasks, where the optimization landscape is largely flat and external rewards are scaree. Inspired by the curiosity-driven behaviors of ani- mals (Schmichuber) 1991), intrinsic motivation methods encourage exploration by rewarding the agent for visiting novel informative states. Count-based approaches (Bellemare et. al. 2016 Tang et. al. 2017
3

---

Published in Transactions on Machine Learning Research (03/2026)
achieve near-optimal exploration in tabular settings but scales poorly to high-dimensional or continuous domains. Prediction-error-busedd methods (Pathak et al. 2017 Burda et al. 2019 train forward or inverse dynamics models and use discrepancies between predicted and observed transitions as intrisic rewards. Zhang
ct al. 2021) proposed rewarding novelty differences between states to encourage breadth-first exploration,
while Wan et al. (2023) scaled observation novelty using mutual information between states and trajecto- ries. Although effective in sparse-reward environments such as MiniGrid (Chevalier-oiivert et al. 2023) or navigation tasks, the applicability of these methods to high-dimentional continuous control task with severe local optima remains not clear. Moreover, balancing extrinsic and intrinsic rewards is non-trivial. MaxIn- foRT. 4Sukhija et al. 2025) mitigates this issue by introducing information-based rewards and automatically tuning the temperature for the transition information gain. They reported superior results in off-policy con- tinuous control lasks with humanoid robots. However, the method incurs significant computational overhead due to model ensembles and target policies, and it is limited to off off-policy settings.
Goal and trajectory planning The Markovian assumption in standard RL algorithms may be subop- timal for tasks involving partial observation or long-horizon dependencies. Recent works have approached exploration a structured, long-term planning problem. Jain et al. (2023) utilized non-Markovian poli- cies that condition past trajectory to maximize stale coverage within limited steps, achieving efficient exploration in gridworld and simple continuous control tasks such as Reacher and Pusher. Hu et al. (2023) leveraged learned world models and planning algorithms to generate exploratory goals with high exploration potential, effectively constructing a goal curriculum. Similarly, Dinz-Bone et al. (2025) quantify the achiev- ability, novelty, and relevance of exploratory goals to guide exploration toward meaningful directions. How-
ever, these approaches generally rely on prior task knowledge and are primarily evaluated in low-dimentional action spaces sparsc-reward environments, such as navigation gridworld tasks.
Multi-modal policy In standard RL implementations such as SAC, the policy is typically modeled as a Gaussian distribution, whose unimodal nature limits the representation of multiple behavioral modes. [Tang & AAgraaal (2020) addressed this issue by discretizing continuous action spaces. With a sufficiently fine granularity (e.g., over 11 bins per action dimension), discrete policies outperformed Gaussian ones on most MuJuCo benchmarks. More recently, Dong et al. (2025) adopted diffusion models for policy represcntation, enabling flexible, multi-modal action distributions. They demonstrated marginal performance improvement MuJuCo tasks. Despite their enhanced representation capacity, these approaches remain constrained by the same limitations inherent to entropy-based exploration discussed earlier. In this work, we provide implementations using both Gaussian and discrete policies. We identify combining diffusion- policies with our framework as a promising direction for future research.
3 Goal Achievement Guided Exploitation
To account for the limitations of entropy regularization, we propose a novel framework named Goal Achieve- ment Guided Exploitation (GAGE). In the optimal case, a policy should not converge before the agent approaches its maximum achievable performance. Therefore, is natural to relate the policy's convergence level to its goal achievement. In this section, we first. define goal achievement. formally, then describe how we use it to construct an adaptive constraint for policy convergence, and finally analyze the benefits of GAGE.
3.1 Goal Achievement
Since an agent trained to maximize the cunulative reward, we define its goal achievement
g(T) = Jn J J= = Σ E (oo,aa)-
T
(1)
By definition, g(x) S 1 and g(T) + 1 as π + T.. The expected return of the current policy Ja can be estimated using the average return of recent rollouts. However, determining the optimal expected return Jn- is sometimes difficult for complex reward functions. In addition, some reward components may not fully align with the task goal, potentially leading to increased g without achieving the actual goal.
4

---

Published in Transactions on Machine Learning Research (03/2026)
To address these issues, examine the structure of the reward function more closely. A typical reward function consists of multiple terms, which can be categorized into gool rewards (Tgoma) and awniliary rewards (Taux). Goal rewards are mandatory and correspond to the intended task objective, such as winning a game or exccuting a robotic behavior. Auxiliary rewards are optional heuristics designed to guide or accelerate learning. We exclude auxiliary rewards when measuring goal achievement for two reasons: (1) they may not align with the true task goal and can lead to suboptimal solutions; (2) their maximum attainable values are often difficult define, whereas the maximum goal reward values are typically available in the task specification. Accordingly, we define the goal achievement for the goal reward as:
((T) = Jgual, 5
T
J Igoal,
Σ t=
E as (o (ggal((s,, az)]
In practice, approximate goal achievement using the moving average of the per-episode ratio between the cumulative goal reward Ei= "gaal,r and its corresponding maximum value > T
k=
goal.t"
Tasks can contain multiple goal reward terms [u et al. 2020 Hayes et al. 2022 In this work, we focus on single-goal settings and leave the multi-objective extension for future research. We primarily consider non- negative rewards. For negative rewards, transformations such as sigmoid or offset can ensure g(T) E [o, 1]. If individual components are unavailable, g(x) can be approximated using cumulative return of total rewards Eq. 1 When the maximum return is unknown, the optimal performance be estimated empirically from observed trajectories, as further demonstrated in Sec. 4.1
3.2 Mitigating Premature Convergence via Action Smoothing
To prevent policy collapse, where the agent prematurely converges to few actions in discrete spaces or an excessively narrow Gaussian in continuous spaces, we introduce an action smoothing technique. It avoids overconfidence by lower-bounding the action space coverage of the policy distribution according to the current goal achievement. Concretely, for each action space type, we select a scalar quantity F(T) that captures action space coverage. We refer to this informally as the flatness of the policy.
We then enforce an adaptive lower bound Fin on the flatness, requiring F(T) 2 Fin throughout training. This lower bound is decreasing function of goal achievement:
Fin(()) = f(g(T)).
As a simple heuristic, we choose be an affine function of the goal achievement, as specified in Eqs. and leaving investigation of alternative mappings for future work. We next specify F(x) and its implementation for continuous and discrete action spaces. Full algorithmic details with different backbone algorithms are outlined in Appendix B
Continuous action space In continuous domains, exploration is typically facilitated by modeling actions Gaussian distributions: p(a s) (N((s), o²). This formulation is used in both stochastic policies, such as SAC and Proxinal Policy Optimization (PPO) (Schulman et al. 2017 Haarnoja et. al. 2018a and deterministic ones such as Deep Deterministic Policy Gradient (DDPG) [illicrap et al. 2016). where Gaussian noise is added for exploration. We employ the standard deviation o of the policy as the flatness measure since larger a directly corresponds to broader action space coverage. The corresponding lower bound øLB (i.e., FLB for the continuous case) is heuristically set as:
(LT(TT) (T) + do.
(2)
where the hyperparameter dg > controls the minimum allowed standard deviation when goal achievement (()) zero. Action smoothing applied by clamping a to dLB, ensuring 2 dLB- When do = 0, the formulation reduces the original bascline algorithms. Although entropy contains similar information standard deviation for Gaussian distributions, directly constraining is more straightforward.
5

---

Published in Transactions on Machine Learning Research (03/2026)
Discrete action space For discrete actions ok, where lo E (1,...,KK) KK), probabilities are computed as:
exp(zx)
p(ax s) = softmax(2)) = E exp(2;)
where = (21....,22k 2x) RK are the logits output from the policy network. Analogous to for Gaussian distribution in continuous spaces, the range of logits reflects the flatness of the categorical distribution. We thus use the negalive logil. range óg as F(x) to ensure that all actions retain minimum probability:
max 22 2j min 2 max 8 wj
(4)
k
and following the same heuristic as in continuous space, the corresponding lower bound 82,1. (i.e., Fin for the discrete case) is set as:
LLL. = (ō.,1 - (2,0g(TT)) + 60,0,
(5)
where 82,0 and ō_,1 are hyperparameters controlling the minimum allowed ó₂ at g(T) = and g(x) = In principle, d,,1- -00 as the policy approaches deterministic. However, this would make numerically difficult to calculate Eq. In practice, we find 62,1 R -15 is sufficient. We implement this constraint using softmaz with adaptive temperature (Asadi & Littman 2017). softmax(z/T), which controls the distribution flatness by adaptively scaling logits before exponentiation. The adjusted logits are computed as:
1.
max
T
122,L
after which action probabilities calculated via softmax(2'). The temperature 7 rescales the logits that 8er 2 62,LB. In practice, enforcing flat distribution with hard constraint the softmax logits alone result in prohibitively large logit magnitudes. To address this issue, we regularize the logits by minimizing:
C.
1=:0
Barpo11= (1=(2)
(6)
t=0
where is regularization coefficient. It's worth noting that label smoothing (Szegody et al. 2016). another technique for distribution flattening, is not suitable for action smoothing. We explain our choice for softmax with temperature in Appendix A.2 in detail.
3.3 Advantages of GAGE
Automated exploration scheduling GAGE enables a dynamic exploration rate without manuel param- eter scheduling, making it easier to tune and generalize. Furthermore, since the constraint depends solely on goal achievement, GAGE promotes more stable learning dynamics that are less sensitive to local optima.
Analytical guarantees for non-vanishing exploration GAGE imposes hard lower bounds the flatness of the action distribution, ensuring that exploration never vanishes. This formulation analytically lower-bounds the probability of all actions for both contu and discrete spaces. For Gaussian policy, this property follows naturally from the lower-bounded standard deviation. For discrete actions, the probability bound after action smoothing can be derived explicitly from Eqs. and
min p(az (s) exp (ming 24)
exp(maxx(c (zx) + 82,LB)
exp 82,LB
Σ
exp(max,(<z) + dL.LB) + (K 1) exp(max, 2x) exp 6 + (K -K-1)
Ei=1 exp 24
Thus, the minimum action probability is adaptively lower-bounded ((T). This mechanism prevents the irreversible loss of potentially optimal behaviors shown in Fig, 7
function of the goal achievement
4 Experiments
This section validates GAGE on robot control problems characterized by high degrees of freedom and local optima, often leading to premature convergence. We show (i) how GAGE achieves superior on- and off-policy exploration efficacy compared baselines, (ii) how GAGE demonstrates strong robustness unknown optimal goals and variations in reward shaping, and (iii) how GAGE generalizes to discretized action spaces.

---

Published in Transactions on Machine Learning Research (03/2026)
Policy Std Episode Returns
40
20
30
20
20
10
15
10
25
1.0
2.0
1.0
0.8
0.6
1.2
0.6
0.6
0.4
0.8
0.4
0.2
0.4
0.2
0.2
0.0
0.5
1.0
Timesteps
leB
Timesteps
Timesteps 1eB
Timesteps
1e8
Timesteps
1eB
PPO
PPO(tuned)
GAGE-0.5
GAGE-0.75
GAGE-1.0
RND
PBT
Schedule(tuned)
Figure Mean episode returns and action standard deviation a over 5 seeds; shaded regions denote one standard deviation. We denote GAGE-005 as using dp 0.5. Top: tasks (Ieft right), Ant Acrobatics, Humanoid Dribbling, Humanoid Cartwheel, Humanoid Tightrope, and Dog (Unitree Go2) Balance Beam. Middle: training curves of episode returns. Bottom: Averaged policy standard deviation ō.
4.1 GAGE for Continuous Action Space
We integrate GAGE into both on-policy and off-policy algorithms by replacing entropy regularization with our adaptive constraint. First, we combine GAGE with PPO, which is widely used to learn challenging robot control tasks due to its benefits from large-scale GPU parallelism [Makoviychuk et al. 2021 To test GAGE under severe local optima issues, we design five highly challenging tasks in IsaacLab (Mittal et al. 2023
Yan al. 2024bb (sce Fig. D. The environment details are provided in Appendix [.1] We then incorporate GAGE into SAC-v2 and evaluate SAC-GAGE, on over 10 HumanoidBench (Sferrazza et al. 2024 tasks.
On-Policy RL with GAGE
We compare PPO-GAGE against: (i) vanilla PPO with default hyperparameters, (ii) PPO with per-task tuned hyperparameters, (iii) Random Network Distillation (RND) (Burda el al. 2019 an intrinsic moti- vated method that encourages exploration of novel states and is commonly used sparse-reward settings; (iv) Population- Population-Based Training (PBT) (Henderson al.) 2018), which trains population of PPO agents with different entropy coefficients in parallel and adapts these coefficients online based episode return performance (details in Appendix C); and (v) hand-designed entropy-coefficient schedules (Appendix C Among the schedule variants we evaluated, cosine decay performed best and is therefore included as our representative scheduling baseline. We include RND to test whether the curiosity-driven method can miti- gate premature convergence dense-reward settings. To obtain strong baselines, we perform grid search over the PPO exploration hyperparameter, the entropy cocfficient, per task and adopt the best-performing values (see Appendix D.2 for details). For RND We follow the hyperparameter settings of the original and subsequent work of Yang al. (2024). PPO-GAGE sets entropy temperature 8 0 and uses the same hy- perparameters vanilla PPO for direct comparison. With action spaces normalized to [-1,1] 1], intuitively, we set 0g E [1.0,0.. 0.75, 0.5) for an ablation study.
Fig. I shows learning curves for episode returns and average o across robot joints. PPO-GAGE with do = 0.5 successfully solved all the tasks, whereas vanilla PPO failed. To ensure a fair comparison with the basclines, in this work, we only report the total episode return including both the goal reward and auxiliary rewards. Hyperparameter optimisation (HPO) over the entropy coeffcient. substantially improves PPO
7

---

Published in Transactions on Machine Learning Research (03/2026)
Eosode Returns
20
40
42
40
18
30
12
0.005
0.02
0.05
12 Action Penalty
12 Action Penalty
L2 Action Penalty
L2 Action Penalty
L2 Action Penalty
(a) Ant Acrobatics
(b) Dribbling
(c) Cartwheel
(d) Tightrope
(e) Balance Beam
Figure 2: Box plots episode returns (excluding action action-penalty values) for tuned PPO versus PPO-GAGE.
The x-axis shows the investigated coefficients of the squared L2 action penalty ||a]e. We aggregate the last 10 episode returns of each seed for 50 data points per box.
Ant Acrobatics, Humanoid Dribbling, Humanoid Cartwheel, and Humanoid Tightrope. However, on the Balance Beam task, HPO yields no noticeable improvement. Among all tasks, PPO-GAGE still wins by clear margins. PPO-GAGE with dg = 0.5 or 0.75 performs reliably across the majority of tasks. In contrast, agents with σo 1.0 exhibit poorer performance, as the excessively large action variance produces highly random behaviors. Hence, for practical use, recommend starting with dg values between 0.5 and 0.75.
The plots reveal that vanilla PPO quickly reduces policies' standard deviation at the start of training, achieving higher rewards by over-exploiting reward components such as energy cost or survival. PPO contin- ues to decrease even after the larget rewards plateau. For instance, the dog robot learns to stabilize on the beam and ceases exploring despite a forward motion target. In contrast, PPO-GAGE sustains exploration and only reduces when the desired behaviors are learned (sce Appondix D.I).
In both the Ant Acrobatics and Humanoid Tightrope environments, we observe that optimally tuned PPO agents exhibit divergent increase in the policy parameter following an initial decay, even task per- formance reaches a plateau. This phenomenon suggests that the agent exploits the entropy regularization term by injecting excessive variance into iask-irreeevant action dimensions, specifically, joints non-critical to maintaining the local optima. These empirical findings underscore a significant objective mismatch: the pressure to maximize entropy drives the policy toward high-variance regimes that are functionally redundant, potentially destabilizing the learning process.
RND succossfully slows down the reduction of relative to PPO. However, the additional exploration does not solve the tasks and can even hurt performance. We hypothesize that RND's novelty-seeking can be counteproduuctive in high-dimensional control where most "novel" states are failed trajectories (e.g., falling down). This result highlights the different focuses between premature convergence and the sparse reward problem. Further experimental results with varying intrinsic reward settings are provided in Fig. 11
Notably, the best fixed entropy coefficient (tuned PPO results) outperforms both PBT-based entropy tuning and the best-performing hand-designed entropy schedule. This suggests that, in our dense-reward control tasks, frequently changing the entropy coefficient during training be detrimental, potentially because it induces a non-stationary regularization and thus a moving optimization target.
Improved Robustness to Reward Shaping
Reward shaping crucial yet delicate, as small changes to reward weights can result in unsuccessful learn- ing. We suppose the sensitivity reward shaping largely arises from premature convergence and conduct experiments to test the robustness of our mothod to variations reward shaping.
The L2 norm of actions is used a penalty term to prevent large actions in robot joints. We vary the action-penalty weight while keeping other weights fixed. The results are illustrated in Fig, 2 To compare across settings, the episode return excludes the action penalty term. PPO with tuned hyperparameters is highly sensitive to the penalty coelficient (e.g., in Dribbling task, the average return at the coefficient of 0.01 is roughly nine times that at 0.04). In contrast, PPO-GAGE outperforms PPO across nearly all settings. It remains robust for action penalty coefficient from 0.005 to 0.04 in every tasks. Both methods struggle when the penalty weight too large (i.e., 0.08 in Dribbling and Balance Beam), suggesting room for future improvement. The full training curves are included in Fig. 13 and Fig. 14
8

---

Published in Transactions on Machine Learning Research (03/2026)
Episode Returns
Ant Acrobatics
Dribbling
Cartwheel
Tightrope
Balance Beam
40-
30
40
20
10
01
6.0
0.5
1.0
1.5
20
Timesteps
Timesteps
Timesteps
Timesteps
Timesteps
0.1x
0.25x
0.5x
1x
1.5x
2x
5x
10x
Figure 3: Sensitivity to the proxy goal estimation: we use & E {0.1, 0.25, 0.5, 1, 1,5, 2,5,100x x the return of the best tuned PPO (black dashed line) as the proxy goal for PPO-GAGE. We set du 0.75 for Balance Beam and Tightrope, and 0.5 for the other three tasks.
Unknown Individual Reward & Optimal Goal
When the agent only receives total rewards and lacks access to individual components, we approximate g(T)) via the Monte Carlo return as Eq. [ Since the optimal return Ju- typically unknown, we estimate it by proxy & obtained from the best performance of reference agent. On our IsaacIab tasks, consider G E {0.1, 0.25, 0.5, 1, 1,,, 2, 10) the best luned PPO ret and train different PPO-GAGE agents. As depicted in Fig. 3 in four tasks, PPO-GAGE fails to learn meaningful policies when given significantly low values of G, i.c. 0.1x, 0.25x. Agents with other goal estimations perform at least as well as tuned PPO, and improves further with a wide range of G values for tasks such as Humanoid Cartwheel, Tightrope, and Dog Balance Beam. When using an overly large target. (e.g., G = 10x), performance degrades as expected: the goal-achievement ratio remains low, so GAGE maintains high exploration lower bound, preventing the policy from converging. This highlights practical limitation of goal-target specification: if G is set far beyond what is allainable under the given reward and training budget, exploration may remain excessive. As rule of thumb, we recommend choosing & within the same order of magnitude as reasonable reference return (e.g,, the best tuned baseline), and increasing it gradually (e.g., 1x 2x) only when training reliably reaches and plateaus near the reference performance.
Off-Policy RL with GAGE
We evaluate SAC-GAGE against SAC-v2 and MaxInfoSAC (Sukhija et al. 2025), the state-of-the-at off- policy algorithms on continuus control. Starting from the SAC-v2 implementation provided by[Sukhija et al. (2055, ; we replace entropy regularization with the GAGE constraint and keep other components unchanged; see Algorithm | for details. Because tanh-squashed Gaussians can cause value overestimation and gradient vanishing near action-space boundaries, we add a policy regularizer analogous Eq. 6 to avoid prohibitively large actor-mean magnitudes. Without access to individual reward components. utilize an estimated total-return target & from the task-specific target return defined in (Serraazaa et al. 2024h to compute goal achievement. The hyperparameters are given in Table 5
We conduct HPO for SAC-v2 and MaxInfOSAC over learning rates and gradient clipping values to obtain strong baselines, as detailed Appendix C The tuned SAC-v2 and Ma achieve similar perfor- mance. We evaluate on the HumanoidBench suite with 12 tasks shown in Fig. 4 Across these tasks, SAC-GAGE achieves comparable final returns on most tasks, while outperforming both baselines on H1- Hurdle-vO, LL-Hiighbar-Simpleeee improves learning speed over the baselines, reaching the same final performance with only 40% of the train- ing steps compared to tuned MaxInfoSAC, which we attribute to its initially higher exploration level.
and H1-Stair-vo. On H1-Walk and H1-Run, SAC-GAGE additionally
Without the adaptive constraint on policy standard deviation applied by GAGE, SAC-v2 and MaxInfoSAC rapidly reduce the policy standard deviation o carly in training and then keep roughly fixed exploration level afterwards. In contrast, SAC-GAGE maintains high variance until returns improve, and then adaptively decreases it in response to goal achievement.
9

---

Published in Transactions on Machine Learning Research (03/2026)
H1 vo
H1 Door-v0
H1 Highhar-Simple vö
H1 Hurdie v0
Episode Returss
1000
0.8
1.0
300
0.9
400
0.9
PS Polcey
600
200
0.8
400
0.7
0.4
0.6
400
0.5
100
200
-0.
0.5
H1-Pole-vO
1e6
H1-Reach-vo
1e6
H1-Run-vO
1e6
112-St--implees
1e6
1000
1000
1.0
Episode Returns
800
0.9
800
0.8
015 Peliey
600
6000
0.8
600
0.7
0.7
400
4000
0.6
0.5
2000
0.6
0.5
0.5
0.4
0.4
4
6
H1-Slide-vO
HI-Stair-V
H1-Stand-v0
H1-Walk-v0
1.0
1.0
1.0
Episode Returns
800
600
800
0.9
Pis Koljod
0.8
400
0.7
0.7
400
0.
0.7
0.6
200
0.5
0.
200
0
0.4
3
Timesteps 1e6
Timesteps SAC-GAGE
1e6
Timesteps
1e5
Timesteps
1e6
MaxInfoSAC
SACV2
Figure 4: Comparison of tuned SAC-v2, MaxInfoSAC and SAC-GAGE HumanoidBench. Mean episode
returns (solid lines) and action a (dashed lines) over 5 seeds; shaded regions denote one standard deviation.
Episade Returs
30
20-
CAGE
GAGE
20-
20-
20
10-
PPO GAGE
10
PPO
10.
PRO
CAGF
GAGF
0.5
Timesteps
Timestops
Timestops
Timestops
Timesteps
(a) Ant Acrobatics
(b) Dribbling
(c) Cartwheel
(d) Tightrope
Balance Beam
Figure 5: PPO vs. PPO-GAGE on discrete action spaces with hyperpameter tuning for both. Mean episode
returns and action over seeds; shaded regions denote one standard deviation.
4.2 GAGE for Discrete Action Space
We further validate GAGE with discrete policies on our IsaacLab tasks by discretizing each action dimension into 11 evenly spaced atomic actions, following Tang & Agrawal (2020). Each dimension is modeled by its own categorical distribution, yielding a factorized policy. We compare PPO-GAGE against the strongest baseline method, tuned PPO, with discrete policies. For PPO-GAGE, we apply action smoothing (see Algorithm 3 in cach action dimonsion. The PPO entropy cocfficient is tuned for each individual task. PPO- GAGE uses the other default hyperparameters from vanilla PPO with its own hyperparameters tuned. The tuned hyperparameters can be found in Table and 3
Discrete PPO with GAGE Fig. presents the learning curves: PPO-GAGE solves all tasks, whereas tuned PPO either has worse performce or even failed learning moving behaviors on Humanoid Cartwheel and Dog Balance Beam. Interestingly, although tuned PPO attains relatively high returns on the Humanoid Tightrope task, its goal achievement value remains low. Fig. 0 further compares the episode returns and goal achievement g for PPO with the two best entropy temperatures / 5.18 x 10-3 and 1.39 10-2 alongside
10

---

Published in Transactions on Machine Learning Research (03/2026)
GAGE (Cartwheel)
GAGE (Balance Beams)
PPO (Balance Beam)
(Cartwheel)
07
It-
-10
Dicrette
110
10
0.1
10-
000
1.0 1.6
2.6
Timestegs
Timesteps
Timestepe
K10"
Timestops
x110
Figure 7: Comparison policies between PPO-GAGE and PPO across two representative joints: left. thigh x (Humanoid Cartwheel, left) and front right call (Dog Balance Beam, right). The heatmaps illustrate the discrete probability distributions over action bins throughout the learning process. The overlaid cyan curves indicate the Goal Achievement metric. Logarithmic scaling is applied to the probabili- ties to highlight the vanishingly small values in collapsed bins.
PPO-GAGE for the Humanoid Tightrope task. For 5.18 10-3, three out of five with differernt sceds learn to survive by remaining stationary and minimizing energy, yielding deceptively high returns. With 3 = 1.39 x 10-2, all agents learn to move but still converge to subopti- mal behaviors, such as unstable motion and excessive energy cost, resulting in lower returns. These obsorvations highlight the vulnerability entropy- based exploration to local optima, a failure mode that GAGE effectively mitigates. On Humanoid Tightrope, the final performance of GAGE with discrete policies surpasses that with continuous ones (Fig. D. primarily due to differences in action penalties. In continuous action spaces, the unbounded nature of the PPO policy allows the agent. to exploit large action magnitudes beyond the physical limits of the robot's 's joints, Although such actions are clipped within the simulation environment, they still incur substantial penal- ties, reducing total reward despite achieving comparable goal achievement.
Goal Achlevement Episode Returns
1P0 F-5.18
30-
10
0.8
0.6
0.4
0.2
Non-vanishing exploration To visualize the evolution of action proba- for the
bility distributions, we selected representative joints: hunanoid in Cartwheel and front right calf for the quadruped in Balance Beam. For each task, we evaluated 11 policy checkpoints saved at equal intervals throughout the learning process. By feeding identical observations to these policies and recording the probabilities for the 11 action bins of the chosen joint, we generated a 11 x 11 probability matrix. These matrices, alongside the corresponding goal achievement values for GAGE and PPO agents, are displayed in Fig. 7 As illustrated, GAGE agents maintain relatively high probabilities across all action bins before securing signifi- cant goal rewards, only concentrating the distributions once performance begins to improve. This sustained variance allows the agents explore a diverse range of behaviors to identify optimal actions. In contrast, despite failing reach high goal achievement, PPO agents suffer from premature distribution collapse the very beginning of the learning process. Consequently, they lose the exploratory capacity to discover optimal behaviours and become trapped in local optima. While entropy regularisation prevents PPO's probabilities from collapsing into a single bin, instead spreading them across a few neighboring actions, it is insufficient to stop the remaining action bins from falling to nogligibly small values.
0.0
0.5
1.0
1.5
Timesteps
1e9
Figure 6: best two PPO agents on Tightrope.
PPO-GAGE vs.
Continuous vs. discrete policies Under the GAGE framework, discrete policies do not always surpass continuous ones and can even be less elficient such as on Humanoid Dribbling, Cartwheel, and Tightrope, contrary to the results of Tang & Agrawal (020). While action discretization with entropy regularization can better mitigate local optima and premature convergence, sacrifices the expressiveness of continuous actions and potentially optimal behaviors. The adaptive exploration strategy in GAGE allows the agent fully leverage the continuous action space, resulting in more efficient learning.
11

---

Published in Transactions on Machine Learning Research (03/2026)
5 Discussion
The results presented in this work demonstrate that Goal Achievement Guided Exploitation (GAGE) pro- vides a principled and effective exploration scheduling strategy for addressing premature convergence in dense-reward reinforcement learning. Across diverse settings, including both discrete and continuous ac- tion spaces, on-policy and off-policy algorithms, and challenging robotic control tasks, GAGE consistently enhances learning performance.
By linking the degree of exploration directly to the agent's goal achievement, GAGE decouples exploration from manually predefined training schedules fixed entropy targets. This allows the agent to adaptively reduce stochasticity as it approaches optimal performance. Empirically, this results in smoother convergence curves and reduced variance across random seeds compared to baselines. Notably, in highly non-convex control tasks such as Dog Balance Beam, GAGE avoids early policy collapse and continues to improve after standard methods saturate. A key strength of GAGE lies its hard constraint formulation. Unlike entropy regularization, which indirectly promotes exploration through soft additive term the optimization objec- tive, GAGE constrains the policy distribution itself through interpretable measures, the policies' standard deviation in continuous spaces and logit range in discrete spaces. This distinction leads to an advantage, ensuring that all actions maintain lower-bounded probability throughout training, thereby preventing the irreversible loss of potentially optimal behaviors.
The experimental results highlight that GAGE is robust to hyperparameter variations and insensitive to reward shaping, two major weaknesses of standard RL algorithms. With the same set of hyperparameters, GAGE achieves superior results compared to finely tuned basclines across different tasks. This property simplifies deployment in real-world systems. Furthermore, the framework performs well even when the optimal goal value is approximated,, suggesting that GAGE can be readily applied in environments with incomplete reward decomposition.
Limitations and future work In this work, GAGE is mainly designed for dense-reward and deter- ministic environments like robot control. As it guides the policy stochasticity, it belongs to undirected exploration (Trun 1992). In sparse-reward tasks with long horizons, its effectiveness could be limited. Despite high policy stochasticity, the agent can still struggle in getting any rewards using only randam exploration. For partially observable environments or competitive multi-agent environments, deterministic policies either cannot break the tie of ambiguity or might be exploited by other agents. The implementation of GAGE in environments demanding stochastic policies is an interesting future topic. Even for the kind of environments we evaluate, GAGE offers aspects for improvement. GAGE defines the same bound for all action dimensions. It can be interesting to investigate the effect of different bounds for different dimensions. Future research can locus on improving the scalability of GAGE and applying it to more complex, dy dynamic, and multi-objective environments. Investigating non-linear relationships between the goal achievement and the exploration metrics could further enhance the method's adaptability to diverse RL problems.
Acknowledgments
We thank Hao Zhu for constructive discussions during the development of GAGE. This work was par- tially funded by the Deutsche Forschngsgemeinschatt (DFG, German Rescarch Foundation.. Project-ID 499552394-SFB1S5 and by grant numbers 428605208 and 539134284 through EFRE (FEIH 2698644) and the state of Waaden-Witttebbbrrg
Co-funded by the European Union
Baden-Württemberg
References
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), ), Proc. of the International Conference on Machine Learning (ICML), volume 97, pp. 151-160. PMLR, 2019.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in AT safety. CoRR, hus/1606.066635 2016.
Kavosh Asadi and Michael L. Littman. An alternative softmax operator for reinforcement learning. In Doina Precup and Yee Whye Tch (ods.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 243- 242.252. PMLR, 06-11 Aug 2017. URL
bttps://proceedingg mlr. wwt..mmlr.rrsssee aiaass/v70/ssadie aaa.aaaaddiaaamm...
Mare G. Bellemare, Sriram Srinivasan, Georg Ostrovaki, Tom Schaul, David Saxton, and Rémi Munos. Unilying count-based exploration and intrinsic motivation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advonces in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 1471 1479, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distil- lation. In Proc. of the International Conference on Leorning Representations (ICLR). OpenReview net, 2019.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jen- nifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing gradient descent into wide valleys. In Proc. of the International Conference Learning Representations (ICLR), 2017.
Maxime Chevalier-Boisverr Bolun Dai, Mark Towers, Rodrigo De Lazcano Perez-Vicente Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and J K Terry. Minigrid & miniworld: Mod- ular & customizable reinforcement learning environments for goal-oriented tasks. In Thirty-seventh Conference on Neural Information Processing Systems Dalasels and Benchmarks Track, 2023. URL https://openrevieet net
http:///peeerrii..e..eeeeroorcca mett/Eorum/iddPFfffsgpn
Karin Dembrower, Peter Lindholm, and Fredrik Strand. A multi-million mammography image dataset and population-based screening cohort for the training and evaluation of deep neural networks the cohort of screen-aged women (CSAW). J. Digil. Imaging, 33(2):408- 413, 2020. doi: 10. 10.1007/110288-00000077
URL http::::/2oa..or//0000000////0/2
Leander Diaz-Bone, Marco Bagatella, Jonas Hübotter, and Andreas Krause. DISCOVER: Automated curric- ula for sparse-reward reinforcement learning. In Proc. of the Conference on Neural Information Processing Systems (NeurIPS), 2025.
Xiaoyi Dong, Jian Cheng, and Xi Sheryl Zhang. Maximum entropy reinforcement learning with diffusion policy. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proc. of the International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pp. 13963 13983. PMLR, 13 19 Jul 2025.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukeuoglu. IMPALA: scalable dis- tributed deep-rl with importance weighted actor-learner architectures. In Jennifer G. Dy and Andreas Krause (eds.), Proc. of the International Conference on Machine Learning (ICML), volume 80 of Proceed- ings of Machine Learning Research, pp. 1406-1415. PMLR, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datascts for deep data- driven reinforcement learning. CoRR, abs/2004.07219, 2020. URL htttp:://ariiii http:://aaxiv..oo/aa//////a 07219
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic; Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause (ods.), Proc. of the International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pp. 1865. PMTR, 2018a.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Ku- mar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/1812.05905,, 2018b.
Conor F. Hayes, Roxana Radulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Math- İeu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Enda Howley, Athirai A. Irisaapaane, Patrick Mannion, Ann Nowé, Gabriel de Oliveira. Ramos, Marcello Restelli, Pe- ter Vamplew, and Diederik M. Roijers. A practical guide to multi-objective reinforcement learning and planning. Auton. Agents Multi Agent Syst., 36(1):26, 2022. doi: 10.1007/110455-02--055533 https://doi. org/ro///// 10. 1007/ 20007//11400 s10458- 022-88-2-22-9522-
URL
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proc. of the AAAI Conference on Artificial Intelligence, pp. 3207-3214. AAAT Press, 2018.
Edward S. Hu, Richard Chang, Oleh Rybkin, and Dinesh Jayaraman. Planning goals for exploration. In Proc. of the International Conference on Learning Representations (ICLR). OpenReview.net. 2023.
Arnav Kumar Jain, Lucas Lehnert, Irina Rish, and Glen Berseth. Maximum state entropy exploration using predecessor and successor representations. In Alice Oh, Tristan Naumann, Amir Globerson, Kale Saenko, Moritz Hardt, and Sergey Levine (ods.), Proc. of the Conference on Newral Information Processing Systems (NeurIPS), 2023.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022. ISSN 1566-2535. doi: https:///00..ogg/1..10/////// 2022. 03.003.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Sil- ver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proc. of the International Conference Learning Representations (ICLR), 2016. URL http: /tappi///axiv...e org/ abs/1509 029711
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance GPU based physics simulation for robot learning. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proc. of the Conference on Neural Information Processing Systems (NeurIPS), 2021.
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 8(6):3740 3747, 2023. doi: 10.1109/LRA.10 200..2222..227000
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning, In Maria- Florina Balcan and Kilian Q. Weinberger (eds.), Proc. of the International Conference on Machine Learn- ing (ICML), volume 48 of JMLR Workshop and Conference Proceedings, pp. 1928-19333 JMT..org, 2016.
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alcch--Buc, Emily B. Fox, and Roman Garnett (eds.), Proc. of the Conference on Neural Information Processing Systems (NewrIPS), pp. 4696 4705, 2019.
Hari Mohan Pandey, Ankit Chaudhary, and Deepti Mehrotra. A comparative review of approaches prevent premature convergence in GA. Appl. Soft Compud., 24:1047 1077, 2014. doi: 10.1016/1AS0C.2211..0
URL ptttps://doi.org///
(016/j.oa.. asoc. 2014. 08.0525
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, 7, volume 70 of Proceedings of Machine Learning Research, pp. 2778 2787. PMLR, 2017. URL http://proceedingg.. mlr. wttppppeee///taaarrrrteeett
14

---

Published in Transactions on Machine Learning Research (03/2026)
Alcksci Petrenko, Arthur Allshire, Gavriel State, Ankur Handa, and Viktor Makoviychuk. Dexpbt: Scal- ing up dexterous manipulation for hand-arm systems with population based training. arv preprint arXiv:2305.12127, 2023.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi- goal reinforcement learning: Challenging robotics environments and request for research. CoRR, abs/1802.0944 2018. URL http://arxiv.. http:/ii...rgg///1/222222 10000000..99444
Juergen Schmidhuber. A possibility for implementing curiosity and boredom in model- bodellbuilding neural con- trollers. In Proc. of the International Conference on Simulation of Adaptive Behavior: From Animals to Animats, pp. 222-277, 1991.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. CoRR, abs/1707.066 00347, 2017.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Picter Abbeel. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. In Dana Kulic, Gentiane Venture, Kostas E. Bekris, and Enrique Coronado (eds.), Proc. of Robotics: Science and Systems (RSS), 2024.
Bhavya Sukhija, Stelian Coros, Andreas Krause, Pieter Abbeel, and Carmelo Sferrazza. Maxinforl: Boosting exploration in reinforcement learning through information gain maximization. In Proc. of the International Conference on Learning Representations (ICLR). OpenReview. net. 2025.
Christian Szegedy, Vincent Vanhoucke, Sergey lofle, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2828-2826. IEEE Computer Society, 2016. doi: 10.1109/CVPR.20116 URL https://doi. wrgooogg//110 1109/CVPR. 2016. 308
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Picter Abboel. #exploration: A study of count-basod exploration for deep reinforcement learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Prec. of the Conference Neural Information Processing Systems (NeurIPS), pp. 2753 2762, 2017.
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In Proc. of the AAAI Conference Artificial Intelligence, pp. 5981-5988. AAAI Press, 2020.
Sebastian B. Thrun. Efficient exploration in reinforcement learning. Technical report, Computer Science Department, Carnegie Mellon University, Pittsburgh, PA, 1992.
Shanchuan Wan, Yujin Tang, Yinglao Tian, and Tomoyuki Kaneko. DEIR: efficient and robust, exploration through setiiiinative-modee based episodic intrinsic rewards. In Proceedings of the Thirty-Second Inter- national Joint Conference Artificial Intelligence, IJCAI, Macao, China, pp. 4289-4298 ijeai.org, 2023. doi: 10.24963 11B/:3///JCAA.223//4232 URL https: /tt////o.orr/00/ htt..o/g//...22///..........// 2023/477
Kaixin Wang, Kuangqi Zhou, Bingyi Kang, Jiashi Feng, and Shuicheng Yan. Revisiting intrinsic reward for exploration in procodurally generated environments. In Proc. of the International Conference on Learning Represenlations (ICLR), 2023.
Yufei Wang and Tianwei Ni. Meta-sac: Auto-tune the entropy temperature of soft actor-critie via. metagra- dient. CoRR, bbs/2207.01932, 2020.
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algo- rithms. Connection Science, 3(3):241-285 -288, 1991.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction- guided multi-objective reinforcement learning for continuous robot control. In Proc. of the International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 10607-10616. PMLR, 2020. URL http::/prooceedingg mlr www.....r.......... www.ppersss///eeeee htwp1.///xx...... html
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi and Roman Garnett (eds.); 2018.
Proc. of the Conference on Neural Information Processing Systems (NeurIPS), pp.
Shengchao Yan, Baohe Zhang, Joschka Boedecker, and Wolfram Burgard. Goal achievement guided ploration: Mitigating premature convergence in learning robot control. In CoRL 2024 Workshop Whole-body Control and Bimanual Manipulation: Applications in Humanoids and Beyond, 2024a.
Shengehao Yan, Baohe Zhang, Yuan Zhang, Joschka Boodecker, and Wolfram Burgard. Learning continuous control with geometric regularity from robot intrinsic symmetry. In Proc. of the IEEE International Conference Robotics and Automation (ICRA), pp. 49-55. IEEE, 2024b.
Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li. Exploration and anti-exploration with distributional random net- work distillation. In Forty-first International Conference on Machine Learning, ICML, 2024, Vienna, Aus- tria, July 21-27, 2024. OpenReview.o .ne, 2024. URL https://pppaneeeeeeeeee//rtrtttd
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E. Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. In Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jemnifer Wortman Vaughan (eds.), Proc. of the Conference on Neurul Information Processing Systems (NeurIPS), pp. 25217 25230, 2021.
16

---

Published in Transactions on Machine Learning Research (03/2026)
A More Background and Related Work
A.1 Premature Convergence
Premature convergence is ubiquitous in reinforcement learning due to the non-convexity of system dynamics, reward shaping, multi-objective, and function approximation error using neural networks. A policy may converge to suboptimal solutions before reaching the desired performance. We briefly clarify the differences between premature convergence and two other related problems in RL, i.c., reward hacking and the sparse reward problem.
In reward hacking, the objective function that the designer writes down admits of some "clever" solutions that formally maximize it but pervert the spirit of the designer's intent (Amodci et al. 2016). For example, cleaning robot can close its sensors to maximize the reward for observing lewer messes. This differs from premature convergence, as agents stuck at suboptimal solutions typically do not maximize the objective function.
Sparse rewards have been regarded as one of the most challenging problems in reinforcement learning (Ladosz) jtt al. 2022). With limited rewards scattered in the vast search space, the result is a primarily flat objective landscape. In such settings, the agent can hardly find the rewards and maximize the objective, no matter whether the policy is converged not. This differs from premature convergence, as agents increase the objective by converging to suboptima. Of course, the challenge is exacerbated by the combination of these two issues, such as the noisy-I'V problem (Burda et al. 2019). However, in this work, we focus on tasks with dense sel Lings and numerous local optima. Note that sparse reward lasks, such as navigation and long-term robot control, can be transferred to dense reward tasks based on domain knowledge. However, auxiliary rewards often introduce new local optima and exacerbate the problem of premature convergence.
A.2 Label Smoothing
0.8
probability
0.6
0.509
0.4
.0.35
0.3
0.27
0.3
0.2
0.19
0
ag
a1
02
d3
40
41
42
43
00
01
02
03
(a) Original distribution
(b) Label smoothing
(c) Action smoothing
Figure 8: categorical distribution and its flattened results using different techniques. The resulting dis- tributions have the same entropy 1.31. (a) Original distribution. (b) Label smoothing has deterministic result calculated with smoothing parameter 0.58. (c) Action smoothing also has deterministic result using temperature 5.34.
Label smoothing (Szegedy al. 20166 is a popular technique for flattening discrete distributions. It is widely used in classification problems to reduce overconfidence through soft learning targets (Mäller al. 2019
It typically mixes the original distribution with uniform distribution: p'(ak s) (1- c)p(aa s) + K
where the smoothing parameter 0 < c. 1. However, it introduces a prior of uniform distribution that is inappropriate for policy learning. As shown in Fig. g actions with the lowest. learned probabilities often lead to obvious penalties or termination states, which the agent should avoid. These actions experience the most significant incrcase through label smoothing, which can potentially lead to failed trials. In con- trast, with action smoothing, the most significant probability increases occur for actions with middle-valued probabilities.
17

---

Published in Transactions on Machine Learning Research (03/2026)
B Algorithm Implementation
We provide the pseudo code for SAC/PPO+GAGE, with Gaussian policy in Alg. 12 and action smoothing of a categorical policy in Alg. 3 to highlight the diffcrences.
We strike through the entropy related components in the original algorithms
Algorithm Soft Actor-Critic (SAC-v2) with Gaussian Policy + entropy regulerization GAGE
Require: enviromment &, discount larget smoothing т, replay size N, batch size B, srnget-entropyy H (eg,-dim(AA), std lower bound JLB, mean action factor Omean: goal achievement update factor À 1: Initialize actor TT(aas) = Mup(s), Es(s)) (Tanh-squashed Gaussian), critics Qo Q2,, target critics
Q0 + 2 Qog + Q0: temperature a- 0, goal achievement, 9=0
2: Initialize replay buffer D +
3: for episode- = 2,,,... do
4: Reset env, get &o
for 0,1,2.... do
σ पक(अग) max oo(ss))
Sample action via reparameterizatiion: e~ N(O,T); u= mo(ss) + +00c; a tanh(u)
8: Execute in &, observe 8t),,d;; store (8t, Te,
rt, 8141 in D
for gradient step = 1,2.... do
10:
Sample minibatch {(8,, as ri, 8 di)) ~D
11:
σ = bø(5)) max(OLB. (s())
12:
Target action and log-prob: e' N(O, = mp(s)) + 1 df -tanh(w);; compute
log T (a(l(()) with tanh correction
Target Q: Q = min (ooz (5% af), Q0, (s{,
13:
Bellman target: y/ ri + 711 d:) (Q: (lggaa(a())t
14:
Critic loss: Lo(0,) B E, (Qo, (s), v1) E {1.2]
15:
16:
a - oa(o,) max (+;))
17:
Update 0,, Og by minimizing Cq
18:
Actor sampling: e~N10, T); (8:) t a; tanh(u); compute log (o(oss)))
19:
Actor loss (reparameterized) augmented by mean action regularization:
1 Cx = B Σ
log (p.|)))))
min Qo, (8i, ap(s:)) +amean Σ Ml (s:)))
k=
20:
Update & by minimizing &x Temperature loss (eptional tuning):
21:
1
log 11 (p[]).)) H)
B
22:
Update-a a by minimizing La
23:
Target update: 0/ +- T 0g + (1 (1-T)0' j (1,2)
24:
end for
25:
if d, then
26:
Update goal achievement: g >g (1 A)ge: where Jo is the episode goal achievement Update the øLB based on the agent's performance: øLB GLB(1 g)
27:
28:
break end if
29:
30:
St +- 8t+1
31:
end for
32: end for
18

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm Proximal Policy Optimization (PPO) Algorithm with Gaussian Policy + entropy GAGE
Initialize policy mean parameters Bu policy standard deviation do, value function parameters do, goal achievement go 0, goal achievement update factor
for iteration k = 1, 2, ... do
Collect set of trajectories {(5,, Te, 8t,1)) by running policy To, (aelse) N(uu, (86). in the environment
4:
for each time step t do Compute advantage estimates A based on value function Vol (st)
5:
end for
Update the policy by maximizing the PPO-CLIP objective with an added entropy term:
min (() Å., clip (N(m (f) ((un, (+:);
N(uo(se)) 02) 1 +e A.
0k+1, 0k+1 arg max 0,0
where sr, (st) is the mean of the Gaussian action distribution, on is the standard deviation (separately learned), and H (TT(oe]s:) is the entropy of the policy, encouraging exploration. The term 3 controls the weight of the entropy regularization.
8:
Update the value function by minimizing the following loss:
Φ arg min E, (Vo(st) - Ra)²
9: Update the goal achievement: Mh+1 = Aga + +1 (1- X)R
10: Lower bound the standard deviation parameter a based on the agent's performance:
max(Ox+1. -00M - 00)
11: end for
Algorithm 3 Action Smoothing Algorithm
Require:
Network outputs (21, 22, =K), goal achievement g(T), hyperparameters 82,0, 82,1 Ensure: Action probabilities p(ax
1: Calculate the original logit difference:
8 = max min(=;).
Compute the upper-bound for the logit difference:
82,BB (82.1 62,0)g(TT) 82.0-
Calculate the temperature:
max(1,
T(g,2)
4: Compute action probabilities using softmax with temperature:
exp(#t/7(0:=)) ΣK exp( <//((.)))
p(ax (s)
for k: 1,2,..., K.
19

---

Published in Transactions on Machine Learning Research (03/2026)
C
Additional Results
In this section, provide more detailed experiment results with continuous action spaces.
Ant Acrobatics
Dribbling
Cartwheel
Tightrope
Balance Beam
Episode Returns
30
40
30
20
20
20
10
20
10
1.0
2.0
1.0
1.2
1.0
Policy Std
0.6
122
0.6
0.0
0.0
0.2
0.4
0.2
0.2
Entropy Coef
10-3
10-9
10:3
10:1
10-3
10-7
10
1011
10
10-7
10-7
10-7
10-
0.7
0.5
1:0
Timesteps
Timesteps
Timesteps
Timesteps
Timesteps
Linear
Cosine
CosWR-1.0
CosWR-0.8
PPO(tuned)
GAGE-0.5
GAGE-0.75
Figure 9: Additional training results of experiments with continuous action space with differerent entropy coefficient schedules. We plot the mean over 5 seeds. The faint area represents one standard deviation
Ant Acrobatics
Dribbling
Cartwheel
Tightrope
Balance Beam
Episode Returns
40
30
30
20
30
20
10
10
1.0
1.0
1.0
2.2
3.0
Policy Std
0.8
0.6
0.8
0.9
0.8
0.6
0.6
0.2
0.2
0.2
0.3
0.0
0.5
10
1.5
Timesteps
Timesteps
Timesteps
Timesteps
Timesteps
Rxed-0.2
Rxed
Unear
Cosine
CosWR
CesWi--0.8
PPO[tuned]
GAGE
GAGE:0.75
Figure 10: Additional training results of experiments with continuous action space with differerent schedules. We plot the mean over 5 seeds. The faint area represents one standard deviation
Std Scheduling vs. Entropy Coefficient Scheduling In contimuous-action PPO, we implement GAGE by directly regulating the Gaussian policy standard deviation a, rather than by applying schedule to the entropy coefficient To justify this choice, perform controlled ablation that applies the same family of common hand-designed time schedules (lincar decay, cosine decay, and cosine anncaling with warm restarts) lo either entropy coefficient or policy sid a, while keeping all other PPO settings fixed. We consider Linear and Cosine decay, which decrease a parameter from a specified start value to an end value over the full training budget, as well as cosine anncaling with warm restarts, denoted CosWR-1.0 and Co.WR-0.8. The warm-restart variants partition training into four equal cycles; each cycle, the value is reset. to the start and annealed to the end, with an optional restart-amplitude decay of 0.8 across cycles
20

---

Published in Transactions on Machine Learning Research (03/2026)
(no decay for CosWR-1.0). Concretely, all entropy-coefficient schedules start at a 0.01 and decay in log scale to a = 10-8 (effectively zero), while all std schedules start at = 0.8 and decay in linear scale to a 00; for clarity, our PPO (tuned) baseline uses a fixed entropy coefficient, selected as the best-performing constant a. The results reveal a clear asymmetry: in our setting, entropy-cocffcient schoduling does not help, every manually designed entropy coefficient schedule underperforms the best fixed (luned) entropy coefficient. In contrast, std scheduling provides a meaningful and reliable exploration-contral interface: several hand-designed schedules are competitive, and cosine decay in particular often matches GAGE on individual tasks. Nevertheless, GAGE achieves the best overall performance across the suite, providing single adaptive schedule that is consistently strong without task-specific schedule selection. Moreover, the std trajectories discovered by GAGE are qualitatively similar to the top-performing manual std schedules; for example, on the Carlwheel lask, GAGE learns a decay profile that lies wetween the tuned fixed-a baseline and the linear canear/ossine std baselines, yielding better performance. Overall, these findings support our design decision to control exploration at the distribution level via o, rather than indirectly through objective-level weighting via the entropy coelficient.
Details of Population-based Training To provide a performance-based automated alternative to hand- designed entropy schedules, we additionall include a Population- Basodd Training (PBT) (Dembrower et al, 2020) bascline that tunes the PPO entropy cocfficient online using population performance. In cach PBT run, we maintain a population of 8 policies trained concurrently and periodically apply selection and mutation, using each member's ' episode return as the optimization objective. To match the overall training budget of our standard runs, the population shares a fixed total amount of environment interaction: environments in parallel and allocate 1024/8 environments to each policy, so that the population collectively consumes the same number of enviromment steps as a single-policy run. PBT updates are performed al a fixed interval equal to 1/100 of the total training horizon (the same fraction for all tasks). At each update, underperforming members are replaced by setter-performing ones (exploit), and their entropy coefficients are perturbed with probability 0.25 (explore) by multiplying them with a random factor sampled uniformly from [1.1, 2.0]. Mutations are applied only when the population has sufficiently diverged in performance, controlled by dispersion thresholds with 0ata- = 0.1 and Oabe = 0.025. The initial entropy coefficients are sampled in log space from 10-6 to 10-3. Reported PBT results are averaged over 5 independent PBT runs with different random seeds. We follow the implementation of Petrenko et al. 2023 and refer readers there for additional details.
run 1024
Intrinsic Reward Weight To evaluate the effect of intrinsic rewards in the proposed challenging control (asks, we trained several RND agents using different weight. combinations for extrinsic and intrinsic rewards: (2.0, 1.0), (2.0, 0.5). (1.0, 1.0), (1.0, 2.0), and (1.0, 4.0). The weight values (2.0, 1.0) are consistent with those used in the original RND work (Burda et al.) 2019] and subsequent research (Yang et al., 2024). Therefore, we also used this ratio for the experiments presented in Fig. D As shown in Fig. I1 none of the R.D agents succeeded in solving the task. Agents with larger ratios of extrinsiccto--inrinsii weights exhibited learning patterns similar to standard PPO, which does not use intrinsic rewards. As the ratio decreased, the agents focused more exploring novel states, indicated by larger standard deviations during training. However, this increased exploration did not contribute to solving the task. Instead, the novelty-based exploration resulted in decreased extrinsic rewards. This phenomenon highlights the distinct focus of our work compared to novelty-based exploration methods. Our work focuses on addressing premature convergence, an issue that is equally important but has been largely overlooked until now. In contrast, curiosity-based mothods primarily tackle sparse rewards. The difference in focus is also reflected in the existing benchmarks for exploration algorithms. Most environments are designed with sparse rewards and moderate local optima, which can be effectively addressed using novelty-based based exploration. For example, environments like Fetch (Plappert et al. 2018), MiniGrid (Chevalier-oisveettt et al., 2023), , AntMaze, and Adroit manipulation tasks (Fu et al. 2020) are 'safc," with sparse termination states or penaltics distributed across the state space. Agents can easily avoid termination and penalty states while exploring for rewards. In such environments, exploring unseen states is a highly effective strategy. However, novelty- based methods struggle in scenarios with more severe local optima. For instance, Noisy-TV has been recognized as a major issue for novelty-based methods, even though it only involves local optima introduced by environment stochasticity. The challenges posed by more severe local optima have not yet been fully explored. In this
21

---

Published in Transactions on Machine Learning Research (03/2026)
Episode Returns
30
Intrinsic Returns
100
Policy Std
1.0
20-
50
10-
1:1
2:0.5
0.5
Timesteps
1e8
Timesteps
Timesteps
(a) Extrinsic reward
(b) Intrinsic reward
(c) Standard deviation
Figure 11: Investigating the effect of novelty-based intrinsic reward to the learning of Dog Balance Beam
task. The curves with legend 1:2 represent the agent trained using extrinsic and intrinsic coefficients of (1.0, 2.0).
Episode Reburns
40-
Episode Returs
30
Episode Returns
1.5
Episodi Returns
1.5
Epicode Returns
88 8
30
10
20
20-
10:
1.0
1c8
Timesteps
Timesteps
1os
Timesteps
leb
Timestops 105
(a) Ant Acrobatics
(b) Dribbling
(c) Cartwheel
(d) Tightrope
(e) Balance Beam
Figure 12: Different Fixed Entropy Coefficient for Continuous Control Tasks using PPO
work, we aim push the boundaries RL exploration rescarch into environments with more challenging local optima issues. The proposed IsaacLab tasks reflect real-world robot control scenarios where optimal behaviors occupy only small portion of the state space, while most of the state space leads to penalties such as falling down wasting energy. This dominant penalizing space creates challenging local optima. In such environments, novelty-based exploration often results in sampling mostly failed trajectories and becoming trapped in local optima.
Episode Returs
Episode Returs
Episode Returns rstumms
Episode Returne 15 28
30.
20
20
20
20:
10
10.
0.3
0.5
1:0
Timesteps
Timesteps
Timesteps
Timestops
Timesteps
(a) Ant Acrobatics
(b) Dribbling
(c) Cartwheel
(d) Tightrope
(e) Balance Beam
Figure 13: Training curves different L2 action penalty coefficients for PPO in continuous control tasks
D Experimental Details
D.1 Tasks Setup
We design five challenging continuous control tasks in IsaacLab. Three robots with many degrees of freedom learn challenging locomotion or dynamic manipulation behaviors. The robots include a humanoid robot with 21 joints, dog robot. (Unitree Go2) with 12 joints, and ant robot with 8 joints. In Table 1 we provide the reward composition of differrnt tasks.
Humanoid Tightrope (HT) The humanoid robot learns side walking for 2m/s on a tightrope, i.c., cylindrical bar with diameter of only 0.1m. This is more challenging than walking forward because balancing with two arms stretched to both sides would be more difficult.
22

---

Published in Transactions on Machine Learning Research (03/2026)
Episode Returns
40-
Episode Returns
Episode Returns
40
Epioade Return
% 8 60
30
20
20-
20
Epicode
20
10-
0.5
0.5
1.6
1.5
2 Timestops
Timesteps
Timestops
Timesteps
Timestops
(a) Ant Acrobatics
(b) Dribbling
(c) Cartwheel
(d) Tightrope
(e) Balance Beam
Figure 14: Training curves of different L2 action penalty coefficients for GAGE in Continuous Control Tasks
Table 1: Reward weights of continuous control tasks. The rewards and penalties from left to right are for de- sired locomotion velocity, environment not terminating, robot orientation, robot distance to the manipulated object, large action commands, energy consumption, joint position close limitations, robot velocity perpendicular to the desired direction, object velocity perpendicular to the desired direction, joint torque, joint acceleration, and action changing rate. The selected goal reward for goal achievement calculation is marked green background.
reward
penalty
U2
alive
orient
dobj
||a|
E
Oimit
F/y
yy.obj
T
0
á
HT
0.5
1.0
1.0
0
0.01
0.05
0.25
1.0
0
0
0
HD
0.3
0.4
1.0
0.2
0.01
0.01
0.25
0
0.5
0
0
HC
2.0
1.0
0.5
0
0.01
0.05
0.25
0
0
0
0
DB
1.0
1.0
1.0
0
0.005
0
1.0
0
1e-6
2.5e-
0.001
AA
1.0
1.0
1.0
0
0.005
0.05
0.1
0
1.0
0
0
0
Humanoid Dribbling (HD)
The humanoid robot learns dribble a football a high speed (3.5m/s).
Additionally, the robot gets random commands for turning the target direction for up grad.
Humanoid Cartwheel (HC) The humanoid robot learns perform cartwheel at speed of 6rad/s. It requires precisely coordinated movement of the whole body, which is more difficult than normal locomotion tasks, which primarily rely on the lower body of the robot.
Dog Balance Beam (DB) The dog robot learns to walk on balance beam at speed of 2m/s. The beam has square cross-section with 0.1m side length. Moreover, the balance beam is tilted for I Frad so that the robot has to climb a slope while balancing.
Ant Acrobatics (AA) The ant robot with four legs learns to balance a pole vertically on its torso while standing on a ball. The pole has a length of 2m. The ball has a diamcter of the same value. Morcover, the robot has to learn to roll the ball forward al a larget speed of Im/s.
D.2 Hyperparameters Optimization and Analysis
GAGE is principled framework which can be integrated into different algorithms. Due to the differences among different algorithms and action spaces, GAGE introduces different sets of hyperparameters to ensure the lower bound of the distribution flatness. In this part, we show the details of hyerparameter optimization in each case and perform an analysis of GAGE's hyerparameters as reference for practitioner.
Default Hyperparameter for PPO We follow the implementation in RSL-RI PPO and reuse their default hyperparameters in the vanilla PPO for continuous control tasks. GAGE shares hyperparameters with vanilla PPO except for its own hyperparameters. We include the default parameters in Table
23

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2:
Tuned hyperparameters for PPO on different tasks
Action Type
AA
HD
HC
HT
DB
entropy coef discrcte
continuous
1.39e-2
7.2e-4
1.93e-3
5.180-3
le-4
5.180-3
5.180-3
1.93c-3
5.180-3
5.180-3
Hyperparameter Optimization for PPO GAGE is addressing the premature convergence by adap- tively changing the lower bound of the policies' standard deviation. To obtain strong baseline to compare in our tasks, we perform a grid search for the entropy coefficient of each continuous or discrete task. We evaluate 8 different entropy coefficients uniformly in log-scale from 1 x 10-4 to 0.1 with 5 seeds. The top-performing hyperparameter setting will then be used for the tuned PPO baseline. We select the best- performing hyperparameter by averaging the last episode relurn over 5 runs with different seeds. Detailed training curves can be found in Fig. 12 The resulting entropy coeffcients are summarized in Table D
Hyperparameter Optimization and Analysis for continuous PPO-GAGE We use PPO as the backbone for all IsaacLab experiments. Our implementation is based on rsl_rl v2.0.0, with the only al- gorithmic modification being the action-smoolhing procedure described in Algorithm 3 For PPO-GAGE, we disable the entropy term by setting the temperature zero; aside from this change, we keep all hy- perparameters identical across methods to ensure fair comparison (see Table -). As shown in Fig. D in continuous-action settings a single default choice of 0.5 performs well across most tasks and is generally sulficient. The only exception is the Balance Beam task, where performance. Based on these results, we recommend σn = 0.5 as practical default for new downstream tasks, with og = 0.75 as a reasonable alternative when stronger late-stage exploration beneficial.
0.75 yields slightly better asymptotic
Hyperparameter for discrete PPO-GAGE We discretize each action dimension into 11 bins, while keeping all other hyperparameters identical to the continuous PPO implementation. Discrete PPO-GAGE introduces three method-specific hyperparameters, (S=,0, 6z,1, Mogat) For each environment, we perform a grid search over 8=,0 E (-3, -4,-5), -5], 82,1 E {-100- -5,,- - 20 and Onggit E {0.001, 0.01, 0.13. For clarity, the ablation in Fig. 15 fixes 5,,1 = 200, as we found it to less influential than the other two parameters. Overall, both 8,00 and Ogogit noticeably affect performance. The best-performing hyperparameter settings are reported in Table 3 On simpler tasks (e.g., Ant Acrobatics), performance relatively insensitive across the grid. In contrast, on tasks that require stronger exploration (e-g., Balance Beam and Cartwheel), larger Ologit tends to yield better performance. As practical guideline, recommend starting with a smaller Mogit and increasing it if learning platcaus carly or converges to suboptimal behavior. Regarding the initial exploration bound 8z less negative value (e.g., -3) generally provides sufficient stochasticity to navigate the reward landscape and discover optimal behavioral modes in stable environments. However, the optimal value is linked to the stability of the task dynamics. In highly unstable scenarios, such as the Dog Balance Beam task, practitioners should prioritize more negative 82,0 (e.g., -5). This more constrained initial bound prevents the agent from frequently sampling 'dangerous' actions that lead to immediate failure, thereby ensuring the agent survives long enough before transitioning to high-velocity gaits.
Table 3: Tuned hyperparameters for PPO-GAGE on different tasks with discrete actions
AA
HD
HC
HT-F
DB
80,0
-4
-4
-3
-4
-5
82,1
-20
-20
-20
-15
-15
Mogit
0.1
0.01
0.1
0.01
0.01
Hyperparameter for SAC-v2, SAC-GAGE, and MaxinfoSAC For the HumanoidBench experi- ments, we build on the public implementation used by Sukhija et al. (2025
To ensure fair comparison,
htttp:://tthuu..o///ubiitt
24

---

Published in Transactions on Machine Learning Research (03/2026)
Ant Acrobatics
Dribbling
Cartwheel
Episode Returns
40
40
20
20
10
20
0
0
0.0
0.5
1.0
1.5
2
2
Tightrope
Balance Beam
Episode Returns
40
40
20
20
0
0
6
8
1
Timesteps
:40
Timesteps
141
6,,0 Apgt 0.001
6, 4, Opoit 0.001
6,.0 = 5, awg: 0.001
6,,0 -3, Abor 0.01
Ōz,5 a 4, aboat 0.01
62 o -, aeoa: 0.01
6,,0 -3, Augn = 0.10
6,,0 = -4, Blogh = 0.10
6,,0 = 5, algg = 0.10
Figure 15: Hyperparameter analysis of discrete PPO-GAGE. We plot the mean over 5 seeds. The faint area represents standard deviation. We fixed the 8z,1 20 as the differences due to this value negligible from our empirical experience.
Episode Returns
500
500
250
0
Sta
0.55
0.55
(tuned) MaxniosAL (uunei)
MaxtriesCC
MastriesAC (unnd)
MastricSAC
MaxinicbAC (tuned)
MaxnlosAL
0.50
0.445
0.45
0.40
Timesteps
166
Timesteps
1c6
Timesteps
1c6
(a) H1-Stand-v0
(b) H1-Walk-v0
(c) H1-Run-vO
Figure 16: Comparison of the original SAC-v2, MaxInfoSAC and tuned versions on a selection of Humanoid-
Bench. Mean episode returns and action a over 5 seeds; shaded regions denote one standard deviation.
we follow the hyperparameters reported by the original authors for the same tasks (H1-Stand-vO, Ial- v0, and H1-Run-vO; see Table 5p. For SAC-GAGE, we disable the entropy term by setting the temperature to zero. In addition, we tuned two generally applicable optimization hyperparameters, the critic learning rate and gradient-norm clipping, and found this substantially improved performance, reducing the critic learning rate from 5 x 10-4 to x 10-4 and enabling gradient-norm clipping with threshold 1.0 (previously disabled). The effect of this tuning is shown in Fig. 16 tuned SAC and MaxInfoSAC improve performance by roughly 2x H1-Stand-vD and Halk-vo and by about 5x on H1-Run-vO. SAC-GAGE introduces three method-specific hyperparameters (Øo. Ouuenn G). For all SAC-GAGE results reported in Fig. 4 we use a single hyperparameter selting across the task set to avoid per-task luning: on = 1.0 and Omoan 0.01, selected as the best overall setting on the same tasks considered by MaxInfoRL, For the praxy goal estima- tion G, we utilize the task-specific target returns provided by Sferrazza et al. (2024); for simplicity, we set & 1.5x the provided target return for all HumanoidBench tasks. To analyze hyperparameter sensitivity, we additionally conduct grid search over do {0.75, 1.0, 1.25) and Omean {0.001, 0.01, 0.1) across all tasks.
25

---

Published in Transactions on Machine Learning Research (03/2026)
As shown in Fig. 17 SAC-GAGE is relatively insensitive within this range, with performance degrading
mainly when meaan = 0.1. Based on these results, use do = 1.0 and Omean = 0.01 as default. values for all tasks, providing a practical starting point and indicating that SAC-GAGE can perform well out of the box.
H1-Crawl-vd
H1-Door-v0
www.-iihhar--iiillene
H1-Hurdie-v0
1000
Episode Returns
300
500
000
200
100
0.4
0.4
0.8
0.0
H2 Pale-vo
1e7
H1- v0
1e7
H1 Run-vO
1e7
H1 Sit Simple-vo v0
1e7
1000
1000
Episode Returns
8000
800
800
800
600
4000
400
200
2000
200
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
H1-Siide-v0
1e7
H1-Stair-vD
1e7
H1-Stand-v0
1e6
1e7
800
Episode Returns
600
800
800
600
800
600
400
200
1.0
0.8
Timesteps
1e7
Timesteps
1e7
Timesteps
1e6
Timesteps
1e6
0.75, 0.001
1.0,
0.001
1,25,
0.001
0g 0.75, Gesee 0.01
1.0, Graan 0.01
Gg m 1.25, Arewn 0.01
0o 0.75, Greesn
Grean
Co Greenn
Figure 17: Hyperparamcter analysis of SAC-GAGE. We plot the mean episode return over 5 seeds. The faint area represents one slandard deviation.
26

---

Published in Transactions on Machine Learning Research (03/2026)
Table 4: Common hyperparameters for PPO(-GAGE)
Hyperparameter
Value
Algorithm
Value loss coefficient
1.0
Clip parameter (e)
0.2
Use clipped value loss
True
Desired KL divergence Discount. factor (7)
0.01
0.99
GAE parameter (A)
0.95
Max gradient norm
1.0
Learning rate
0.0005
Number learning epochs
Number mini-batches
4
Learning rate schedule
Adaptive
Policy
Activation function
ELU
Actor hidden dimensions
[400, 200, 400] [400, 200, 100]
Critic hidden dimensions
Initial noise standard deviation
1.0
Runner
Number sleps per environment
Number of environments
1024
Empirical normalization
Falsc
RND
Intrinsic Reward coefficient
Extrinsic Reward coefficient
2
Intrinsic Reward Normalization
yes
Table Hyperparameters for SAC & MaxInfoSAC & SAC-GAGE
Hyperparameter
SAC SAC/tuned /unted
MaxInfoSAC/tumed
SAC-GAGE
hidden dimensions discount (7)
[512, 512]
[512, 512]
512, 512]
0.99
0.99
0.99
tau
0.005
0.005
0.005
target update period
1
target entropy
dim(
- di)
backup entropy
true
true
actor
0.0005
0.0005
0.0005
critic
2.0005/0.0000
0.0005/0.0001
0.0001
temperature
0.0005
0.0005
initial temperature ensemble Ir
1.0
0.0005
information gain temperature Ir
0.0005
gradient norm clip initial std lower bound do mean action factor Omean
None/15 1.0
None/1.0 /1.0
1.0
1.0
0.01
proxy goal estimation G
1.5x ..5xhenccmark value
Table 6: HumanoidBench task-specific target return
Sferrazza et al.
2024
Crawl
Door
Highbar
Hurdle
Pole
Reach
Run
Sit
Slide
Stair
Stand
Walk
700
600
750
700
700
12000
700
750
700
700
800
700
27


Reviews:
[{"review_id": "fhT6Xyq6D8", "reviewer": "Reviewer_7HFz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an adaptive exploration method in reinforcement learning (RL). The authors propose Goal Achievement Guided Exploitation (GAGE), a framework that regulates exploration based on the agent’s goal achievement. Starting from the classical maximum entropy (MaxEnt) principle, the paper highlights its implicit and ambiguous nature regarding exploration. This leads the authors to define a new objective for goal achievement, $g(\\pi) = J_{\\mathrm{g},\\pi} / J_{\\mathrm{g},\\pi_\\ast}$ based on auxiliary reward function $r_\\mathrm{g}$. Using a variant of measures defined via $g(\\pi)$, the GAGE method successfully guides agent policies to accomplish discrete and continuous control tasks, including those with very long behavioral sequences. The paper provides extensive empirical studies demonstrating the effectiveness of GAGE.", "claims_and_evidence_yn": "No", "claims_explanation": "While I do think this can be a promising direction regarding reward engineering in goal-oriented RL with good performance, I also find its claims regarding MaxEnt RL to be puzzling or ambiguous.\n* I do not particularly agree with its main claim, the abstract notion that the maximum entropy does not solve (or cause) premature convergence. Theoretically, maximum entropy aids RL optimization by encouraging exploration while guaranteeing the uniqueness of the solution. Empirically, RL algorithms often get stuck in local minima due to realistic constraints, such as the vastness of the solution space and the instability of estimating moving targets.  In other worlds, goal-oriented approaches like GAGE likely succeed in practice not because they rectify a fundamental limitation of MaxEnt, but because they inject useful, stable prior knowledge (e.g., enforcing a humanoid robot to proceed with specific velocities). I believe the authors mistakenly frame MaxEnt RL as having a fundamental, universally agreed-upon flaw in terms of exploration without providing convincing theoretical consideration for such a limitation.\n* I do not understand the implication of example Fig 1, which is used to criticize the entropy measure. It may be true that two distributions have the same entropy; entropy is an indirect signal for exploration, whereas exploitation of returns takes precedence. The combination of reward and entropy typically balances this to ensure sufficient exploration. I totally agree $r_g$ is far more direct and stable, but this is not because it acts as a universally superior approach to exploration. Rather, it simplifies the original problem by providing guidelines for exploitation drawn from prior knowledge. I encourage the authors to clearly specify exactly which concept they are criticizing regarding the theoretical or empirical expectations of entropy-based exploration.\n* I do not understand the term flatness throughout the paper. Are there citations for this specific definition? I encourage the authors to provide a detailed explanation of how flatness helps readers evaluate whether an action distribution is good. I also find it difficult to digest the concept of “sufficient exploration” as presented, specifically why establishing lower bounds on action probability is linked to such sufficiency.", "audience_yn": "Yes", "audience_explanation": "Goal-oriented RL is one of the most practical methods for enabling agents to learn sophisticated action sequences. The general performance benefits demonstrated by GAGE are impressive, and I find the experiments were well-conducted.", "requested_changes": "* Please clarify exactly what issue GAGE attempts to solve regarding maximum entropy reinforcement learning.\n* I suggest the authors focus more on the identification of useful priors in robot reinforcement learning and provide a generalization of their findings that aims solely at exploitation (or goal-guidance) rather than framing it as a fix for entropy-based exploration.  \n* Provide either theoretical arguments or convincing evidence (e.g., toy experiments, citations) to support the central claim that MaxEnt causes the specific convergence issues described.", "broader_impact_concerns": "N/A", "additional_comments": "The symbol g in $g(\\pi)$ and $J_{\\mathrm{g},\\pi}$ are used in similar contexts but with slightly different meanings. Please use distinct symbols to clearly distinguish each concept to avoid confusion.", "extra_scores": {}, "extra_text": {}}, {"review_id": "sLmkjtzvBu", "reviewer": "Reviewer_nw2b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Goal Achievement Guided Exploitation (GAGE), a method for regulating exploration in reinforcement learning by enforcing adaptive lower bounds on policy stochasticity based on a measure of goal achievement. Instead of adding entropy regularization to the reward, GAGE directly constrains the policy distribution—via the standard deviation in continuous action spaces and the logit range in discrete ones—ensuring that exploration does not vanish until the agent approaches its target performance.\nThe method is integrated into both on-policy (PPO) and off-policy (SAC) algorithms and evaluated on a suite of challenging dense-reward robotic control tasks, including custom IsaacLab environments and HumanoidBench. The experiments demonstrate that GAGE improves robustness and learning stability compared to PPO baselines and avoids certain forms of premature convergence observed with entropy-based exploration.\nKey strengths include a clear critique of entropy regularization, an interpretable goal-performance-based exploration mechanism, and extensive experiments on difficult control tasks.\nKey weaknesses are insufficient comparison to strong maximum-entropy and exploration-scheduling baselines, and an overstatement of conceptual novelty relative to what is effectively a performance-based exploration scheduling method.", "claims_and_evidence_yn": "No", "claims_explanation": "While the experimental results convincingly show that GAGE outperforms PPO and avoids premature convergence in the selected tasks, the paper’s central claims go beyond what the evidence supports.\nThe work is framed as a fundamental alternative to maximum entropy reinforcement learning, yet:\n\n- First, the paper positions GAGE as an alternative to maximum entropy reinforcement learning, yet the strongest empirical gains are shown against PPO, which is not a principled MaxEnt RL algorithm. Comparisons against SAC and MaxInfoSAC—the most relevant MaxEnt-based baselines—show similar final performance, with improvements largely limited to learning speed.\n\n- Second, the method effectively implements adaptive variance/entropy scheduling, but is not compared against other scheduling strategies (manual annealing, performance-based schedules, KL-constrained methods, or learned schedules).\n\n- Third, the paper criticizes newer entropy-based methods (e.g., SAC-v2) on the grounds that entropy tuning is difficult, yet does not evaluate SAC-v2 empirically, while GAGE itself introduces several hyperparameters that require tuning. This asymmetry undermines the argument that GAGE meaningfully reduces tuning burden relative to modern MaxEnt approaches.\n\n- Fourth, although the paper discusses intrinsic-reward–based exploration methods, it uses Random Network Distillation (RND) as the representative baseline. RND is explicitly designed for sparse-reward exploration, and its poor performance in dense-reward robotic control tasks is well known. Using RND in this setting does not provide a meaningful comparison and weakens claims about outperforming intrinsic motivation methods.\n\n- Finally, a key assumption of the method is access to a well-defined goal reward or a reliable approximation of the optimal goal return. In many real-world or complex RL tasks, separating goal rewards from auxiliary shaping terms—or estimating a meaningful proxy for the optimal return—is non-trivial. While the paper explores heuristic approximations, the sensitivity of GAGE to mis-specified or poorly estimated goal achievement is not thoroughly analyzed. This requirement limits the general applicability of the method beyond carefully designed dense-reward robotic control settings.\n\nAs a result, the experimental evidence does not fully substantiate the paper’s stronger claims about replacing or fundamentally improving upon maximum entropy RL.", "audience_yn": "Yes", "audience_explanation": "The paper addresses an important and underexplored issue—premature convergence in dense-reward continuous control—and proposes a practical, interpretable mechanism that many researchers in reinforcement learning and robotics would find interesting. The empirical results on challenging humanoid and locomotion tasks, as well as the critique of entropy-based exploration, are likely to be informative to a subset of the TMLR audience.\n\nHowever, interest alone does not outweigh the current limitations in positioning and evaluation.", "requested_changes": "Below is a list of proposed adjustments, indicating which are critical for acceptance and which would strengthen the paper.\n\n# Critical Changes (Required for Acceptance)\n1. Stronger and more appropriate baseline comparisons\nThe method should be compared against:\n- entropy annealing schedules,\n- performance-based entropy or variance schedules,\n- KL-constrained policy optimization methods (e.g., MPO-style constraints),\n- and other adaptive exploration mechanisms beyond PPO-style entropy bonuses.\nWithout these comparisons, it is not possible to assess whether GAGE offers advantages beyond well-designed scheduling heuristics.\n\n2. Clearer positioning of GAGE as an exploration scheduling method\nThe paper should either:\n- substantially strengthen evidence that GAGE outperforms strong MaxEnt methods (e.g., SAC variants) in final performance, tuning difficulties with a fair comparison, or\n- explicitly reframe GAGE as a principled exploration scheduling strategy rather than a replacement for maximum entropy RL.\n\n3. More cautious claims regarding MaxEnt RL\nThe critique of maximum entropy RL should be moderated to reflect that modern methods (e.g., SAC-v2) already address some of the stated issues pragmatically. Claims should be aligned more closely with empirical findings.\n\n4. Reconsideration of intrinsic-reward baselines\nUsing RND as the primary intrinsic-reward baseline in dense-reward continuous control tasks is not appropriate.\nEither more suitable intrinsic exploration methods should be evaluated, or claims regarding intrinsic-reward approaches should be substantially toned down.\n\n# Non-Critical but Strengthening Changes\n5. Sensitivity analysis of GAGE hyperparameters\nA more systematic study of sensitivity to σ₀, goal smoothing, and proxy goal estimation would strengthen claims of robustness and ease of tuning.\n\n6. Clarification of scope and limitations\nThe paper should more explicitly state that GAGE is designed for dense-reward, single-goal tasks and discuss how it might (or might not) extend to sparse, multi-goal, or stochastic environments.\n\n7. Additional theoretical discussion\nWhile full convergence guarantees are not required, discussion of potential failure modes (e.g., over-exploration, misestimated goal achievement) would improve completeness.\n\n8. Clarification of scope\nThe paper should more explicitly limit its claims to dense-reward, single-goal robotic control tasks, and discuss potential failure modes outside this regime.", "broader_impact_concerns": "The submission does not raise immediate or severe ethical concerns. The proposed method focuses on improving exploration strategies in reinforcement learning and is evaluated primarily in simulated robotic control environments.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nQybXt7oUD", "reviewer": "Reviewer_PA8K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission proposes a reinforcement-learning framework called Goal Achievement Guided Exploitation (GAGE) that aims to mitigate premature policy convergence by linking how much the policy is allowed to collapse to an explicit goal-achievement measure. Goal achievement is defined as a ratio between the current policy’s expected return and an optimal (or maximum) expected return, with a practical refinement that measures goal achievement using goal-reward terms while excluding auxiliary shaping rewards, and approximates it with a moving average over per-episode ratios. The framework replaces maximum-entropy regularization with adaptive hard constraints on policy flatness: lower-bounding Gaussian policy standard deviation for continuous actions, and lower-bounding a discrete policy’s action probabilities via logit-range–based smoothing for discrete actions, with the intent of guaranteeing non-vanishing exploration until near-goal performance is reached. The paper instantiates GAGE in on-policy PPO (PPO-GAGE) and off-policy settings, and emphasizes fair comparisons by disabling the MaxEnt term (entropy temperature set to zero) while keeping other backbone hyperparameters aligned with the baseline where applicable.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core claims are supported by a clear motivation, formal definitions of goal achievement and the associated exploration constraint, and empirical results on both on-policy and off-policy backbones. In particular, the paper motivates why entropy regularization can both change the optimization objective and fail to guarantee sufficient exploration, then proposes a goal-achievement-conditioned hard constraint that yields interpretable control of exploration. The experiments on challenging robot-control tasks and HumanoidBench generally align with the stated claims, and the appendix provides useful implementation and tuning details. That said, reporting a few benchmark-specific details (e.g., how the goal-reward maximum or proxy target is instantiated per environment, and a concise summary of tuning budgets for compared off-policy baselines) would further strengthen reproducibility and make the evidence easier to audit.", "audience_yn": "Yes", "audience_explanation": "The submission targets a central design choice in modern reinforcement learning—maximum-entropy regularization—and offers an alternative mechanism that is simple to integrate with common algorithms (e.g., PPO and SAC) and applicable to both continuous and discretized action parameterizations. Researchers working on exploration, policy collapse/premature convergence, and practical robustness in robot control (including reward-shaping sensitivity) are likely to find the perspective and results interesting.", "requested_changes": "Critical changes\n1.\tPlease explicitly report, for each benchmark, the exact normalization constant or proxy target used in practice (including its numeric value and where it comes from, e.g., task configuration or reference-agent performance), and clarify how the computation is handled under early termination or variable episode lengths.\n2.\tFor Figure 6(b) and 6(d), the curves do not clearly plateau by the end of the reported budget. Please clarify whether the intended takeaway is final performance or fixed-budget learning, and report budget-aware summaries (e.g., area under the learning curve and last-window averages). Extending training to show near-asymptotic behavior would further strengthen the claim if the paper emphasizes final performance.\nNon-critical changes\n3.\tProvide a more systematic sensitivity analysis for the new method-specific hyperparameters (beyond the existing ablations), and summarize recommended defaults and common failure modes. This would strengthen the robustness claim and reduce the perceived tuning burden for practitioners.\n4.\tFor completeness, briefly summarize the tuning search space/budget used for the off-policy baselines (e.g., how many configurations were tried, what ranges were used, and the selection criterion). The tuned hyperparameters are already reported; this addition would simply make the comparison easier to audit.\n5.\tFigure 6(a) does not display the timestep scale factor (for example, 1e8) that appears in other panels.\n6.\tImprove figure captions to explicitly state whether plotted curves correspond to total return or goal-related return, and remind the reader of the exact goal-achievement computation, since this quantity is central to the mechanism and claims.\n7.\tAdd a brief practitioner-oriented note summarizing when the method is expected to be most beneficial (for example, dense-reward tasks with pronounced local optima) and any anticipated limitations (for example, unclear goal reward decomposition or unknown goal maxima), to improve adoption and reduce misuse.\n8.\tI also noticed a few minor typos/grammar issues (e.g., “incoporates” instead of “incorporates”), which can be easily fixed", "broader_impact_concerns": "None identified beyond standard considerations for deploying exploratory RL in the physical world; the paper’s experiments are in simulation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)