Master Skill Learning with Policy-Grounded Synergy of LLM-based Reward Shaping and Exploring
Authors: Yanbin Chang
Junfan Lin
Jie Jiang
Runhao Zeng
Changxin Huang
Jianqiang Li
Venue: ICLR 2026 Poster (2026)
Keywords: Robot Skill Acquisition
Dexterous Manipulation
Automatic Reward Design
Primary area: applications to robotics, autonomy, planning

Abstract: The acquisition of robotic skills via reinforcement learning (RL) is crucial for advancing embodied intelligence, but designing effective reward functions for complex tasks remains challenging. Recent methods using large language models (LLMs) can generate reward functions from language instructions, but they often produce overly goal-oriented rewards that neglect state exploration, causing robots to get stuck in local optima. Traditional RL addresses this by adding exploration bonuses, but these are typically generic and inefficient, wasting resources on exploring task-irrelevant areas. To address these limitations, we propose Policy-grounded Synergy of Reward Shaping and Exploration (PoRSE), a novel and unified framework that guides LLMs to generate task-aware reward functions while constructing an abstract affordance space for efficient exploration bonuses. Given the vast number of possible reward-bonus combinations, it is impractical to exhaustively train a policy from scratch for each configuration to identify the best one. Instead, PoRSE employs an in-policy-improvement grounding process, dynamically and continuously generating and filtering out reward-bonus pairs along the policy improvement process. This approach accelerates skill acquisition and fosters a mutually reinforcing relationship between reward shaping, exploration and policy enhancement through close feedback. Experiments show that PoRSE is highly effective, achieving significant improvement in average returns across all robotic tasks compared to previous state-of-the-art methods. It also achieves initial success in two highly challenging manipulation tasks, marking a significant breakthrough.

Published as a conference paper at ICLR 2026
MASTER SKILL LEARNING WITH POLICY-GROUNDER SYNERGY OF LLM-BASED REWARD SHAPING AND EXPLORING
Yanbin Chang"; Junfan Lin", Jie Jiang', Runhao Zeng', Changxin Huang"! Jianqiang Li 'Shenzhen University. "Peng Cheng Laboratory, 'Shenzhen MSU-BIT University
changyanbin22236en 2410673027@maile szu. edu. edu.cn, huangchx@szu.co edw. cn, eetessuuedu..c
szu. szu.edu. en, 1inj p@pcl. acl.aaccc.. en, en, zengrh@ ambu. www..edu.uu......a.mmmmmmm..a... edu. cn,
ABSTRACT
The acquisition of robotic skills via reinforcement learning is crucial for advanc- ing embodied intelligence, but designing effective reward functions for complex tasks remains challenging. Recent methods that use large language models generate reward functions from language instructions can reduce manual effort, but they often produce overly goal oriented rewards that neglect state exploration and cause agents to get stuck in local optima. Traditional RL addresses this by adding exploration bonuses, but these are typically generic and inefficient, wast- ing resources exploring task-irreeevant areas. To address these limitations, we propose Policy-grounded Synergy of Reward Shaping and Exploration (PoRSE), unified framework that uses large language models to generate task-aware goal rewards and to construct an abstract affordance state space that guides structured and relevant exploration. PoRSE integrates these components through an in-policy improvement process that continuously proposes, evaluates, and filters reward and exploration configurations without requiring policies to train from seratch. This synergy enables more efficient policy learning and stabilizes continual optimiza- tion. Experiments across 24 robotic manipulation and locomotion tasks show that PoRSE consistently outperforms prior state-of-the-ar LLM-based reward-design -dssggn methods and achieves the first successes on several previously unsolved challeng- ing tasks.
INTRODUCTION
Training embodied robots with deep reinforcement learning (DRL) has achieved promising results in acquiring specific skills Celik al. [2024); [Tang al. (2024). with applications in autonomous driving Coelho et al.] (2024). robotic arm manipulation Wen et al. (2025), and legged locomotion control Liang et al. (2024). For example, in legged robot walking tasks. speed rewards encourage forward movement, while stability rewards penalize posture deviations Liang et al. (2024). How- ever, designing effective reward functions typically requires extensive domain knowledge and man- ual tuning by human experts, making the process time-consuming and labor-intensive Booth et al. 2023).
The recent success of large language models (LLMss Liang et al. 2023); Minaee et al. (024)):
Brohan et al. (2023) and their ability to perform logic reasoning opened new possibilities for reward design in robotic skill learning Zeng et al. (2024); Fu et al. (2024); Venuto et al. (2024). Language to Reward (L2R) Yu et al. (2023) generates modular reward function compositions based on natural language instructions and then uses Model Predictive Control (MPC) to solve robot actions online. Eureka Ma et al. (2023) employs an evolutionary search framework to iteratively optimize reward functions, outperforming human-designed rewards across various robotic tasks. To enhance training efficiency, ROSKA Huang et al. (2025) proposes reward-policy co-evolution strategy that leverages
"Equal Contribution
1 "oorrsponndig Author

---

Published as a conference paper at ICLR 2026
Eureka
ROSKA
PORSE (Ours)
Generation
Gener ortin
Generation
Geator Rrward
Guil-riend Reare
IIM Design
Gealeriaated Howard Fasntias
Affertance State Happing Fexcden
Design
Dedig
Foodbtack
RI Training
Feedbock
Training
Expleraice
Performance
LLM Trudu ell
Trais from Scratch
Train from Previsms Pelicy
Perforttance
Evaluation
Evaluaaioo
Pulicy
Evalaation
Figure 1: Comparison of PoRSE with prior methods. Eureka Ma et al. (2023) uses LLMs to design reward functions and optimizes them via RL feedback. ROSKA Huang et al. (022 co-evolves rewards and policies based on historical policies. Our PoRSE guides LLMs to create task-aware rewards and constructs an abstract functional space for efficient exploration.
the knowledge of previously trained policies without the need to start from scratch when rewards are adjusted.
Current LLM-driven approaches, though advanced, rely solely on task- sesk-specifii textual descriptions for reward design, leading to goal-oriented preferences while neglecting state exploration Chenn et al. (2022) ; Devidze et al. 20022). This limitation is particularly problematic in nigh-degree-o-ffreedee
robotic tasks, such as dexterous manipulation involving object throwing and catching, where the policy search space is vast and target states sparse, causing agents get stuck in local optima. While exploration bonuses can partially address this issue Pathak et al. (2017); Tang et al. (2017), existing methods Ostrovski et al. (2017) do not link exploration to task objectives, resulting in in- efficient exploration of irrelevant states Wang et al. (2024). Moreover, conventional RI. methods require manual tuning of the trade-off ratio between goal-oriented behavior and state-space explo- ration for each task, with this ratio fixed from the start of policy training regardless of subsequent policy progress.
To tackle the previously discussed challenges, we introduce groiuddg Synergy of Reward Shaping and Exploration (PoRSE), novel, unified framework, as shown Fig. I PoRSE em- powers I.MSs to create task-specific reward functions and simultaneously constructs an abstract affordance state space (AFS) for efficient exploration. In particular, PoRSE goes beyond traditional task-unrelated exploration bonus methods. It harnesses LLMs to design an affordance state space, which condenses high-dimensional environmental states into low-dimensional, discrete space rele- vant to the task hand. By tracking the frequency of visits to these task-relevant AFS, we formulate a curiosity-die exploration bonus that is tightly aligned with the task objectives as well as the policy behavior.
Given the near-infinit, combinatorial space of goal-oriented reward functions and affordance state spaces, exhaustively training policies for each combination find the best combination well its grounding
trade-off ratio infeasible. To address this, PoRSE employs an process (IPG), which leverages real-time policy performance feedback guide LLMs in refining reward-bonus configurations and dynamically balancing their trade-off's Firstly, IPG incorporates an LLM-bootstrapping dlimintionneexpanioon filtering mechanism efficiently search for optimal reward-bonus combinations and ratio trade-offs across policy improvement stages. Such filter- ing mechanism is achieved via coordinate optimization fixing one variable while optimizing an- other. By doing so, it simplifies the complex and intertwined search process, greatly reducing the search space and difficulty. Secondly, to effectively integrate the new knowledge acquired dur- ing the IPG process into the existing policy, IPG adopts policy fusion approach similar to that in ROSKA Huang et al. (2025). However, we enhance it by using fast LLM- optimization to streamline the determination of the fusion ratio, eliminating the need for cumbersome Bayesian optimization.
Our PoRSE framework masters skill acquisition by integrating gral-oriented reward functions and affordance state space into the IPG process, creating a self-reinforcing cycle where reward shap- ing, exploration, and policy refinement enhance each other. Experiments on 24 robotic tasks, from
2

---

Published as a conference paper at ICLR 2026
dexterous manipulation to legged locomotion, show PoRSE achieves significant average return im- provements and gains breakthroughs in two previously unsolved complex manipulation tasks, setting new milestones in robotic skill acquisition.
2 RELATED WORK
Reward Design for Robot Skill Learning. Designing effective reward functions remains a core challenge in reinforcement learning (RL) Eschmann] (2021). Traditional approaches heavily rely on manual design, requiring extensive domain expertise Booth et al. (2023); Eschmann (2021). Re-
cent studies have explored leveraging large language models (LLMs) to automate reward design Kwon et al. 2023); Zhou et al. (2024) Cao et al. (202)). The Eureka framework Ma et al. (2023)
employs LLMs to generate executable reward codes, achieving human-level performance across diverse robotic tasks. ROSKA Huang al. (2025) enhances data efficiency through reward-policy co-evolution. REvolve Hazra et al. (2024) proposes an evolutionary framework that leverages LLMs and human feedback to iteratively refine reward functions. However, the rewards designed by exist- ing methods lack a certain exploration mechanism. contrast, our method introduces an efficient exploration reward that dynamically adjust between rewards the policy is optimized, achiev- ing efficient training of the policy.
Curiosity-drive exploration. In sparse reward tasks, direct policy learning faces challenges of insufficient exploration Ng et al. (1999); Hare (2019); Li et al. (2020). Traditional solutions rely on manually designing dense reward functions, but this process is both time-consuming and brittle. Count-bass intrinsic reward mechanisms incentivize exploration by rewarding less-visited states based on visitation frequency statistics Kolter& & Ng (200)); Tang et al. (2017). Existing methods set exploration bonuses independently of task objectives, causing agents to waste resources exploring task-irrelevant states. In contrast, PoRSE employs task-relevant state exploration rewards, effec- tively addressing this limitation by directing exploration toward states that genuinely contribute to task performance.
PRELIMINARY
RL-based Robot Skills Acquisition. Skill learning for robots via reinforcement learning can be formalized as a Markov Decision Process (MDP), where the agent interacts with the environment E. The MDP is defined by quintuple A, R. 7). consisting of the state space S, action space A, transition probability P, reward R. and discount factor E (0, I]. At each time step 4, the robot observes state 8, E S. selects action de E A according policy π (*:), transitions 8t+11 P(-(st, ot), and receives reward F2- = R(st. ae, St+1). The return for state st is the cumulative "-- discounted reward:
The objective of policy optimization is to maximize the expected cumulative return. We define 8p as the policy parameters after p updates: 6p = I(R, fb. Pb. where 8g are the randomly initialized policy parameters, and I(R, 0. p) represents the policy improvement process given reward function R over environment steps.
Reward Function Generation via LLMs. Large Language Models (LLMs) have shown strong capabilities in code generation and reasoning for robotic learning tasks, enabling automated reward function generation reinforcement learning. The Eureka framework Ma al. (202)) uses LLMs to design reward functions Rg based on task goal descriptions La and environment code Le- Eureka employs inlliiiiteaaiinn optimization approach: R best from the previous iteration and the sparse reward evaluation V V(T)) of the trained policy are fed back to the LLM to generate optimized reward functions, as shown in Eq. equation
the n-th iteration, the best reward function
R LCM((L, Ie. Ra V(T)),
(1)
where V(T) is a sparse reward array reflecting the policy's 's optimization trend. Each generated reward function trains a policy from scratch for Tmax epochs, and the reward function yiclding the highest task performance is selected as the best for this iteration, shown in Eq. equation? 2
Rn best argmaxV (I G, Tmax))
(2)
Through this iterative process, the LLM improves reward functions based on feedback, enhancing robotic skill learning performance.
3

---

Published as a conference paper at ICLR 2026
Policy-Grounded Synergy
Shaping Exploring
Training
Preeph dxppodeed:: hadanee
Weighting Parameter
Rignt
Reward Fonction
Eig sceiice and Cuall ercert of Rowar)
CLLM
Saara
That Dnicription
Prabiation
Rigul
A
Pulley
Boat
Grak-erkentdd
Afferdane: State Expleration Benmns
Rewards
Pa 12--
Initialize Policy Network
+
Update O
Affordance State Mapping Function
Best Polke 93
Afferdance Space
counts
bonus R2
Polley fusion
Pakks
comnts
Evaluation of the Best Pollcy
cad la pollpclicc
us mastmally relecumn oor sal.
Trorept depporterh
Gral- Oalee--riieeed Rewards
Vettran crmngg discce
lourd
Feedback
Explocation
Goal-oriented Reward F Function
to
Fronge in Siepesker To
tuke as offective key
8.0
6:41
lokar prockull alla
Figure 2: Overview of PoRSE. an affordance mapping function for exploration bonuses. These rewards are dynamically combined to optimize policies. An iterative feedback loop continuously refines rewards and affordance state space, creating a co-evolutionary system.
It leverages LLMs to generate goal-oriented rewards while building
4 METHOD
The methodology section organized follows: Sec introduces abstract affordance state space for an efficient stale-xxploration bonus, which is combined with goal-oriented rewards for pol- icy training. Sec 4.2 proposes a dynamic exploration-exploitttioo adjustment mechanism guided by policy feedback. To avoid costly training of numerous reward-bonus combinations, we employ policy inheritance and a tournament elimination mechanism. See [433 the interplay between goal-oriented behavior, state exploration, and policy learning, showing how their collaborative opti- mization accelerates skill acquisition. The overall framework of the method shown in Fig. The following sections detail each component.
4.1
GOAL-REELVVNT EXPLORATION BONUS GENERATION
4.1.1 AFFORDANCE STATE VISITING COUNT EXPLORATION BONUS.
To prevent the robot from exploring irrelevant states and ensure efficient policy optimization, we introduce a task-relevant exploration bonus based on a curiosity-driven
Pathak et al. (2017);
Tang et al. (2017). As shown in Fig. [ we map the robot's state space S to a low-dimensional Affordance State Space (AFS) S. using a mapping function M S + S.. Each dimension of S. quantifies the distance between the robot's current state and the goal state based specific behavioral affordances. For example, in the DoorOpenInward task, S, = [s s/1, where s is the Fuclidean distance the door handle, and s/, the door's angular displacement. We design the exploration bonus by computing the agent's visitation frequency to each state the AFS. Frequently visited states receive lower bonuses, while less-explored states receive higher bonuses. To facilitate counting, we discretize the continuous AFS using a function D(S)), resulting in C' discrete states So,d = D(S)). Accordingly, our exploration bonus R. defined as follows:
A
(s0)
(3)
V Er 1(8 = s
4

---

Published as a conference paper at ICLR 2026
Here, Ei=
% counts the number of times the state Sid has been visited, and À is the weight parameter that regulates the exploration bonus.
However, defining the affordance state space for each robotic task undoubtedly time-consuming and labor-intensive. To address this, we leverage the task parsing capabilities of LLMs, using task descriptions as instructions to guide the model in automatically setting the state mapping function M for each robotic task. This can be defined as: M = LLM([a, Ie).
Given that the exploration bonus inherently grounded in the affordance state space, which closely associated with the task, this effectively encourages the agent explore task-relevant states while mitigating excessive exploration of goal-irrelevant states. We then combine the exploration [
bonus with the goal-oriented reward Rg from Eq equation as follows:
the total reward function is formulated
Riocal (s) = R + R Rg E.
(4)
4.1.2 REWARD-BONUS REFINEMENT.
To enhance the quality of goal-oriented reward and state-exploration bonus, following the previous method Eureka Ma al. (2023), we feed the evaluation data of the policy back to LLMs and perform multi-iteration refinement of the reward and bonus. Specifically, we adopt the iterative optimization approach shown Equ. (T) to optimize the goal-oriented reward function Rg. For the exploration bonus, we similarly guide LLMs to adjust the mapping function M of the AFS based on policy
feedback. The key difference lies in the LLM prompt design: instead of using the best reward function from the previous iteration instructions, Mbest from the previous iteration as reference sample. Formally:
the best AFS mapping function
M LLM((L,, Ie, Mes V(0)).
(5)
This iterative process enables co-evolution between the reward functions and mapping functions, progressively improving the exploration efficiency of the RL agent.
4.2 A.-POLICY-IMPRRVEMEEN GROUNDING PROCESS (IPG)
To balance the state-exploration bonuses Re and goal-oriented rewards Rg for policy improvement, we employ a weighted summation approach, formulating the total reward as follows:
R ,3 * R + (2 3) , E [0. 2], Rg E R
(6)
where 3 serves as the weighting parameter to balance the contributions of the two components. Con- sequently, the weighting parameter 3 plays critical role in balancing the agent's 's exploration and exploitation behaviors. To determine an proper value for 3, we leverage the data analysis capabilities of LLMs to evaluate the policy performance feedback V(0). thereby generating a set of potential values: B = LLM(Ta, V(0)). B denotes set comprising K distinct (3 values, formally defined
as: B
[BT 5g }R)
4.2.1 LLM- BOOTSTRAPPING CEEIINATTOO-EXPASSIIN FILTERING (LEF).
Since each 3 consumes resources during the continuous optimization of the policy, we introduce an elimination mechanism to reduce the computational resources required for policy optimization, as shown in the top of Fig. _ We define an exploration-gow reward pair as a combination of a state- exploration bonus R., a gral-oriented reward Rg. and a reward weight coefficient 3.
of such ... (#N. Re,N Ry,v)}. During the parallel training of N exploration-goa reward pairs, we conduct an elimination process after every H epochs of all policy training, removing the lowest-performing pair based on the current policy's performance. the L-SHADE algorithm
pairs is denoted as N = [(81,R,,1 RR,,,R,,))... Rg.1).--
This process is similar to the linear population reduction mechanism incorporate expansion Tanabe & Fukunaga (2014). To further improve the quality of B, we
mechanism inspired by the PSO algorithm Priyadarshi & Kumar (0025). Specifically, after the elimination process, the best-performing reward-bome bonus combination among the survivors is used as the base for expansion, introducing / new mutated values. This cycle of elimination and expansion alternates iteratively, enabling continuous exploration for better / configurations.
5

---

Published as a conference paper at ICLR 2026
1
1
1
1
1
1
0.5
0.5
0.5
0.5
0.5
0
MTS
Pon
DeorOpenOutward
Scissors
DoorCloselmeard
BottleCap
GrespAndPlace
CatchOver2
Over
1
-1
-1.3
0.5
0.5
0.5
0.5
-0.5
5
LIfUnderamm
CatchUnderamm
BlockStack
CatchAbreest
Anymal
Quadoopter
Humanoid
FrankaCabinet
Sparse
Human
Eureka
ROSKA
PoRSE
Figure 3: MTS Performance Comparison Moderate- Difficulty Manipulation Tasks. Compared to other baselines, PoRSE achieved the best results on 15 tasks.
Moderate-Difficulyy
4.2.2 LEVERAGE PREVIOUS KNOWLEDGE VIA POLICIES INHERITANCE.
To address the inefficiency retraining policies from scratch for each reward-bonus refinement iteration, we adopt a strategy similar to ROSKA Huang al. (2025), blending the best previous policy parameters duest with stochastic policy frandom-
0y(a)= a Obest + (1-a) Oaadom::
where E [0, 1] is the fusion ratio. ROSKA uses Bayesian optimization find the optimal o, which is computationally expensive due to repeated policy training and data collection.
Instead, dynamically adjust using the LEF method: Cneew LCM((Ia V(O)), where a is op- timized via elimination and mutation strategies. This approach avoids the need for collecting ratio- performance sample pairs, reducing computational cost while maintaining an effective exploration- exploitation balance.
4.3 POLICY-GROUNDED SYNERGY OF REWARD SHAPING AND EXPLORATION
the PoRSE framework, goal-oriented rewards and state-exploration bonus, and policies are syner- gistically optimized to facilitate efficient robot skill acquisition. The goal-oriented reward (Ry) pro- vides precise guidance for achieving task objectives through reward signals. The state-exploration bonus (R.) is designed within an affordance state space, encouraging the policy to explore unvisited state regions. These two components are fused using a weighting coefficient B, and dynamically adjust the weighting coefficient 3 based the real-time performance of the policy. When the pol- icy's performance stagnates or deteriorates, 3 increases to emphasize the exploration bonus, helping the policy escape local optima. Conversely, when the policy demonstrates effective optimization and progressively achieves the task objectives, 3 decreases to strengthen the goal-oriented reward, accelerating the attainment of the target state. Through iterative cycles of synergistic optimization among these three components, the PoRSE framework enables efficient learning of complex skills.
5 EXPERIMENT
5.1 EXPERIMENTAL SETTINGS
Training Setting, In all experiments, we used the PPO algorithm Schulman et (2017): Makovi ichuk& & Makoviychuk (202T) for policy training with the DeepSeck- V3 model Liu et al. (2024). Our approach runs N iterative rounds, each generating
6 reward-mapping pairs for train- ing. In each round, after every H 500 epochs, the lowest-performing policy is eliminated. At the 1500-epoch mark, J 5 mutated pairs are introduced, and the lowest-performing pairs are eliminated. Subsequently, every H 500 epochs, the worst pair is removed. Only one policy completes the full 3000 epochs per round. We alternate between reward coefficient 3-based and policy inheritance coefficient a-based mutation searches across iterations. For comparison, Eureka and ROSKA also generate reward functions per round over rounds, with each trained for 3000 epochs. Sparse rewards and human-designed rewards follow the same setup. We record the best

---

Published as a conference paper at ICLR 2026
1
MTS
0.5
0.5
0.5
0
0
Doortooseuuwardd
Kettle
SwinCCup
TwoCatch
PushBlock
DeorOpeninward
ReOrienation
Switch
Sparse
Human
Eureka
ROSKA
PoRSE
MTS Performance Comparison on Hard Manipulation Tasks. PoRSE achieved the best results all difficult tasks.
Figure 4:
result per experiment using 5 random seeds. The computational cost for our method, Eureka, and ROSKA is nearly identical across experiments. Additional settings are detailed in the appendix.
Benchmark. We conducted experimental evaluations on 24 robotic skill learning tasks across two simulation environments: Bi-DexHands Chen et al. (202)) and Isaac Gym Makoviychuk et al. (221). Specifically, 20 tasks originated from Bi- Bi-DexHands, while the remaining 4 were from Isaac Gym. Bi-DexHands serves benchmark environment for dexterous bimanual manipulation tasks, focusing evaluating robotic systems" capability to perform complex manipulation using both hands. In contrast, Isaac Gym provides a benchmark for continuous control robotics, en- compassing skill learning tasks involving legged robots and manipulator arms, among other robotic platforms. order to improve readability, we have adopted abbreviations for some tasks, and the complete task names provided in the appendix.
Evaluation Metrics. In our experiments, we utilized Maximum Training Success (MTS) as the principal evaluation metric. MTS quantifies the highest sparse reward attained during training, serv- ing as a critical indicator of policy efficacy.
Baseline Methods. We compared the performance of method against four baseline approaches: Sparse rewards Ma et al. (2023). human-designed rewards Makoviychuk et al. (2021), Eureka Ma (0023) and ROSKA Huang et al. (2025). Sparse Rewards refers to reward settings that express the task objective,an and are defined in EurekaMa et al. (023). Human Expert-Designed Rewards are carefully designed by human experts, and more refined compared to sparse rewards. Eureka is general framework that automatically generates reward functions leveraging LLM
al. (2023). ROSKA robot skill learning framework [Huang et al. (2025). PoRSE is our method.
5.2 COMPARISON RESULTS.
We categorized the 24 reinforcement learning tasks in this study into two difficulty levels-moderate and basd on training scores using human-designed rewards. Results for each level are pre- sented in separate tables.
Performance Comparison on Moderate-Difficuly Manipulation Tasks. Experimental results in Fig. 3 demonstrate PoRSE's superiority in Maximum Training Success (MTS). PoRSE achieved perfect MTS (1.000) in 5/16 tasks (e.g., Pen, Scissors), outperforming the human baseline by 1.6%- 3.5%. In GraspAndPlace, PoRSE reached MTS 0.984 (25.3% better than human baseline 0.785). The only exception was Humanoid locomotion, where Roska (8.917) slightly outperformed PoRSE (8.454, +5.2%). due bipedal tasks favoring gral-oriented rewards. However, PoRSE excelled in Anymal (MTS -0.012, +42% vs. human baseline -0.021). Overall, PoRSE achieved leading performance 15/16 tasks, confirming effectiveness across diverse robotic scenarios.
Performance on Hard Manipulation Tasks. In challenging manipulation tasks, PoRSE demon- strated superior performance through its integrated optimization mechanism, as shown Fig. achieved perfect MTS (1.000) in DoorCloseOutward and Kettle tasks, outperforming Eureka (0.553, 0.742). As shown Fig. PoRSE attained a significant improvement over the human baseline in DoorOpenInward, and the first solve TwoCatch (MTS 0.349). In BlockStack. PoRSE achieved MTS 0.753, significantly surpassing ROSKA (0. 148). These results validate PoRSE's fectiveness in sparse reward scenarios and high-dimensional action spaces.

---

Published as a conference paper at ICLR 2026
Iteration
Iteration
Iteration
1.0
0.8
ROSES hutse
M
0.4
M
0.2
Litunderarm
0.0
1500
3000
3000 Training Epoch
1500
1500
3000
1500
3000
1500
3000
Training Epach
Training Epoch
Training Epoch
Training Epoch
Iteration
Iteration
Iteration
Iteration
E.WOOAA
RUSGA
Surcess
My Ma
TwoCatch
1500 Training Epoch
1500 Trainingg Epoch
Training Epoch
Training Epoch
Training Epoch
Figure 5: Results show that all methods can guide the policy optimize in simple tasks, but PoRSE achieved the best results. For complex tasks, only PoRSE can effectively guide policy optimization, and both Eureka and ROSKA methods are ineffective.
1.0
Eureia
0.2
PoRSEE0.::1.5)
Success Rate
PoRSEE008:1122
Success Rate
Po
20.00
50.05
P0.05(1.500.3)
LLMCount
PoR15E110:1.00
0.4
LLMCount
Po
WM
0.2
0.0
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
Figure 6: Training curves of different 3 values. It is evident that only by employing appropriate ratios can the policy be successfully trained attain better performance.
Iterative Training Performance Comparison. As shown Fig. we compare PoRSE's with Eureka and ROSKA on LiftUnderarm (simple) and TwoCatch (complex) tasks. In LiftUnderarm, PoRSE reached 0.97 success rate by Iteration (1500 episodes), outperforming Eureka (0.6) and ROSKA (0.8) and converging faster. In TwoCatch, where Eureka and ROSKA failed (success=0), PoRSE achieved a 0.4 success rate by Iteration 5, with a tenfold increase from Iteration 1, highlight- ing its effectiveness in handling sparse rewards and refining policies.
5.3 ABLATION STUDIES
This section conducts ablation experiments investigate: (1) How goal-oriented rewards, state-exploration bonuses, and policy inheritance PoRSE, and their indi- vidual contributions? (2) The roles of reward-bonus coefficient and policy inheritance coefficient a, and whether their alternating iteration is necessary?
Ablation Study Questions.
(3) How do different 3 values affect policy optimization, and the relative importance of reward components?
(4) PoRSE's iterative reward optimization process necessary? (5) How robust is PoRSE when the Affordance State Space is noisy or randomly assembled?
Component-wise Ablation Analysis. This ablation study examines the roles of goal-oriented re- wards (R,), iitooraatiooooiinnttt bonuses (Ra), and dynamic policy fusion inheritance (9 (ffasio)) in PoRSE. As shown in Tab. removing Ry significantly degrades performance in precision tasks like TwoCatch (0.349 + 0.190) while retaining reasonable performance in manipulation tasks like FrankaCabinet (0.883, -7.7%). Removing R. reduces performance in exploration-requirig tasks, with BlockStack dropping from 0.753 to 0.393 and TwoCatch from 0.349 to 0.193. Eliminating 8 /uusion causes severe performance drops in long-tem optimization tasks, with FrankaCabinet de- creasing from 0.957 to 0.671 and TwoCatch plummeting from 0.349 to 0.017. The synergistic effects of all components critical, removing any single component reduces PushBlock success
8

---

Published as a conference paper at ICLR 2026
Table 1: The MTS results show that both the architectural components and optimization strategy
contribute to the final performance.
Anymal
Quadcopter
Franka
Method
MTS
Gap
MTS
Gap
MTS
Gap
PoRSE w/o Rg
0.128
0.116
-..40
+ 0.0.6
0.883
4 0.074
PoRSE w/o R
0.097
0.085
0.038
4 0.024
0.912
+ 0.045
PoRSE w/o Orusion
0.346
0.334
-0.034
0.020
0.671
i 0.286
PoRSE w/o Ryatio
0.066
0.054
-0.019
0.005
0.946
+ 0.011
PoRSE w/o Oratio
-0.020
0.008
-0.015
0.001
0.940
+ 0.017
PoRSE
-0.012
-0.014
0.957
Table 2: The MTS results show that both the architectural components and optimization strategy contribute to the final performance.
BlockStack
PushBlock
TwoCatch
Method
MTS
Gap
MTS
Gap
MTS
Gap
PoRSE w/o Rg
0.328
0.425
0.295
0.023
0.190
$ 0.159
PoRSE w/o R.
0.393
4 0.360
0.306
4 0.011
0.193
4 0.156
PoRSE w/o Orusion
0.603
0.150
0.236
0.082
0.017
4 0.332
PoRSE w/o Ryatio
0.296
0.457
0.243
0.075
0.297
4 0.052
PoRSE w/o fratio
0.590
0.1363
0.309
4 0.009
0.276
4 0.073
PoRSE
0.753
0.318
0.349
Table 3: MTS comparative results of PoRSE and LLMCount baselines across six representative manipulation tasks, PoRSE achieves apparent improvement in task success rates.
Method
Pen
TwoCatch
Franka
BlockStack
PushBlock
DoorOpenlnward
LLMCount
0.412
0.000
0.706
0.140
0.127
0.025
PoRSE
1.000
0.349
0.957
0.753
0.318
0.250
rates by 7.2%-25.8%% while the complete PoRSE framework achieves optimal performance through collaborative optimization.
Optimization Strategy Ablation Study. This study investigates the effects of alternating reward fusion (8) and policy inheritance (a) coefficients during iterative optimization. Fixing either co- efficient degrades MTS across tasks. As shown in Tab. and Tab. 2 with fixed policy fusion inheritance (PoRSE w/o B the "TwoCatch" MTS drops from 0.349 to 0.276 due to reduced balance, as seen in the
adaptability. Fixing reward fusion (Rruio) disrupts exploration exploration-exploiaatii "PushBlock" task where MTS falls from 0.318 to 0.243. The complete PoRSE method, alternat- ing both coefficients, achieves optimal reward-policy coordination, delivering the best performance across all tasks.
Ablation Analysis of 3. We evaluated different reward ratios (R., : R.) for DoorOpenInward and TwoCatch tasks. Configurations included Eureka -nlyy, LLMCoun et al. (2024) (Ra- only), and PoRSE ratios (0.5:1.5, 0.8:1.2, 1.2:0.8, 1.5:0.5. 1.0:1.0,, all trained for 3000 epochs. As shown in Fig. 6 results show single-reward approaches failed completely (0% success), while optimal fusion ratios achieved significant improvements. PoRSE (1.5:0.5) reached 97% success on MborOpenInward and 25% TwoCatch, outperforming prior methods (~ 0%).

---

Published as a conference paper at ICLR 2026
Table 4: MTS Performance Comparison of PoRSE-AFS Random and Baseline Methods on Repre- sentative Manipulation Tasks (mean = standard deviation).
Task
BlockStack
PushBlock
LiftUnderarm
Sparse
0.000 + 0.001
0,003 0.004
0.000 0,000
Human
0.600 0.229
0.011 0.003
0.348 0.140
Eureka
0,254 = 0.119
0.025 0.008
0.739 + 0.166
Roska
0.148 0.154
0.069 0.049
0.608 0.211
PoRSE-AFS-Random
0,680 = 0,080
0.324 = 0.034
0,802 + 0,029
PoRSE
0.753 0.210
0.378 0.022
0.952 0.015
Ablation Study on Reward Refinement.
This study examines the impact of reward and bonus refinement during policy optimization. Us- ing LLMCount Sarukkaii et al. (2024) as baseline, which generates a fixed exploration bonus, we compare its performance against PoRSE. As shown in Tab. 3 results show PoRSE significantly outperforms LLMCount across all tasks, demonstrating that static rewards inadequately address complex skill learning needs. For example, in the TwoCatch task, LLMCount's fixed bonuses failed to adapt to policy performance changes, while PoRSE's collaborative optimization loop dynami- cally adjusted R. and Rg ratios, effectively guiding training through different learning stages and highlighting the benefits of dynamic reward refinement.
Ablation Study on AFS Robustness. This study evaluates the impact of replacing the LLM- generated task-aware AFS with randomly assembled AFS. As shown in Tab. TO the variant FoRSE-AFS-Randoon remains clearly stronger than Sparse, Human, Eureka, and ROSKA on the representative manipulation tasks BlockStack, PushBlock, and LiftUnderarm. For example, it achieves 0.680 + 0.080 on BlockStack versus 0.254 + 0.119 for Eureka and 0.148 + 0.154 for ROSKA, and reaches 0.324 + 0.034 PushBlock where the best baseline is 0.069 + 0.049. Al- though is below the full PoRSE that uses task-relevant AFS, the elimination mechanism prunes unhelpful dimensions and the LLM adapts the exploration coefficient, which preserves strong perfor- mance despite noisy AFS and confirms that PoRSE is robust to imperfect affordance specifications.
CONCLUSION
We propose groiudd Synergy of Reward Shaping and Exploration (PoRSE), novel frame- work for balancing reward design and state exploration robotic skill learning. PoRSE integrates LLM-generated affordance state spaces with dynamic curiosity-driven bonus mechanism, en- abling efficient exploration aligned with task objectives without manual reward tuning. The in- ioprey-improvemmnt grounding process (IPG) optimizes reward configurations via real-time policy feedback, using fast LLM-aided methods instead of manual Bayesian optimization. Experiments diverse robotie tasks show PoRSE's superiority, achieving significant performance gains state-of-the-ar methods. By unifying reward shaping, exploration, and policy refinement in a self- reinforcing loop, PoRSE sets paradigm for autonomous robotic skill acquisition.
7 LIMITATION
Similar to other LLM-based generation approaches, our method is also subject to the inherent in- stability of LLM outputs. Even with identical prompts, the quality of generated code can vary, and non-executable code may occasionally be produced. This is common limitation LLMS rather than a specific issue of our approach. Nevertheless, with the rapid advancement and continuous iteration of LLM technology, stability in reward design and code generation is expected to gradually improve, and LLM capabilities in these areas will continue to strengthen. To ensure fair compar- ison in this paper, we have implemented the following measure for Eureka, ROSKA, LLMCCunt, and PoRSE: all six functions generated in each iteration undergo code correctness verification. This ensures that every generated function is executable, thereby mitigating result deviations caused by L.LM instability and maintaining the reliability and faimess of the comparative evaluation.
10

---

Published as a conference paper at ICLR 2026
8 ACKNOWLEDGE
This work is supported in part by the National Natural Science Foundation of China (No. 62403325, No. 62325307, No. 62527809, No. 62203134, No. 62373258, No. 62506180), part
by the Natural Science Foundation of Guangdong Province (No. 2023B1515120038) in part
by Shenzhen Science and Technology Innovation Commission (No. 20231122104038002 No.
KIZDD20309231 1380010044 No. JCYI220440811141622880 13141628038, No. KIZID20239921 112292231111112222 in
part by the Shenzhen Key Industry R&D Program Project (No. ZDCY20250901 102300001), part
by China Postdoctoral Science Foundation (No. 2025M771522), in part by the Major Key Project of PCL (No. PCL2024A04, No. PCL2025A17).
REFERENCES
Serena Booth, W Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of and---r reward design: misdesign through overfitting and invalid task speci- fications. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. Conference on robot learning. PMLR, 2023.
Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, and Yun Li. Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods. IEEE Transactions Neurul Networks and Learning Systems, 2024.
Onur Celik, Aleksandar Taranovic, and Gerhard Neumann. Acquiring diverse skills using curricu- lum reinforcement learning with mixture experts. arXiv preprint arXiv:2403. 06966, 2024.
Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Song-Chun Chun Zhu, and Yaodong Yang. Towards human-leved bimanual dex- terous manipulation with reinforcement learning. Advances in Neural Information Processing Systems, 35, 2022.
Daniel Coelho, Miguel Oliveira, and Vitor Santos. Rifold: Reinforcement learning from online demonstrations in urban autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Exploration-guidde reward shaping for reinforcement learning under sparse rewards. Advances in Neural Information Processing Systems, 35, 2022.
Jonas Eschmann. Reward function design in reinforcement learning. Reinforcement learning algo- rithms: Analysis and Applications, 2021.
Yuwei Fu, Haichao Zhang, Di Wu, Wei Xu, and Benoit Boulet. Furl: Visual- languaee models as fuzzy rewards for reinforcement learning. arXiv preprint arXiv:2406.000 00645, 2024.
Joshua Hare. Dealing with sparse rewards in reinforcement learning. 1rXiv:1910.09200 2019.
arXiv preprint
Rishi Hazra, Alkis Sygkounas, Andreas Persson, Amy Loutfi, and Pedro Zuidberg Dos Martires. Revolve: Reward evolution with large language models using human feedback. arXiv preprint arXiv:2406.01300 2024.
Changxin Huang, Yanbin Chang, Junfan Lin, Junyang Liang, Runhao Zeng, and Jianqiang Li. Effi- cient language-instructed skill acquisition via reward-policy co-evolution. In Proceedings of the AAAI Conference Artificial Intelligence, 2025.
J Zico Kolter and Andrew Y Ng. Nayr-bayesian exploration in polynomial time. In Proceedings of the 26th annual international conference machine learning. 2009.
11

---

Published as a conference paper at ICLR 2026
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303.0000) 2023.
Jing Li, Xinxin Shi, Jiehao Li, Xin Zhang, and Junzheng Wang. Random curiosity- driven explo- ration in deep reinforcement learning. Neurocomputing, 2020.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation. IEEE, 2023.
William Liang, Sam Wang, Hung-Ju Wang, Osbert Bastani, Dinesh Jayaraman, and Yecheng Jason Ma. Eurekaverse: Environment curriculum generation via large language models. arXiv preprint arXiv:2411.01775. 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. 2024.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 12931,,.........1111111111111 2023.
Denys Makoviichuk and Viktor Makoviychuk. rl-games: A high-performance framework for rein- forcement learning. Denys88/rl_gamet 2021.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint
1tn::1108.147000 2021.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Am- atriain, and Jianfeng Gao. Large language models: A survey, arXiv preprint arXiv:2402.06196. 2024.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. Ieml. Citeseer, 1999.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference machine learning. PMLR, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning. PMLR, 2017.
Rahul Priyadarshi and Ravi Ranjan Kumar. Evolution of swarm intelligence: a systematic review of particle swarm and ant colony optimization approaches in modern research. Archives of Compu- fational Methods Engineering, 2025.
Vishnu Sarukkai, Brennan Shacklett, Zander Majercik, Kush Bhatia, Christopher Ré, and Kayvon Fatahalian. Automated rewards via llm-generated progress functions. arXiv preprint arXiv:2410.09187, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063447 2017.
Ryoji Tanabe and Alex S Fukunaga. Improving the search performance of shade using linear popu- lation size reduction. In 2014 IEEE congress on evolutionary computation. IEFE, 2014.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schul- man, Filip Tururkk, and Pieter Abbeel. exploration: study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Yao Tang, Zhihui Xie, Zichuan Lin, Deheng Ye, and Shuai Li. Learning versatile skills with cur- riculum masking. In The Thirty- Annual Conference on Neural Information Processing Systems, 2024.
12

---

Published as a conference paper at ICLR 2026
David Venuto, Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, and Ankit Anand. Code as reward: Empowering reinforcement learning with vlms. arXiv preprint arXiv:2402.04764, 2024.
Yiming Wang, Kaiyan Zhao, Furui Liu, et al. Rethinking exploration in reinforcement learning with effective metric-based exploration bonus. Advances in Neural Information Processing Systems, 37, 2024.
Yongyan Wen, Siyuan Li, Rongchang Zuo, Lei Yuan, Hangyu Mao, and Peng Liu. Skilltree: Ex- plainable skill-based deep reinforcement learning for long-horizon control tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huee Lee, Montse Gonzalez Are- nas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, al. Language to rewards for robotic skill synthesis. arXiv preprinl arXiv:2306 2tt.:2370008 2023.
Yuwei Zeng, Yao Mu, and Lin Shao. Learning reward for robot skills using large language models via self-alignment. arXiv preprint arXiv:2405.00 162 2024.
Hao Zhou, Chengming Hu, Ye Yuan, Yufei Cui, Yili Jin, Can Chen, Haolun Wu, Dun Yuan, Jiang, Di Wu, et al. Large language model (Ilm) for felecommunication:: A comprchensive survey principles, key techniques, and opportunities. IEEE Communications Surveys Tutorials, 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.I ALGORITHM DESCRIPTION
In this section, present the Policy-grounded Synergy of Reward Shaping and Exploration (PoRSE) framework, as detailed in Algorithm
The algorithm operates through N 5 iterations of policy optimization. In each iteration, the large language model (LLM) generates K = 6 combinations containing goal-oriented reward functions R and state-exploration affordance mapping functions M where n denotes the iter- ation index. The exploration bonus calculated using a parameter λ=0.01 to scale the reward derived from state visitation counts. Each candidate combination trains parallel policies via Prox- imal Policy Optimization (PPO) for up to Tmxx 3000 epochs, using dynamically fused rewards R = B6 R + (2 (2-3h) Re Resplore Here, the reward coefficient 3 (default: balances goal-oriented rewards and state-exploration bonuses, while the policy inheritance ratio (default: 0.8) controls fusion with prior policies. During training, an LLM-boostrapping gimination-expansion mecha- nism prunes candidates: the worst-performing policies are removed every 500 epochs, while the 1500-epoch mid-stage, PoRSE generates J = mutated variants of either 3 (odd iterations) or or (even iterations) expand the candidate pool. After mutation, the bottom 6 policies are eliminated, and pruning continues every 500 epochs until only the best policy test remains at 3000 epochs.
Algorithm 1
Synergy of Reward Shaping and Exploration (PoRSE)
Require: Task description Is. environment code Le, iterations N = 5, candidates per iteration K 6, max epochs Tmmx 3000, initial policy 0o
1: for 71 = 1 to N do
2: if
1 then
3:
// Reward and mapping functions initialization via LLM
4: R LLM(Ta, Ta), M LLM(Ta, Ta)
5: else
6:
// Reward and mapping functions refinement via LLM
7: R2+1 + LLM((la, Rhess ((0hes)), M + LLM(La, Mheso V V(Ohes))
8:
end if
9:
I/ Construct goal-oriented rewards and state-exploration bonuses do
for k = R < (s, R \/ count(D(MM
11:
12: end for
I/ Parallel policy training using fused rewards
14:
for (R g.k: R.,k) E N do
R + 3 + (2 R) g,k- 04 + aaffe + (1 a000 (Default a and (3)
16:
Train 8f with PPO
end for // A.L.-bboostrrapping Elimination-expaasion Filtering for 1=500, 100.,,............ do
18:
20:
if 1500 then
21:
Generate J = 5 mutated a or 3 via PSO,
Remove the worst policy 0,
22:
else Remove the worst policy 0
23:
24: end if
25: end for
26:
end for
27:
return Obest
In later iterations (n 2 2), the LLM refines new reward functions R and affordance mappings M based sparse reward evaluations V(en from the current best policy. This cycle repeats for all 5 iterations, with 3 and a alternately mutated to diversify exploration. The final output is an optimized policy Prea that synergistically balances goal achievement and state-space exploration.
14

---

Published as a conference paper at ICLR 2026
A.2 EXTENDED EXPERIMENT RESULTS
A.2.1 DETAILED EXPERIMENT RESULTS
As shown in Tab. 5 and Tab. 6 present the detailed results of all experimental data involved in this paper in the form of mean + standard deviation, with each result derived from experiments with 5 different random seeds.
Table 5: MTS performance comparison on moderate-di diftteelffff tasks. MTS (mean + std) showing PoRSE's superiority over baselines in manipulation and locomotion tasks.
Task name
Sparse
Human
Eureka
Roska
PoRSE
Pen
0.061=0.108
0.984=0.016
0.634+0.460
0.111=0.077
1.000=0.000
DoonOoenOutward
0.000#0.000
0.813=0.418
0.971=0.061
0.638+0.303
1.000=0.000
Scissors
0.387+0.529
0.996#0.001
1.000=0.000
0.992#0.011
1.00040.000
DoorClosennward
0,400=0.547
1.000=0.000
1.000=0.000
1.000=0.000
1.000=0.000
BottleCap
0.901=0.033
0.886#0.056
0.424+0. 145
0.348+0.175
0.988+0.011
GraspAndPlace
0.001
0.78540.298
0.66740.079
0,284+0.084
0.98440.008
CatchOver2
0.000*0.000
0.85420.013
0.001#0.001
0.100#0.221
0.973#0.008
0.019#0.042
0.925=0.011
0.725+0.377
0.579#0.388
0.965#0.013
LiftUnderarm
0.000=0.000
0.348 140
0.739+0. 166
0.608+0.211
0.952#0.015
CatchUnderarm
0.000#0.000
0.54420.250
0.00040.000
0,271
0.89420.038
BlockStack
0.000*0.001
0.600#0.229
0.254 119
0.148+0.154
0.753#0.210
CatchAbreast
0.000#0.000
0.369=0.213
0.000#0.000
0.000#0.000
0.745=0.073
FrankaCabinet
0.000#0.010
0.100=0.050
0.778+0.177
0.850=0.329
0.957=0.04p
Anymal
-0.863+0.017
-0.021=0.005
-0,23540.306
-0.314+0.540
-0.01240.005
Quadcopter -1.382+0.083
-0.067=0.029
-0.023+0.007
-0.019#0.007
-0.01420,005
Humanoid
5.691+0.880
7.235+1.152
6.534+1.933
8.917=0.512
8.454=0.343
Table 6: MTS performance comparison on hard manipulation tasks. Results demonstrate PoRSE's breakthroughs in hard skill learning tasks where existing methods fail completely.
Task name
Sparse
Human
Eureka
Roska
PoRSE
DlorCloseuuwward
0,279=0.348
0.244=0.424
0.553+0.340
0.49140.467
1.000=0.000
Kettle
0.001#0.002
0.046=0.077
0.742+0.419
0.534+0.448
1.000=0.000
SwingCup
0.001+0.003
0.025=0.026
0.353+0.407
0.076+0.043
0.995=0.002
TwoCatchUnderarm
0.000=0.000
0.000=0.000
0.001#0.001
0.000*0.000
0.349+0.063
PushBlock
0.003+0.004
1.0.000
0.025+0.008
0.069#0.049
0.378=0.022
OporOwenlwward
0.000*0.000
0.004=0.004
0.007=0.010
0.002+0.002
0.283+0.338
ReOrientation 0.021+0.004
0.028=0.003
0.107:0.019
0.060E0.027
0.149=0.026
Switch
0.000+0.000
0.000=0.000
0.000+0.000
0.000*0.000
0.000=0.000
A.2.2 RESULT COMPARISON HNS
In our experiments, we introduce the Human Normalized Score (HNS) as the second evaluation metrics. HNS assesses method's performance relative human-engineered reward functions and is computed as:
MTSMethod
MTS Sparse
Sparse
HNS
(8)
MTS Haman
MTS Spurse
15

---

Published as a conference paper at ICLR 2026
Human Normalized Score
Anymal
FrankaCabinet
Quadcopter
Humanoid
Pen
LiftUnderarm
SwingCup
CatchUnderarm
3
CatchAbreast
BlockStack
PushBlock
Kettle
BottleCap
CatchOver2
Closelnward
Scissors
OpenOutward
GraspAndPlace
Over
CloseOutward
Openlnward
ReOrientation
TwoCatch
Switch
Sparse
Human
Figure 7: HNS score comparison across 24 robotic tasks. PoRSE significantly outperformed other methods in 23 (all 24) robot skill learning tasks, achieving the best experimental results.
where Mecho represents the score achieved by particular method, MTS denotes the base- line sparse reward, and MTSSHumn corresponds to expert human performance.
From the experimental results and the corresponding visualization chart, with Human (human- engineered rewards, HNS 1) and Sparse (sparse reward baseline, HNS 0) as references, the proposed method (Ours/PoRSE) demonstrates significant advantages in most tasks: On the one hand, in tasks requiring complex operations or fine-grained control (c.g.. FrankaCabinet, SwingCup, PushBlock, Kettle). the HNS of Ours far exceeds that of Human (for instance, the HNS of Ours in SwingCup reaches 41.45 1..42, which is much higher than Human's 's and the performance of other meth- ods), demonstrating that can learn better reward strategies than buman-engineered ones and drive policies to achieve performance far beyond expert-level. On the other hand, in relatively simple tasks or those with mature solutions (e.g., Quadcopter, Scissors), the HNS of Ours is comparable to that of Human other state-of-the-an methods, reflecting its adaptability tasks. In contrast, comparative methods Eureka and ROSKA even yield negative HNS in tasks like BottleCap, while Ours still achieves positive and excellent performance this further reflects that Ours superior in the stability and effectiveness of reward learning, can guide policy learning more efficiently, and possesses stronger capabilities of performance generalization and optimization in various robotic manipulation tasks.
different types of
A.2.3 DETAILED EXPERIMENTAL RESULTS LLMCOUNT
We conducted extensive experiments on the LLMCoun Sarukkai et al. (024)) method on 24 rein- forcement learning tasks, with 5 different random seed results collected and averaged for each task. The detailed comparison between LLMCount method and PoRSE method is as follows.
As shown in Tab. 7 the PoRSE method significantly outperforms the LLMCount benchmark method on 23 of the 24 robot skill learning tasks. Specifically, in the Quadruped Robot Motion Control task (Anymal), PoRSE improves the Mean Training Success (MTS) from -1.22 LLMCount -0.012, which is close to the expert human-designed level of -0.021. As for the TwoCatch task, which re- quires precise operation, L.LMCount fails completely (MTS=0), while PoRSE achieves a success rate of 0.349 through the dynamic reward weight adjustment mechanism, indicating that can ef- fectively solve the exploration-exploitatii comprehensively validate the combined advantages of the PoRSE framework by synergistically op- timizing the goal-oriented reward and state-explone exploration bonus mechanisms.
dilemma under sparse rewards. The experimental results
A.2.4 MAPPING FUNCTION ABLATION EXPERIMENT AND ANALYSIS
To evaluate the actual contribution of the Mapping Function to the PoRSE framework, we employed alternative mapping function-simmaatt simhash Tang et al. (2017). This function directly map similar high- high-dimensional environmental state information into the same interval, serving as the foundation for count-based rewards. Unlike the Mapping Function in this paper, simhash lacks any semantic
16

---

Published as a conference paper at ICLR 2026
1
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
SwingCup
Over
Salssors
Closein
OpenOut
Openin
CloseOut
LiftUnder
SimHash
PORSE
Figure 8: Success rate comparison of SimHash and PoRSE methods
Table 7: MTS Comparison of LLMCount and PoRSE methods, PoRSE method achieved higher MTS on 23 tasks.
Method
Anymal
Franka
Quadcopter
Humanoid
LifUnderaarm
Pen
I.MCouut
-1.226
0.706
-0.092
7.737
0.649
0.412
PoRSE
-0.012
0.957
-0.014
8,454
0.952
1.000
Method
SwingCup
CatchUnder
CatchAbreast BlockStack
PushBlock
Kettle
LLMCount
0.993
0.000
0,238
0.140
0.127
1.000
PoRSE
0.995
0.894
0.745
0.753
0.378
1.000
Method
BottleCap
CatchOver2
CloseIn
Scissors
OpenOut
GraspPlace
LLMCount
0.985
0.387
1.000
1.000
0.998
0.906
PoRSE
0.988
0.973
1.000
1.000
1.000
0.984
Method
Over
CloseOut
OpenIn
ReOrientation
TwoCatch
Switch
LLMCount
0.594
0.905
0.025
0.098
0.000
0.000
PoRSE
0.965
1.000
0.283
0.149
0.349
0.000
Table 8: MTS Comparison of SimHash and PoRSE methods, PoRSE method achieved higher MTS robot skill learning tasks.
Method
SwingCup
Over
Scissors
CloseIn
OpenOut
OpenIn
CloseOut
LiftUnder
SimHash
0.454
0.782
1.000
1.000
1.000
0.000
0,958
0.000
PoRSE
0.995
0.965
1.000
1.000
1.000
0.250
1.000
0.952
connection to the task goal and relies solely on the similarity of high- high-ddmennional environmental state data partition intervals and perform counting. Thus, this comparative experiment is designed to validate the effectiveness of the Mapping Function adopted in this paper.
As shown in Tab. experiments reveal that the SimHash mapping function, devoid of task mantics, underperforms PoRSE across all tasks. Semantic state abstraction via language mod- els markedly enhances exploration efficiency. PoRSE's task-relevant state mapping directs agents to focus on key behavioral features, unlike SimHash's count-based mechanism reliant high- dimensional state similarity, which struggles to escape local optima. This validates the necessity of task-driven abstract state space design for complex skill learning.
A.2.5 LLM ABLATION EXPERIMENT AND ANALYSIS
To verify the robustness of our framework to the output quality of large language models (LLMs), design this ablation study. Specifically, replace the LIM used in the original method with
17

---

Task Published as a conference paper at ICLR 2026
more lightweight GPT-4o-mini model (denoted PoRSE-GPT-4o-min and compare its perfor- mance with the original method and other baselines across multiple tasks. The core purpose of this experiment is to investigate whether the proposed co-optimization paradigm can still maintain its effectiveness when the quality of the underlying L.M changes, thereby demonstrating that the advantage of the framework does rely specific high-performance L.M,, but rather its inherent and dynamic optimization mechanism.
Table 9: Performance Comparison of PoRSE- PORSE-GPT--A GPT-4o-mini with Baselines on Representative Tasks
Task
BlockStack
PushBlock
LiftUnderarm
CatchUnderarm
Sparse
0.000 = 0.001
0.003 0.004
0.000 + 0.000
0.000 + 0.000
Human
0.600 + 0.229
0.011 0.003
0.348 0.140
0.544 + 0.250
Eureka
0.254 + 0.119
0.025 0.008
0.739 = 0. 166
0.000 + 0.000
Roska
0.148 = 0.154
0.069 0.049
0.608 0.211
0.271 = 0.370
PoRSE-GPT-4o-min
0.618 + 0.165
0.360 = 0.051
0.869 = 0.07
0.907 = 0.015
PoRSE
0.753 0.210
0.378 0.022
0.952 0.015
0.894 + 0.038
As can be observed from the experimental results (shown in the table below), even when using the less performant GPT-4o-mini model, PoRSE-GPT-4o-iint significantly outperforms all baseline methods except the original PoRSE on the four tasks: BlockStack, PushBlock, LiftUnderarm, and For instance, in the PushBlock task, its performance (0.36=0.051) far exceeds that of Roska (0. (0.069+0.049) 049) and Eureka (0.025+0.008 ).00); in the CatchUnderarn task, its success rate even reaches 0.907=0.015. This fully demonstrates that although the initial output quality of the LLM decreases, the strategy can still be guided to converge to high-performance level through multiple rounds of dynamic optimization within the framework. This result strongly verifies the robustness of the framework to the variability of LLM outputs, and its core value lies establishing a "goal- exploration-policy"" co-optimization paradigm that is not strongly tied specifie LLM and can continuously self self-improve.
A.2.6 AFS ABLATION EXPERIMENT AND ANALYSIS
Table 10: MTS Performance Comparison of
and Baseline Methods Rep-
resentative Manipulation Tasks (mean + standard deviation)
BlockStack
PushBlock
LiftUnderarm
Sparse 0.000 0.001
0.003 0.004
0.000 0.000
Human
0.600 # 0.229
0.011 0.003
0.348 0.140
Eureka
0.254 0.1
0.025 0.008
0.739 0.666
Roska
0,148 0.154
0.069 0,049
0.608 0.211
FORSE-AFS-Randee
0.680 0.080
0.324 0.034
0.802 0.029
PoRSE
0.753 0.210
0.378 0.022
0.952 0.015
To verify the fault tolerance of the PoRSE framework to invalid/inefficient affordance state space (AFS) dimensions generated by large language models (LLMs), well the core supporting role of the built-in elimination mechanism in task adaptability, this ablation study designs a control group with "randomly assembled AFS" (denoted as PORSE-AFS-Rande framework leverages LLMs to generate task-relevant AFS dimensions for guiding efficient explo- ration, in practical applications, L.LMs may generate goal-irrelevant dimensions due to task com- prehension biases or lead to degraded AFS quality due to initial design flaws. To simulate this worst-case scenario, the experiment constructs invalid AFS by randomly splicing environmental state variable dimensions. The aim is to explore whether the framework can dynamically screen dimensions valuable for policy optimization through the elimination mechanism, while verifying whether the framework can still maintain performance superior to baselines even when the initial AFS quality is poor-thereby demonstrating its core advantage of strong robustness that does not rely on specific high-quality I.IM outputs.
Although the original PoRSE
18

---

Published as a conference paper at ICLR 2026
As shown in Table 10 even with the randomly assembled invalid AFS its Maximum Training Success (MTS) on three representative tasks (BlockStack, PushBlock, and LiftUnderarm) is still significantly superior to that of baseline methods including Sparse Hu- man, Eureka, and Roska. Specifically, in the BlockStack task, the MTS of DASF--FSSRRaddo (0.680=0.080) is 2.67 times and 4.59 times that of Eureka (0. (0.25420.113) 119) and Roska (0. (0.14420..544 1544, respectively, In the PushBlock task, its performance (0. (0.324+0.034) far exceeds all baselines (the highest-performing baseline, Roska, only achieves 0.069=0.049). In the LiftUnderarm task, its MTS (0.802+0.009) is also better than that of Eureka (0.739 166) and Roska (0. (0.608+0.211), Mean- while, although
is slightly inferior to the original PoRSE (which relies on valid AFS generated by LLMs), during the experiment, the framework dynamically eliminates in- valid dimensions through the elimination mechanism and adaptively reduces the exploration reward coefficient via the L.L to avoid interfering with policy training. Ultimately, it still maintains signif- icant performance advantages in the scenario of random AFS, fully verifying the framework's ' fault tolerance to AFS dimension invalidity and the critical role of the elimination mechanism.
A.2.7 DIVERSITY OF DIMENSIONS IN AFS ABLATION EXPERIMENT AND ANALYSIS
Table 11: Maximum Training Success (MTS) Comparison of PoRSE and PoRSE-Prompt on AFS Dimension Diversity Experiments
Method
FrankaCabinet
OpenInward
OpenOutward
PoRSE-Prompt
0.951 = 0.031
0.264 + 0.044
1.000 + 0.000
PoRSE
0.957 + 0.049
0.283 + 0.386
1.000 + 0.000
To investigate the regulatory effect of Prompt design on the scope of affordance state space (AFS) dimensions generated by large language models (LLMs), as well as the impact of including dimen- sions not directly related to the task goal on policy training, this ablation study designs a "Prompt- guided group" (denoted as PoRSE- Prompp) In the original PoRSE framework, the AFS dimensions generated by LLMs are mostly centered on the task goal (e.g.. focusing on task-directly relevant dimensions such as "door handle distance" in door control tasks). However, practical applica- tions, it is necessary to verify two key points: first, whether I.MMs can expand the scope of AFS dimensions (to include dimensions not directly related to the task, such as "cxploration of the un- known position of the handle") through Prompt guidance; second, whether this dimension diversity will interfere with task completion. The core motivation of the experiment is to clarify the abil- ity of Prompt design to regulate AFS composition, while verifying the framework's 's compatibility with AFS dimension redundancy -i..., whether mask-irrelevan dimensions will impair task success rate-thereby providing experimental support for the flexibility of AFS design.
As shown in Table 11 there is no significant difference in Maximum Training Success (MTS) be- tween PoRSE-Prompt and the original PoRSE across three representative tasks (FrankaCabinet, OpenInward. and OpenOutward). Specifically, in the FrankaCabinet task, the MTS of PoRSE- Prompt (0.951 only shows slight decrease of 0.6 percentage points compared to the orig- inal PoRSE (0.957=0.045) 049). In the OpenInward task, its MTS (0.264+0.044) is close to that of the original PoRSE (0.283+0.386) with weaker data volatility. Notably, in the OpenOutward task, both methods achieve a 100% success rate (1.0=0.0). These results indicate that after adding guiding statements (e.g., "explore the unknown position of the handle") to the Prompt, the AFS generated by LLMS includes dimensions not directly related to the task; however, these redundant dimensions do not interfere with policy training, as reflected by the insignificant change in task success rate. This finding not only verifies the regulatory role of Prompt design in the scope of AFS dimensions but also proves that the PoRSE framework has good compatibility with AFS dimension diversity, providing an experimental basis for the flexible design of AFS in subsequent studies.
A.2.8 BIN COUNTS ABLATION EXPERIMENT AND ANALYSIS
To clarify the rationality of the "bin count" parameter in the exploration bonus generation process and address potential ambiguity regarding this parameter, this ablation study conducts verification on the bin count during Affordance State Space (AFS) discretization. As shown in the main text,
19

---

Published as a conference paper at ICLR 2026
Table 12: Maximum Training Success (MTS) Comparison of PoRSE with Different Bin Counts for Exploration Bonuses
Method
BlockStack
PushBlock
LiftUnderarm
PoRSE (1000 bins)
0.753 = 0.210
0.378 = 0.022
0.952 0.015
PoRSE (2000 bins)
0.708 = 0.056
0.335 = 0.103
0.950 0.023
PoRSE (3000 bins)
0.763 = 0.266
0.321 = 0.027
0.922 0.079
the calculation of the exploration bonus (Rt)) relies on converting the continuous AFS into discrete states via the discretization functionD(S..), and the bin count directly determines the discretization gramularity. Although the default bin count mentary materials, the impact of this parameter on performance had not been verified. Three bin counts (1000, 2000, and 3000) were selected as variables in the experiment, with core motivations as follows: on the one hand, to confirm whether the bin count significantly affects the task success rate (Maximum Training Success, MTS) and avoid result biases caused by subjective parameter lection; on the other hand, to provide experimental evidence for adopting 1000 bins as the default setting, ensuring the scientific validity and reproducibility of the framework parameters.
explicitly stated as 1000 in the previous supple-
As shown in Table 12, different bin counts have no significant impact on the MTS of PoRSE across the three tasks (BlockStack,, PushBlock, and LiftUnderarm). Specifically, in the BlockStack task, the MTS values of PoRSE with 1000 bins (0.753=0.210), 2000 bins (0. (0.708=0.056), and 3000 bins (0.763++0.66) fluctuate within the range of 0.708 to 0.763, with no obvious performance superi- ority or inferiority. In the PushBlock task, although PoRSE with 1000 bins achieves the optimal performance (0.378=0.022). the performance gaps between 2000 bins (0.335#0. 103) and 3000 bins (0.02)+0.0277 are all within 0.05, and there is no statistical significance in these differences. In the LiftUnderarm task, the MTS values of all three settings maintain high level of 0.922 to 0.952, with only PoRSE with 3000 bins (0.922+0.077 (0079) showing slight fluctuations due to excessively fine bin- ning granularity. This result confirms that the bin count within the range of 1000 to 3000 does not exert critical impact on policy training effectiveness. Therefore, selecting 1000 bins the default setting not only meets the requirement of AFS discretization but also avoids computational resource waste caused by excessive bins, achieving a balance between performance and efficiency.
A.2.9 NORMILIZATION ABLATION EXPERIMENT AND ANALYSIS
Maximum Training Success (MTS) Comparison of PoRSE with Different Reward Coeffi- cient Sum Scales
Table 13:
Method
BlockStacc
PushBlock
LiftUnnerarm
PoRSE (Normalized-to--1
0.744 + 0.201 0.753 + 0.210
0.353 + 0.077
0.957 + 0.020
PoRSE
0.378 + 0.022
0.952 0.015
To explain the rationality of the design where "the sum of coefficients is fixed to 2" in the total reward formula and verify the impact of reward coefficient scale selection performance, this ablation study designs a control group with "coefficient sum normalized to 1" (denoted as PoRSE- Normalized-to- 1). As shown in the main text, the total reward formula of the PoRSE framework is R = 3 R + (2 - B) R2 (where 3 is the weight of the exploration bonus R., 2 - 3 is the weight of the goal-oriented reward Rg. and the sum of the coefficients is fixed to 2). This design is essentially a matter of scale selection. The experiment has two core motivations: first, to verify whether there is a significant difference Maximum Training Success (MTS) when the sum of coefficients is "I" versus "2", eliminating the risk of scale dependence; second, from the perspective of optimization efficiency, to prove that "independent optimization of two coefficients (including a optimization (fixed scale, only
scale variable)" is mathematically equivalent to "single-coeffice optimizing 3)". If the scale has no significant impact, the scale variable can be omitted and only a single coefficient optimized, greatly simplifying the framework's optimization process.
20

---

Published as a conference paper at ICLR 2026
As shown in Table 13 the impact of the coefficient sum being 2 (original PoRSE) versus (PoRSE- Normalized-to- on MTS is minimal. Specifically, in the BlockStack task, their MTS values are 0.753=0.210 and 0. 0.744=0.201 respectively, with difference of only 0.009 and no substamtial perfor- mance gap. In the PushBlock task, although the original PoRSE (0.378+00.022) is slightly superior to the normalized group (0.353=0.077), the gap is within 0.03 and there is no statistical significance. In the LiftUnderarm task, the normalized group nal PORSE (0.952=0.0155. This result fully confirms that the sum of reward coefficients, within the scale range of "1" or "2", does not exert critical impact on policy training effectiveness. Therefore, the design of the original PORSE-" "fixing the sum to 2 and only optimizing the single coefficient - only ensures performance stability but also avoids computational redundancy caused by multi-variable optimization, achieving a balance between performance and optimization efficiency.
even slightly outperforms the origi-
A.2.10 COMPONENTS OF PORSE ABLATION EXPERIMENT AND ANALYSIS
To verify the necessity of the three core components of the PoRSE framework-godde goal-oriented re- ward, exploration bonus, and reward-policy co-optimization and to demonstrate that the frame- work is not simple superposition of existing modules, this ablation study systematically com- pares PoRSE with representative methods in the field (Eureka, ROSKA, ROSKA+CE) in terms of component composition and performance. As stated in the main text, the core innovation of PoRSE lies in using "dynamic policy feedback" as a link to integrate the three components into an organic whole, rather than treating them isolated parts. In contrast, existing methods ei- ther lack exploration bonuses (Eureka, ROSKA) or only achieve simple component aggregation through "traditional count-based exploration (CE) ROSKA" (ROSKA+CE), with neither realiz- ing inter-component synergy. Three BlokksBloocSttckk PushBlock, and fnftunddrramm--veerin both "complex manipulation" and "high-efficiency exploration requirements" were selected for the experiment. The core objectives are: to prove the simplicity (only three core components) and neces- sity of PoRSE's component composition through performance differences caused by the presence or absence of components; and to highlight the critical role of "component synergy" (rather than "simple superposition") in achieving performance breakthroughs by comparing with ROSKA+CE.
Table 14: Component Composition Comparison of PoRSE with State-of-the-Art Methods
Methods
Goal Reward
Exploration Bonus
Reward-Police Co-optimization
Eureka
v
x
x
ROSKA
V
X
ROSKA+CE
V
PoRSE
As seen from the component composition [Table[1,, PoRSE adopts minimalist thrre-component architecture," with each component being indispensable for efficient skill learning. Firstly, the goal- oriented reward serves as the foundation for task guidance-all all compared methods include this component (marked with V.. and its absence would result in no clear optimization direction, a consensus in robotic RL tasks. However, Eureka, which relies solely on goal-oriented rewards (lacking exploration and co-optimization), only achieves an MTS of 0.025=00.08 in the PushBlock task, struggling to escape local optima. Secondly, the exploration bonus is key to efficient explo- ROSKA, which lacks this component, achieves an MTS of 0.608+0.211 in the UidrUnnd task, significantly lower than PoRSE (0. (0.95200.01) which includes an exploration bonus. Although ROSKA+CE adds an exploration bonus, it adopts traditional count-based exploration irrelevant to the task, leading to performance far inferior to PoRSE. Thirdly, reward-polic co-optimization is central to performance enhancement- Eurcka, without co-optimization, cannot dynamically adjust the adaptability between rewards and policies, resulting in the worst performance in complex tasks (e.g.. BlockStack). These three components together form a minimalist yet necessary component system for PoRSE; the absence of any one component makes efficient skill learning unattainable.
The core advantage of PoRSE lies not in the "presence or absence" of components, but in the mutu- ally reinforcing effect formed between components through "dynamic policy feedhack" effect
21

---

Published as a conference paper at ICLR 2026
Table 15: MTS Performance Comparison of PoRSE with State-of-the-Art Methods (mean + standard
deviation)
Methods
BlockStack
PushBlock
LiftUnderarm
Eureka
0.254 0.119
0,025 = 0.008
0.739 0.166
ROSKA
0.148 + 0.154
0.069 + 0.049
0.608 = 0.211
ROSKA+CE
0.217 0.056
0.129 = 0.040
0.664 0.072
PoRSE
0.753 0.210
0.378 + 0.022
0.952 0.015
clearly reflected in the experimental results. On the one hand, goal-oriented rewards anchor the direction for exploration bonuses: PoRSE's exploration bonus is based on task-relevant Affordance State Space (AFS) generated by L.LMs, rather than the "goal-agnostic count-based exploration" of ROSKA+CE. This focuses exploration resources task-critical dimensions (e.g., "block stacking height" in BlockStack), avoiding ineffective exploration. On the other hand, exploration bonuses expand the optimization space for goal-oriented rewards: in the PushBlock task, PoRSE discov- ers the "optimal state for bimanual collaborative pushing" through exploration bonuses, improving the optimization efficiency of goal-oriented rewards. MTS (0.378+0.022)) is 2.93 times that of ROSKA+CE (0..29.0.040) Meanwhile, seward-police co-optimization (via the IPG process) dynamically balances the two components: it adjusts the 33 coefficient based on real-time policy performance, enabling adaptive matching between goal-oriented behavior and exploration as the policy evolves. This positive cycle of "goal-anchored exploration, exploration feeding back to goal- oriented rewards, and co-optimization maintaining balance" allows PoRSE to significantly outper- form existing methods relying on simple component superposition across all tasks, fully verifying the ireelaceabiliity of component synergy.
A.2.11 ABLATION EXPERIMENTAL DROP RESULTT
Table 16 summarizes the percentage performance drop each ablation relative the full PoRSE configuration across six representative tasks. The largest degradations arise when the fusion mech- anism (0rusion) is removed, with particularly severe declines on locomotion tasks such as Anymal () 2783.33%) and Quadcopter (J. 142.86%) and marked decrease on the long horizon sparse ma- nipulation task TwoCatch (4 95. 9.). This pattern indicates that policy grounded fusion central to PoRSE, since the dynamic blending of goal oriented rewards ( (R,) and exploration bonuses (R,) stabilizes learning and sustains final performance. Eliminating either (Rg) or (Re) also harms out- comes across domains. On Anymal the absence of (R₂) leads to (1 966.67%) and the absence of (Re) to (L 708.33%), which highlights the complementary roles of goal shaping and count based exploration provided by the AFS space and the IPG loop.
The ratio related terms (Ryaio) and (Bratio) have a milder yet meaningful effect. Their removal pro- duces moderate declines manipulation tasks, for example BlockStack (4 60.70%) without (Raaio) and PushBlock (4. 23.58% suggesting that ratio based normalization helps calibrate reward magni- tudes and maintain comparability of AFS counts when scenes involve multiple objects changing contact phases. Overall, the ablations support three conclusions that are consistent with the design goals of PORSE. The policy grounded fusion is the dominant contributor to robustness and efficiency, both (Ry) and (Re) are necessary to realize the intended synergy, and the ratio based components further stabilize learning in compositional settings.
A.2.12 REVOLVE-AUTO EXPERIMENT CONFIGURATION AND ANALYSIS
To ensure fair comparison with PORSE, the Retolve-Aut(Hazza et al. (2024) variant was aligned with PoRSE's core experimental settings: it generates 6 reward functions per iterative round and runs for 5 total iterative rounds. Table, 17 (Experimental comparison of PoRSE with SOTA frame- works including REvolve) supplements the experimental results of the REvolve framework four representative tasks (Humanoid, Liftunderarm, BlockStack, TwoCatch) to further verify PoRSE's superiority over state-of-the-art (SOTA) reward-design frameworks. For fair comparison, made
22

---

Published as a conference paper at ICLR 2026
Table 16: Performance drop (%) of each ablation compared with the full PoRSE framework.
Method
Anymal
Quadcopter
Franka
BlockStack PushBlock
TwoCatch
PoRSE w/o Rg 966.7%
185.7%
7.740%
56.44%
7.230%
45.56%
PoRSE w/o R.
708.3%
171.4%
4.700%
47.80%
3.770%
44.70%
PoRSE w/o Otusion
2783%
142.8%
29.89%
19.92%
25.79%
95.13%
PoRSE w/o Rratio 450.0%
35.71%
1.150% 1.780%
60.70%
23.58%
14.90%
PoRSE w/o ratio
66.67%
7.140%
21.64%
2.830%
20.92%
Table 17: MTS (mean + std) across four tasks.
Method
Humanoid
Liftunderarm
BlockStack
TwoCatch
Bureka
6.534 1 1.933
0.739 1 0.166
0.254 0.119
0.001 1 0.001
ReVolve
7.894 + 0.635
0.814 + 0.079
0.342 0.054
0.000 + 0.000
PoRSE
8.454 + 0.343
0.952 0.015
0.753 0.210
0.349 + 0.063
key adjustments the original REvolve setup: since the original REvolve relies on human partici- pation in its reward optimization loop, we removed human intervention and migrated the complete Auto-REvolve framework into the Eureka pipeline (denoted as Revolve-Auto.. This adaptation retains the evolutionary processes and prompt information exactly described in the original RE- volve paper, while aligning evaluation metrics (Maximum Training Success, MTS) with those used for PoRSE.
The table includes three methods for comparison: Eureka (LLM-drriven reward generation), Revolve-Auto (human-free REvolve adaptation), and PoRSE (our proposed framework). The sults show that PoRSE comprehensively outperforms the other two methods across all tasks: In the Humanoid task, PoRSE achieves MTS of 8.454+0.343. surpassing Revolve-Auto (7.894+0.635 and Fureka (6.534+1 ,933); in Liftunderarm, PoRSE's MTS (0.952+0.015) outperforms Revolve- Auto (0.814+0.079) and Eureka (0.739=0.166) by notable margin; in BlockStack. PoRSE's per- formance (0.753+0.210) is more than twice that of Eureka (0.25420.11) and nearly double that of Revolve-Aute (0.342=0.054); most notably, in the highly challenging TwoCatch task (requiring synchronized bimanual coordination), PoRSE achieves a breakthrough MTS of 0.349+0.063 while both Revolve-Auto and Eureka fail to make meaningful progress (MTS 0 and 0.001+0.001, re- spectively).
This performance gap stems from core design differences: Revolve-Auto, like the original RE- volve, focuses primarily on evolutionary goal-oriented reward design but lacks task-aware explo- ration mechanisms, limiting its effectiveness in high-dimensional or sparse-reward tasks. In contrast, PoRSE's
Grounding (IPG) mechanism dynamically adjusts rewards and ex- ploration bonuses based on real-time policy performance, while its Affordance State Space (AFS) enables relevaelevant exploration ttese synergistic designs drive sustained policy optimization.
A.2.13 INTUITION FOR LL
This section elaborates on the reasoning underlying the large language model (LeMM-predicted pol- icy fusion coefficient a and reward weight coefficient & in the PoRSE framework, as well as the numerical evolutionary trends of these coefficients across training iterations.
The intuition for leveraging LLMs to predict a and ( originates from the model's inherent reasoning and data analysis sapabiliiee--stengee validated in prior work focused on LLM-driven decision- making [1], self-composed reasoning structures [2], and enhanced mathematical reasoning via code integration [3]. In the PoRSE framework, we input time-series training data (e.g., Maximum Train- ing Success (MTS) over epochs) of the current policy to the LLM, allowing the model to assess the policy's 's training state (c.g.. whether is trapped in a local optimum or nearing convergence). Rather than generating single numerical value for and 3, the L.LM produces set of candidate
23

---

Published as a conference paper at ICLR 2026
outputs; this multi-candidate design mitigates biases from isolated predictions, thereby improving the stability of coefficient selection. The full prompt template guiding LLM predictions of a and documented later in this section.
To characterize the evolutionary patterns of a and B (as noted in the main text L302-304), we col- lected their values across training iterations for representative robotic tasks. Since PoRSE alternates between optimizing a (policy fusion coefficient) and 3 (reward weight coefficient) across iterations, tracked 6 total iterations: data were gathered Iterations 1, 3, while 3 data were collected at Iterations 0, 2, 4. The collected values for each task are presented in Table 18
Tasks
Iterl
Iter3 a
Iter5
IterO /
Iter2 /
Iter4 B
Humanoid
0,142
0.473
0.815
1.725
1.363
0.784
Anymal
0.137
0.468
0.809
1.572
1.191
0,793
Quadcopter
0.214
0.562
0.863
1.734
1.462
0.831
BlockStack
0.218
0.571
0.872
1.673
1.184
0,842
Liftunderarm
0.275
0.644
0.893
1.845
1.763
0.872
CatchAbreast
0.281
0.652
0.901
1.881
1.282
1.064
PushBlock
0.332
0.715
0.934
1.823
1.371
0.893
UndCaarn
0.341
0.723
0.942
1.942
1.684
1.085
Table 18: Numerical values of a and 3 across training iterations for representative tasks.
Analysis of Table T8 reveals consistent evolutionary trends across tasks: a values increase mono- tonically across iterations, indicating that PoRSE increasingly prioritizes inheriting parameters from the pererrrrfrming historical policy as training progresses--strenthhennii -ttrngtteenigg policy consistency and leveraging prior effective learning progress. In contrast, 3 values decrease across iterations, flecting a strategic adjustment: early stages use higher 3 to emphasize exploration bonuses (aiding escape from local optima), while later stages reduce 3 to prioritize goal-oriented rewards (refining task completion performance).
A.3 EXPERIMENTAL SETUP
In this section, present the detailed experimental parameter setup for all methods in this paper, provide an introduction to the IsaacGym tasks, outline the computing infrastructure, and detail the calculation of total training epochs (TTE). The PoRSE code will be released at the following URL: https://annoymoont how.eymouuuuoou....ooomoomm sci wwt.:nscccceeeeeeeeeeeeee
A.3.1 DETAILED EXPERIMENT PARAMETER SETTING
Table 19: Comparison of key parameters among different methods in the paper.
Reward Functions
Total Training
Max Policy Training
Method
Iterations
per Iteration
Epochs (TTE)
Epochs per Iteration
PoRSE
5
90,000
3,000
ROSKA
90,000
3,000
Eureka
90,000
3,000
LLMCount
6
90,000
15000
The specific configurations for all methods in this paper are follows:
ROSKA: The ROSKA DOSKAHuamd Huang et al. (2025) method also conducts 5-iteration optimization, gener- ating 6 reward functions in each iteration and training the corresponding policies in parallel. To ensure fairness in computational resources, we increased the number Bayesian optimization sam- ples from the default 12 to 14, while other parameters remain per the original paper.
24

---

Published as a conference paper at ICLR 2026
Anymal
FrankaCabinet
Quadcopter
Humanoid
Pen
LifUnndeaam
SwingCup
CatchUnderarm
CatchAbreast
BlockStack
PushBlock
Kettle
BottleCap
CatchOver2
Closelmward
Scissors
OpenOutward
GraspAndPlace
Over
CloseOutward
Openinward
ReOrientation
TwoCatch
Switch
Figure 9: The 24 reinforcement learning task example images selected the experimental section of this paper.
Euereka: The EurekaMa et al. (2023) method employs the same 5-iteration structure, generating 6 reward functions per iteration and allocating 3000 training epochs to each policy for thorough optimization.
LLMCount:
Sarukkaii et al. (2024) method used in the ablation study generates 6 progress functions zero-shot through the large language model, with each function allocated 15,000 training epochs to ensure balanced computational resources.
The LLMCoun Sarukkai
Human and Sparse: For the Human(Makvvyycuuk et al. (2021) and Sparse al. (2023) meth- ods, we used their default experimental configurations, conducting policy training for 3000 epochs.
PoRSE: The detailed parameter configuration of our method has been explained in section
A.3.2 CALCULATION TTE
have detailed statistics on the TTE for the different methods discussed in this paper. For each task, epoch the statistical unit to measure each method. For the same reinforcement learning task, the policy will interact with the environment in the same epoch for the same number of environment steps. We count the total number of epochs spent on each method during the iteration process to ensure fairness in computing resources.
All compared methods-Fureke ROSKA, ILMCount, and PoRSE are designed lar computational budget, each totaling 90,000 training epochs. While their approaches differ in iteration counts, reward function generation, and progressive training schemes, they maintain com- putational parity to ensure fair comparison in evaluation.
simi-
A.3.3 COMPUTING PLATFORM
All experiments in this study were conducted on computer system rumning Ubuntu 22.04.4 LTS, equipped with eight RTX 4090 GPUs. Since the training time required for training on different tasks is different, for intuitive comparison, we use the total training epochs in each task for comparison. The detailed comparison of Total Training Epochs(TTE) calculations between Eurcka, ROSKA, and PoRSE methods is follows.
A.3.4 BENCHMARK DESCRIPTION
In this paper, experiments involve 24 robot skill learning tasks, including 4 tasks from IsaacGym and 20 tasks from Bi- Bi-DexHands. The environment task names, task description, and task fitness function F of the 24 tasks are shown in the table 20
25

---

Published as a conference paper at ICLR 2026
Table 20: The introduction to 24 tasks in the experiments of this paper.
IsaacGym Environments
Environment Name(obs dim, action dim)
Task description
Task fitness function F Quadcopter (21, 12)
To make the quadcopter reach and hover near fixed position -cur-dist
FrankaCabinet (23, 9)
To open the cabinet door
1 [cabinet.pos > 0.39]
Anymal (48, 12)
To make the quadruped follow randomly chosen Jr and yaw target velocities
(Linveleerror angvelerror)
Humanoid (108, 21)
To make the humanoid run as fast as possible
cur dist prev prev_dist dist
Over (398, 40)
This class corresponds to the HandOver task. This environment consists of two shadow hands with palms facing up. opposite each other, and an object that needs to be passed. In the beginning, the object will fall randomly in the area of the shadow hand the right side. Then the hand holds the object and passes the object to the other hand. Note that the base of the hand is fixed. More importantly, the hand which holds the object initially can directly touch the target, nor it directly roll the object to the other hand, so the object must be thrown up and stays in the air in the process
[dist < 0.03]
DoorCloseInward (417, 52)
This class corresponds to the DoorCloselnward task. This environment requires a closed door to be opened and the door can only be pushed outward initially open inward. Both these two environ- ments only need do the push behavior, so relatively simple.
[door handle dist
0.5]
DoorCloseOutward (417, 52)
This class corresponds to the DoorCloseOutward task. This environment also requires a closed door to be opened and the door only be pushed inward or initially open outward, but because they can't complete the task by simply pushing, they need to catch the handle by hand and then open or close it, so it is relatively difficult.
[door handle dist
0.5]
DoorOpenInward (417, 52)
This class corresponds the DoorOpenInward task. This environment also requires an opened door to be closed and the door can only be pushed inward or initially open outward, but because they task by simply pushing, they need catch the handle by hand and then open or close it, so it is relatively difficult.
can't complete
[door handle dist 0.5]
26

---

Published as a conference paper at ICLR 2026
DoorOpenOutward (417,52)
This class corresponds the DoorOpenOutward task. This environment requires an opened door to be closed and the door can only be pushed outward or initially open inward. Both these two environments only need do the push behavior, so it is relatively simple.
(door_handleedi
0.5]
Scissors (417, 52)
This class corresponds to the Scissors task. This environment involves hands and scissors; we need to use two hands to open the scissors.
[dof-pos 3]
SwingCup (417, 52)
This class corresponds to the SwingCup task. This environment involves two hands and dual- handled cup; we need to use two hands to hold and swing the cup together.
1 [rot_diss < 0 785]
Switch (417,52) 52)
This class corresponds to the Switch task. This environment involves dual hands and a bottle; we need dual hand fingers to press the desired button.
[1.4 4 (left.switch.2
right.switch.
0.05]
Kettle (417, 52)
This class corresponds to the PourWater task. This environment involves two hands, kettle, and a bucket; we need to hold the kettle with one hand and the bucket with the other hand, and pour the water from the kettle into the bucket. the practice task in Isaac Gym, we use many small balls simulate the water.
bucket kettle.spout < 0.05]
LiftUnnerarm (417,
This class corresponds to the LiftUnderarm task. This environment requires grasping the pot handle with two hands and lifting the pot to the designated position. This environment is designed to simulate the scene of lift in daily life and is practical skill.
[dist 0.05]
Pen (417,52) 52)
This class corresponds to the Open Pen Cap task. This environment involves hands and pen; we need to use two hands to open the pen cap.
1 [5
pen cap pen body! > 1.5]
Bottle Cap (422, 52)
This class corresponds to the Bottle Cap task. This environment involves two hands and bottle; we need hold the bottle with one hand and open the bottle cap with the other hand. This skill requires the cooperation of two hands to ensure that the cap does not fall.
[dist > 0.03]
CatchAbreast (422, 52)
This class corresponds to the Catch Abreast task. This environment consists of two shadow hands placed side by side in the same direction and an object that needs be passed. Compared with the previous environment which is more like passing objects between the hands of two people, this environment is designed to simulate the two hands of the same person passing objects, requiring more hand translation and rotation techniques.
[dist 0.03]
CatchOver2Underaam (422, 52)
27

---

Published as a conference paper at ICLR 2026
This class corresponds to the Over2Underarm task. This environment is similar to Catch Under- arm, but with object in each hand and the corresponding goal the other hand. Therefore, the environment requires two objects to be thrown into the other hand simultaneously, which demands higher manipulation techniques than single-object environments.
[dist < 0 0.03]
CatchUnderarm (422, 52)
This class corresponds to the Catch Underarm task. In this task, two shadow hands with palms facing upwards are controlled to pass object from one palm to the other. The difficulty is increased by unfreezing the hands' translation and rotation degrees of freedom the action space.
[dist < 0.03]
ReOrientation (422, 40)
This class corresponds to the ReOrientation task. This environment involves two hands and two objects. Each hand holds an object and need to reorient the object to the target orientation.
[rot_dist 0.1]
GraspAndPlace (425, 52)
This class corresponds to the GraspAndPlace task. This environment consists of dual hands, object, and a bucket. The task requires picking up the object and placing it into the bucket.
1 [ [[block bucket
< 0.2]
BlockStack (428, 52)
This class corresponds to the Block Stack task. This environment involves dual hands and two blocks; we need stack the blocks into stable tower structure.
[goalldist.1 < 0.07 and goal.dist.2 < 0.07 and 2.dist_1 < 0.05]
PushBlock (428, 52)
This class corresponds
PushBlock task. This enviromment involves two hands and two blocks; need to use both hands to reach and push each block to its designated goal position simultanc- ously.
[left_dist <= 0.1 and right.dist <= 0.1]
TwoCatchUnderamm (446, 52)
This class corresponds to the TwoCatchUnderarm task. This environment extends the Catch Under- arm task with dual-object manipulation: each hand holds object and must throw it to the opposite hand's target position, requiring synchronized bimanual coordination.
1 [goal dist 1 + goal dist 2 0.06]
A.3.5 ENVIRONMENT STEP AND PPO UPATE COUNTS
Table 21 (Training budget and compute parity across tasks) is designed to verify the consistency of training resources and computational costs across PoRSE and baseline methods (Eureka, ROSKA), ensuring the fairness of training efficiency comparisons. The table includes 5 representative robotic tasks (Humanoid, FrankaCabinet, Anymal, Quadcopter, Liftunderarm) and the average metrics across all tasks, with columns covering key training budget indicators and GPU time costs.
In terms of training budget uniformity: The "Total epochs" column shows that all tasks maintain fixed total of 90,000 training epochs for all methods, which aligns with the earlier statement that "PoRSE incorporates mutation and elimination mechanisms but still maintains the same total train- ing epochs other basclines." " For "Total steps" and "Total PPO updates" (core indicators of training resource input), tasks (Humanoid, FrankaCabinet. Anymal, Quadcopter) share identical values (1,440,000 env steps and 7,200,000 PPO updates), while Liftunderarm uses half the steps
28

---

Published as a conference paper at ICLR 2026
Table 21: Training budget and compute parity across tasks.
Tasks
Total env steps
Total PPO
Total epochs
GPU time (Eureka)
GPU time (ROSKA)
GPU time
updates
(PoRSE)
Humanoid
1,440,000
7,200,000
90,000
7.0
7.1
7.0
FrankaCabinet
1,440,000
7,200,000
90,000
4.5
4.4
4.6
Anymal
1,440,000
7,200,000
90,000
3.5
3.5
3.6
Quadcopter
1,440,000
7,200,000
90,000
3.5
3.4
3.4
Liftuddrarm
720,000
3,600,000
90,000
5.6
5.6
All tasks
5.5
5.7
5.8
(720,000) and updates (3,600,000) this difference is task-specific (due Liftunderarm' lower state dimensionality) and applied uniformly to all methods, ensuring no bias in resource allocation.
In terms of computational cost parity: The "GPU time [h]" columns for Eurcka, ROSKA, and PoRSE demonstrate minimal differences across tasks. For example, in the Humanoid task, GPU times are 7.0h (Eureka), 7.1h (ROSKA), and 7.0h (PoRSE); in FrankaCabinet, they 4.5h, 4.4h, and 4.6h respectively. Even at the average across all tasks, the gap remains negligible (5.5h for Eureka, 5.7h for ROSKA, 5.8h for PoRSE). This confirms that PoRSE does incur additional computational overhead despite its mutation and elimination mechanisms. Combined with the earlier note that "PoRSE's algorithmic complexity scales linearly with the number of iterations (O(n))", the table further validates that PoRSE achieves superior performance without sacrificing training efficiency.
A.4 DISCUSSION
A.4.1 DISCUSSION ON DEPLOYMENT
The deployment of the PoRSE framework on physical robots is centered on the core design principle of "decoupling of state representation in the training phase and decision- flcision-making in the deployment phase," " ensuring the framework can adapt to physical systems without relying on structured state inputs from simulation environments. The Affordance State Space framework- - functions exclusively in the training phase, where it is used to construct task-relevant exploration bonus models and low-dimensional state representations learning. When the policy is deployed on physical robots, however, it can directly take raw sensor data from the robot as inputs for pocisiinnmaking typical examples of such raw sensor data include RGB-D images captured by vision cameras and torque and angle information from joint proprio- ceptive sensors. Discrepancies
core component of the
facilitate efficcintt policy
dynamics and sensor noise between simulation and real-world environments can be addressed through transfer learning: specific approach for this adaptation is to fine-tune the trained policy with a small amount of physically collected data. For core AFS features potentially required in real-world scenarios, examples of these features include relative dis- tances between objects and grasping posture angles, and mature existing perception technologies offer efficient solutions to obtain them. For instance,intel RealSense series depth cameras can be used to accurately estimate the spatial positions of objects, while tactile sensors at the robot end or joint torque feedback extraction modules need to be designed. If real-time reward adjustment is required during physical deployment, a lightweight AFS can also be dynamically constructed based on the aforementioned perception data. Furthermore. to enhance the framework's 's adaptability to unstructured real-world scenarios, common cases of such scenarios include multi-object manipulation in cluttered environ- ments and object occlusion scenarios. Lightweight dision-language models can be integrated into the existing framework pipeline future work. Representative models of these lightweight vision- language models are CLIP and Grounding DINO, and their integration enables the system to parse task semantics from real-time visual observations of physical robots. This parsing process helps generate reward rules and AFS dimensions that better align with real-world needs, thereby laying technical foundation for the application of the PoRSE framework on various physical robots. Exam- ples of these physical robots include robotic arms and dexterous hands.
infer grasping stability this means additional complex feature
29

---

Published as a conference paper at ICLR 2026
A.4.2 DISCUSSION ON REPRODUCIBILITY
The reproducibility of the PoRSE framework ensured through multi-dimeesionall design and infor- mation transparency, providing clear pathway for the verification and extension of subsequent search. At the experimental level, the framework achieves stable performance improvements across 24 tasks covering dexterous manipulation and legged locomotion. The effectiveness of core mech- anisms, such as AFS construction and IPG dynamic optimization, has been validated through con- sistent experimental results, avoiding reproducibility biases caused by task specificity. The settings of key parameters are clearly justified: for example, AFS's default number of bins is set to 1000, this number used for exploration bonus calculation, and ablation experiments have confirmed that 2000 or 3000 bins have no significant impact on performance; configurations like the number of IPG iterations and the number of reward functions per iteration are detailed in the supplementary materials, and parameter sensitivity analysis results are also provided alongside to guide adaptive adjustments under different scenarios. At the implementation level, the framework will open-source complete code and experimental configuration files, including standardized prompt templates for templates require no task-specific tuning,
interactions between LLMs and the framework-suun and only task goal descriptions and environmental state variable codes need to be modified setup scripts for simulation environments like Isaac Gym and Hands, and complete process code for policy training and evaluation. This ensures researchers can quickly reproduce the experimental environment. Meanwhile, to address LLM dependency, the framework has verified that lightweight models like GPT-4o-mini can replace the original LLM while maintaining performance advantages, reducing reproducibility barriers caused by model access restrictions or computational resource lim- itations. In addition, core modules, such as the screening mechanism for reward-exploration com- binations and policy fusion logic, are implemented through modular design, with each component having independent functions and clear interfaces. Researchers can verify the functionality of each module individually, further enhancing the traceability of the reproducibility process.
A.4.3 DISCUSSION ON THE CENERALITY OF AFS
The generality of the Affordance State Space (AFS) in the PoRSE framework stems from the dual guarantee of "semantically driven design + dynamic adaptive mechanism," enabling it to function stably across different types of robotic tasks without extensive customization. From the perspective of interaction logie, AFS construction adheres to standardized prompt template specifications: when interacting with LLMs, only task goal descriptions and environmental state variable codes need to be replaced, while core components such as semantic mapping rules and dimension selection logic remain fixed. No task-specific prompt optimization is required, avoiding repeated development costs caused by task differences. Its built-in dynamic climination mechanism further enhances task adapt- ability: this mechanism continuously tracks the actual contribution of each AFS dimension policy performance, automatically selecting the exploration directions most aligned with task requirements in the current training phase. Even for unseen new tasks, such as collaborative multi-object sort- ing, or high-complexity tasks, such as dexterous hand grasping of irregularly shaped objects, it can quickly identify key state dimensions without manual intervention in dimension selection. Mean- while, AFS exhibits strong fault tolerance toward invalid dimensions generated by LLMs: when LLMs propose rask-irrelevant dimensions due to semantic understanding biases, the framework dy- namically reduces the weight of such dimensions through policy performance feedback, and com- bines adaptive adjustment of the exploration reward coefficient / to minimize the interference of ineffecttvee exploration
training. Experiments show that even when using randomly assembled AFS dimensions this setup is to simulate extreme noise scenarios- framework's performance on typical tasks such as BlockStack and PushBlock remains significantly superior to that of tra- ditional baseline methods. Furthermore, the consistent performance of AFS across different task types, including dexterous manipulation tasks like LiftUnderarm and legged locomotion tasks like Anymal walking, confirms that it can not only adapt to differences in task semantics but also be compatible with the state space characteristics of different robotic systems, providing core support for the task expansion of the framework.
A.4.4 DISCUSSION ON EXPANDING PORSE.
The method extensions of the PoRSE framework focus on addressing the adaptability challenges of complex real-world scenarios and further enhancing the framework's scalability, efficiency, and
30

---

Published as a conference paper at ICLR 2026
stability, with several promising directions worth exploring. First, expanding the Affordance State Space (AFS) to multi-modal state representations is a key riooriy-byy integrating multi-source sensory data such RGB images, depth information, and tactile feedback, and combining vision- language models, such as CLIP or Grounding DINO, which are referred to as VLMs for short, the framework can automatically extract task-relevant affordance features from unstructured environ- mental inputs. For example, in cluttered domestic scenes with object occlusion, multi-modal AFS can fuse visual semanties, which involves identifying "mug" vs. "book", and tactile cues, which entail distinguishing "hard" vs. "soft" surfaces, to avoid over-reliance on pre-defined environmental semantics, enabling more robust exploration for dexterous manipulation tasks. Second, extending grounding process- refferrd to IPG-to hierarchical reinforcement learning architectures, or HRL for short, can effectively tackle long-horizon tasks that require se- quential subtask decomposition, such as the "fetching ingredients, cutting, and cooking" process for a kitchen robot. By embedding IPG's dynamic reward-cxploration optimization into each sub- task layer, the framework can adjust the trade-off between global goal alignment and local subtask exploration- for instance, prioritizing exploration in the "cutting" subtask to find optimal knife an- gles while maintaining goal orientation in the "fetching" subtask, thereby improving the scalability of complex sequential tasks. Third, enhancing the elimination-exxaasionn mechanism with meta- learning strategies can reduce computational overhead: by leveraging experience from historical tasks, the framework can meta-learn the optimal mutation rate adaptation rule-for simple single- object grasping tasks, low mutation rate used to stabilize policy optimization, while for complex multi-object collaborative tasks, higher mutation rate is adopted to explore diverse heward-bonur combinations, minimizing trial-and-error costs. Finally, integrating uncertainty quantification into LLM-generated reward functions can mitigate instability in policy updates: by introducing Bayesian LLMs to estimate the confidence of reward rules-a typical case is that LLM-generated rewards in novel tasks with ambiguous semanties have lower confidence. Based on this, the framework can dy- namically adjust the weight of exploration bonuses whhuess-henn reward uncertainty is high, it increases exploration collect more state feedback for reward refinement; when uncertainty is low, it fo- cuses on goal-oriented optimization opiinization-avoddiinn policy oscillation caused by unreliable reward sig- nals. These extension directions mutually supportive: multi-modal state representations provide accurate state foundations for hierarchical RI,, while meta-learning and uncertainty quantification jointly enhance the framework's efficiency and stability, ultimately enabling PoRSE to adapt to more complex real-world scenarios such as industrial assembly and home service robotics.
the --policy-improveneee
1
1
1
1
1
FranccCCatrett
Lintnddeamm
Pan
Swingdap
Calchinder
CarchAlrea
Brocitadk
PranBlack
Kerta
BottwCap
CanchDer22
1.
0
10
0.5
Scasore
Opentitt
Crmpftice
OcerOur
Opents
RuCrienattonn
TwoDarth
Switch
Quadeopter
Humanood
LLMCouut
PORSE
Figure 10: MTS Comparison of LLMCount and PoRSE methods, PoRSE method achieved higher MTS on 23 tasks.
A.4.5 DISCUSSION ON BROADER IMPACT.
The development of PoRSE has broader implications for the field of robotics and AI. On the posi- tive side, PoRSE's ability to efficiently acquire complex robotic skills using LLM--gnnrated reward functions and directed exploration can accelerate the deployment of robots various industries, in- cluding manufacturing, healthcare, and logistics. This can lead to increased automation, improved productivity, and enhanced safety in work environments. Moreover, the reduced need for manual reward design makes robotic skill learning more accessible to researchers and practitioners without extensive domain knowledge. However, there are also potential negative impacts and ethical con- siderations. The reliance L.LMs raises concerns about the transparency and interpretability of the
31

---

Published as a conference paper at ICLR 2026
reward functions and exploration strategies. There is risk of inheriting biases present in the LLM training data, which could lead to unfair or unsafe robotic behaviors.
A.5 LIMITATION
Similar to other works utilizing large language models (LLMs) for content generation, our approach is also affected by the inherent instability commonly seen in LLM outputs. Even with identical prompts, the quality of generated functions can vary significandly, and in some cases, non-executable code may be produced. This instability is prevalent limitation of L.LSs rather than an issue unique to our approach, which can lead to a situation where, in each generation iteration (N=6 functions), only part of functions may be successfully usable for guiding policy optimization. However, with the rapid development and iterative updating of LLM technology, it is anticipated that the instability in reward design and code generation will be progressively mitigated, and the capabilities of LLMs in these aspects will be steadily enhanced. By adopting the latest LLMs, frameworks like ours have the potential to achieve better robot skill learning outcomes.
To address this common limitation and ensure the fairness of comparison results in this paper, we have implemented the following measures for the Eureka, ROSKA, LLMCount, and PoRSE meth- ods. For each iteration of functions generated by the L.M,, we conduct code running correctness tests ensure that all six functions generated in each iteration of the above methods can run cor- rectly. This helps avoid result deviations caused by the instability of LLM responses and ensures that the comparison results are fair and reliable.
A.6 PROMPT DETAIL.
Reward Function Initial Prompt Example
You are a reward engineer trying to write reward functions to solve reinforcement learning tasks as effective as possible. Your goal is to write a reward function for the environment that will help the agent learn the task described in text. Your reward function should use useful variables from the environment as inputs. As an example, the reward function signature be: [task_reward, wwwwarddiggnatureess.. _strnng Since the reward function will be decorated with @ @torch.jit.scrr please make sure that the code is compatible with TorchScript (c.g., use torch tensor instead of numpy array). Make sure any new tensor or variable you introduce on the same device as the input tensors.
The output of the reward function should consist of two items: (1) the total reward, (2) dictionary of each individual reward component.
The code output should be formatted as python code string: """python...
Some helpful tips for writing the reward function code:
(1) You may find it helpful to normalize the reward to a fixed range by applying transforma-
tions like torcheexp to the overall reward or its components
(2) If you choose transform reward component, then you must also introduce a temper- ature parameter inside the transformation function; this parameter must be named variable in the reward function and it must not be an input variable. Each transformed reward com- ponent should have its own temperature variable
(3) Make sure the type of each input variable is correctly specified; a float input variable should not be specified as Tech.Tensor
(4) Most importantly, the reward code's input variables must contain only attributes of the provided environment class definition (namely, variables that have prefix self.). Under no circumstance can you introduce new input variables.
Reward Function Refinement Prompt Example
Please carefully analyze the policy feedback and provide new, improved reward function that can better solve the task. Some helpful tips for analyzing the policy feedback:
(1) If the success rates are always near zero, then you must rewrite the entire reward function (2) If the values for certain reward component are near identical throughout, then this means RL is not able to optimize this component as is written. You may consider
32

---

Published as a conference paper at ICLR 2026
(a) Changing its scale the value of its temperature parameter
(b) Re-writing the reward component
(c) Discarding the reward component
(3) If some reward components' magnitude is significantly larger, then you must re-scale its value proper range
Please analyze each existing reward component in the suggested manner above first, and then write the reward function code.
Mapping Function Initial Prompt Example
You are a reinforcement learning engineer trying to write mapping functions solve rein- forcement learning tasks as effectively as possible. Your goal is to identify the variables for the environment that are maximally relevant in the task described in text. Some tasks may have only single stage, and some tasks may have two separate stages.
You will be provided with definition of the observation space for reinforcement learning environment, and also provided with small set of helper functions that can be used to transform the variables the observation space. Write a function that returns variable most associated with task for each stage of the task.
This function can take as input any member of self defined in compputeoosservationn, and can apply any of the helper functions to any variables from self.obs_bbu to generate new derived features (ex. computing the distance between object and goal). If a single stage requires multiple mapping variables, average the variables.
Also return a bool for each variable that is True if task goal requires the variable to increase, and False if it requires the variable to decrease.
Mapping vars name stores the string names of the corresponding variables in order.
Mapping Function Refinement Prompt Example
Please carefully analyze the policy feedback and provide an improved mapping function to better solve the task. Here some tips for analyzing the policy feedback:
the success rates are consistently near zero, you must rewrite the entire mapping function.
If the values of mapping variable remain nearly identical throughout training, it means that the variable failed to effectively capture the task progress dynamics of the agent
You may consider:
(a) If one of the mapping variables in the current mapping function changes weakly during training and may not effectively quantify task progress, you should analyze its sensitivity and focus on whether the variable strongly correlated with the task goal?
(b) If one of the mapping variables has not changed all during the training process, indicating that the mapping variable cannot reflect the progress of the task. Please redesign more sensitive alternative variable, requirement:
1. Strong correlation with task objectives
2. can be decomposed into multi-stage indicators
(c) If one of the mapping variables does not sigmificantly contribute to task performance, please analyze its necessity. If there redundancy, please propose alternative mapping variables or directly eliminate the mapping variable
First, analyze each existing mapping variable using the guidelines above, then write an optimized mapping function.
Policy Fusion Search Prompt
You are an expert in reinforcement learning, skilled in analyzing training data for policy models.
As we know, in reinforcement learning, the policy continuously learns under the guidance of rewards to eventually achieve the task objective.
Currently, we train a policy over multiple stages. In each stage, corresponding rewards are used to guide the learning process. However, directly using the parameters of policy trained
33

---

Published as a conference paper at ICLR 2026
with the rewards of the previous stage the initial parameters for the next stage often leads to learning issues due to changes in the rewards.
To mitigate this problem, sum of the parameters of the entire policy model and a random model:
policy fusion method. Specifically, perform weighted
fusion = a policy + (1 - a:) raadom.
The current problem is determining the fusion ratio a. will provide you with the training data of the policy model from the previous stage. Please analyze the training results to determine the appropriate fusion ratio for the policy model.
Your task is to effectively analyze the training data of the current policy model. Here are some analysis techniques:
1. If the policy scores from the previous stage are consistently close to 0, it indicates that the rewards in the previous stage were ineffective in guiding the policy to achieve the reinforce- ment learning task objective. In this case, the fusion ratio should be set lower to ensure higher plasticity the policy model, allowing it to better adapt to the rewards in the next stage.
2. If the policy scores from the previous stage consistently increase, it indicates that the policy model well-guided by the rewards achieve the task objective. In this case, the fusion ratio a should be set higher to ensure the policy retains more effective experience.
3. If the policy scores from the previous stage remain constant but that the rewards did not improve the existing policy. In this case, the fusion ratio should be slightly reduced to increase the model's plasticity.
zero, indicates
Since a single fusion ratio may not be completely accurate, you will have five opportumities design this parameter. In these five attempts, you should:
1. Propose the most likely optimal fusion ratio a. 2. Ensure the ratios a are distributed evenly as possible in the range from avoiding clustering all five fusion ratios within narrow range.
reward balance Search Prompt
You are an expert in reinforcement learning, skilled in analyzing training data for policy models. As we all know, the training of a policy model requires effective guidance from rewards. At different stages of policy model training, adjustments should be made based on the characteristics different tasks.
Currently, we have two types of rewards jointly guiding the training of the policy model, so the coefficient ratio between these two rewards crucial.
The first reward is the dense reward designed by the large model B..MM_Rewrrd_Ree reward directly comes from the reward function designed by the large model, and its primary focus is always achieving the task objective guide policy training. However, whether the reward function designed by the large model can effectively guide the policy training uncertain. But the goal of this reward is known: to always guide the policy to achieve the corresponding reinforcement learning task objectives.
This
The second reward is the wragress-based exploration reward (Progress Reward Ratio). By utilizing the progress function designed by the large model for different reinforcement learn- ing tasks, the progress function converts nigh-dimensional environmental state information into low-dimensional progress variable information related to task completion.
This progress variable measures the task completion progress. By dividing the continuous progress variable into multiple numerical intervals, exploration rewards are based on the number of visits each interval. That is, the fewer visits to a certain progress interval, the higher the exploration reward. Conversely, the more visits, the lower the reward.
This encourages the policy model continuously advance the task completion progress.
34

---

Published as a conference paper at ICLR 2026
Now, I will provide you with the training data of the current policy model recent period. This training data includes each component of the reward function designed by the large model, each progress variable of the progress function, and the current task scores.
These data are presented in array format. During this training period, your task to analyze the training data effectively. Here are some analysis techniques:
1. If the current reinforcement learning task scores remain nearly the same, it indicates that the reward function designed by the large model cannot effectively guide the policy achieve the task objective. Therefore, the coefficient of the LLM Reward component should be slightly reduced.
the reinforcement learning task scores increase gradually during the training process, it indicates that the reward function designed by the large model, combined with the explo- ration reward, can effectively guide the policy to achieve the task objective. In this case, the coefficient of the LLM Reward component can be slightly increased.
Please determine the optimal ratio of these two reward components based on the information provided. The coefficient range for both rewards should be controlled between and 2. Since single attempt may not be precise, you will have five opportunities to design parameters.
these five attempts, you must not only propose the most likely optimal reward parameter ratios but also maintain relatively uniform search distribution, avoiding concentrating all five designs same parameter interval.
35


Reviews:
[{"review_id": "kzQFV364KZ", "reviewer": "Reviewer_KXzj", "summary": "This paper proposes PoRSE, a framework that unifies LLM-based reward shaping with task-relevant exploration for robot skill learning. An LLM generates (1) goal-oriented rewards and (2) an affordance state space (AFS) that maps high-dimensional states to discrete, task-meaningful features; a count-based bonus over AFS promotes exploration. PoRSE dynamically balances rewards and bonuses via a weighting β and accelerates training with an in-policy-improvement grounding (IPG) process. Across 24 tasks (Bi-DexHands, Isaac Gym), PoRSE outperforms sparse, human, Eureka, and ROSKA baselines, with ablations validating each component and robustness to different LLMs.", "questions": "1. I am wondering about the training efficiency. E.g., for one epoch, how much time is needed for the proposed PoRSE and other baselines?\n\n2. How sensitive is PoRSE to the quality of the LLM outputs? Does fine-tuning or prompt engineering significantly alter results?\n\n3. How do the LLMs perform with increasing task complexity or policy dimensionality? E.g., for long-horizon tasks or when envs get more complex.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. PoRSE provides a unified and generalizable structure that integrates reward shaping and exploration within an LLM-guided loop.\n\n2. Extensive Experiments: Evaluated across 24 robotic skill tasks (Bi-DexHands and Isaac Gym) with strong quantitative results — outperforming baselines in 23/24 tasks.\n\n3. Clear Practical Value: Avoids costly full retraining via policy inheritance, making it more computationally efficient.\n\n4. Comprehensive Ablations: Studies on β, α, mapping functions, and LLM robustness (e.g., GPT-4o-mini variant) show the framework’s adaptability and resilience.", "weaknesses": "1. The reliance on specific LLM outputs (e.g., DeepSeek-V3) introduces reproducibility and bias issues, not fully explored in the paper.\n\n2. Limited Real-World Evaluation: While the appendix outlines a feasible sim-to-real adaptation (fine-tuning with small datasets, RealSense/tactile sensors, and lightweight VLMs for affordance cues), no empirical real-robot results are yet presented. Thus, deployment remains conceptually described rather than experimentally validated.\n\n3. The paper asserts “nearly identical” cost to Eureka/ROSKA, but PoRSE trains K candidates per round with elimination/expansion and mid-round mutations (J new variants at 1500 epochs), plus alternating searches over β and α. This likely alters wall-clock and sample budgets vs. baselines. Provide exact environment steps, PPO updates, and GPU hours for each method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LWrEGA7FCJ", "reviewer": "Reviewer_JaxW", "summary": "PoRSE is a framework for robotic skill learning that synergizes LLM-generated goal-oriented rewards with task-aware exploration bonuses via an affordance state space (AFS), using in-policy grounding (IPG) for dynamic refinement and policy fusion. It iteratively filters reward-bonus pairs and adjusts trade-offs based on policy feedback, evaluated on 24 simulated tasks (IsaacGym, Bi-DexHands) with PPO, achieving superior MTS/HNS over baselines like Eureka, ROSKA, and LLMCount, including breakthroughs in hard manipulation tasks.", "questions": "1.What are the total LLM token costs per task/iteration, and how does efficiency scale to longer horizons?\n2.Are there evaluations on open-source LLMs (e.g., Llama) or weaker models beyond GPT-4o-mini?\n3.How robust is AFS to multi-modal inputs like vision; could it integrate VLMs?\n4.In cross-task extensions (A.4.4), what specific adaptations are needed for real environments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Effectively addresses exploration gaps in LLM rewards by integrating task-relevant AFS bonuses, with IPG enabling adaptive synergy without exhaustive retraining.\n2.Strong results across 24 diverse tasks, outperforming SOTA on 23/24 vs. LLMCount and achieving first successes in sparse-reward challenges like TwoCatch.\n3.Comprehensive ablations validate components (rewards, bonuses, fusion), robustness to LLM variants (e.g., GPT-4o-mini), and parameters (bins, normalization), with fair compute parity.\n4.Practical insights into policy-reward co-evolution, supported by visualizations and multi-seed stats.", "weaknesses": "1.Limited to simulation and no real-robot validation.\n2.High LLM dependence (GPT-4o) without cost analysis—multiple iterations with 6 rewards each could be token-expensive.\n3.Baselines like Eureka/ROSKA use same LLM but may not be optimally adapted; sparse/human rewards are weak strawmen in some tasks.\n4.AFS discretization assumes low-dim abstractions work universally, but ablations show sensitivity to randomness or prompts.\n5.Claims of \"breakthroughs\" in hard tasks lack comparison to non-LLM exploration methods like RND or ICM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EcJLBE7Mwg", "reviewer": "Reviewer_LtXN", "summary": "Building on LLM-based reward with evolutionary optimization over reward code (i.e., EUREKA), this method improves exploration by having an LLM compress the state space into a low-dimensional representation and add a count-based novelty bonus. The LLM then balances exploration and goal-oriented rewards by producing a set of weights based on the policy performance. To avoid a combinatorial sweep over weight–reward pairs, weak candidates are pruned early while promising ones are lightly refined. Finally, the policy is retrained with the refined reward (warm-started from the previous policy), and the LLM decides how to mix the new and prior rewards based on observed results.", "questions": "- Q1. L200 says it maps the robot’s state to a low-dimensional space, but the example (“distance to the door handle”) depends on the environment state. Is the mapping’s input the robot state only, or the full environment state?\n- Q2. Results are shown up to 5 iterations. How does performance evolve with more iterations—does it converge or diverge? With more iterations, can it solve harder tasks such as “ReOrientation” or “Switch” in Fig. 4?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper tackles a core RL challenge—exploration—by combining LLM with a classical novelty bonus. The approach is clear and leverages both strengths: LLMs can keep improving, while novelty rewards remain general and broadly applicable.\n- Presents rich analyses that identify which and how introduced components contribute.", "weaknesses": "- **Intuition for LLM-chosen $\\alpha$**\n  + The rationale for how the LLM predicts  $\\alpha$ isn’t clear. Could authors elaborate on the intuition, and if applicable, include the related prompt, as well as show how  $\\alpha$ evolves during training?\n\n- **Trends for $\\beta$**\n  + L302–304 describe how $\\beta$ would evolve with policy performance, but no experimental evidence is shown. Could authors support this? Having a plot or table showing performance vs. $\\beta$ over training would give clarification.\n\n- **Clarification on metric**\n  + The main paper reports only the best reward, which raises a concern for me that the variance may be high, given that the method introduces multiple coefficients. However, later I found that Table 3 shows reasonable standard deviations, so I suggest either including a reference in the main results to guide readers.\n  + As a minor suggestion, adding performance drop percentages to Table 1 would make it easier to see which components are most critical. Ultimately, up to the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cKiSUV5iBN", "reviewer": "Reviewer_kT8z", "summary": "The paper introduces PoRSE (Policy-grounded Synergy of Reward Shaping and Exploration), a framework for robotic skill learning that integrates LLM-generated reward design with efficient, task-relevant exploration. Existing LLM-based methods like Eureka and ROSKA focus too narrowly on goal rewards or reuse past policies but neglect structured exploration. PoRSE overcomes this by constructing an Affordance State Space (AFS) which is a low-dimensional, task-aware abstraction of the environment where exploration bonuses are assigned based on visitation frequency.", "questions": "3.1 Why absent comparison with state-of-the-art frameworks? Can you add comparisons with further baselines?\n\n3.2 At 1500 epochs after adding J=5, how many epochs do the mutants train before any ranking; is J=5 total or five per survivor?\n\n3.3 Which β formulation produced for the results based on point 2.4?\n\n3.4 Can you provide evidence that pruning the lowest-performing candidate every 500 epochs truly leads to worse final performance?\n\n3.5 Could you provide further clarification on Table 3, as mentioned in point 2.3?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.1 The paper’s idea of guiding exploration with a task-aware state mapping instead of generic novelty is clear and makes sense. The ablations confirm that this mapping improves learning stability and performance.\n\n1.2 The experiments span a wide range of tasks and are supported by detailed ablations and additional appendix results, which make the findings more convincing and strengthen the overall paper.", "weaknesses": "2.1 The results are incomplete. The paper lacks comparison and positioning against state-of-the-art frameworks, such as the REvolve, which likewise uses LLMs in reward design, and should therefore have been included.\n\n2.2 Section 5.1 tournament selection is unclear. You start K=6; at 500/1000 you prune to 4; at 1500 you add J=5 mutants; then you say “remove 6” at 1500. It is not stated whether the five new candidates receive any training before ranking, nor whether J=5 is total or “per survivor.” As written, untrained mutants seem pruned alongside half-trained survivors. This needs a further clarification. \n\n2.3 In Table 3 (Appendix A.2.1), MTS is reported as $\\text{mean} \\pm \\text{std}$ of the per-seed maximum. For manipulation tasks this “maximum” represents a success rate $\\in [0,1]$, while for locomotion it is a return (unbounded). The caption incorrectly labels all rows as “success rates,” which conflates metrics and makes cross-row comparisons invalid. Moreover, the table selectively bolds PoRSE even when other methods achieve identical or better results (e.g., 1.000 ties), which is misleading.\n\n2.4 β mixing is inconsistent. In the main document at Section 4.2 the authors use β with (2−β) but in Appendix A.1 uses β with (1−β).\n\n2.5 Pruning every 500 epochs assumes early rank predicts who will still be bad at 3000. The paper does not support this claim with empirical results.\n\n2.6 Several interesting results exist in the Appendix which are not referenced in the main paper.\n\n2.7 Abstract reads like an extended introduction; it mixes motivation, method and broad claims. This could be tightened.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)