Progress Reward Model for Reinforcement Learning via Large Language Models
Authors: Xiuhui Zhang
Ning Gao
Xingyu Jiang
Yihui Chen
Yuheng Pan
Mohan Zhang
Yue Deng
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Large Language Models
Reward Design
Primary area: reinforcement_learning
TL;DR: We construct a Progress Reward Model with convergence guarantee for Reinforcement Learning via Large Language Models.

Abstract: Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. 
Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. 
However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks.
In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. 
For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. 
For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization.
Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance.

Progress Reward Model for Reinforcement Learning via Large Language Models
Xiuhui Zhang, Ning Gao, Xingyu Jiang, Yihui Chen, Yuheng Pan, Mohan Zhang, Yue Deng"
Beihang University
37 Xueyuan Road, Haidian District, Beijing (zhangziuhui, gaoning_ai, bby3aar edu. cn {21375399, fzx12003, zmh666, (reng) edu. cn
Abstract
Traditional reinforcement learning (RL) algorithms face significant limitations handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-basee haustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a com- plex task is decomposed into series of simple manageable subtasks, with subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-base reward shap- ing, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results robotics control tasks demonstrate that our approach outperforms both ALM-based planning and reward methods, achieving state-of-the-an performance. The code is available at https://gii higgttttu....... cww/deng-/oe////oeo/e/eco..o. ai lab/PRM4RI.
methods require extensive human feedback ex-
Introduction
Reinforcement learning (RL) has shown exceptional potential in solving sequential decision decision-making making problems. However, in real-world scenarios involving complex, long-term tasks with sparse rewards, agents face significant challenges in planning and decision-making due to insufficient feedback signals I 2 Traditional RL frameworks address these challenges primarily by focusing on two strategies: task decomposition and reward shaping. Hierarchical R 4 simplifies decision-making by autonomously decomposing complex tasks into hierarchical subtasks. while reward shaping techniques provide dense training signals through the reconstruction of reward functions, such as inverse 5 6 7 and intrinsic reward mechanisms(S) [ Despite their advantages, both approaches heavily rely expert data for priors [ supervision [0], and the large-scale acquisition of such data is often challenging, limiting the scalability and applicability of these methods.
Recent advancements in large language model (LLM) research have made significant progress in areas such as dialogue [2], sequential decision-making [13] [4], and coding [5] 16, [7]. By
leveraging the extensive and diverse human knowledge acquired during the pre-training phase, LLMs
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Consequently, promising approach overcome the challenges faced by traditional RI. methods [20] To tackle the issues of long-term planning and sparse rewards, previous LLM-based approaches have evolved into two main paradigms: I.LM as planner (2 22 23 and LLM as rewarder [24] [6] as illustrated in Figure [ As a planner, LLM decomposes complex, long-term tasks into a series of short-term, goal-oriented subtasks, incorporating subtask priors into the decision-making process. As rewarder, LLM constructs a dense reward function for sparse- reward tasks, providing guidance to the underlying policy based on the meaningful priors encoded within the LLM.
possess a broad range of world knowledge and emergent reasoning abilities 19,
LLM-Augmented RL has emerged
Original Task
V
LLM--Aggmestatio
Sabrask
Sabtask
GAP: NO Sabrask Prioes
Subtack
Subeack
Training Low-Level Control Policy
Spurie Reward
LLM-Augmcoied Beward
Fine-Grained Progres Reward Madel
Policy with Prior perssprion
Policy without Prior
por
(b) LLM Rewarder
Fromework
Results Metoworld Menkkill
Figure Quick view. (a), (b) and (c) shows the structual similarities and differences between LLM--Augeettd methods. (d) is the average evaluation results on Metaworld 27 and Maniskill
However, LLM as a planner primarily focuses on high-level planning. In the absence LLM- augmented dense reward guidance, task decomposition -elling solely natural-language task priors-is insufficient to support the effective learning of subtasks [2 [22] Previous approaches have alleviated this limitation by constructing pre-trained policy libraries 21 incorporating traditional motion planning to link subtasks [22], yet these methods lack flexibility and efficiency due to their non-end-to-end nature. On the other hand, LLM rewarder provides low-level guidance. However, without subtask decomposition, constructing a reward function for an entire long-term, complex task remains a significant challenge for LLMs 30 31]. Previous methods have enhanced the quality of LLM- generated reward functions by incorporating human feedback [[44 or evolutionary search algorithms[25) 26). but these approaches still require extensive search and significant human expert assistance. As shown in Figures Dp and [p, both of these approaches focus on single aspect of RL, neglecting the importance of integrating both high-level planning and low-level reward shaping.
To address the challenges of long-term planning and sparse rewards in reinforcement learning more efficiently, we propose novel framework, Progress Reward Model for Reinforcement Learning (PRM4RL), which integrates both high-level long-term planning and low-level dense rewards to provide comprehensive solution, illustrated in Figure Te. In high-level planning, the task is decomposed into subtasks, each with termination conditions and fine-grained progress evaluations. For low-level reward generation, we utilize the progress to construct an efficient dense reward signal with theoretically grounded optimality and convergence guarantees, In practice, our pipeline decomposes complex task into a series of manageable, goal-oriented subtasks, each associated with natural language subtask prior. For each subtask, we build fine-grained progress function that dynamically tracks the execution of the task. Inspired by the potential field theorem, we regard the progress function as potential function and design Progress Reward Model with optimality and convergence guarantees. The evaluation results in Metaworld and Maniskill 28) as shown in Figure Id demonstrate the effectiveness of our proposed framework, with significant improvements in success rate and reduced I.I. calls, outperforming existing methods in both performance and efficiency.

---

2 Related Works
2.1 LLM for Planning
Large language models (LL.S), enriched with diverse world knowledge and human priors during their pretraining phase, possess the ability to understanding and reasoning [88 19
17
16
11). When
provided with relevant task information, LLMs are able to decompose complex, long-term tasks into simpler, short-term tasks, each focused on specific objectives [3] 33). Numerous approaches have leveraged Task and Motion Planning (TAMP) to realize such task decomposition in robotics However, when using RL for low-level control strategies, a significant gap arises between high-level planning and low-level control- to efficiently leverage the natural language plans generated by LLMs to guide policy training? SayCan [2] addresses this challenge by utilizing set of pretrained policies, where the LLM simply selects the appropriate policy based on the plans. Plan-Seq-ILeam [22] uses traditional motion planning to link subtasks, alleviating the burden on RI. policies. In text-based environments, ELLM [3 trains RL. policies by describing state transition in natural language and computing the similarity between the executed actions and the plan, using this similarity as a reward signal. However, this approach is computationally expensive and not suitable for robotics tasks. In contrast, our method constructs an efficient and dense reward signal through the Progress Reward Model (PRM), effectively bridging the gap between high-level planning and low-level policy training.
scenes 34 35
2.2
LLM for Reward Generation
Reward functions are the primary mechanism to guide RL training 36 37 leading to numerous approaches that utilize L.LMs to generate executable reward functions that provide dense rewards. Text2Reward 24] uses LLMs to generate executable python functions based on goal described in natural language, with human involvement in the loop. Eureka 25 extends this paradigm by introducing evolutionary search and automatic reward refinement through LLMs. CurricuLLM[[38 translates natural language description of tasks in executable task code, including the reward code and goal distribution code. REvolve 26 translates implicit human knowledge into explicit reward functions for training RL with human feedback. However, these methods treat the task as a whole and do not account for the potential stage-wise features of complex tasks. As a result, the generation process of reward function tend to be highly complex and often require human feedback [24] (66 or extensive search (2]. In contrast, approach simplifies the reward generation process by decomposing the task into short-term, simple subtasks, providing more specific and clear guidance for each stage of the task.
3 Preliminary
In standard reinforcement learning [RL) tasks, the problem is typically modeled as a Markov Decision Process (MDP), denoted by M = (S, A. T, R. 7, Po). Here, S and A represent the state and action spaces, respectively, is the discount factor, and Po is the distribution dynamics of environment are captured by the transition function T : S x A x S [0, 1], which defines the probability of transitioning from one state to another given a specific action. The reward function S x A > R quantifies the immediate reward received for taking an action in a particular state. A trajectory = (8o: ag To...., 8T, @T, rr) describes the sequence of states, actions, and rewards encountered by an agent following a decisienn making policy π. At each time step t, the agent selects an action a, according to a ~ (-))s)), and the environment responds by transitioning to the next state St+1 T- ).). The state-action value function (Q-function) (s, a) represents the expected cumulative return when taking action in state s and subsequently following policy m:
initial states. The
T R(B dets) 8: = 8, a: = a
Q" (s. a) = -E
Σ
k==
The objective in RL is for the agent to learn policy cumulative return defined as:
S A that maximizes the expected
T
((T) = Erop(rim)
Σ
r' R(s,, ar)
(2)
3

---

Task
000
Subtaska Decomposition
Progress Function ?
6C Satoak
Refice the angrest por paan FLtrAI
"her mi "Nove gail"
correst (eng
T. Rig distrernation dIAt)
Sarall
Grspafeer
Peture cnnticion for value kinn shorwicti
Caloadee the pragres ceret gropping
2.0iueeee onlern tya pester the thmraaae
wo..so.LLLe PELoo.pa wes.sagalllaaga.... Strstt servest
enject"
"Savdracment Infor
C2ferco to grovsing the sdject
"Taak Descriprtent
<LP 193 mmgge
carret, Hog e
"Sserructionate ofte pither pryer
SAtek
sud daa bugee geat, chenkk
(n dn ni put dutt custroonatio
(a) Step-Ry-step Thinking
Progress Reward Model
Env
Subtask Prior
Subcask
Subtack
9
Action de
propeo
Progress Function
Main Pragress (1))
Sanmaa "Gropp oljer"
Seboos
Progress roward:
Augmented Policy p(als,prior)
rabtoak sip
(o(s.)) 9(6)
Saccess
Success reward:
Sentence Encoder
Ty ((6) Fyacer
Reward F,
Priee pra
Embeeddinggilann.ss((.....
(b) PRM Augmented RL Policy
Figure An overview of our framework. (a) We prompt the LLM using Chain-of-Thought approach in Python code format. The reasoning process is provided as comments (e.g., using #). while the intermediate steps and final response are represented as executable Python statements. (b)
Our framework augments RL in two ways: subtask prior and the PRM rewards.
4 Method
Our framework, as illustrated in Figure 2 bridges the gap between high-level planning and low-level training. We begin by decomposing a complex long-term task into a series of short-term subtasks, with a determination function V to classify the current subtask (Section 4.1. Next, we construet potential based Progress Reward Model (PRM) derived from the progress function 4 which generates effective reward signals that are proved to have convergence and optimality guarantees (Section 4.2 Finally, the policy is augmented using both the PRM reward and the subtask priors (Section
LLM usage pattern. In our framework, we prompt an LLM to decompose the tasks and generate the determination function y and progress function 4. Considering that the intermediate inference steps and the required final response are primarily represented Python code (as shown later Section inspired from PAL (3)], we require the LLM to generate programs as intermediate steps, as shown in Figure 2.. Specifically, we instruct the LLM generate natural language reasoning steps using comment syntax (e.g., "#' in Python), while the code itself is generated normally. Through this program-aide aided function in a single interaction.
4.1
of-Thought reasoning, the LLM can generate both the plan and progress
4.1 Subtask Decomposition
Decomposing long-term task into series of short-term, manageable tasks provides the policy with subtask priors and alleviates the complexity of constructing the progress function. Existing methods 21 22 have demonstrated that, given sufficient environment-related information, LLMs are capable of task decomposition. In our framework, we leverage an L.M to break down the task into series of subtasks, each contributing to the success of the overall task. We construct Pythonic representation prompt that includes semantically rich contextual information such state

---

Algorithm PRM4RL
Require: pythonic LLM prompt prompt, sentence encoder E, policy To, MDP M (S,A, T, R. Pu), RL optimization algorithm Optimize_Algo
// Requeest LLM
plan_liss (s),, ((s)) = LLM (prompp))
11 Augment State Space
P :prior = El E(plon_lis(
I1 Augment Reward Function
R RPRM = (8(+1) ((s)) ( (($++1) Pbonus
Optimizing policy under augmented MDP
M' = (S + A, T, RPRM Po)
0" Optimize__Alo((A)
space, environmental descriptions, and task descriptions (detailed in Appendix A.5 providing high-dimensional abstraction of environment-speeific information.
While decomposing the subtasks, the LLM is also required to generate a natural language form subtask prior, which is description of the corresponding subtask. To ensure consistency and generalization across the robotics control domain, we designed a fixed pattern for the subtask priors. Specifically, instruct the LLM to output each subtask prior in "verb noun" format (e.g.. 'Move to object' or 'Grasp object'), significantly enhancing the generalization ability of the trained policy. The expected output from the LLM is Python list, called plan_ list, shown in Figure (II), where each element corresponds to subtask prior.
However, guiding the policy with subtask priors presents another challenge: real-time tracking of which subtask the policy is currently exccuting. Previous approaches require calling the LLM at each timestep, either to regenerate real-time plan [23] or to check the completion of ongoing subtasks (2) leading to significant time and computational overhead.
Our key insight is that, with thorough understanding of the state space, we can determine which subtask the policy in based solely on current state. For the previously generated plan_list, we instruct the LLM to identify the completion condition for each subtask and locate the relevant features the state space. We then construct subtask determination function V (s), where given state s, will return the index of current subtask in plan list. As shown in Figure (III), the LLM ultimately generates an efficient, low-cost logical chain that ensures rigorous subtask determination.
4.2 Potential-Basee Progress Reward Model
Our framework aims construct dense and efficient reward signal to guide the training of low-level policies. While previous approaches 24 25 26 directly design reward function, we choose to model the progress of task execution instead. This approach is more task-relevant and provides stronger theoretical guarantees on optimality and convergence as proved later.
With the decomposition of subtasks, the focus of our progress function shifts from a long-term, complex task to short-term, simpler subtasks. This transformation simplifies task comprehension and the design of usable metrics. Based on the established subtask determination logic V we instruct the LLM further define progress function for each subtask, and then integrating these fine- grained sub-pre progress indicators into an overall evaluation of the task progress (s)), as shown in Figure (III.. The progress function offers precise and efficient method for monitoring overall task execution.
Inspired by potential-based framework, we treat the progress function (s) as a form of potential and construct the Progress Reward Model (PRM) using the temporal difference form:
RCRM 7 (s+++)) v(st) + I(s++1) ) Thoous
where R.PM is the dense reward given by PRM at timestep t, y is the discount factor, and the indicator function I equals 1 when 8t+1 meets the success conditions and 0 otherwise. (($++1) and (s,)) represent the progress at timesteps t ++1 and t, respectively. The PRM reward can be regarded as consisting of two parts: sparse success reward and a potential-based reward shaping term. Following 40), we present the following theorem:
5

---

Theorem 4.1 (Policy Invariance). Define an MDP M = (S,A,T, ", Po). where R = II Tbonus represents the sparse reward when the task is considered successful. Giving a potential- -based reward shaping function as follows:
F ( ((s))
Let another MDP M' = =,,,,T, optimal policy in M and vice versa.
Po). then every optimal policy in M' will also be an
The proof of Theorem 4.1 is provided in Appendix A.1
Theorem 4.1 indicates that RPRM does not distort the learning of the policy by introducing "deceptive" or "short-sighted" behaviors due to reward shaping. The optimal policy M' remains focused on maximizing the original task objective, Lc. maximize success rate. Based on Theorem we have the following corollary:
Corollary 4.2 (Convergence Guarantee). The introduction of potential-based reward shaping will not affect the convergence of reinforcement learning algorithms.
Since the PRM reward has theoretical convergence guarantee, we proceed demonstrate its efficiency in convergence, following the work of [4]] (Detailed proof provided in Appendix A.2).
Theorem 4.3 (Convergence Efficiency). Consider two MDPs: M (8, A, T, R F, 7, po) with Q-function Q and M' = (S,AT R, Po). If we initialize the Q-function in M' as follows:
(((s,a) a) = Qo(s, a) + +s))
Then for all timesteps 2 and (s, a) E S A: Qi(s, a) Qr a) + $(s)
where Qt and Q2 are the Q-functions of MDP M and M' respectively.
The mathematical equivalence between PRM and Q-value initialization, as established in Theorem provides two fundamental insights into why PRM accelerates convergence. First, it demonstrates that PRM effectively provides prior knowledge-baset initialization for value functions, directly reducing the initial approximation error IQu Q*|- |. Second, the proof formally justifies that any well-designed potential function (s) serves as an actionable curriculum, where the shaping term ( (s) progressively guides exploration toward high-valus value regions without altering optimal policies.
These theoretical results establish PRM not only heuristic reward design but also as mathemati- cally grounded approach to guide the policy with optimality and convergence.
4.3 Reinforcement Learning Augmentation
Giving an MDP M = A, R, 7, Po) our framework augment it through both high-level planning and low-level reward shaping.
At the high level, the policy cannot directly process the natural language form of the plan_list. To address this, employ SimCSE[42), sentence encoder E, to embed the subtask prior. Specifically, the subtask prior is expressed as:
prior = E(plan_lis((
where subtask prior the embedding of current subtask corresponding state s. The state space is
then augmented to S + P. At the low level, we use the reward R PRM from Section 4.2 to efficiently guide the learning of policy. Finally, the augmented MDP expressed as:
M' = (S + P,A,T, PRM 77.P)
This formulation integrates both high-level planning through the subtask prior and low-level control via the PRM reward to guide policy learning. Consequently, algorithms to optimize the policy under the augmented MDP M'. In this paper, we use Soft Actor- Critic (SAC) 43 and Proximal Policy Optimization (PPO) (4
utilize existing RL optimization
5
Experiment
In this section, we evaluate PRM4RI. on diverse suite of long-term, sparse reward RL tasks, testing its ability provide high-level plan and guide low-level training.

---

700
171
130
5.5
3.0
1.0
1250
035 8.0
031
13%
235
13
130
960
536
013
579
150
338
1.0
12.
Our
ELLM
Orack
1250
Figure 3: The learning curves comparison methods on Metaworld and Maniskill measured by success rate, All experiments are conducted with 5 random seeds.
5.1 Experiment Settings
The framework is implemented based on the Stable Bassllnes framework 45), ensuring consistency across all methods. We employ the same RL algorithm with identical training parameters for all methods. For all L.-M-related requirements, we utilize OpenAl's GPT-4o Details of the hyper- parameters provided in Appendix A.3 To ensure robustness and reliability, all experiments conducted with five different random seeds.
Baselines. ELLM(CCM23)) [23 utilizes LLM high-level planner, embedding subtask priors into the policy. serves as an example an LLM-planner approach. T2R(CLL224) [24] combines LLMs and human feedback to generate dense reward functions for policy training. We use it as an example of an LeW-rewarder approach. Oracle is the expert-written dense reward function provided by the environment. The details of the baseline settings provided in Appendix A.2.2
Environments MctaWorld[27
an open-source simulated benchmark features a Sawyer robot interacting with tabletop setup that includes drawer, window, ball, faucet, door and many objects. Maniskill 28 is an advanced robotics simulation platform designed for high-fidelity manipulation tasks. Further details about the environment and the tasks are provided in Appendix
5.2 Evaluation Analysis
5.2.1 Outperform previous LLM approaches
As shown in Figures and|4 our method consistently outperforms the comparison methods across nearly all environments, demonstrating faster convergence and superior performance. We observe the following advantages: (1) In complex long-term tasks such as MW-pick-out-of-hooll and MS- LiftPegUUprihh,, the baseline methods achieve less than 15% success, while our method achieves nearly 100% success, highlighting our ability to effectively solve challenging tasks. (2) When compared to ELLM, our approach demonstrates significantly improved convergence speed, which underscores the efficiency of the Progress Reward Model in providing dense, theoretically guaranteed rewards for the low-level control policy. This results in more stable and faster learning compared to traditional L.LM-based planning approaches.
(3) comparison with T2R, the subtask decomposition in our framework plays critical role in simplifying the coding process. By breaking down complex
7

---

ASCORZ
Our
ELLM
T2r
NW buttomn
windew
MM
MM Aauad
MW
1:44
5.80
5.00
place
MW MW-owwop
VS MS-FashCate
NS-Pal
HS-LiCubb
MS-Reedubee
Figure 4: The evaluation results of comparison methods Metaworld(With prefix 'MW") and Maniskill(with prefix 'MS') measured by success rate. All experiments are conducted with 5 random seeds.
tasks into more manageable subtasks, our method reduces the difficulty of understanding environment dynamics and designing useful metrics, enabling us solve complex tasks that T2R fails address. This advantage is particularly evident in tasks where the environment is more intricate and requires clear separation of the planning and reward stages.
(a) Training curves of ablatioo experiment.
114
Wilverdda--pp
MWdduccm-ppen
Wibpice chiicee
(h) Evaluation resalts of ablation experiment.
wat, prior
Our w.o, potencial reward
Our wab, prm
Oracle
Figure 5: The evaluation results of ablation study on Metaworld measured by success rate. All experiment conducted with 5 random seeds.
5.2.2 Ablation Study
To further validate the effectiveness the two augmentations in our framework, we conduct an ablation study by constructing three additional baselines:
The results are shown in Figure We observe the following findings:
(1) Compared with 'our w.o. prm', replacing the PRM reward with the oracle reward leads significant drop in performance. This demonstrates that, without an efficient and effective reward signal, directly feeding the subtask prior to the policy does not improve its performance. In most cases, the training curves and evaluation results of 'our w.o. prm' are similar to those of the oracle, with some instances where adding the subtask prior even causes slight performance decrease.
(2) The performance of 'our w.o. prior' shows that abandoning the subtask prior results in slower convergence in the training curves. This highlights the role of the subtask prior in enabling efficient convergence for complex, long-term tasks. Moreover, 'our w.o. prior' still outperforms T2R, further demonstrating the effectiveness of the PRM reward in guiding the policy.

---

w.o. PRM reward (denoted as 'our w.o. prm ): The PRM reward is replaced with the oracle reward, while retaining the subtask prior.
w.o. subtask prior (denoted as 'our w.o. prior"): No subtask prior is provided, while retaining the PRM reward.
w.o. potential-basee reward shaping (denoted as 'our w.o. potential reward"): ):
360
The progress function is used directly as the reward, while retaining the subtask prior.
Uhe permii
(3)The performance of 'our w.o. potential reward" further demonstrate the effectiveness of our potential-baset reward model, where directly using progress as reward signal lead to significant drop in performance. We also analyzed the anomalous behavior of 'our w.o. potential reward" where the success rate initially rises but then declines training pro- gresses. As shown in Figure 66 the episode reward increases, while the success rate paradoxically decreases. This suggests that the using progress function directly as reward function suffers from inefficiencies, as there are instances where the task fails despite receiving high reward, leading the policy to learn an undesirable direction. In contrast, our method
Figure 6: Training details of our and our w.o. poten- tial reward.
ensures stable convergence, with the PRM providing well-structured and task-relevant rewards with optimality and convergence guarantees.
In summary, the two enhancements in our framework the subtask prior and the PRM reward- work synergistically to produce a "1+1 > 2" effect. Without the subtask prior, convergence speed is hindered, and without the reward signal, providing effective and efficient guidance becomes challenging.
1.0
1.00 1400
0,90
0.66
0.08
0.0
0.79
0.8
0.49
0.6
0.4
0.36
0.16
0.00
Rgp
Original Task
Unsean Task 1
Unseen Task 2
Unseen Task
Unseen Task 4
Our
w.e, prior
ELLM
Figure 7: The evaluation results of generalization experiment. The results are averaged on 5 random seeds and 100 evaluations per seed.
5.2.3 Generalization Results
To evaluate the generalization capabilities of the trained policy, we test the trained policy's perfor- mance on unseen tasks MetaWorld, Additional results and detailed information regarding the tasks provided in Appendix A.2.1 As shown in Figure 7 while most methods perform well on the original task, we observe performance decline on the unseen tasks. In comparison to other methods, our framework experiences the least performance loss, demonstrating relatively strong generalization abilities. Notably, when comparing to our "w.o. prior' (our framework without the subtask prior), observe significant drop in performance unseen tasks. This highlights the critical role of the subtask prior enhancing generalization. The "verb + noun" pattern we designed effectively captures commonalities across different tasks, acting powerful prior that guides the policy to perform well on unseen tasks. Thus, the subtask prior serves as robust tool for improving the generalization of reinforcement learning policies across wide range of tasks.
6 Conclusion
In this paper, we introduced the Progress Reward Model for Reinforcement Learning (PRM4RL),, a novel framework that integrates high-level planning and low-level reward shaping to address the challenges of long-term planning and sparse rewards in reinforcement learning, At the high level,

---

decompose complex task into manageable subtasks with subtask prior. At the low level, we design Progress Reward Model with optimality and convergence guarantee. The experimental results demonstrate the effectiveness and efficiency of our method, significantly improving both convergence speed and the ability to solve complex, multi-stage tasks.
Limitations. Similar to T2R(24) and other methods (23, 21 22 that depend on LLMs for planning or coding, our approach could be affected by LLM hallucinations during reasoning 47). Besides, while our method demonstrates strong performance across various tasks, its applicability to domains outside of robotics remains unexplored. We anticipate that further optimization and adaptation of the framework will be necessary to address diverse environments and tasks.
Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (Grant No. 6..6200010,, Grant No.62325101).
10

---

References
[1] Richard Sutton, Andrew Barto, et al. Reinforcement learning: An introduction, volume MIT press Cambridge, 1998.
[2] Gabriel Dulac-Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110(9))2419222
2021.
[3] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
[4] Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hicrarchical rein- forcement learning: A comprehensive survey. ACM Computing Surveys (CSUR), 54(5):1-35, 2021.
[5] Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
[6] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pages 4433-143.. Chicago, IL. USA, 2008.
[7] Chelsea Finn, Sergey Levine, and Picter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
[8] Shanchuan Wan, Yujin Tang, Yingtao Tian, and Tomoyuki Kaneko. Deir: efficient and robust exploration through discriminative-model--bee episodic intrinsic rewards. arXiv preprint arXiv:2304.10770, 2023.
[9] Anna Penzkofer, Simon Schaefer, Florian Strohm, Mihai Bâce, Stefan Leutenegger, and Andreas Bulling. Int-hrl: towards intention-based hierarchical reinforcement learning. Neural Computing and Applications, pages 1-12, 2024.
[10] Xinlei Pan, Eshed Ohn-Bar, Nicholas Rhinehart, Yan Xu, Yilin Shen, and Kris M Kitani. Human-interactiive subgoal supervision for efficient inverse reinforcement learning. arXiv preprint arXiv: 180011000888 08479, 2018.
[11] Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. A survey recent advances in -lmed multi-tune dialogue systems. arXiv preprint arXiv:2402. 180133 2024.
[12] Ethan Brooks, Logan Walls, Richard L Lewis, and Satinder Singh. Large language models can implement policy iteration. Advances in Neural Information Processing Systems, 36:30349- 30366, 2023.
[13] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632-19642, 2024.
[14] Yuanzhao Zhai, Tingkai Yang, Kele Xu, Dawei Feng, Cheng Yang, Bo Ding, and Huaimin Wang. Enhancing decision-making for I1m agents via step-level q-value models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27161-27169, 2025.
[15] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
[16] Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, et al. Codescope: An execution-based multilingual mul- titask multidimensional benchmark for evaluating 1lms on code understanding and generation. arXiv preprint arXiv:2311.08588 2023.
[17] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
11

---

[18] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICI.R, 2023.
[19] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint urXiv:2501.1294: 2025.
[20] Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, and Yun Li. Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods, 2024.
[21] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on robot learning, pages 287-318. PMLR, 2023,
[22] Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, and Ruslan Salakhutdinov. Plan-seq- learn: Language model guided rl for solving long horizon robotics tasks. arXiv preprint arXiv:2405.01534, 2024.
[23] Yuqing Du, Olivia Watkins, Zihan Wang. Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforvement learning with large language models. In International Conference on Machine Learning, pages 8657-8677. PMLR, 2023.
[24] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Automated dense reward function generation for reinforcement learning. In International Conference on Learning Representations (ICLR), 2024 (07/05/2024- 1 11/0/22024 Vienna, Austria), 2024.
[25] Yecheng Jason Ma, William Liang, Guanzhi Wang. De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 1293,3 2023.
[26] Rishi Hazra, Alkis Sygkounas, Andreas Persson, Amy Loutfi, and Pedro Zuidberg Dos Martires. Revolve: Reward evolution with large language models using human feedback. arXiv preprint arXiv:2406.01309 2024.
[27] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019.
[28] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li. Tongzhou Mu, Nan Xiao, Arnay Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskil13: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. Robotics: Science and Systems, 2025.
[29] Archiki Prasad, Alexander Koller, Mareike Hartmann. Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models. arXiv preprint a 11:223/110070 1.0577,, 2023.
[30] Vishnu Sarukkai, Brennan Shacklett, Zander Majercik, Kush Bhatia, Christopher Ré, and Kayvon Fatahalian. Automated rewards via Ilm-generated progress functions, 2024.
[31] Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. Dric: Reinforcement learning with dense rewards from Ilm critic. arXiv e-prints, pages 24X1 2024.
[32] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks, 2023.
[33] Quan Yuan, Mehran Kazemi, Xin Xu, Isaac Noble, Vaiva Imbrasaite, and Deepak Ramachandran. Tasklama: Probing the complex task understanding of language models, 2023.
12

---

[34] Yongchao Chen, Rujul Gandhi, Yang Zhang, and Chuchu Fan. N12t:: Transforming natural languages temporal logics using large language models. arXiv preprint arXiv:2305.07766. 2023.
[35] Yongchao Chen, Jacob Arkin, Charles Dawson, Yang Zhang, Nicholas Roy, and Chuchu Fan. Autotamp: Autoregressive task and motion planning with ilms as translators and checkers. In 2024 IEEE International conference on robotics and automation (ICRA), pages 6695-6702. IEEE, 2024.
[36] Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Informativeness of reward functions in reinforcement learning, 2024.
[37] Jonas Eschmann. Reward function design in reinforcement learning. Reinforcement learning algorithms: Analysis and Applications, pages 25-33, 2021.
(38) Kanghyun Ryu, Qiayuan Liao, Zhongyu Li, Payam Delgosha, Koushil Sreenath, and Negar Mehr. Curricullm: Automatic task curricula design for learning complex robot skills using large language models, 2025.
[39] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aide aided language models. In International Conference on Machine Learning, pages 10764-10799. PMLR, 2023.
[40] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. In Ieml, volume 99, pages 278-287. Citeseer, 1999.
[41] E. Wiewiora. Potential baase shaping and q-value initialization are equivalent. Journal of Artificial Intelligence Research, 19:205-208, September 2003.
[42] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings, 2022.
[43] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018
[44] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[45] Antonin Rafin, Ashley Hill, Adam Gleave Anssi Kanervisto, Maximilian Ernsstus, and Noah Dormann. baselneas: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 2021.
[46] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
[47] Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint arXiv:2404.0097), 2024.
13

---

A Appendix
A.1 Theoretical Proof
Theorem A.1 (Policy Invariance). Define MDP M (S,A, R. Y,p), where Rd = I(s++1)
Tbonus represents the sparse reward when the task is considered successful. Giving a potential-basee reward shaping function as follows:
F = (5441)) (ss))
another MDP M' (S, R+ + Po), then every optimal policy M' will also be an optimal policy in M and vice versa.
Proof. For the original MDP M, we know that its optimal Q-function Vús satisfies the Bellman Equations
(8, = -E n(s,a,8')+ + max
ya (dr('',a')
Some simple algebraic manipulation then gives
Qm((,,) (bs) - -Es R(s, [(ss,,,)) a') 7 ((8) + + max (Qin(s',a') 4(s'))
If we now define ( a) - (s) ((s) and substitute that and F(s,a,8') = 70(66) =(0(())))=0((()))) (()) back into the previous equation, we get
(8,
(s, E, A(s,a,s') a(s. +(ss.as))+((s.s.)))) + max (ss
deE
= E. R'(s,,,s') + ( Qa ''En
But this is exactly the Bellman equation for M'. For the undiscounted case, we moreover have Qur a) = Qir a) ((5) = 0 = 0. So, QM (s, a) satisfies the Bellman equations for M', and must in fact be the unique optimal Q-function. Thus, (s) = (s, = Qir (s) and the optimal policy for M' therefore satisfies.
*p (8) E arg max aEA
(M,
arg max a
Qi (s, a) (s))
(5)
arg max Q (8) asA
* and is therefore also optimal in M. To show every optimal policy in M is also optimal in M', simply apply the same proof with the roles of M and M' interchanged (and using the shaping function This completes the proof.
Theorem A.2 (Convergence Efficiency). Consider two MDPs: M (8, A,T, R F, po) with Q-function Q and M' (S,A, T, R, 7: Po). Ie we initialize the Q-fnction M' as follows:
('(s,a) a) = =o((s +(8)
Then for all timesteps 2 0 and (8, E x A:
(s Qi(s, a) + ((s)
(7)
where Qi and Q2 are the Q-functions of MDP M and M' respectively.
14

---

Proof. We prove the equivalence by induction on the update steps.
Base 0): By initialization definition:
()(s,0) a) = Qo(s, +s)) V(s,a)
(8)
Inductive step: Assume (s a) = Qx + ((s) holds for all (s, a) at step k. Consider the
Q-learning update for both agents when transitioning from (s,a) to (s', a," with reward r:
MDP M:
Qk+1(5; = Qu(ss a) ak r F(s, + max Qx (x(
= Qu((8 a) + ar r+ (>0((5)) (()) + ma Qa(s', m
- Qu(s,))
In MDP M':
Q2+1 (2. on m ( .') (s(
= =QQ(ss + (s)) ax y Q4
[0) + (Qu(s))
Qx(s.a) ((8) an r+ max Q (8' + Qx(s,,)) (())
Simplify the difference:
(s+1(s,a) Qu+i(s, ((s) + an 70(5') {ss) an (70(5') {ss)
(s)
Thus (5, Qx+1(s, + (s) maintains the invariant. By induction, the equivalence holds for all 1200 0.
A.2
Experiment Details
A.2.1
Additional Generalization Results
1.0
O38 1.00
0.8
0.52
2.44
0.6
0.4
9,26
B,14
0.26
0.10
0.0
2.00 3.00
0.00
Original Task
Unsean Task 1
Unseen Task
Unseen Task
Unseen Task
Our
Qur re
ELLM
12R
Oracle
Figure 8: The evaluation results of generalization experiment.
We conduct additional generalization evaluations. Specifically. for the evaluation results shown in Section 5.2.3] the original task tittawoold--fuce--opee the unseen tasks are 'Metaworld- drawer-close" deetaworid-button-presss respectively, For the results shown in Figure A.2.II the original task is pMetaworld-pick-place" the paaeworl-piickplaeeeeee
sittawoordd-uutto-rreeeee 'Metawortd-coffie-bbttoo
unseen tasks are "Metaworld-drawer-cooss Metaaworddhannlle handle-press', respectively. We test the trained policy in evaluation tasks, and the results averaged over 5 seeds and 100 test per seed.
'Metaworld-sweep.
A.2.2 Baseline Settings
(ILM(ICMI.23) utilizes an L.LM high-level planner, embedding subtask priors into the policy. Since ELLM has not been previously tested on Metaworld and Maniskill, we re-implement
15

---

the experiments. For goal embedding, we gave the same subtask prior as in our framework to the policy; For reward, we then gave the subtask success reward. which is also goal-related in its original application. We also use our V for determining current subtask to avoid enormous LLM calls.
Tex22eearr((CLL22) (24 utilizes an LLM to write dense reward function. As T2R has evaluate its performance Metaworld and Maniskill, we directly reuse its generated reward functions. If there are few-shot and zero-shot versions of reward functions, we always choose the few-shot version for better performance. Specifically, for -rrtwoorddddaawe--losss' 'Metaworld wowolld-windwwoooo seetaworld-button-eoooo we reuse its generated reward functions. For other tasks that are not orginally evaluated in T2R, we generate a reward function following its code using the same GPT-4o LLM.
window-open'
"eetaworddsswep-inng
Aletaworidddoor-close
"Maniskill-Picke
A.3 Hyper-Parameter Details
In this section, we provide the hyper-parameter details used for reward function generation and reinforcement learning backbones. For progress function generation, we base on GPT-4o. In the experiments of the main body, the temperature of sampling is set to 0.7 for each experiment.
For reinforcement learning training, we employed implementation 43 from Stable-Baselines]) 45 and list the hyper-parameters in Table A.3 and
open-source PPO and SAC
Table Hyper-parameter of SAC algorithm applied each task.
Hyper-parameter
Value
Discount factor 7
0.99
Target update frequency
2
Learning rate
3 x 10-4 44
Train frequency
Soft update
5 x 10-3
Gradient steps
Learning starts
4000
Batch Size
512
# of layers
3
Hidden units per layer
256
Initial temperature
0.1
Rollout steps per episode
500
Table 2: Hyper-parameter of PPO algorithm applied to each task.
Hyper-parameter
Value
Discount factor of epochs per update
0.8
Learning rate # of environments
10-4
Batch size
400
Target KL divergence
0.1
of layers
Hidden units per layer
256
of steps per update
3200
We utilize 4 NVidia Geforce RTX-3090 graphic cards, 128 core CPUs, and 256 GiB memory server for RL training. The time required for training policy is approximately I hours per task (5 seeds running simultaneouly) for MerlW and between 10 hours for ManiSkill, varying with the task's s difficulty.
sttable- baselines stable- baselines3
v2.6.0 (MIT License), code
available at https://github. www.eo..o///...eee RM/
16

---

A.4
Tasks
In this section, we provide full list tasks within each simulation environment, accompanied by their corresponding language instructions. Across all tasks, we follow the default settings of their
A.4.1 Metaworld
In METAWORLD (7 environment, we use DoF Sawyer robot arm with a fixed base to complete tabletop tasks. For all tasks, the observation space is combination of the 3D position of the robot end-effector, a normalized measurement of gripper openness, the 3D position of the manipulated object, the quaternion of the object, all of the previous measurements, and the goal position. The environment adopts end-effector delta position control, which means the action space consists of the change of the end effector's 3D position, well the normalized torque the gripper should apply. For all tasks, the initial and target positions of the manipulated object and the initial joint positions of the robot variable.
button-press
door-close
drawer-close
faucet-open
Window-open
sweep-lnto
pick-place
pick-out-of-hole
sweep
push
Figure 9: Tasks in Metaworld.
The task descriptions provided in the following Table A.4.I
Table 3: Task list of Metaworld
Task
Task Description
button-press
Press button in y coordination.
door-close
Close door with revolving joint by pushing the door's handle.
drawer-close
Close a drawer by its handle.
faucet-open
Rotate the handle counter-clockwise.
window-open
Push and open sliding window by its handle.
sweep-into
Sweep puck from the initial position into hole.
pick-place
Pick up an object and move it to the goal location.
pick-out-of-hole sweep
Pick an object out
hole and move the goal location.
Sweep puck off the table.
push
Push object to the goal location.
A.4.2 Maniskill
MANISKILL environment uses DoF Franka Panda as the default robot arm. For all tasks, the observation space consists of robot proprioception information (e.g. current joint positions, current joint velocities, robot base position and quaternion in the world frame) and task-specific information (e.g. goal position, endeffector position). We use end-effector delta pose control mode for this
Metaworld v2.0.0 (MIT License). code available http:://ggttuboccom//aaaaaaafaat
Metaworld
"Maniskil3 v3.0.0 (Apache-2.0 2.0 License), code available http:r///ghhhbbbooo///eelll/////
17

---

environment, which controls the change of 3D position and rotation (represented as an axis-angle in the end-effector frame). For all tasks, the initial and target positions of the manipulated object, the initial joint positions of the robot arm and physical parameters (e.g. friction and damping coefficient) are variable.
LiftCube
PullCube
PushCube
PickCube
prightt
Figure 10: Tasks in Maniskill.
The task descriptions are provided in the following Table A.4.2
Table 4: Task list of Maniskill
Task
Task Description
LiftCube
Pick up cube and lift up by 0.2 meters.
PullCube
Pull cube to the goal position.
PushCube
Push cube the goal position.
PickCube
Pick up cube A and move it to the 3D goal position.
LirtPegUpright
Move a peg laying on the table to any upright position on the table.
A.5
Prompt Details
To ground LLMs into robotics simulation environments, following Text2Reward we propose a Pythonic prompt, which can be abstracted by the experts who developed the environment. This class-like prompt is a more compact representation than simply listing all environment attributes linearly, which can delete redundant information and save more tokens, and this Pythonic prompt can also better bootstrap Python code generation. More specifically, our prompt leverages python class, class attribute, python typing and comments to recursively define the environment. For simplicity, here we prompt the progress function to return the current subtask index together with the main progress.
Here provide example of the zero-shot prompt for Metaworld manipulation tasks:
Basic Prompt provides the foundational context for the task. The prompt establishes the user's role as expert robotics, reinforcement learning, and code generation, focusing the development of a Python function called 'progress function' for RL. The objective is to help generate code that tracks progress during task execution, guiding the learning of low-level policy.
Basic Propmt
You are an expert in robotics, reinforcement learning and code generation. We are going to robot arm to complete given tasks. Now want you help me write python function named 'progress function' of reinforcement learning.
Environment Information defines the environment in which the robot operates. The BaseFiv class describes the robot and interaction with objects in the environment, including their positions and states. The Robot and RigidOhject classes further define the robot's end effector, object positions, and other relevant parameters. This context provides essential details for the task, enabling the LLM to make informed decisions about task progress and control.
18

---

Environment Information
class BaseEnv (gyn. env):
relf.robot robot Robot the robot in the environment
self.obj RigidObject the first object in the environment up.ndarray [(3,)] # indicate the 3D position the goal
self golf.goal.position
uear_object = float (tcp_to_obj <= 0.3)
success float tobgeto_target <= 0.07)
class Robot:
self ee ee__osiiionn position of effector
ndarray [(3,)] indicate the 3D
self handaadiiig pos: np. adarray [(3,)] # indicate the initial
3D position of effector
self.tap self, ee pe_position [o, o, 0.045] indicate the 3D position of effector's tool center point
class RigidObject:
self position up np. ndaraay [(3,)] indicate the position of the rigid object
self qul..quatennion of the rigid object
nd.rrar [(4,)] indicate the quaternion
self. obj_ooojjiiit_po 3D position of the rigid object
np. ndarray [(3,)] indicate the initial
Task Information provides description of the task to be completed by the robot. The task description, which needs to be filled in by the user, is crucial for decomposing the task into manageable subtasks.
Task Information
Fill in the task description here. (e.g. In pick-place, the robot need to pick up the object and move it to the goal position.)
Basic Instructions provide guidelines for writing the code necessary for the task. The instructions emphasize the importance of using existing Python packages only when necessary, avoiding the invention of new variables or attributes, and incorporating step-by-step reasoning with clear comments. These instructions ensure that the code remains efficient, understandable, and aligned with the task's requirements.
Basic Instruction
You are allowed to use any existing python package if applicable. But only use these packages when it's really necessary.
Do not invent any variable or attribute that not given.
Think step by step, add comment for reasoning and thought when you write code.
Reasoning Instructions outline step-by-step procedure for task decomposition and the generation of key functions.
Reasoning Instructions
Follow the steps below:
a. Decompose the task and generate plan_lis Jist
a.1 Decompose the given task into series of subtasks. Each subtask should be described in
'verb + noun' form. tips: 'move to object', "grasp object', 'move to goal'...
a.2 Generate python list named plan list where each subtask is element of the plan list.
19

---

b. Write 'determination, function' for determine which subtask are in based Environ- ment Information above.
b.1 For each subtask in the plan_list, consider when each subtask is considered completed, express the condition in python code form. Typical completion conditions: 'Movve' com- pletion: distance under threshold Baahhldd-Grrsppp -'Grasp' completion: object's z-coordinate higher than its initialize z-coordinate.
b.2 Write 'determination ffunction using 'if" and "elif" categorize the current subtask based on Environment Information consider the logic of determining subtasks completion, using 'if not' 'elif not' if needed.
b.3 The eetermination_funntiiee
return the current subtask's 's index in plan list.
Write 'progress_ ffnnctio__
c.1 For each subtask, construct a progress metric named "subprogress' indicating the progress of the corresponding subtask. The "subprogress" should be range from 0 (no progress) to (completed) Typical subprogress metric: the distance between the robot's gripper and target object normalized by its initial distance;
c.2 Based on the "determination function', corrriinttioouuuctttonnn, add the 'subprogress' computation codes in corresponding 'if" or 'elif" block.
c.3 Add the 'main progress' computation codes each subtask's 'if' or "clif" block. 'main progress' is the progress of the entire task. typically, main progress be calcu- lated by adding the current subtask idx and current subprogress, regarding all completed subprogress as '1'. The completed progress be set to '1' without computing again. c.4 'progress function' return the paign_ro and the current subtask's 's index in plan_liat list.
A.6 Example of Generated Progress Function
In this section, help better understand the generation ability and results, we provide zero-shot generated reward function on Metaworld, The response of the prompt defined in Appendix A.5 is as follows:
LLM response in Metaworld-pick-plarn
import numpy np
Plan the task into subtasks def generato_plam_Iiste
# Task decomposition: [verb + noun]
plan_ list -
'moveutouobject',
'grasppobject',
'move_tougoal',
'releaseuobject?
return plan_list
Determining the current subtask based on environment information
def octermination_fuctiio
plan_list).
Decompose the task completion into conditions for each aubtask
Subtask 1: 'move object up linalg norm (env robot. tep env. env.obj position) 0.3: return 0 # Still moving tovards the object
Subtask 2: 'grasp object'
elif env. obj positionn [2] bav.obj.co obj. inj_ittjjiiit_po [2] return Object not yet grasped
env.
# Subtask 3: move to goal>
20

---

elif np. linalg norm (env obj. position env sav.goal_position) 0.07:
return 2 Still moving tovards the goal
Subtask 4: rreeeaee object else:
return Task is completed
Progress computation for each subtask def fenrrtsosfunctioneeeo (env): plan_list generate_plan.istt0
subtask_idx
plan_list)
Initialize variables
main.progress 0.0 # Overall task progreas
subprogress 0.0 Progress the current subtask
Subtask 1: 'move to object?
if subtask_idz == 0:
# Progress of moving towards the object
initial_distance mp. norm (env robot. tcp env. obj_iijjinit_pos
parrent_distance np wew.linalg.... norm cenv.robot.. tep env.
obj.position
subprogress (current_distance
Saitial_dsstanee
main_progress subtask_idx + subprogress Task progress is the sum of subtasks
Subtask 'grasp object'
elif subtask,idx 1:
# Progress of grasping the object (object lifted above initial position)
subprogress
se_position [2] obj
obj_init_pos [2]) 0.5 Normalize based on height
main.progress subtask_idx + subprogress
Subtask 3: 'move goal
elif subtask_idx 2:
# Progress of moving towards the goal position = np. lilalllgggg.. norm norm(evv.o))) objj position env.obj_init_pos)
current.distanee.oogg.ll mp. app.inaag..o norm (env obj
position env. subprogress anv.goal.position)
Ccurrent_distancetooogooo
initial disiielldittanc_tooooaa main.progress subtask_idx + subprogress
Subtask 4: 'release object'
elif subtask_idz == 3:
# Task is complete, no further progress is needed subprogress - 1.0
main.progress 1.0
return main_progrest aubtask_ids
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes].
Justification: Please refer Section for limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
22

---

Justification:
Please refer to Appendix A. for the proof of the theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes].
Justification: Please refer Appendix A.3 for experiment information.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes].
Justification:
We will provide the code in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwticccguie//eeeeeeeeeis CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: Please refer to Appendix A.3 and A.4 for training and test details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: Please refer to the results in Section 5 for error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
24

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [No].
Justification: Partially. We report the type of compute workers, memory, time of execution in Section A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp::/neurips. http///rrrrppeecpperreeeeaa/aaaa wttppub.icccccicccccueeeoooo/oee
Answer: [Yes].
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
This paper pose no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: Please refer Appendix A.3 and A.4 for detailed information of used assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: Included in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
27

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes].
Justification: Please refer Section for the usage of LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
28


Reviews:
[{"review_id": "VvLgv4ebYF", "reviewer": "Reviewer_MTbw", "summary": "The paper introduces PRM4RL, a framework that first asks an LLM to break a long-horizon robotics task into natural-language subtasks, then derives a per-subtask progress function Φ and turns its temporal difference into a potential-based shaping reward. The authors prove policy-invariance and convergence, embed the current sub-goal description into the state, and train SAC/PPO policies on Meta-World and ManiSkill. Experiments show higher success rates and fewer LLM calls than SayCan-style planners or Text2Reward-style rewarders.", "questions": "Besides the points of Weaknesses:\n- Please compare to a hand-coded distance-to-goal or potential-based reward to isolate LLM contributions.\n- What is the actual token and wall-clock cost per episode for PRM4RL? How does performance scale to 7B-parameter models?\n- Does the same Φ generalize if the task dynamics change (e.g., object mass or friction)?\n- How does the method detect or recover from a misleading or suboptimal Φ generated by the LLM?", "limitations": "The paper included limitations in the conclusion section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**  \n  - Blends LLM-based planning and reward shaping in a way that’s conceptually sound and empirically useful.\n  - Implementation is practical—program-aided prompting keeps LLM outputs structured and reproducible.\n  - Outperforms published LLM-augmented RL methods on multiple simulated manipulation tasks, with clear ablation.\n  \n**Weaknesses:**  \n  - Potential-based shaping for hierarchical RL is well studied ([1], [4]), and recent LLM-based curricula like CurricuLLM ([3]) and ICIRA-24 skill shaping ([2]) tackle closely related problems but are not cited.\n  - Only simulated robotic arms with proprioceptive state; no vision, no physical robots, and limited ablations (e.g., no hand-crafted potential or naive reward as control). Meanwhile, the submission does not provide codes to reproduce the experiments.\n  - No analysis of how robust Φ is to environment changes.\n\n**References**  \n- [1] “Policy Invariance under Reward Transformations.” ICML 1999.  \n- [2] “Utilizing Large Language Models for Robot Skill Reward Shaping in RL.” ICIRA 2024.  \n- [3] “CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Models.” ICRA 2025.  \n- 4] “Potential-Based Reward Shaping for Hierarchical Reinforcement Learning.” IJCAI 2015.", "was_revised": true, "final_justification": "I appreciate authors for their further feedback. I would be happy to increase my score and encourage authors to put the new results in the revisions.", "rebuttal": "# Response to Reviewer MTbw\n\nWe thank Reviewer MTbw for their thoughtful review. We are grateful for their recognition that our framework is **\"conceptually sound and empirically useful\".** In response to their valid concerns, especially regarding the evaluation, we have conducted **new experiments** and provide detailed clarifications below. We hope this new evidence will resolve the main issues raised and demonstrate the significance of our contributions.\n\nDue to the character limit, we have kept our responses as concise as possible, but we warmly welcome the opportunity to elaborate on any of these points in greater detail during the discussion period. \n\n---\n\n### **Weakness 1**\n> **Reviewer's Comment:**\n> \"Potential-based shaping for hierarchical RL is well studied ([1], [4]), and recent LLM-based curricula like CurricuLLM ([3]) and ICIRA-24 skill shaping ([2]) tackle closely related problems but are not cited.\"\n\n**Our Response:**\n\nThank you for highlighting these important connections and references.\n\n* **On Our Novelty:** We agree that our work builds upon the strong foundation of potential-based reward shaping. Our novelty is twofold:\n  1. We are **the first framework to introduce and formalize** potential-based reward shaping within LLM-augmented RL, providing proven **convergence guarantees**.\n  2. We introduce a **highly efficient structure** that uses a single LLM call to synergistically generate the entire hierarchical setup: the plan, the potential function (Φ(s)), and a novel, zero-cost subtask tracker (Ψ(s)).\n  \n  Therefore, while the underlying theory is classic, our contribution is its novel application and the creation of an efficient, automated framework. The significant performance of this principled integration is validated by our ablation studies.\n\n* **On Citations:** We appreciate the references and will add a discussion of of these relevant works to our Related Works section to better contrast our approach.\n\n### **Weakness 2**\n> Reviewer's Comment:\n> \"Only simulated robotic arms with proprioceptive state; no vision, no physical robots, and limited ablations (e.g., no hand-crafted potential or naive reward as control). Meanwhile, the submission does not provide codes to reproduce the experiments.\"\n\n**Our Response:**\n\nWe thank the reviewer for these detailed points.\n\n1. **On Code Availability:** We would like to respectfully clarify that our **full codebase was provided in the supplementary material**. We apologize if this was not sufficiently clear and will state it more prominently in the final version.\n\n2. **On Ablations and Baselines:** We have conducted new experiments with additional hand-coded baselines. For the sake of brevity, we present new results and analysis in our response to **`Question #1`**.\n\n3. **On Experimental Scope:** \nWe would like to clarify that our framework is indeed **compatible with visual inputs**. In modern robotics, proprioceptive state often coexists with vision as \"privileged information\" during training. A key future direction is to use our state-based PRM4RL to generate high-quality, dense rewards to **guide and accelerate the training of a vision-based policy**. Our work provides the algorithmic foundation for this future direction.\n\n### **Weakness 3**\n> Reviewer's Comment:\n>  \"No analysis of how robust Φ is to environment changes.\"\n\n**Our Response:**\n\nA new experiment has been conducted to analyze the robnustness. We used the **exact same Φ function** to train three separate policies from scratch in three environments **with different physical dynamics.** \nThe results are summarized below:\n\n| Training Env | Final Success Rate | Steps to Convergence |\n|:-:|:-:|:-:|\n| Original Env  | 1.00  |  $\\approx$ 1.25M     |\n| Object Mass x1.5  | 0.99   |  $\\approx$ 1.60M     |\n| Friction Param x1.5 | 1.00  |  $\\approx$ 1.10M     |\n\nThe policies trained with the same Φ function achieve near-perfect success in all conditions, which demonstrates that **Φ is a robust reward signal across a range of environment conditions,** not overfit to specific dynamics.\n\n---\n\n### **Question 1**\n> **Reviewer's Question:**\n> \"Please compare to a hand-coded distance-to-goal or potential-based reward to isolate LLM contributions.\"\n\n**Our Response:**\n\nWe introduce two hand-coded baseline in new experiments:\n\n1. Hand-coded Potential-Based Reward: An expert-designed, potential-based reward requiring significant manual engineering following [1].\n\n2. Hand-coded Distance-to-Goal (Oracle): The default environment reward function, which acts as a naive hand-coded distance-based signal.\n\n| Method| MW-drawer | MW-window | MW-faucet | MS-PickCube | MS-PullCube |\n|:-:|:--:|:--:|:-:|:-:|:-:|\n| Our Method | 1.00      | 1.00      | 1.00      | 0.99        | 1.00        |\n| Hand-coded potential-based reward  | 1.00      |  0.99     |  1.00     |  1.00       |  1.00       |\n| Hand-coded distance-to-goal reward (Oracle)| 0.75      | 0.51      | 0.26      | 0.19        | 0.13        |\n\nThe results show a clear hierarchy: naive Distance-to-Goal rewards fail on complex tasks, while our automated `PRM4RL` consistently **matches the performance of a meticulously hand-engineered potential-based reward.** This powerfully demonstrates that our framework successfully automates expert-level reward design.\n\nReference: [1] Ng, Andrew Y., Daishi Harada, and Stuart Russell. \"Policy invariance under reward transformations: Theory and application to reward shaping.\" Icml. Vol. 99. 1999.\n\n### **Question 2**\n> **Reviewer's Question:**\n>  \"What is the actual token and wall-clock cost per episode for PRM4RL? How does performance scale to 7B-parameter models?\"\n\n**Our Response:**\n\n1. **Cost Analysis:** \n   \n   Our method's cost is a single, upfront expense, not a recurring per-episode cost. The specific numbers for this one-time call are detailed below (Metaworld-pick-place task, GPT-4o).\n\n    | Method      | LLM Calls | Token Cost (Input / Output) | Wall-Clock Cost  |\n    |:-:|:-:|:-:|:-:|\n    | Our Method  | 1  | 875 / 628 Tokens | 9.776 seconds|\n\n    What is crucial to understand is that this \"One-Time Generation\" represents **a fundamentally new and more efficient LLM Usage Pattern** compared to prior paradigms, as the table below shows, our method achieves the **best performance with the lowest cost**.\n    \n\n    | Method| LLM Usage Pattern | Total LLM Calls(Typical 1M-step Training) | Average Success Rate on Metaworld / Maniskill |\n    |:-:|:-:|:-:|:-:|\n    | Our Method  | One-Time  | 1  | 0.99 / 0.98         |\n    | Text2Reward | Iterative, Human-in-the-Loop Refinement | N (N $\\geqslant$ 1 cycles)                 | 0.62 / 0.26         |\n    | ELLM        | Continuous, Per-Timestep Planning       | $\\approx 1000000$                          | 0.68 / 0.38         |\n\n2. **Scaling to Smaller Models:** \n   We conducted new experiments with a open-source 8B model (Qwen3-8B). We evaluated both the model's ability to generate a valid function (\"Generation Success Rate\") and the performance of the policy trained with that function (\"Task Success Rate\").\n   \n   | LLM|  Generation Success Rate| Task Success Rate(if generated) |\n   |:-:|:-:|:-:|\n   | GPT-4o   |1.0 | 1.00 |\n   | Qwen3-8B |0.9 (9/10 runs)| 0.99|\n   \n   These results show that our structured prompting makes the generation task feasible even for a much smaller 8B model, which produced high-performing reward functions with high reliability. \n\n### **Question 3**\n> **Reviewer's Question:** \n>  \"Does the same Φ generalize if the task dynamics change (e.g., object mass or friction)?\"\n\n**Our Response:**\n\nWe respectfully refer you to our **response to `Weakness #3`** above.\n\n### **Question 4**\n> **Reviewer's Question:** \n>  \"How does the method detect or recover from a misleading or suboptimal Φ generated by the LLM?\"\n\n**Our Response:**\n\nWe interpret a \"misleading or suboptimal\" generation as a form of LLM hallucination, which is a known and fundamental challenge for all LLM-based methods. Our primary approach is not passive recovery, but **proactive mitigation** through a two-pronged strategy:\n\n1. **Framework Structure:** By **decomposing a complex, long-horizon problem into a series of simple, manageable subtasks**, we fundamentally reduce the complexity of the code generation and task understanding challenge. This is **empirically validated** by our method's success on multi-stage tasks (e.g., **1.00 success** on `MW-pick-place`) where the monolithic Text2Reward approach fails (**0.00 success**).\n\n2. **Structured Prompting:** For each simple subtask, our **structured, Pythonic prompt** communicates the task in a format that code-specialized LLMs are best at understanding. By providing a clear, class-based \"API\" of the environment information (see `Appendix E`), we **minimize ambiguity** and guide the LLM to generate code within this well-defined, familiar structure. **A new ablation experiments** confirms its importance: replacing it with a standard natural language prompt dropped the generation success rate to just 30%, demonstrating our approach is crucial for reliability.\n   \n   In practice, we found this two-pronged approach to be highly reliable, leading to the successful generation of valid, high-quality functions for all 15 diverse tasks in our experiments.\n\n* **Limitations and Future Work:** We see great potential in **synergizing our method with iterative refinement systems** like Eureka [1]. PRM4RL could provide **a high-quality \"perfect initialization\"** for such systems, drastically reducing their search space. We believe this combination is a promising direction for future work.\n  \n  References: [1] Ma, Yecheng Jason, et al. \"Eureka: Human-level reward design via coding large language models.\" arXiv preprint arXiv:2310.12931 (2023)."}, {"review_id": "W8mAUlcZIz", "reviewer": "Reviewer_ccSg", "summary": "This paper presents the Progress Reward Model for Reinforcement Learning (PRM4RL), a novel framework that combines high-level task decomposition with low-level dense reward shaping to enhance RL learning efficiency in sparse reward tasks. The approach leverages LLMs to automatically generate subtask priors, a progress function for each subtask, and a determination function that tracks the agent’s position within a hierarchical task structure. With potential-based reward shaping, the method provides dense, informative feedback while preserving policy optimality and ensuring convergence by theoretical analysis. Extensive experiments on complex robotic manipulation tasks MetaWorld and Maniskill demonstrate that PRM4RL consistently outperforms previous LLM-based pure planning or pure reward-generation approaches in both learning performance and sample efficiency.", "questions": "See weakness 1 and 2.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper effectively combines the capabilities of LLMs as both a planner and a reward shaper, providing fine-grained task decomposition and reward guidance. It proposes an efficient method to minimize LLM calls by extracting both the task plan and logical functions within a single code-format prompt, instead of heavily per-step realtime LLM query.\n2. The theoretical analysis demonstrated for optimal policy preservation and convergence with the proposed potential-based reward shaping, proving that the designed rewards will not lead to policy degradation or instability.\n3. Extensive experiments show clear improvement over baselines, including ELLM (pure planner) and T2R (pure rewarder), particularly in sample efficiency and in solving long-horizon, sparse reward tasks where baselines often fail. The ablation study is well-justified to show the benefits of designed modules.\n\nWeakness:\n1. Although the framework is more efficient in LLM call frequency, it remains unclear if the LLM prompt design is robust across highly diverse environments or tasks and whether significant prompt engineering is needed for each new domain. It is not clear whether this fewer query method sacrifices the robustness of intensive real-time call methods to rare situations and tasks not seen during training.\n2. In the ablation experiment in Fig. 5, there is almost no difference in the impact of removing the prior on performance, and it only slightly improves the training efficiency. Does it mean that the benefit of priors has been somehow encoded into the PRM, so the prior does not actually have a significant effect then. Conversely, this also indicates that the effect of PRM depends heavily on the design of the prior decomposition subtask. Can ablation experiments be added to better explain the relationship between prior and PRM?\n3. The current formulation of PRM and plan lists rely on structured, geometry-driven metrics (e.g., position, distance related tasks). I'm interested in whether the progress function can generalize to tasks requiring complex reasoning or abstract metrics, such as improving movement fluency or something else. Can PRM be extended to evaluate progress in such scenarios?", "was_revised": true, "final_justification": "I think the author's response solved most of my problems. Although the high-level ideas of this article belong to the same family as Eureka and text2reward and other series of works, I think the methods and experiments of this article are still decent methods with practical significance. I hope the author can add the concerns mentioned by all reviewers to the limitation or related discussions. I would like to raise my rating to 5.", "rebuttal": "# Response to Reviewer ccSG\n\nWe sincerely thank Reviewer ccSg for their thorough and positive review. We are particularly grateful for their clear and insigtful recognition of our framework's key strengths, including **the effective and efficient integration of planning and reward shaping**, **the strong theoretical guarantees**, and **the well-justified experimental results**. The reviewer's thoughtful questions regarding robustness, the role of the prior, and generalization to abstract metrics are highly insightful, and we welcome this opportunity to elaborate on these points and discuss the future potential of our framework.\n\n---\n\n### **Weakness 1**\n> **Reviewer's Comment:** \n> \"Although the framework is more efficient in LLM call frequency, it remains unclear if the LLM prompt design is robust across highly diverse environments or tasks and whether significant prompt engineering is needed for each new domain. It is not clear whether this fewer query method sacrifices the robustness of intensive real-time call methods to rare situations and tasks not seen during training.\"\n\n**Our Response:**\n\nWe thank the reviewer for these crucial questions about the practicality and robustness of our framework. We address each of the three points below.\n\n1. **On Prompt Robustness for Diverse Tasks:** Our prompt is designed to be highly robust within a given domain by using a structured, class-based representation of the environment. As detailed in `Appendix E`, we define the environment's components using Python classes like `BaseEnv` and `Robot`. This structured format is highly effective as it communicates the task in a language that code-specialized LLMs excel at understanding. This ensures that for any new manipulation task, the **core prompt structure remains unchanged;** often, only the **one-line natural language task description needs to be updated.** The successful application of this prompt structure **across all 15 diverse tasks** is strong evidence of its robustness.\n\n2. **On Prompt Engineering for New Domains:** \nThe reviewer's point on prompt engineering is insightful. Our class-based design clarifies what this \"engineering\" entails. \nTo adapt our framework to a completely new domain (e.g., from robotics to navigation), an expert would need to define a new set of classes that describe that domain's state and action spaces (see `Appendix E`). We argue that this is a **systematic, one-time effort to create a domain-specific API**, rather than the iterative work often associated with \"prompt engineering.\" Our work provides a clear template for this robust approach.\n\n3. **On the Efficiency vs. Robustness Trade-off:** \n   Our framework is designed to strike a highly practical balance that delivers both state-of-the-art robustness and feasible efficiency. The reviewer correctly identifies \"intensive real-time call methods\" as a theoretical alternative, but we argue this paradigm is **computationally prohibitive for standard RL training.** A typical training run of millions of steps would require an equal number of LLM calls, incurring infeasible latency and financial cost. The table below provides a clear cost-benefit analysis against the baselines from our paper:\n\n   | Method      | LLM Usage Pattern | Total LLM Calls(Typical 1M-step Training) | Average Success Rate on Metaworld / Maniskill |\n   |:-:|:-:|:-:|:-:|\n   | Our Method  | One-Time  | 1  | 0.99 / 0.98         |\n   | Text2Reward | Iterative, Human-in-the-Loop Refinement | N (N $\\geqslant$ 1 cycles)   | 0.62 / 0.26         |\n   | ELLM        | Continuous, Per-Timestep Planning       | $\\approx 1000000$ | 0.68 / 0.38         |\n\n   As the table shows, our approach is not just orders of magnitude more efficient, but it also delivers superior final performance.\n\n   Furthermore, regarding **robustness on \"tasks not seen during training,\"** our generalization experiments (`Section 5.2.3`, line 280~290, Figure 7) show that our framework's structured priors enable significantly better zero-shot performance on unseen tasks than baselines.\n\n   Finally, to **ensure robustness and mitigate LLM hallucination,** we employ a **proactive, two-pronged mitigation strategy:**\n   * **Framework Structure:** Our framework decomposes complex problems into simple subtasks, fundamentally reducing the generation difficulty. This is **empirically validated** by our method's success on multi-stage tasks (e.g., 1.00 success on MW-pick-place) where monolithic approaches like Text2Reward fail (0.00 success).\n   * **Structured Prompting:** Our structured, class-based prompt provides a clear \"API\" for the LLM, which is crucial for reliability (Appendix E). **A new ablation** confirms this: replacing our structured prompt with a standard natural language prompt caused the generation success rate to plummet to just 30%.\n### **Weakness 2**\n> **Reviewer's Comment:** \n> \"In the ablation experiment in Fig. 5, there is almost no difference in the impact of removing the prior on performance, and it only slightly improves the training efficiency. Does it mean that the benefit of priors has been somehow encoded into the PRM, so the prior does not actually have a significant effect then. Conversely, this also indicates that the effect of PRM depends heavily on the design of the prior decomposition subtask. Can ablation experiments be added to better explain the relationship between prior and PRM?\"\n\n**Our Response:**\n\nWe thank the reviewer for this close reading of our ablation study and for asking for a deeper explanation of the prior's role. Our existing experiments demonstrate its key contributions as follows:\n\n\n* **Impact on Convergence Speed:** As the reviewer noted, the prior improves training efficiency. The training curves in Figure 5a show that the policy without the subtask prior (`Our w.o. prior`) exhibits **slower convergence.** The prior provides essential high-level context that guides the policy search more efficiently.\n\n* **Critical Role in Generalization:** The prior's most critical contribution is revealed in our generalization experiments(Section 5.2.3, line 280~290, Figure 7). The results are summarized as follows. \n  \n  | Method         | Original Task | Unseen Task 1 | Unseen Task 2 | Unseen Task 3 | Unseen Task 4 |\n  |:--------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n  | Our Method     | 1.00          | 0.96          | 0.90          | 0.78          | 0.76          |\n  | Our w.o. prior | 1.00          | 0.86          | 0.46          | 0.70          | 0.28          |\n  \n  We must first sincerely apologize for a confusing caption in the original Figure 7, where the label `Our-re` was used for our model without the prior (`Our w.o. prior`). When correctly interpreted, the results are unambiguous: the `Our w.o. prior` model shows **a significant drop in performance on all unseen tasks.** This demonstrates that the prior, encoded in our \"verb + noun\" format, is crucial for the policy to generalize its learned skills.\n\n**Synergy between Prior and PRM:** These results reveal that the prior and the PRM are synergistic, not redundant. The `PRM` provides the dense, low-level reward signal (what to do now), while the `prior` embedding provides the policy with high-level contextual information (what is the current abstract goal) . It is this context that enables both faster learning and, most importantly, better generalization.\n\nWe believe our existing experiments already demonstrate this crucial relationship. We will correct the caption in Figure 7 and revise the text in Section 5.2.2 to articulate this synergy more explicitly in the camera-ready version\n### **Weakness 3**\n> **Reviewer's Comment:** \n> \"The current formulation of PRM and plan lists rely on structured, geometry-driven metrics (e.g., position, distance related tasks). I'm interested in whether the progress function can generalize to tasks requiring complex reasoning or abstract metrics, such as improving movement fluency or something else. Can PRM be extended to evaluate progress in such scenarios?\"\n\n**Our Response:**\n\nThis is an excellent and truly forward-thinking question. We are genuinely grateful to the reviewer for highlighting this future direction, which pushes beyond the standard geometric goals and explores the broader potential of our framework.\n\nThe reviewer is correct that our current experiments focus on tasks where progress is intuitively measured by geometric metrics. However, the PRM4RL framework is **not inherently limited to these metrics.** The core mechanism is prompting an LLM to write a Python function for Φ(s) based on the provided state space description. If the state space included more abstract features related to movement quality, our framework is well-equipped to generate a corresponding progress metric.\n\nFor example, to address the reviewer's insightful suggestion of **improving movement fluency,** we could include the robot's joint jerk (the time derivative of acceleration) in the state observation. Then, we could add a natural language instruction like: \"Minimize the integral of the squared jerk during the movement.\" in the prompt. The LLM could then generate a subprogress function that take movement fluency into consideration.\n\nThe reviewer's suggestion has inspired us to recognize that the true strength of our framework lies in its ability to **translate any high-level, language-described goal**—be it geometric, physical, or abstract like 'fluency'—**into a computable reward signal.** Exploring these more nuanced objectives is a key avenue for future research, for which our framework provides a strong and flexible foundation.\n\nWe again thank the reviewer for this insightful and forward-thinking question. It has sparked what we feel is a valuable discussion, and **we would be very happy to engage in further dialogue on this topic during the upcoming discussion period.**"}, {"review_id": "2a8wJumeiE", "reviewer": "Reviewer_nbNu", "summary": "The paper proposes PRM4RL, a framework that combines high‑level task decomposition and low‑level dense reward shaping to tackle long‑horizon, sparse‑reward robotics tasks. An LLM first decomposes a complex task into a list of natural‑language subtasks, then auto‑generates a subtask determination function that maps raw states to the current subtask index, and a progress function whose temporal difference serves as a potential‑based shaping term. The resulting progress reward model provides dense rewards for the task. Empirical results on 10 MetaWorld and 5 ManiSkill manipulation tasks show faster convergence and higher success rates than prior LLM‑planner and LLM‑rewarder baselines, often matching or surpassing expert “oracle” rewards and requires fewer LLM calls.", "questions": "N/A", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The proposed method bridges planning and reward shaping. Unlike previous work that addresses only one side, the proposed method uses the LLM once to obtain both a high‑level plan and a dense reward, reducing engineering effort.\n- Results on 15 challenging manipulation tasks are good. The proposed method has near‑100 % success on cases where previous methods have < 40 % performance. The proposed method also converges 2‑3× faster than baselines.\n- The ablation is insightful. Removing either subtask priors or the PRM reward degrades performance. \n- The learned policies generalize better to unseen tasks than baselines.\n\nWeaknesses\n- The proposed method seems like a combination of LLM-planner and Text2Reward.\n- The scope is narrow. All experiments are simulated robotics with dense state access. The trained policy can only do one task. This is very different from recent vision‑language-actions models.\n- The proposed method, similar to previous methods, relies heavily on environment knowledge. \n- Text2Reward functions are reused from prior work, which is done two years ago with a much weaker language model.", "was_revised": true, "final_justification": "The authors addressed most of my concerns regarding novelty and comparison to previous work, so I increased my rating.", "rebuttal": "# Response to Reviewer nbNu\n\nWe sincerely thank Reviewer nbNu for their concise and detailed review. We are especially grateful for the reviewer's thorough summary of our work's strengths and found our results on 15 challenging tasks to be **\"good\"** with **\"near-100% success\"** where prior methods fail, **\"2-3x faster\"** convergence, and **better generalization**. We also appreciate that they found our ablation study **\"insightful\"**.\n\nGiven this strong positive assessment of our empirical contributions, we understand that the reviewer's primary concerns are the conceptual weaknesses raised. We welcome this opportunity to address each of these points directly.\n\n---\n\n### **Weakness 1**\n> **Reviewer's Comment:** \n> \"The proposed method seems like a combination of LLM-planner and Text2Reward.\"\n\n**Our Response:**\n\nWe appreciate the reviewer situating our work with respect to these two important paradigms. However, we respectfully argue that PRM4RL is not a simple combination, but rather a **novel, unified framework** that synergistically integrates planning and reward shaping. Our novelty is twofold:\n\n1. **Theoretically-Grounded Reward:**  \n   PRM4RL is the **first framework to introduce and formalize the use of potential-based reward shaping** within LLM-augmented RL. Unlike prior heuristic approaches, our method constructs a reward model with **proven convergence guarantees,** ensuring robust and optimal policy learning.\n\n2. **A Synergistic and Efficient Structure LLM Usage Pattern:** \n   \n   We introduce a new, more efficient, and effective LLM usage pattern—**our \"One-Time Generation\" structure**—which requires only a single LLM call to generate the entire hierarchical setup. The following table contrasts this synergistic pattern with prior paradigms, highlighting the immense benefits in both efficiency and final performance:\n   \n    | Method      | LLM Usage Pattern  | Total LLM Calls(Typical 1M-step Training) | Average Success Rate on Metaworld / Maniskill |\n    |:-:|:-:|:-:|:-:|\n    | Our Method  | One-Time | 1 | 0.99 / 0.98 |\n    | Text2Reward | Iterative, Human-in-the-Loop Refinement | N (N $\\geqslant$ 1 cycles) | 0.62 / 0.26  |\n    | ELLM(LLM Planner) | Continuous, Per-Timestep Planning | $\\approx 1000000$  | 0.68 / 0.38  |\n\nIt is this combination of **theoretical guarantees and a novel, synergistic structure**—validated by our ablation studies —that makes our framework fundamentally more than a simple combination and represents our core contribution.\n\n### **Weakness 2**\n> **Reviewer's Comment:** \n> \"The scope is narrow. All experiments are simulated robotics with dense state access. The trained policy can only do one task. This is very different from recent vision language-actions models.\"\n\n**Our Response:**\n\nWe thank the reviewer for raising this important point, as it allows us to clarify the specific research paradigm our work addresses and how it differs from the large-scale Vision-Language-Action (VLA) models.\n\n1. **Clarifying the Research Paradigm:** Our work addresses the classic but critical challenge of **how to make Reinforcement Learning agents learn efficiently** in complex, long-horizon tasks with sparse rewards. Therefore, we followed standard practice in the RL community by using **a simple policy architecture (a 3-layer MLP)** and focused on its ability to master a single difficult task at a time. This is fundamentally different from the VLA paradigm, which focuses on achieving broad generalization through **large-scale imitation learning with complex architectures, massive parameter counts, and large-scale training.** The table below highlights this architectural difference:\n   \n   | Method                         | Main Policy Structure | Hidden Dim | Policy Params       |\n   |:------------------------------:|:---------------------:|:----------:|:-------------------:|\n   | Our Method                     |  3-Layers MLP         | 256        | $\\approx 0.0002$ B  |\n   | VLA (e,g., $\\pi_0$ [1])        | 18-Layers Transformer | 2048       | $\\approx 3.3$ B     |\n\n   This vast difference in scale underscores that our work and VLAs are designed to solve problems from different domain.\n\n2. **On Generalization to Multiple Tasks:** We would like to clarify the point that \"the trained policy can only do one task\". We highlight our **zero-shot generalization results on unseen tasks** from Section 5.2.3, Figure 7 and Appendix B.1, Figure 8.\n\n   | Method|faucet-open(original)|drawer-close|button-press|coffee|\n   |:-:|:-:|:-:|:-:|:-:|\n   | Our Method     | 1.00| 0.96 | 0.90| 0.78 | \n   | Our w.o. prior | 1.00| 0.86 | 0.46| 0.70 | \n   |     ELLM       | 0.94| 0.56 | 0.54| 0.50 |\n   | Text2Reward    | 0.98| 0.58 | 0.38| 0.40 | \n\n    These results show that under the same 3-layer MLP policy structure, our trained policies demonstrate **significantly better generalization capabilities** compared to all baselines. This success is largely attributed to our **\"verb + noun\" subtask prior**, which enables the policy to transfer learned skills.\n   \n3. **A Synergistic Future with VLA Models:** Finally, we see these two research paradigms as complementary, not competitive. Our framework, which excels at **generating high-quality rewards for complex tasks,** could be invaluable for **efficiently fine-tuning a generalist VLA model** for a specific, novel task where expert data is scarce, bridging the gap between broad knowledge and specialized skills.\n\nWe will expand on these clarifications in the final manuscript to better frame our work's contributions.\n\nReference: [1] Black, Kevin, et al. \"$\\pi_0 $: A Vision-Language-Action Flow Model for General Robot Control.\" arXiv preprint arXiv:2410.24164 (2024).\n\n### **Weakness 3**\n> **Reviewer's Comment:** \n> \"The proposed method, similar to previous methods, relies heavily on environment knowledge.\"\n\n**Our Response:**\n\nThe reviewer is correct that our method requires environment knowledge, which is true for most current approaches that apply LLMs to robotics. We view this not as a mere reliance, but as a necessary **\"grounding\"** of the LLM's abstract world knowledge into the specific physics and embodiment of the agent.\n\nOur framework is designed to make this grounding process as systematic and efficient as possible. Specifically:\n\n* **Structured Grounding:** As detailed in `Appendix E`, we achieve this grounding through a **structured, class-based Pythonic prompt.** By defining the environment with classes like `BaseEnv` and `Robot`, we provide the LLM with a clear, semi-formal \"API\" of the world it needs to reason about.\n* **One-Time Effort:** A key advantage of our framework is that this grounding process happens **only once** when designing the initial prompt. This is far more efficient than methods that might require continuous re-grounding or prompting during the learning process.\n\nThus, while environment knowledge is necessary, our framework structures this knowledge in a reusable way and minimizes the interaction cost to a single upfront step.\n\n### **Weakness 4**\n> **Reviewer's Comment:** \n> \"Text2Reward functions are reused from prior work, which is done two years ago with a much weaker language model.\"\n\n**Our Response:**\n\nThis is a very fair and insightful point regarding the importance of a strong baseline. We thank the reviewer for raising this, as it highlights an area where our manuscript was not sufficiently clear.\n\nWe sincerely apologize for the ambiguity in our experimental description (`Appendix B.2`). We would like to clarify this important detail: for all tasks not originally evaluated in the Text2Reward (T2R) paper, we followed their methodology but **used the same state-of-the-art LLM, GPT-4o, that our own method uses.** Our intention was to create the strongest possible baseline and ensure a direct, 'apples-to-apples' comparison between the frameworks themselves, controlled for LLM capability.\n\nTo make this comparison concrete, we highlight a few representative results from our experiments:\n\n| **Task**                 | **Our Method** | **Text2Reward With GPT-4o** |\n|:------------------------:|:--------------:|:---------------------------:|\n| Maniskill-PullCube       | 1.00           | 1.00                        |\n| Metaworld-push           | 0.99           | 0.32                        |\n| Maniskill-LiftPegUpright | 0.96           | 0.02                        |\n| Metaworld-pick-place     | 1.00           | 0.00                        |\n\nThese results reveal a clear pattern. While the T2R framework can succeed on simpler, single-stage tasks (e.g., `MS-PullCube`), it struggles significantly on complex, multi-stage tasks that require hierarchical reasoning (e.g., `MW-pick-place`, `MS-LiftPegUpright`). Attempting to generate a single, monolithic reward function for such tasks is far more prone to failure, even for a powerful LLM like GPT-4o.\n\nThe fact that PRM4RL still significantly outperforms this GPT-4o-powered T2R baseline—especially on complex, multi-stage tasks where it fails completely—therefore strongly supports our central claim: the **architectural advantages** of our unified, hierarchical framework are the key driver of its success, not simply the power of the underlying LLM.\n\nWe are grateful to the reviewer for pushing us to make this crucial detail more transparent. We will revise the experimental section in the camera-ready version to state this explicitly and remove any ambiguity."}, {"review_id": "IyueZYFlAE", "reviewer": "Reviewer_DqUw", "summary": "**Summary**\n\nThis paper introduces **PRM4RL (Progress Reward Model for Reinforcement Learning)**, a novel framework designed to tackle the challenges of long-term, sparse-reward tasks in reinforcement learning. The core idea is to integrate the strengths of large language models (LLMs) in high-level planning with dense, low-level reward shaping to enable more efficient policy learning.\n\nAt the **high level**, PRM4RL decomposes complex tasks into subtasks using natural language priors and evaluates progress through a fine-grained progress function. At the **low level**, it constructs a dense reward signal inspired by potential-based reward shaping, leveraging the progress function as a potential function to provide theoretical guarantees on convergence and optimality.\n\nThe approach bridges the gap between previous LLM-based methods that focused either on planning or reward shaping alone. Extensive experiments on robotics control tasks (MetaWorld and ManiSkill) demonstrate that PRM4RL achieves state-of-the-art performance, improving task success rates and reducing reliance on frequent LLM calls.\n\nOverall, PRM4RL provides a unified and theoretically grounded framework that enhances both planning and policy optimization in complex environments.", "questions": "### **Questions and Suggestions for the Authors**\n\n1. **Clarification on Progress Function Construction**  \n   Could the authors provide more details on how the fine-grained progress functions are constructed across different tasks? Specifically, are they hand-crafted per task, or is there an automated or generalizable method for defining them? The generality of the method would impact its applicability to new domains.  \n   _Stronger evidence of automation or generalization would positively impact the evaluation score._\n\n2. **Effect of LLM Errors on Performance**  \n   Given the reliance on LLMs for both task decomposition and prior knowledge extraction, how robust is the method to LLM hallucinations or errors in subtask generation? Have the authors evaluated how performance degrades when LLM output is noisy or incorrect?  \n   _A discussion or ablation showing robustness to LLM noise would strengthen the paper._\n\n3. **Computational Efficiency and LLM Usage**  \n   While the method claims to reduce LLM queries, could the authors provide quantitative data on LLM calls and overall compute cost compared to baselines? How does the method scale with more complex tasks or larger LLMs?  \n   _A clearer analysis of compute/efficiency tradeoffs would make the contribution more practically compelling._", "limitations": "Yes, the authors have explicitly acknowledged two key limitations:\n\nLLM Hallucinations: \n\nThey note that, similar to prior work, their framework may be affected by hallucinations in large language models during reasoning and task decomposition.\n\nDomain Generalization: \nThe current evaluation is limited to robotic control tasks, and the method's applicability to domains beyond robotics (e.g., NLP, games, real-world planning tasks) remains unexplored.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Quality**\n\n**Strengths:**  \n- The paper addresses a well-known and practically significant challenge in reinforcement learning: sparse rewards in long-horizon tasks.\n- The proposed framework, PRM4RL, integrates high-level task decomposition with low-level reward shaping using a theoretically grounded approach based on potential-based reward shaping.\n- The empirical evaluation is thorough, covering multiple benchmarks (MetaWorld, ManiSkill) and demonstrating improved performance over both LLM-as-planner and LLM-as-rewarder baselines.\n\n**Weaknesses:**  \n- It is unclear how sensitive the progress function construction is to task design or domain-specific heuristics. This could impact reproducibility and generalization.\n- Some ablation studies are missing—for example, analyzing the effect of planning quality or progress estimation accuracy on final performance.\n\n---\n\n### **Clarity**\n\n**Strengths:**  \n- The paper is generally well-written and easy to follow, especially in describing the motivation and contrast with prior work.\n- Figures are informative and help visualize the proposed framework and comparisons.\n\n---\n\n### **Significance**\n\n**Strengths:**  \n- PRM4RL proposes a unified framework that advances the integration of LLMs into RL in a more holistic manner than prior one-sided approaches.\n- It contributes to an important research direction—LLM-augmented RL—by demonstrating how progress tracking can inform both planning and reward shaping.\n- Potential impact is high in robotics and other real-world domains where sparse reward signals are common.\n\n**Weaknesses:**  \n- Current experiments are limited to robotics. Broader applicability in domains like navigation, games, or instruction following remains unexplored.\n- The reliance on pre-trained LLMs might limit deployment in resource-constrained settings.\n\n---\n\n### **Originality**\n\n**Strengths:**  \n- The idea of using a fine-grained **progress function** to bridge high-level planning and low-level reward shaping is novel and compelling.\n- The paper contributes a new hybrid paradigm to the LLM-RL literature, going beyond the typical planner/rewarder dichotomy.\n\n**Weaknesses:**  \n- While the integration is novel, the individual components (e.g., reward shaping, LLM planning) draw heavily on existing literature.", "was_revised": true, "final_justification": "The author’s response addressed the majority of my questions and concerns; I am maintaining my original score.", "rebuttal": "# Response to Reviewer DqUw\n\nWe sincerely thank Reviewer DqUw for the thorough and supportive review. We are grateful for the recognition that our framework is **\"novel and compelling,\"** contributes a **\"new hybrid paradigm\"** to the literature, and provides a **\"unified and theoretically grounded\"** solution to a significant challenge in RL. We welcome this opportunity to provide the requested clarificationsto the insightful questions below.\n\n---\n## **Regarding Weaknesses**\n### **1. Weakness from Quality Section**\n> **Reviewer's Comment 1:** \n> \"It is unclear how sensitive the progress function construction is to task design or domain-specific heuristics.\"\n\n**Our Response:** We address the two aspects of sensitivity below:\n* **Robustness to Task Design:** Our framework's robustness to diverse task designs is empirically validated by its successful generation of high-quality functions for **all 15 diverse tasks** in our experiments. By combining the LLM's world knowledge with our structured environment description, our method can reliably generalize across a variety of tasks.\n* **Sensitivity to Domain-Specific Heuristics:** Our framework is designed to minimize sensitivity to ad-hoc heuristics by handling domain knowledge in a systematic way. We systematically encode the necessary domain knowledge into a **reusable, pythonic, class-based \"API\"** (e.g., `BaseEnv`, `Robot`) that code-specialized LLMs are adept at processing (see `Appendix E` ). This structured approach ensures that the generation process is reproducible and not dependent on fragile, task-specific prompt engineering.\n\n> **Reviewer's Comment 2:** \n> \"Some ablation studies are missing—for example, analyzing the effect of planning quality or progress estimation accuracy on final performance.\"\n\n**Our Response:** This is a very insightful suggestion. While directly ablating the \"quality\" or \"accuracy\" of LLM-generated components is a challenging research problem, we believe a compelling proxy for the **\"effect of planning quality\"** can be found in our existing comparison against the Text2Reward (T2R) baseline.\n* Our PRM4RL operates on a **\"high-quality\" plan,** where a complex task is decomposed into a sequence of subtasks.\n* Text2Reward, by contrast, operates on a **\"low-quality\" plan,** by treating the task as a single, monolithic block.\n  \nWe highlight some results here:\n|Method|MW-button-press|MW-pick-place|MS-LiftPegUpright|\n|:-:|:-:|:-:|:-:|\n|Our Method|1.00|1.00|0.96|\n|Text2Reward|1.00|0.00|0.02|\n\nThis significant performance gap on multi-stage, complex tasks(`pickplace`, `LiftPegUpright`) powerfully demonstrates the critical **impact of high-quality planning on final performance.** We agree with reviewer that a more detailed analysis of both planning quality and progress estimation accuracy is a valuable direction for future work.\n### **2. Weakness from Significance Section**\n> **Reviewer's Comment 1:** \n> \"Current experiments are limited to robotics. Broader applicability in domains like navigation, games, or instruction following remains unexplored.\"\n\n**Our Response:** We thank the reviewer for this insightful comment. We focus on robotics as it is an ideal testbed for the long-horizon, sparse-reward RL challenges our framework is designed to solve. **The core principle of our work, however, is domain-agnostic:** it applies to any task that can be logically decomposed into sub-goals whose progress can be measured from state variables.\n\nFor instance, in games like `Minecraft`, our framework could auto-generate a dense reward signal to guide an agent through a complex crafting tree (e.g., find wood -> craft table -> craft pickaxe), a classic challenge in that domain. We believe these are promising future directions and will add a discussion of this broader potential to our paper.\n\n> **Reviewer's Comment 2:** \n> \"The reliance on pre-trained LLMs might limit deployment in resource-constrained settings.\"\n\n**Our Response:**\n\nWhile the base LLM itself can be resource-intensive, our framework is extremely **resource-efficient** at training and deployment time.\nThis is achieved through our **\"One-Time Generation\"** paradigm, where the LLM is invoked only once before training begins. During the training process, the agent is guided exclusively by the generated, lightweight Python function, **with no further LLM calls required.**\nPlease refer to **`response to Question #3`** for a comprehensive, data-driven analysis of this efficiency.\n### **3 Weakness from Originality Section**\n> **Reviewer's Comment:** \n> \"While the integration is novel, the individual components (e.g., reward shaping, LLM planning) draw heavily on existing literature.\"\n\n**Our Response:**\n\nWe are grateful for the **reviewers' recognition on our novelty and originality.** We completely agree that our primary novelty lies in the **synergistic integration** of theoretically-grounded concepts. \n\nOur contribution is the creation of what the reviewer aptly describes in the review as a **\"new hybrid paradigm\"**: an automated systemthat that connects high-level planning and low-level reward generation, distinguished by its theoretical guarantee. We are grateful for the recognition of this central contribution.\n\n---\n## **Regarding Questions**\n### **Question 1: Clarification on Progress Function Construction**\n**Our Response:**\n\nThank you for this excellent question, as it allows us to clarify a core novelty of our framework. To be clear, the progress functions Φ(s) are **not hand-crafted,** but are **fully and automatically generated by our LLM in a zero-shot manner.** This is achieved via our structured, Pythonic prompt (see `Appendix E`), where we define the environment with pythonic classes like `BaseEnv` and `Robot`, provide the LLM with a clear, semi-formal API of the world it needs to reason about.\n\nTo generate a function for a new task, a user only needs to provide a high-level, natural language task description. The LLM then uses this structured context to write the complete, fine-grained progress function. A concrete example of a progress function Φ(s) generated with this **automated, zero-shot process** is provided in `Appendix F`. The successful **application across 15 diverse tasks** in our paper is the empirical validation of this generalizable method.\n\n### **Question 2: Effect of LLM Errors on Performance**\n**Our Response:**\n\nOur primary approach to LLM hallucinations is not passive, reactive recovery, but **proactive mitigation** through a two-pronged strategy:\n1. **Framework Structure:** By **decomposing a complex, long-horizon problem into a series of simple, manageable subtasks**, we fundamentally reduce the complexity of the code generation challenge. This is validated by our method's success on multi-stage tasks (e.g., 1.00 success on `MW-pick-place`) where the monolithic Text2Reward approach fails (0.00 success).\n2. **Structured Prompting:** For each simple subtask, our **structured, Pythonic prompt** communicates the task in a format that code-specialized LLMs are best at understanding. By providing a clear, class-based \"API\" of the environment (see `Appendix E`), we **minimize ambiguity** and guide the LLM to generate code within this well-defined, familiar structure. **A new ablation experiments** confirms its importance: replacing it with a standard natural language prompt dropped the generation success rate to just 30%, demonstrating our approach is crucial for reliability.\n\nIn practice, we found this two-pronged approach to be highly reliable, leading to the successful generation of valid, high-quality functions for **all 15 diverse tasks** in our experiments.\n\nAs a **new ablation for robustness to LLM noise,** we increased the LLM's sampling temperature to 0.9. The generation success rate remained 100% and the resulting policy's performance was unaffected, confirming our framework's robustness.\n\n* **Limitations and Future Work:** We see great potential in **synergizing our method with iterative refinement systems** like Eureka [1]. PRM4RL could provide **a high-quality \"perfect initialization\"** for such systems, drastically reducing their search space. We believe this combination is a promising direction for future work.\n  \n  References: [1] Ma, Yecheng Jason, et al. \"Eureka: Human-level reward design via coding large language models.\" arXiv preprint arXiv:2310.12931 (2023).\n### **Question 3: Computational Efficiency and LLM Usage**\n**Our Response:** \n1. **Cost Analysis:** \n   Our method's cost is a single, upfront expense: the one-time call consumes 875 input tokens and 628 output tokens, with a wall-clock time of 9.776 seconds. \n\n    Our \"One-Time Generation\" is a fundamentally new and more efficient LLM Usage Pattern. The table below provides a clear cost-benefit analysis. Our method achieves the **best performance with the lowest cost**.\n\n    |Method|LLM Usage Pattern| Total LLM Calls(Typical 1M-step Training) |Average Success Rate on Metaworld/Maniskill |\n    |:-:|:-:|:-:|:-:|\n    |Our Method|One-Time|1|0.99 / 0.98|\n    |Text2Reward|Iterative, Human-in-the-Loop Refinement |N (N $\\geqslant$ 1 cycles)|0.62 / 0.26|\n    |ELLM|Continuous, Per-Timestep Planning|$\\approx 1000000$|0.68 / 0.38|\n2. **Scalability Analysis:**\n   * **With Task Complexity:** Our hierarchical approach ensures that even for a complex task, the LLM only needs to generate progress metrics for simple, decomposed subtasks. The successful application of long-horizon tasks like `MS-LiftPegUpright` is empirical evidence of this scalability.\n   * **Scaling with LLM Size:** Our framework has a high performance ceiling (benefiting from larger models like GPT-4o) and a high floor. Our new experiments confirm its robustness to scaling down, showing that even a smaller, open-source 8B model (Qwen3-8B) can reliably produce high-performing policies (90% generation success rate; 0.99 task success rate)."}]

Author rebuttal:

Meta-review: This work makes an attempt to develop the reward model in RL by LLMs. The work includes both a theoretical guarantee and an algorithmic structure. All reviewers agree that the work provides significant contributions to the community. I therefore recommend acceptance.
Decision: Accept (poster)