Title: Progress Reward Model for Reinforcement Learning via Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Large Language Models

Reward Design
Abstract: Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. 
Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. 
However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks.
In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. 
For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. 
For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization.
Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Summary**\n\nThis paper introduces **PRM4RL (Progress Reward Model for Reinforcement Learning)**, a novel framework designed to tackle the challenges of long-term, sparse-reward tasks in reinforcement learning. The core idea is to integrate the strengths of large language models (LLMs) in high-level planning with dense, low-level reward shaping to enable more efficient policy learning.\n\nAt the **high level**, PRM4RL decomposes complex tasks into subtasks using natural language priors and evaluates progress through a fine-grained progress function. At the **low level**, it constructs a dense reward signal inspired by potential-based reward shaping, leveraging the progress function as a potential function to provide theoretical guarantees on convergence and optimality.\n\nThe approach bridges the gap between previous LLM-based methods that focused either on planning or reward shaping alone. Extensive experiments on robotics control tasks (MetaWorld and ManiSkill) demonstrate that PRM4RL achieves state-of-the-art performance, improving task success rates and reducing reliance on frequent LLM calls.\n\nOverall, PRM4RL provides a unified and theoretically grounded framework that enhances both planning and policy optimization in complex environments.\n\nSTRENGTHS AND WEAKNESSES:\n### **Quality**\n\n**Strengths:**  \n- The paper addresses a well-known and practically significant challenge in reinforcement learning: sparse rewards in long-horizon tasks.\n- The proposed framework, PRM4RL, integrates high-level task decomposition with low-level reward shaping using a theoretically grounded approach based on potential-based reward shaping.\n- The empirical evaluation is thorough, covering multiple benchmarks (MetaWorld, ManiSkill) and demonstrating improved performance over both LLM-as-planner and LLM-as-rewarder baselines.\n\n**Weaknesses:**  \n- It is unclear how sensitive the progress function construction is to task design or domain-specific heuristics. This could impact reproducibility and generalization.\n- Some ablation studies are missing—for example, analyzing the effect of planning quality or progress estimation accuracy on final performance.\n\n---\n\n### **Clarity**\n\n**Strengths:**  \n- The paper is generally well-written and easy to follow, especially in describing the motivation and contrast with prior work.\n- Figures are informative and help visualize the proposed framework and comparisons.\n\n---\n\n### **Significance**\n\n**Strengths:**  \n- PRM4RL proposes a unified framework that advances the integration of LLMs into RL in a more holistic manner than prior one-sided approaches.\n- It contributes to an important research direction—LLM-augmented RL—by demonstrating how progress tracking can inform both planning and reward shaping.\n- Potential impact is high in robotics and other real-world domains where sparse reward signals are common.\n\n**Weaknesses:**  \n- Current experiments are limited to robotics. Broader applicability in domains like navigation, games, or instruction following remains unexplored.\n- The reliance on pre-trained LLMs might limit deployment in resource-constrained settings.\n\n---\n\n### **Originality**\n\n**Strengths:**  \n- The idea of using a fine-grained **progress function** to bridge high-level planning and low-level reward shaping is novel and compelling.\n- The paper contributes a new hybrid paradigm to the LLM-RL literature, going beyond the typical planner/rewarder dichotomy.\n\n**Weaknesses:**  \n- While the integration is novel, the individual components (e.g., reward shaping, LLM planning) draw heavily on existing literature.\n\nQUESTIONS:\n### **Questions and Suggestions for the Authors**\n\n1. **Clarification on Progress Function Construction**  \n   Could the authors provide more details on how the fine-grained progress functions are constructed across different tasks? Specifically, are they hand-crafted per task, or is there an automated or generalizable method for defining them? The generality of the method would impact its applicability to new domains.  \n   _Stronger evidence of automation or generalization would positively impact the evaluation score._\n\n2. **Effect of LLM Errors on Performance**  \n   Given the reliance on LLMs for both task decomposition and prior knowledge extraction, how robust is the method to LLM hallucinations or errors in subtask generation? Have the authors evaluated how performance degrades when LLM output is noisy or incorrect?  \n   _A discussion or ablation showing robustness to LLM noise would strengthen the paper._\n\n3. **Computational Efficiency and LLM Usage**  \n   While the method claims to reduce LLM queries, could the authors provide quantitative data on LLM calls and overall compute cost compared to baselines? How does the method scale with more complex tasks or larger LLMs?  \n   _A clearer analysis of compute/efficiency tradeoffs would make the contribution more practically compelling._\n\nLIMITATIONS:\nYes, the authors have explicitly acknowledged two key limitations:\n\nLLM Hallucinations: \n\nThey note that, similar to prior work, their framework may be affected by hallucinations in large language models during reasoning and task decomposition.\n\nDomain Generalization: \nThe current evaluation is limited to robotic control tasks, and the method's applicability to domains beyond robotics (e.g., NLP, games, real-world planning tasks) remains unexplored.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes PRM4RL, a framework that combines high‑level task decomposition and low‑level dense reward shaping to tackle long‑horizon, sparse‑reward robotics tasks. An LLM first decomposes a complex task into a list of natural‑language subtasks, then auto‑generates a subtask determination function that maps raw states to the current subtask index, and a progress function whose temporal difference serves as a potential‑based shaping term. The resulting progress reward model provides dense rewards for the task. Empirical results on 10 MetaWorld and 5 ManiSkill manipulation tasks show faster convergence and higher success rates than prior LLM‑planner and LLM‑rewarder baselines, often matching or surpassing expert “oracle” rewards and requires fewer LLM calls.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed method bridges planning and reward shaping. Unlike previous work that addresses only one side, the proposed method uses the LLM once to obtain both a high‑level plan and a dense reward, reducing engineering effort.\n- Results on 15 challenging manipulation tasks are good. The proposed method has near‑100 % success on cases where previous methods have < 40 % performance. The proposed method also converges 2‑3× faster than baselines.\n- The ablation is insightful. Removing either subtask priors or the PRM reward degrades performance. \n- The learned policies generalize better to unseen tasks than baselines.\n\nWeaknesses\n- The proposed method seems like a combination of LLM-planner and Text2Reward.\n- The scope is narrow. All experiments are simulated robotics with dense state access. The trained policy can only do one task. This is very different from recent vision‑language-actions models.\n- The proposed method, similar to previous methods, relies heavily on environment knowledge. \n- Text2Reward functions are reused from prior work, which is done two years ago with a much weaker language model.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the Progress Reward Model for Reinforcement Learning (PRM4RL), a novel framework that combines high-level task decomposition with low-level dense reward shaping to enhance RL learning efficiency in sparse reward tasks. The approach leverages LLMs to automatically generate subtask priors, a progress function for each subtask, and a determination function that tracks the agent’s position within a hierarchical task structure. With potential-based reward shaping, the method provides dense, informative feedback while preserving policy optimality and ensuring convergence by theoretical analysis. Extensive experiments on complex robotic manipulation tasks MetaWorld and Maniskill demonstrate that PRM4RL consistently outperforms previous LLM-based pure planning or pure reward-generation approaches in both learning performance and sample efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper effectively combines the capabilities of LLMs as both a planner and a reward shaper, providing fine-grained task decomposition and reward guidance. It proposes an efficient method to minimize LLM calls by extracting both the task plan and logical functions within a single code-format prompt, instead of heavily per-step realtime LLM query.\n2. The theoretical analysis demonstrated for optimal policy preservation and convergence with the proposed potential-based reward shaping, proving that the designed rewards will not lead to policy degradation or instability.\n3. Extensive experiments show clear improvement over baselines, including ELLM (pure planner) and T2R (pure rewarder), particularly in sample efficiency and in solving long-horizon, sparse reward tasks where baselines often fail. The ablation study is well-justified to show the benefits of designed modules.\n\nWeakness:\n1. Although the framework is more efficient in LLM call frequency, it remains unclear if the LLM prompt design is robust across highly diverse environments or tasks and whether significant prompt engineering is needed for each new domain. It is not clear whether this fewer query method sacrifices the robustness of intensive real-time call methods to rare situations and tasks not seen during training.\n2. In the ablation experiment in Fig. 5, there is almost no difference in the impact of removing the prior on performance, and it only slightly improves the training efficiency. Does it mean that the benefit of priors has been somehow encoded into the PRM, so the prior does not actually have a significant effect then. Conversely, this also indicates that the effect of PRM depends heavily on the design of the prior decomposition subtask. Can ablation experiments be added to better explain the relationship between prior and PRM?\n3. The current formulation of PRM and plan lists rely on structured, geometry-driven metrics (e.g., position, distance related tasks). I'm interested in whether the progress function can generalize to tasks requiring complex reasoning or abstract metrics, such as improving movement fluency or something else. Can PRM be extended to evaluate progress in such scenarios?\n\nQUESTIONS:\nSee weakness 1 and 2.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PRM4RL, a framework that first asks an LLM to break a long-horizon robotics task into natural-language subtasks, then derives a per-subtask progress function Φ and turns its temporal difference into a potential-based shaping reward. The authors prove policy-invariance and convergence, embed the current sub-goal description into the state, and train SAC/PPO policies on Meta-World and ManiSkill. Experiments show higher success rates and fewer LLM calls than SayCan-style planners or Text2Reward-style rewarders.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**  \n  - Blends LLM-based planning and reward shaping in a way that’s conceptually sound and empirically useful.\n  - Implementation is practical—program-aided prompting keeps LLM outputs structured and reproducible.\n  - Outperforms published LLM-augmented RL methods on multiple simulated manipulation tasks, with clear ablation.\n  \n**Weaknesses:**  \n  - Potential-based shaping for hierarchical RL is well studied ([1], [4]), and recent LLM-based curricula like CurricuLLM ([3]) and ICIRA-24 skill shaping ([2]) tackle closely related problems but are not cited.\n  - Only simulated robotic arms with proprioceptive state; no vision, no physical robots, and limited ablations (e.g., no hand-crafted potential or naive reward as control). Meanwhile, the submission does not provide codes to reproduce the experiments.\n  - No analysis of how robust Φ is to environment changes.\n\n**References**  \n- [1] “Policy Invariance under Reward Transformations.” ICML 1999.  \n- [2] “Utilizing Large Language Models for Robot Skill Reward Shaping in RL.” ICIRA 2024.  \n- [3] “CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Models.” ICRA 2025.  \n- 4] “Potential-Based Reward Shaping for Hierarchical Reinforcement Learning.” IJCAI 2015.\n\nQUESTIONS:\nBesides the points of Weaknesses:\n- Please compare to a hand-coded distance-to-goal or potential-based reward to isolate LLM contributions.\n- What is the actual token and wall-clock cost per episode for PRM4RL? How does performance scale to 7B-parameter models?\n- Does the same Φ generalize if the task dynamics change (e.g., object mass or friction)?\n- How does the method detect or recover from a misleading or suboptimal Φ generated by the LLM?\n\nLIMITATIONS:\nThe paper included limitations in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}