Title: Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

reasoning

planning

reinforcement learning
Abstract: Large language models (LLMs) excel in tasks like question answering and dialogue, but complex tasks requiring interaction, such as negotiation and persuasion, require additional long-horizon reasoning and planning. Reinforcement learning (RL) fine-tuning can enable such planning in principle, but suffers from drawbacks that hinder scalability. In particular, multi-turn RL training incurs high memory and computational costs, which are exacerbated when training LLMs as policies. Furthermore, the largest LLMs do not expose the APIs necessary to be trained in such manner. As a result, modern methods to improve the reasoning of LLMs rely on sophisticated prompting mechanisms rather than RL fine-tuning. To remedy this, we propose a novel approach that uses goal-conditioned value functions to guide the reasoning of LLM agents, that scales even to large API-based models. These value functions predict how a task will unfold given an action, allowing the LLM agent to evaluate multiple possible outcomes, both positive and negative, to plan effectively. In addition, these value functions are trained over reasoning steps rather than full actions, to be a concise and light-weight module that facilitates decision-making in multi-turn interactions. We validate our method on tasks requiring interaction, including tool use, social deduction, and dialogue, demonstrating superior performance over both RL fine-tuning and prompting methods while maintaining efficiency and scalability.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes leveraging goal-conditioned value functions to guide the reasoning process of LLM agents, with a framework that scales to large API-based models. During the offline training phase, training samples are summarized and embedded to train a lightweight goal-conditioned Q-function. At inference time, a natural language critic utilizes this value function to generate feedback in natural language, helping the LLM agent refine its reasoning. Extensive experiments across diverse benchmarks demonstrate the method’s effectiveness compared to traditional RL training and task-specific prompting approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Pros**\n\n1. The paper is clearly written and easy to follow.\n2. The proposed method is evaluated on three datasets spanning different domains and compared against a variety of baselines, demonstrating strong performance.\n3. The method shows promising efficiency.\n\n**Cons**\n\n1. The authors claim that their method benefits from training a goal-conditioned Q-function to score the likelihood of different future outcomes. A natural baseline would be to directly prompt an LLM (e.g., GPT-4) to predict probabilities of future outcomes while keeping all other components unchanged, to verify whether the trained Q-function provides an advantage.\n2. The experiments exclusively use GPT-4 as the decision-making LLM. The framework should also be evaluated on open-source models such as LLaMA or Qwen to better validate its generality—especially given that RL-based baselines use smaller models like GPT-2.\n3. The authors should include ablation studies on the number of goals $n$ and the number of refinement stages $m$ to analyze their individual contributions.\n4. Recent reasoning benchmarks such as AIME, GPQA, and Humanity's Last Exam also require multi-step reasoning, which aligns well with the paper's focus on long-horizon reasoning tasks where the goal corresponds to the final answer. This framework could achieve greater impact by demonstrating improvements on these more challenging reasoning benchmarks.\n\nQUESTIONS:\nSee Cons above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose to use goal-conditioned value functions to guide the long-horizon planning of LLM agents in domains like tool use, social deduction. Specifically, they train a light-weight (2-layer) goal-conditioned value function using offline Reinforcement Learning, and use it to guide the LLM policy to take actions during inference -- by (1) scoring their actions, (2) feeding the scores back to enable the LLM to self-correct. The authors demonstrate superior performance across a range of long-range tasks compared to established baselines like ReAct and Reflexion. Moreover, the authors show how their method performs better than and is more efficient (during inference) than baselines that integrate search (like MCTS) with LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:  \n* The paper is generally well-written and easy to follow.  \n* The authors compare their method against a wide range of baselines, including both general (e.g., ReAct, Reflexion) and domain-specific methods (e.g., AgentQ), and show strong performance improvements.\n\nWeaknesses:  \n* **Dependence on specially prepared offline data.**  The method relies on offline data that includes thoughts and step-level goals, which don’t exist by default. These annotations must be generated using a large language model (here, GPT-3.5), adding a significant preprocessing step.\n* **Domain-specific training required** In L 283: 284, the authors state -- *However, it is important to note that such methods use domain-specific prompting mechanisms and do not naively generalize to other task* --  But your proposed method also requires training with domain-specific data. The authors state that in the future, one could use the reasoning abilities of LLM agents. Wouldn't this be the same as the Self-Plan baseline? Moreover, fine-tuning them on offline data would be even more expensive -- going against very motivation of the paper. This looks like a chicken-and-egg dilemma.  \n* **Lack of more targeted ablations.** The ablation results show that using step-level goals to compute value and feeding it back into the policy LLM gives a large boost. To this end, it would be useful to see if similar improvements occur when adding step-level goals to existing baselines like ReAct, Self-Plan, or Reflexion. For example, Prompt ReAct to generate a step-level goal along with the thought, or in Self-Plan, generate multiple responses with goals before selecting the best one -- this would essentially be the same as the future work that authors propose to address the domain-specific pretraining of value function. A similar one could be used for Reflexion.  Finally, the proposed method bears a high overlap with heuristic planning with LLMs like [3] that uses pretrained value function (Pay) to guide the search. This could be a potential baseline, too.  \n* **Time comparison missing.** There is only a brief mention of runtime trade-offs (lines 286–288), but no clear analysis. It would be helpful to compare AgentQ with Mixtral-7B using MCTS with n=5 or GPT-4 generating 5 responses every time-step. Note that the proposed method also requires (1) to augment the offline data by running a GPT-3.5 model for thought and step-level goal-generation, (2) to compute embeddings of the (s, a, g) pairs using another LLM.\n\n[1]  SayCanPay: Heuristic planning with large language models using learnable domain knowledge, Hazra et al., AAAI 2024\n\nQUESTIONS:\nQuestions:\n* What is SR in Tables 1,2?\n\nMajor Points:  \n* The authors have cited the wrong paper for their offline RL approach (Reference 12). It is crucial that the authors pay attention to this. The correct Implicit Q-Learning paper is [2].  \n* Please clearly define the notations used in Lines 162–163 -- like the distinction between $\\overset{\\wedge}{V}$ and $V$ is unclear. Without a precise definition or explanation of these terms, the paper is not self-contained.\n\nMinor points:   \n* L233: natural language value -> natural language string  \n* In general, it would be useful for the reader if the authors clearly state that every rollout/trajectory also has an overall task/goal that is then further subdivided into sub-goals. A GA-MDP formulation could be used for the same [3]\n\n[2] Offline reinforcement learning with implicit q-learning, Kostrikov et al., 2021  \n[3]  Goal-Conditioned Reinforcement Learning: Problems and Solutions, Liu et al., IJCAI 2015\n\nLIMITATIONS:\nPlease see Weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Planning with a Natural Language Critic (PNLC), a method that involves training a goal-conditioned Q-function Q(s, a_thought, g), where g is a goal (a natural language summary of a state after s) on a held-out dataset D, which is a relatively small network trained on language model embeddings. At inference time, an LLM uses Q to solve problems by sampling thoughts and goals from an initial state, evaluating Q on them, and reflecting in order to come up with other candidate actions, etc. The paper demonstrates good performance relative to a number of prompting and fine-tuning baselines and demonstrates the utility of each component of the method through ablations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+Clarity: the paper is fairly easy to understand, with generally clear methods and results and good examples provided throughout.\n\n+Quality: The work uses a substantial collection of relevant baselines on well-chosen benchmarking environments.\n\n+Significance: the work shows considerable performance gains over baselines and ablations.\n\n+Originality: to my knowledge, this precise method hasn’t been used before.\n\nWeaknesses:\n\n-Significance: I have some concern about precisely how to think about these comparisons. PNLC learns from an offline dataset of examples. As far as I can tell, none of the non-finetuning baselines make use of these data — it would be helpful to have versions that make at least some use in prompting (e.g. with few-shot examples). Were these included? The baselines that do fine-tuning appear a bit underpowered…the ArCHers use GPT-2, whereas PNLC gets to use GPT-3 representations, which is a huge difference. As far as I can tell, the only other baseline that does finetuning is Agent Q, which uses a Mistral model and doesn’t get access to GPT-4.\n\n-Originality: As I said above, I don’t know any method that does precisely this; however, the paper omits a lot of relevant related work (While e.g. DeepSeek r1 [4] and Self-Taught Reasoner [3] and its follow-up work finetune the language model directly, and hence can be considered unfair baselines with much higher train-time computation budgets, it feels pretty funny to omit these sorts of ideas in related work.) including works that come quite close. Indeed the original GSM8K paper [1] discusses training verifiers to be used at inference time, and a lot of very popular work e.g. [2] builds upon this.\n\n-Clarity (minor): a lot of important details (what the offline datasets are, how finetuning baselines are treated) are reserved entirely for the supplementary. Not a huge deal, but I feel like the paper would be easier to read if some more of these details were included earlier.\n\n[1] Cobbe, Karl, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert et al. \"Training verifiers to solve math word problems.\" *arXiv preprint arXiv:2110.14168* (2021).\n\n[2] Hosseini, Arian, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. \"V-star: Training verifiers for self-taught reasoners.\" *arXiv preprint arXiv:2402.06457* (2024).\n\n[3] Zelikman, Eric, Yuhuai Wu, Jesse Mu, and Noah Goodman. \"Star: Bootstrapping reasoning with reasoning.\" *Advances in Neural Information Processing Systems* 35 (2022): 15476-15488.\n\n[4] Guo, Daya, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" *arXiv preprint arXiv:2501.12948* (2025).\n\nQUESTIONS:\n-Did any of the prompting baselines get access to any of the datasets generated for training of the main method?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Planning with a Natural Language Critic (PNLC), a scalable approach to enhance the reasoning and planning capabilities of LLM agents in multi-turn, interactive settings without requiring fine-tuning of the base model. PNLC leverages offline goal-conditioned RL to train lightweight value functions that predict the likelihood of achieving positive or negative outcomes based on proposed high-level strategies. These value functions act as natural language critics during inference, enabling LLM agents to refine their reasoning by evaluating hypothetical future scenarios without the need for expensive inference-time search. The method outperforms state-of-the-art approaches across benchmarks in web navigation, social deduction games, and persuasive dialogue tasks, while requiring significantly lower computational costs. PNLC’s modular design makes it scalable to frontier LLMs like GPT-4, but the approach is limited by the need for task-specific value functions and reliance on the base LLM's domain knowledge.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths:**  \n1. **Scalability to Frontier LLMs**:  The method scales effectively to advanced models like GPT-4, which are typically constrained in their ability to perform RL fine-tuning due to API and computational limitations.  \n\n2. **Strong Performance Across Tasks**:  PNLC demonstrates consistent success across diverse tasks requiring interactive decision-making and reasoning, such as web navigation, social deduction, and persuasion.  \n\n---\n\n### **Weaknesses:**  \n1. **Missing Baselines and Limited Critic Design**:  PNLC employs a lightweight two-layer perceptron for goal-conditioned value prediction, which may have limitations in capturing complex relationships. Exploring the use of LLMs as critic agents in a zero-shot setting could provide valuable insights into performance improvements.  \n\n2. **Structured Subgoal Reflection**:  The generation of subgoals during inference appears to mimic a manually-designed self-reflection process. Including experiments to directly compare PNLC with a baseline that uses these structured subgoals as self-reflection materials would strengthen the paper’s claims and highlight the added benefits of PNLC's approach.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}