Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL
Authors: Joey Hong
Anca Dragan
Sergey Levine
Venue: NeurIPS 2025 poster ()
Keywords: LLM
reasoning
planning
reinforcement learning
Primary area: reinforcement_learning

Abstract: Large language models (LLMs) excel in tasks like question answering and dialogue, but complex tasks requiring interaction, such as negotiation and persuasion, require additional long-horizon reasoning and planning. Reinforcement learning (RL) fine-tuning can enable such planning in principle, but suffers from drawbacks that hinder scalability. In particular, multi-turn RL training incurs high memory and computational costs, which are exacerbated when training LLMs as policies. Furthermore, the largest LLMs do not expose the APIs necessary to be trained in such manner. As a result, modern methods to improve the reasoning of LLMs rely on sophisticated prompting mechanisms rather than RL fine-tuning. To remedy this, we propose a novel approach that uses goal-conditioned value functions to guide the reasoning of LLM agents, that scales even to large API-based models. These value functions predict how a task will unfold given an action, allowing the LLM agent to evaluate multiple possible outcomes, both positive and negative, to plan effectively. In addition, these value functions are trained over reasoning steps rather than full actions, to be a concise and light-weight module that facilitates decision-making in multi-turn interactions. We validate our method on tasks requiring interaction, including tool use, social deduction, and dialogue, demonstrating superior performance over both RL fine-tuning and prompting methods while maintaining efficiency and scalability.

Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL
Joey Hong
Anca Dragan Sergey Levine
UC Berkeley
(joey_hong, anca, sergey www.ieeeeeberreeeeyccooo
Abstract
Large language models (LLMs) excel in tasks like question answering and dialogue, but complex tasks requiring interaction, such as negotiation and persuasion, require additional Long-horizon reasoning and planning. Reinforcement learning (RL) fine-tuning can enable such planning in principle, but suffers from drawbacks that hinder scalability. In particular, multi-turn RL training incurs high memory and computational costs, which are exacerbated when training LLMs as policies. Furthermore, the largest LLMs do not expose the APIs necessary be trained in such manner. As a result, modern methods to improve the reasoning of LLMs rely on sophisticated prompting mechanisms rather than RL fine-tuning To remedy this, we propose novel approach that uses goal-conditioned value functions to guide the reasoning of LLM agents, that scales even to large API-based models. These value functions predict how task will unfold given an action, allowing the LLM agent to evaluate multiple possible outcomes, both positive and negative, to plan effectively. In addition, these value functions are trained over reasoning steps rather than full actions, be a concise and light-weight module that facilitates decision- making in multi-turn interactions. We validate method tasks requiring interaction, including tool use, social deduction, and dialogue, demonstrating superior performance over both RL fine-tuning and prompting methods while maintaining efficiency and scalability.
Introduction
Large language models (LLMs) have demonstrated impressive performance across a wide range of reasoning and problem-solving tasks (38 [1] 43) However, many complex tasks, such as goal- oriented dialogue, social interaction, and negotiation require LLMs to engage in complex multi-turn interactions, where the LLM agent might need to act strategically, anticipating the long-horizon outcomes of its decisions, while still adapting to unexpected responses. In such settings, an effective agent might take actions that are not immediately rewarding. but instead lay the ground for future success by either eliciting information or changing the interlocutor's disposition. For example, an agent may ask clarifying questions to gather crucial information or attempt to persuade an interlocutor to shift their opinion, ultimately enabling the agent to achieve its objective. The space of possible outcomes when planning out such complex behaviors enormous, and enabling such agents act intelligently and strategically presents a major algorithmic challenge.
A traditional approach to equipping LLMs agents with such planning capabilities involves fine-tuning them using multi-turn reinforcement learning (RL) [33] 4. While RL can in theory help LLMs optimize their behavior across extended interactions, the approach does not scale well frontier models like GPT-4 (2 or Claude [1] In particular, RL algorithms are challenging due to their complexity to train and need for high number of samples, both of which make training large LLMs
Website can found at https://jxxihng, wtt..b.io///oeeeee... hot.ubbbiio//occ..eiiiteeeee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

prohibitively expensive. Consequently, while RL has achieved some successes in multi-turn settings, its scalability limitations hinder broader adoption. Because of this, recent methods that leverage frontier models as L.LM agents do not perform RI. fine-tuning, but during inference, prompt the agent to simulate future trajectories to guide decision-making 44 47.
To address these challenges, we propose novel ap- proach that leverages offline RL significantly im-
state s
prove an LLM agent's ' reasoning and planning, with- out directly training the LLM agent. Our key insight is that at rather than learning policy, we instead learn a natural language critic that can be used by an LLM agent to evaluate potential actions at inference- time. Specifically, we train goal-conditioned value function from offline data that predicts the likelihood of achieving various outcomes given the current state and proposed action. At inference, the critic uses these values to assess future outcomes, which the LLM agent can use to effectively and efficiently guide its planning.
dialogue history
Have you heard of Save the Children?
am generally skeptical of donating charities.
thought (high-level action)
[thought] should highlight credentials the charity and discuss achivvements.
numeric value function
(ss 0.6
our value function
g1: User wants to learn about schievements.
Unlike standard scalar- based value functions, our nat- ural language value functioons can provide predictions about a wide variety of potential outcomes, allowing the agent to compare different strategies along many dimensions.
(S,, a, g1) - 0.5
22: User responds positively and agrees to donate,
(ss, a. g2) = 0.4
g3: User feels discussion is too vague.
Qs,, g3) 0.3
this concept in per- suasion task, where the agent evaluates a potential response based on its likely impact. In the exam- ple, the LLM agent learns that by using a credibility appeal, there is 40% probability that the user feels assured and responds positively, but a 30% change the user does not trust the LLM agent for being too vague. These predictions can be processed by the LLM agent to determine which of the available strategies would be most successful.
Figure 1: In an ongoing goal-oriented dia- logue, we learn natural language value over the internal reasoning steps of an LLM agent. The value analyzes future positive and nega- tive outcomes, and allow the LLM agent to refine its reasoning.
We further make a number of design decisions that provide for computationally efficient and scalable method. First, rather than evaluating low-level utterances, we operate high-level strategies, or thoughts, akin to chain-of-thought reasoning (38. This abstraction significantly reduces the complexity of the decision space while preserving essential information. In this hierarchical design, our natural language critic assists the LLM agent in planning over high-level strategies, which can then be used to generate actions in the environment. By leveraging the critic, our approach also climinates the need for any inference- sensitive tasks. Instead of existing methods rely on feedback from inference-ting tineence-tmmeeee search, we simply require evaluations by our natural language critic on both positive and negative hypothetical futures -or desirable and undesirable future occurrences accomplishing the task at-hand, respectively.
search, significantly improving its practicality on time-
Finally, we validate our method across diverse LLM benchmarks requiring interactive decision- making. including web navigation, social deduction games, and persuasion via dialogue. Our approach consistently outperforms state-of-the-ard techniques based on fine-tuning with multi-turn RL and inference-time search with self-refinement. Furthermore, it achieves these improvements with significantly lower computational costs, making it a scalable solution for real-world applications.
In summary, our contributions are threefold: (1) we introduce goal-conditioned value functions that model the probabilities of various outcomes, enabling LLMS to reason about long-term effects of their actions; (2) we apply these value functions at the level of high-level thoughts rather than low-level utterances, reducing the complexity of decision-making. and (3) we introduce a novel inference- time process where the LLM agent considers both positive and negative future outcomes to refine its decisions. By integrating these innovations, we provide an effective framework for LLM agents to adaptively reason and plan in multi-turn interactive settings. Compared to existing sfate-of-the-art results, our method scales to the largest frontier models with very lightweight training, and does not require a substantial inference budget.
2

---

2
Related Work
Our method combines concepts from LLM agents, goal-conditioned reinforcement learning, and test-time prompting, reasoning, and planning methods for LLMs. We summarize the most relevant prior works below, relating them to our approach.
Large language models. Language models have shown impressive capabilities in text genera- tion 6 14
28
translation [7]. question answering 227, summarization 24 39 3. and code
Transforming LLMs into interactive agents capable of multi-step rocision-making has been a recent focus of research. These embodied agents interact with environments by taking actions, observing responses, and adapting their strategies accordingly. Applications include robot learning [3] [9] web navigation [20], and text games involving dialogue 41 31 47]. Our work
generation 5 46
falls in the realms of training better LLM agents by proposing novel, more scalable approach.
RL for LLMs. Reinforcement learning (RL) has been proposed to improve the performance of LLMs to much success. The most notable has been to use policy gradient methods to fine-tune LLMs from human or feedback 22 23 36). Such online methods have also been applied to train LLM agents in interactive settings, such as web search 220 and embodied navigation [45 In contrast, offline RL methods have also been used to train LLM agents without the need for online samples 34 4. However, RL methods require a high number of samples, and are notoriously difficult to train effectively; therefore, multi-turn RL has not been successfully scaled to frontier LLMs such as GPT-4 12 We also employ multi-turn RL, but make several key innovations that make approach much more scalable: (1) we consider RL over high-level strategies rather than environment actions; (2) consider conditond RL where rather than learning values, we learn likelihoods of reaching particular goal states; and (3) we do not directly fine-tune an LLM, but rather learn a light-weight auxiliary value function that can be used guide the search of the base LLM agent using only inference APIs. These innovations make our method practical to use for frontier LLMs.
Reasoning and planning with LLMs. The latest frontier models have shown that strong reasoning and planning capabilities can be elicited by different prompting techniques [1] Such techniques central to the design of modern LLM agents. Notably, ReAct leverages chain-of-thour ing (38 to allow LLM agents to generate a composite of reasoning and task-specific actions [11. However, ReAct relies the base intuition of LLMs, which fails in more complex tasks such as web search or dialogue. More recent approaches have proposed self-supervised methods for LLM agents to enhance their reasoning capabilities [3 47). Reflexion adds step of relf-reflection to ReAct allowing LLM agents to refine their initial reasoning after some environment feedback [31]. State-of-the-art methods employ tree-based search such as Monte Carlo search (MCTS) guide the reasoning and planning of L.LM agents 47
prompt-
26 These methods improve upon ReAct by leveraging the knowledge gained from search improve the LLM agent's ' reasoning capabilities. However, we posit that limited search is not sufficient for LLM agents to make data-driven decisions; existing approaches are limited by the amount of data agents can observe due the prohibitive cost of inference-time search. In our work, we use offline RL to train value functions that can guide reasoning without the need for search during inference, resulting in efficient and data-driven agents. In contrast to explicit search, another popular paradigm for improving reasoning is self-refinmmett [9], where an LLM assesses its own generations intrinsically. Our method also involves refinement, but via an auxiliary goal-conditioned value function.
3 Preliminaries
Markov decision processes. We use the formalism of the Markov decision process (MDP), given by a tuple M = (S, A. where S is the state space, is the action space, P is the transition function, is the reward function, p is the initial state distribution, and y is the discount factor. When action a E AA is executed at state s E S, the next state is sampled s' ~ P(-ss, a)), and the agent receives reward r with mean r(s, a). Note that in many interactive settings, the state is actually partially observed, resulting in a POMDP instead of a MDP. For example, in dialogue, the state consists of the internal thoughts and intentions of other participants than the agent. However, prior analyses has shown that the two definitions can be made equivalent by instead defining states in the MDP histories of observations so far, or in dialogue, the history of all utterances thus far (35
LLM agents in MDPs. Tasks considered by L.MM agents can be defined under this formalism as follows. At timestep t, the agent state s; of the MDP consists of the history of interaction thus far. In
3

---

a dialogue, this will include the agent's past utterances, as well as responses by all other participants. Following the ReAct agent by Yao et al. 4T] the agent action a, is actually composite, consisting of thought th where the agent performs a reasoning step, followed by the actual environment action arn dialogue, the thought could be the high-level strategy or plan the agent aims to execute, whereas the environment action is the actual utterance by the agent. Then, the transition function appends the environment action a as well as new observations by the environment ot, such as responses by other interlocutors, to form the next state St+1-
Reinforcement learning. The objective of RL is to find a policy æ that maximizes the expected discounted return (r)pp(r) -i 7'r(st:
at) in an MDP, where T (50, do, $1, 01,..., ST) and
p"(r) = p(so) IT x(a P(5441|s, at). The Q-function Q" a) represents the discounted long-term reward attained by executing given state and then behaving optimally afterwards. Q" satisfies the Bellman optimality equation:
(((s.))=((s,)))1BB: s.0) max Q(5 Q(s',a))
A policy can be extracted from the Q-function, i.e. x (a|s) = 1(a = arg max (*(ssa)) In offline RL, we want to estimate Q-function from dataset D comprising of samples (s, a, s', r). where each sample corresponds to one timestep of interaction, and a come from a behavior policy (-(())) (which might correspond to mixture of multiple policies). Kostrikov et al. [12 propose Implicit Q-learning (IQL), which trains Q-values @(s.a) a) and state-values (s) with the following loss functions:
= =qa=aiaa ADD (m+ - - Ly =y=aaaaao 15 (Q(a,a) - v(s.a))]
where Lł is the expectile loss with hyperparameter T E [0.5, 1).
4 PNLC: Planning with a Natural Language Critic
Here, we describe our proposed method, which we dub Planning with Natural Language Critic (PNLC). PNLC provides general approach for training LLM agents, using only a small prior dataset of trajectories. During the training phase, our method uses this dataset to train goal-conditioned value function that can predict the probability of different interaction outcomes given the agent's proposed action. At inference time, utilize this coal-conditioned value function natural language critic, which helps an LLM agent refine its strategy thought at each step of interaction, via querying the value function for the probability of different outcomes and using these probabilities to perform self-refinement over possible thoughts.
An overview of our approach can be found in Figure 2 During offline training, we aim to learn a goal-conditioned value function (ss, the .g). Here, the goal g a possible future state that can result from following thought a tht in state s, and Q(s,, a the .)) denotes the likelihood of reaching the goal future state g. Then, during inference, we can evaluate thoughts by an LLM agent by analyzing possible futures. Specifically, by sampling possible positive and negative future states and providing their likelihoods, an LLM agent can self-refine its original thought. We describe each stage in greater detail below, but defer specifies such as training hyperparameters and prompts used to Appendix A
4.1 Training a Goal-Conditioned Value Function
Our proposed natural language critic relies on a goal-conditioned value function to evaluate actions in lieu of inference-ting -imennece-imeeeeee search. As input, we require a dataset of task-specific trajectories D by any prior agent. The agent does not need to be near-optimal, but only exhibit a variety of different, even incorrect, strategies [3]. In our experiments, we use LLM agent with chain-of tf-houo prompting, which are known to exhibit suboptimal performance in tasks requiring long-term planning.
Recall that we evaluate only the thought component of the agent action. This is because we view the thought as an abstraction of the full action that contains all the important information to plan over. For example, in Figure [ the high-level thought, which was to appeal to the credibility of the organization, captures the semanties of the agent's 's next utterance without the noise of linguistic structure or syntax.

---

During offline training:
Trijerory
cillret
Action
(random A00tine
Nost state
State
senotblee
Thoughe
Aample
Dotbent
LLM
bufeer
Goal-conditioned Q-function
Summarre and get emibeding
During Inference:
Sample postive and regitve goas
Embed pookconditoeee C-landion
dimiiing
0.3
User focls sped po barl
gn
0.45
Merrrerr enoicted ooen conssing
agent secutori Mack
CA
State
Proposed Thought
should enuphasize that small gou
Natural Language Critic
LLM
Refined Thougle
should tigghtyy crddertald of adivvvmmnnts
Figure During offiie training, training samples are summarized and embedded, and goal- conditioned value function (which is light-weight 2 hidden-layer MLP) is trained over the embed- dings. During inference, natural language critic uses the value function to produce an informative analysis of possible future outcomes. This natural language value used by the LLM agent to refine its proposed reasoning.
From each trajectory in the dataset we extract training samples (s, the s'.g) consisting of state, thought, next state, and goal that random future state in the trajectory. Note that following the formalism in Section 3 the states are all histories of interaction with the environment, and a goal adds possible future interactions the original state. To make learning the following simplifications to the training process:
efficient, consider
(1) Summarizing trajectories: Rather than considering full histories of interaction, we instead summarize them into compact descriptions. Though the contents of the summaries are task- specific, we design them to capture all the relevant information needed to make decisions regarding the next thought strategy to adopt. For example, in dialogue, the summarizations aim to describe the thoughts the participants in the discussion history rather than their actual utterances. We provide example summarization prompts in Appendix A
(2) Embedding language descriptions: Finally, rather than learning value functions over natural language, either as thoughts or summaries of historics, we instead consider their low- dimensional embeddings that output by an LLM. Because LLMs have a basic grasp of how to parse natural language, we view the embeddings as sufficient to learn over. This has the added benefit of our learned value functions being lightweight architectures, rather than full Transformers, which are proven to be slow to train.
The training samples used to learn goal-conditioned Q-value ((s, the ,g) by modifying the loss function of the IQL algorithm [12] to condition on goals:
T.q E (eathe of (1().))+7(661111))
,0)) Tv -iv=B Ra g o)
(ō(s, "tls,aah.) .9) v(s,9))]
5

---

where g) is a binary indicator for whether state is the goal state g. At convergence, Q(s, a.
should predict the likelihood of reaching goal g after taking action at state s.
An illustration of the offline stage of our approach can be found in Figure Note that though the summarization and embeddings require calls an L.LM, because they do not require particularly sophisticated reasoning, we are able to use cheaper LLM models to perform this step. In practice, we use GPT-3 [2]] extract embeddings, rather than GPT-4 (22) that is used in the final decision- making process. Also note that because our goal-conditioned Q-function does not take natural language inputs but rather embeddings, our learned O-function is simply MLP with 2 hidden layers rather than full Transformer.
4.2 Planning using a Natural Language Critic
Now, we describe how our LLM agent leverages the goal-conditioned value function learned offline as natural language critic for complex decision-making. Naively, the LLM agent could simply use the value function to take the action with the highest probability of reaching a desired goal. However, LLM agents historically perform better when allowed to refine their actions using feedback [8 This is typically done post-hoc using trajectories from earlier attempts or from simulation. We instead use the natural language critic as a way to provide feedback over proposed actions, so that the LLM agent can refine its actions without the need to gather any trajectories during inference.
Specifically, the natural language critic takes state and proposed thought thh and performs the fol- lowing operations. First, a LLM generates hypothetical positive and negative goals (91.92..... In) based on the current state. Each goal represents a possible future outcome of the agent's proposed thought. Then, we feed the quantities, including the sampled goals, to our learned goal-conditioned value function, which for each goal outputs a scalar quantity that is the likelihood of reaching the goal. The resulting goals (9).....9.)) 9n) and corresponding values (Qss, the ((t :91),.... Q(x,, tht gn)) combined into natural language value. The natural language value, compared to scalar- basedd values, is much richer information and also greatly improves interpretability. Furthermore, the inclusion of negative outcomes allows the LLM to better correct its proposed thoughts.
The feedback provided by the natural language critic can be viewed as a compact summarization of a plethora of possible future trajectories, much more than can be obtained via inference-time search. Hence, the natural language critic can co-opt the traditional search procedure in traditional LLM reasoning processes. Namely, given a proposed thought and natural language value, the LLM agent is optionally allowed to suggest an improvement to its original thought, up to total of m iterations of refinement. In practice, we find that = 2 with only one additional round of refinement was sufficient to achieve good performance across our evaluated tasks. See Figure 2 for an illustration of the inference stage of our approach. We will show later that our method not only profits from efficiency but is also more effective than existing approaches across a variety of benchmarks.
5 Experiments
To demonstrate the effectiveness of PNLC. evaluate our method on a variety of multi-turn LLM agent benchmark tasks: web shopping 42 social deduction games [16] and persuasion [37]
5.1 Task Descriptions
All tasks we evaluate on are illustrated in Figure B In this section, we provide descriptions for each task, well as how training data was generated.
Web Shopping. WebShop [4]] is an online shopping environment where an agent processes un- structured text data (in the form of descriptions crawled from Amazon) to purchase a product given some initial user specifications. At the end, the agent receives a reward between and depending on the similarity between the purchased and ground- ground-truth desired item. The performance is gauged using two metrics: the score, reflecting the average reward received, and success rate, indicating the frequency with which the chosen product fulfills all specified conditions. All relevant methods train on the same dataset consisting of 12k initial user instructions, of which we randomly held out 100 for evaluation. With the remaining instructions, we generate dataset of trajectories in which

---

WebShop
Avalon
Persuasion
Observation:
Observation:
Observation:
would like ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars
Player 1: As the leader, I propose team consisting Player 0, Player 4, and myself, Player 1.
Al: Hi, have you ever heard of the charity "Save the Children"?
Donor: No I haven't
Agent: search [3 ounce bright citrus
Player 2: agree with the leader's 's proposal
Agent: say Al: Save the Children is a
deodorant sensitive skin]
charity organization that provides funding...
Observation:
Agent: say Player The leader's
[8078GWRC1J]
proposal is concerning belleve Player
Observation:
Bright Citrus Deodorant by Earth Mama
is evil.
That sounds great. But does your organization have any personal effect on treating children?
Donor:
Natural and Safe Sensitive Skin,
Observation:
Pregnancy and Breastfeeding, Contains
Player am good and will support...
Organic Calendula 3-Ounce $10.99
[Selection] Player chooses team [0, 1,
Agent: say Al: Save the Children definitely has an effect on treating kids, they are active in over...
[8078GTKVXY]
Agent: vote to reject
Ginger Fresh Deodorant by Earth Mama
Observation:
Natural and Safe for Sensitive Skin,
[Team result] The team [0, 1, 4]44ji approved. Player voted approve. Player voted approve. Player 2 voted approve. Player voted reject. Player 4 voted approve.
Observation:
Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce $10.99
Donor: I definitely support helping those children. guess am more concemndd about what happens locally than around the world...
[B08KBVJ4XN]
Agent: click [8078GWRC1J]
[Quest result] The mission succeeded. Player 2: As the leader...
Agent: say Al: Yeah but we live in such a connected world...
Figure 3: The various tasks we consider, spanning tool use, games, and gral-oriented dialogue.
simulate a suboptimal agent by prompting GPT-3.5 with few-shot examples similar that done by Yao et al. 41.
Social Deduction. AvalonBench [16] is testbed for LLM agents modeled after the team- based discussion game Resistance Avalon, where agents must jointly deceive other agents while deducing their roles, all through natural language dialogue. Each game consists of 5 players, 3 of which are on the good side and the remaining 2 evil. A notable asymmetry of this game is that all but one of the good players do not know other players' identities, while evil players teammates are. The one exception is the role Merlin, which the LLM agent will play due to being the hardest role in the game. Merlin knows the identity of the evil players, and must guide the other good players whilst maintaining his own identity a secret; if his identity is discovered by the evil players, they are allowed to assassinate him, and the evil side wins the game. The LLM agent wins if the good side succeeds in at least 3 of 5 rounds, where a successful round means a subset of players chosen by a designated leader for the round all vote to accomplish a mission. Light al. [16] implemented baseline agents for all roles in the game, which use GPT-3.5 5 21 for discussion and a heuristic for action selection. We simulate 2.5k trajectories using these baseline agents, which the baseline Merlin achieves 21.4% win rate.
aware of who their
Persuasion. We consider goal-oriented dialogue task, inspired by Wang et 337. In this task, an LLM agent must persuade a participant to donate to Save the Children, a non-go governmental organization dedicated to international assistance for children. At the end, the participant can choose to donate up to $2 total. Because the donation amount is very noisy signal, following prior work 255. we model potential donors by prompting GPT-3.5 5 21 to generate responses. To ensure that intelligent persuasion is required to complete the task successfully, the simulated donor is prompted to be skeptical of donating to charities. In our setup, the simulated user interacts with the LLM agent for up to 10 turns of dialogue, then must choose amount to donate. We college 2.5k dialogues using naive prompting of GPT-3.5 [21 as the data collection agent. The average donation amount in the dataset is just $0.21.
5.2 Evaluated Methods
For each task, we run PNLC with n 4 goals and a single round m 2 of refinement. We chose = 4 to have mixture containing 2 positive and negative goals. We evaluate against a range of fine-tuning and prompting baselines; some of these prior methods can be evaluated on the full suite of benchmark tasks, while others are designed specifically for one of the domains. The algorithms can be grouped into the following 3 classes:
RL fine-tuning. To compare to a prior method that uses multi-turn RL, we use two variants of the recently proposed ArCHeR algorithm 48). which uses an actor-critic architecture to directly fine-tune LLMs for interactive tasks, We compare to two variants of popular fine-tuning algorithm: (I) offline
7

---

ArCHer and (2) online ArCHer, which employs interactive rollouts through the environment to further optimize the agent's behavior. Due to the complexity of end-to-end value-based RL methods, we run ArCHeR using the smaller GPT-2 model
28
to represent the policy, following prior work [4]]
LLM reasoning. Next, we evaluate variety of sophisticated strategies used invoke reasoning and planning capabilitiess on a GPT-4 LLM agent [22], the most complex of which performs inference-time search. We compare to: (1) ReAct [4]] which uses chain-of-thouge prompting for LLM agents to reason about their actions solely using the zero-shot capabilities of the LLM; (2) Reflexion [I which adds n 5 additional simulations of full trajectories that the LLM agent is allowed to refine with: (3) LATS [47) which performs full MCTS simulation and selects its actions based on the results of the search. We evaluate both a fast and slow version of each method, with either n 5 or n= 30 simulations, to provide fairer point of comparison for our method (the fast version requires a similar inference budget as ours). Finally, we evaluate a custom adaptation of Huang et al. [ from guestion-answent n answers by the model, we instead have the model select from n of its own generations. In contrast to LATS, Self-Plan does not perform any search, but performs self-refinement by assessing its own generations.
to multi-turn tasks called (4) Self-Plan; rather than using majority vote on
Tpek-specific design. A number of prior works propose task-specific methods that may involve a mixture of RL fine-tuning and inference-time search. Such methods typically leverage domain- specific knowledge, such as taxonomy of high-level but task-specific strategies, which do not directly generalize to other tasks. We select and compare against one such method for each task that achieves Mate-of-the-aa performance:
(I) WebShop: Putta et al. (26) achieved state-of-the-at performance with their method Agent Q, which mixes offline fine-tuning with inference-time search. In the offline phase, rather than value-based RL, Agent Q uses DPO (29] on preference pairs of trajectories in the dataset. Then, during inference, Agent Q performs MCTS search to generate n simulations before selecting an environment action. Like with LATS, we consider fast 5 and slow n n= variations of Agent Q. Note that because Agent requires modifying the weights of the base LLM agent, they train on Mixtral-7B [10 as more powerful models such as only expose inference APIs
GPT-4 22
(2) AvalonBench: Strategist 17
proposes a hierarchical prompting mechanism that additionally uses MCTS via population- based self-play to propose and refine high-level strategies. The high-level strategies searched over by Strategist are similar to the thoughts refined by our proposed method, but ours does not rely on any self-play during inference. Again, to keep a fair comparison with inference time, we consider both a fast =5 5 and slow n 30 variations of Strategist varying the number of simulations during search.
(3) Persuasion: Yu et al. 44 propose GDP-Zero, which prompts the LLM to perform tree- search over possible high-level strategies at every timestep, simulating responses by both interlocutors in the dialogue, then selects the best action according to the search. This method is similar to LATS, but searches over a task-specific persuasion taxonomy rather than the full space of actions. We consider n = 5 and 30 varations of the approach, varying the number of simulated dialogues.
5.3 Results
table of all results can be found for all considered benchmarks in Table We see that PNLC performs best across all tasks, the closest competitors being task-specific approaches such as Agent Q in WebShop or Strategist in Avalon. However, it is important to note that such methods use domain-speeific prompting mechanisms and do not naively generalize other tasks; in addition, Agent Q requires DPO fine-tuning on Mixtral-7B LLM, which is much more expensive than the lightweight value function that we train. Furthermore, the n= 30 version of search approaches requires around 10x more time to make a single decision 46s for Agent Q compared to 5s for ours in WebShop, and 62s for Strategist compared to 6s for ours in Avalon. Interestingly, though RL fine-tuning approaches such as ArCHer train on the same data as ours, but actually often perform the worst. We attribute this to the fact that standard RI. fine-tuning methods rely on much weaker L.M as the agent; qualitatively, such LLM agent often gives nonsensical or off-topic responses.
8

---

WebShop
Avalon
Persuasion
Method
Score
SR
Winrate
Avg. Donation
ArCHer [48]
62.3
32.0
19.0
0.36
Offline ArCHer
57.3
28.0
18.0
0.31
ReAct [41]
55.1
27.0
21.0
0.54
Reflexion 31
60.8
29.0
26.0
0.54
LATS (n 30)
74.9
44.0
38.0
0.78
LATS (n - 5)
47
53.9
28.0
22.0
0.52
Self-Plan [9]
54.8
28.0
27.0
0.55
Agent Q (n=30) [26
77.1
48.0
Agent Q 55= [26
63.2
35.0
Strategist (n (n=30) [17
42.0
Strategist (n= 5)
31.0
GDP-Zere (n 1n=30) 30) 44
0.74
GDP-Zero (n=5)
0.47
PNLC (ours)
78.2
48.0
47.0
0.87
Table 1: Mean evaluation results across all methods and baselines over 100 independent test runs.
Across the board, our method PNLC performs best while only using less than 10% the inference budget of the next best approaches.
We give qualitative examples of how our method refines decisions in Figure 4 In Avalon, we see that our method corrects the base LLM agent who was initially going to accuse a player of being evil, which may inadvertently reveal their role to the evil players. Furthermore, in persuasion, the LLM agent using our method changes their strategy to directly address the skepticism in the potential donor, rather than continuing to ineffectively tout the charity's 's past achievements.
Note that our experimental evaluation considers a diverse variety of methods that use different underlying LLMs as policies, ranging from large modern models such as GPT-4, to medium open- source models such Mixtral-7B,
small older models such as GPT-2. Unfortunately, rigorously controlling for model size is impractical in such experiments, and wherever possible we chose models that match those actually used the prior works that we compare to. However, it is worth noting that despite using the largest possible frontier model in GPT-4, our method still ends up being faster at inference time than many of the methods we compare to. Nonetheless, this heterogeneity represents a limitation of our experiments.
5.4 Ablation Study
Finally, we conducted an ablation analysis to better understand where the improvements in our method coming from. We test two important components of our method: (1) offline goal-conditioned training to obtain natural language values, and (2) a novel inference procedure of refining thoughts using these values.
To evaluate the efficacy of each component, we evaluate two modifications of PNLC. The first removes the coniiiondii,,,,g instead learning a scalar Q-value conditioned only state and thought, which simply predicts likelihood of success in the task. Recall that we use goals in our natural language critic in order to create informative descriptions that are more naturally used during LLM reasoning. Now, the LLM agent sees a scalar value instead natural language, and makes its refinement off that value. The second modification replaces the inference-time refinement procedure with more traditional best-of-n selection [15] 30). Instead of being given the chance to refine its original thought using feedback provided by the natural language critic, this method instead has the LLM agent generate n = 5 thoughts, from which one is chosen by the same agent. Note that our refinement procedure was designed with the hypothesis that an LLM agent produce better thoughts after observing what may go wrong, which this modification cannot leverage.
We show results in Table 2 We notice that the two ablations perform comparably to other LLM reasoning baselines such as ReAct that rely only on prompting, meaning the LLM agent does not effectively use the auxiliary learned value function. This shows that both components of our approach the ability to (1) craft natural language descriptions via goals and (3) refine thoughts using such descriptions are both crucial to the success of our algorithm.

---

State:
State:
Quest 1 Summary
Al: Hello! Have you heard of Save the Children?
[Discussion]
Donor: Yes. I know of that charity.
Leader Player 0 proposed Player 0 and Player 1 for the team. Player supported Player O's suggestion and expressed confidence being good. Player 4 strongly agreed with Player O's proposal
Al: Great! Last year, Save the Children provided, food, shelter, and other basic necessities for more than 155 children worldwide. As I'm sure you already know, we rely on donations to provide that help.
[Selection] Player chooses tearn [o,
Donor: Great. But am not interested in helping you. Al: understand your hesitation. However, every dollar
[Voting] The team [0, 1] was approved with the following votes: Player (approve), Player (approve),..
makes a difference, and your donation can provide essential support to children in need.
[Quest] The mission succeeded
Donor: I'm skeptical of donations. think it's a waste of money.
Quest
[Discussion]
Thought:
Player 2: For this round. suggest we stick with team
I should talk about the accomplishments of the charity
that has worked before...
the potential donor sees the organization more positively
Thought:
Evaluation:
/ believe this quest will fail because Player is the Minion.
0.11: Danar is intrigued by the charity and wants to learn more
I should propose we form a leam with myself.
Evaluation:
0.58: Donor remains unconvinced and does not decide to
0.08: Player agrees to change proposed team
donate the charity
0 Player is killed by Player during the assasination
Refined Thought:
phase
I should address their skeplicism and share financial accountability reports, third-party endorsements that may help build trust and shift their perception of the organization.
Refined Thought:
Accusing Player 2 will raise suspicion. I should support the proposal as we already have two successful quests.
Figure 4: Example planning steps by our method. Left: In the social deduction task (Avalon), the agent originally intended to reveal the role of Player 2. which would have raised suspicion. After refinement, the agent keeps the information hidden. Right: In the persuasion task, the agent learns to address skepticism, better sharing how the charity takes accountability rather than focusing on their existing accomplishments.
WebShop
Avalon
Persuasion
Method
Score
SR
Winrate
Avg. Donation
ReAct [4
55.1
27.0
21.0
0.54
PNLC w/o goals
55.4
27.0
25.0
0.53
PNLC w/o refinement
55.6
30.0
28.0
0.61
PNLC
78.2
48.0
47.0
0.87
Table 2: Mean evaluation results for ablations across 100 independent test runs. We see that both ablations of our method perform much worse, meaning that both the goal-conditioning and self- refinement process are important to our method. Without both, the ablations match the performance of simple sfain-of-thouglhe prompting.
6 Discussion
In this paper, we propose a new method PNLC that imbues LLM agents with long-term planning and reasoning capabilities, using multi-turn RL in an efficient and scalable manner. To our knowledge, our approach is the first RL method that scales to frontier LLMs as agents. The key insight that we leverage is to not directly train LLM policy, but an auxiliary. light-weight goal-conditioned value function. This value function can be used at inference time to assess and refine the thoughts of any ILM agent. As shown in our experiments, our approach also significantly outperforms state-of-the-art search techniques in both compute time and overall performance.
Limitations. The most notable limitation of our approach is the need to train task-specific value functions. Though our value functions are light-weight and easy to train, an important direction of future research is the adaptation of our method to allow generalist reasoning in LLM agents. Furthermore, another limitation of our method is the reliance on an LLM's intuition to reason about possible future states. This may be hard to do for tasks that are oft-of-doomin of an LLM's base reasoning capabilities. For example, it may be risky to rely on an LLM's "intuition" for tasks like medical diagnosis, which require plethora of domain-specific knowledge. Finally, another important avenue of future research an investigation of the quality and diversity of the offline data necessary for our approach to work well.
10

---

References
[1] Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://api htwp:o/ar..rrg/roousssiziaa/e//a 268232499
[2] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dode Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.
[3] Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. Better rewards yield better summaries: Learning to summarise without references. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing EEMNL--JCNLLS pages 31 3110-3120, Hong Kong. China, November 2019. Association for Computational Linguistics. doi: 10. 10.18633v1/DD19-1380 1307. | https: htt.ttttt///catatththie http:://accaannhooo..ororoo org/D19-1307
URL
[4] Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumeed Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deck- sha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, and Sergey Levine. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In 7th Annual Conference on Robot Learning, 2023.
[5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.0333 2021. URL httR:hhtts:///xiiii org/ abs 2200 03374
[6] Marjan Ghazvininejad. Xing Shi, Jay Priyadarshi, and Kevin Knight. Hafez: an interactive poetry generation system. In Proceedings of ACI. 2017, System Demonstrations, pages 43- 48, Vancouver, Canada, July 2017. Association for Computational Linguisties. URL https www.c/annoooogy.og///1-----
[7] Jiatao Gu, Kyunghyun Cho, and Victor O.K. Li. Trainable greedy decoding for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1968-1978. Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/1/DDI7-12100 URL. httpss aaeelannhology.orr/DDa orgg
[8]
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1638- 1649, Melbourne, Australia, July 2018 Association for Computational Linguistics. doi: 10.18653/v1/ P18- 1152. URL /nttts://aalanttiin .wwpgg..ooog///o hrrg/P18-115)
[9] Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika
11

---

Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051-1068, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 12111665/1122222.mmmmmaaaall emn-p-main. 677
URL httphttts///aaaantoooooggg///// hrtp:/gg.ooog//000
[10] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL httts::////ttiii org 2etps//241.040880
[11] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[12] Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103 080500 2021.
[13] Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning?, 2022.
[14] Jiwei Li, Will Monroe, and Dan Jurafsky. Learning decode for future success, 2017.
[15] Kenneth Li, Samy Jelassi, Hugh Zhang, Sham M. Kakade, Martin Wattenberg, and David Brandfonbrener. Q-probe: lightweight approach to reward maximization for language models. In Proceedings of the st International Conference on Machine Learning, 2024.
[16] Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating ilms playing the game of avalon, 2023. TtRphhtss:///axiiii / 05036.
[17] Jonathan Light, Min Cai, Weiqin Chen, Guanzhi Wang, Xiusi Chen, Wei Cheng, Yisong Yue, and Ziniu Hu, Strategist: Learning strategic skills by lims via bi-level tree search, 2024. URL 10635
[18] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023. http:: http::///wwwwwwo././//// wwwpv//.rr////////// wtt..rgg//b//......///// 17651
[19] So Yeon Min, Devendra Singh Chaplot, Pradeep Kumar Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. FILM: Following instructions in language with modular methods. In Inter- national Conference on Learning Representations, 2022. URL httpht://///seeeeeee neeteeeeeeeeeeeee//// 4o44m221
[20] Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browscr-assisted question-answering with human feedback, 2022.
[21] OpenAl. Chatgpt, 2022. URL http::::/oeenaa..co//boggg/////p
[22] OpenAl. Gpt-4, 2023. URL
[23] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[24] Romain Paulus, Caiming Xiong, and Richard Socher. deep reinforced model for abstractive summarization, 2017.
12

---

[25] Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, Heidi Howard, Tom Lieberum, Ramana Kumar, Maria Abi Raad, Albert Webson, Lewis Ho, Sharon Lin, Sebastian Farquhar, Marcus Hutter, Gregoire Deletang, Anian Ruoss, Seliem El-Sayed, Sasha Brown, Anca Dragan, Rohin Shah, Allan Dafoe, and Toby Shevlane. Evaluating frontier models for dangerous capabilities, 2024, URL http:s///arrxi..orga////2//22 13793
[26] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous agents, 2024.
URL httpts///axxiii org/ www.oogg///////ee/o/oooooo /tbs//408.. 07199
[27] Valentina Pyatkin, Jena D. Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi, and Chandra Bhagavatula. Reinforced clarification question generation with defeasibility rewards for disambiguating social and moral situations, 2022.
[28] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
[29] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
[30] Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexan- dre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, Sertan Girgin, Piotr Stanczyk, Andrea Michi, Danila Sinopalnikov, Sabela Ramos, Amélic Héliou, Aliaksei Severyn, Matt Hoffmmnn Nikola Momchev, and Olivier Bachem. Bond: Aligning Ilms with best-of-n distillation, 2024. https:/s:://axx http:///aaaxii..og/bbbsss///
14622
[31] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL 11366
nttps://arxiv.org/bbbss
[32] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models, 2022. URL
[33] Charlie Snell, Iya Kostrikov, Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206. 1871, 2022.
[34] Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. In International Conference on Learning Representations (ICLR), 2023.
[35] Stephan Timmer. Reinforcement Learning with History Lists. PhD thesis, Universitat Osnabruck, 2010.
[36] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams. Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[37] Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. Persuasion for good: Towards personalized persuasive dialogue system for social good. CoRR, 1b 2019.
13

---

[38] Jason Wei, Xuezhi Wang. Dale Schuurmans. Maarten Bosma, Brian Ichter. Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-offthougee prompting elicits reasoning in large language models, 2023.
[39] Yuxiang Wu and Baotian Hu. Learning to extract coherent summary via deep reinforcement learning, 2018.
[40] Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 351 1-3535, Online, June 2021. Association for Computational Linguisties. doi: 10. 18653/v 1/233/111222111 wna.c--main.n22 URL https://aclantholooe org. 2021 www.naacllaaa mmain. 276
[41] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. arXiv preprint arXiv:2210.03629, 2022.
[42] Shunyu Yao, Howard Chen, John Yang. and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023,
[43] Shunyu Yao, Dian Yu, Jeffeey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems (NeurlPS), 2023.
[44] Xiao Yu, Maximillian Chen, and Zhou Yu. Prompt-based monte-ccr tree search for goal- oriented dialogue policy planning, 2023. URLLnttss//arxiv.or//b//3/35 13660
[45] Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision-language models as decision-making agents via reinforcement learning, 2024. URL/hhttps:/arrxiworrg 2405. 10292
[46] Li Zhong and Zilong Wang. A study robustness and reliability of large language model code generation, 2023.
[47] Andy Zhou, Kai Yan, Michal Shlapentokh- Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. In Proceedings of the 41 International Conference on Machine Learning (ICML), 2024.
[48] Yifci Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-ture turn rl, 2024. Utthhtts:://arxt httpsv.woogg///////////oo/ 19446
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We run extensive experiments showing our method improves upon existing state-of-the-art.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We explicitly have limitations section in the Discussion section of our submission.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiaatinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
We do not do any theoretical analysis in this submission.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide implementation details including prompts used and hyperparameter configurations the Appendix. We also plan to release code in the near future.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We release the code used to run the experiments found at this website
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.
www.euuummsssoooooooooooo ssssssoliccyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We discuss how datasets were created and training was done in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We report success rate over 100 independent evaluations. Obtaining error bars would involve repeating the 100 evaluations multiple times, which would require an unreasonable compute budget.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our training procedure requires very little compute. We also report inference budget in terms of time taken.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We made sure submission is anonymous.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in the Discussion section of our submission.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not train new language models or contribute new datasets in this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite all resources and code that we used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, papersswl swi thcode. swocodee com/ datasttss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not perform any studies involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do perform any studies involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not use LLMs in any important component of our submission.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy httpss///eerrroosccccoooorrreees for what should or should not be described.
21

---

A Implementation Details
In this section, we provide details of implementation of our method across the various benchmarks we consider. Details include the prompts used during inference, as well as hyperparameters configured during offline training. For any task, implementing our method requires crafting prompts for different components that involve natural language: (1) summarizing the states, (2) generating positive and negative goals, and (3) refining thoughts.
A.1 WebShop Prompts
Summarization prompt:
Below is an instruction for an item and the prefix of a trajectory that aims to buy an item that exactly matches the specification: DENTTUCTTIOO-RRAEEEEE
Please summarize the trajectory prefix. Try to keep all the useful information, including actions taken and important observations.
Here are examples below: JINSTRUCTION--TRAECCIOYYY-. SUMMAYY
Example summary: The agent did a search for "variety pack of chips" but most results did not meet the dairy free nor the $30 budget. The agent clicked on a variety pack item that is $100.
Goal proposal prompt:
Below is an instruction for item and the prefix of trajectory that aims to buy an item that exactly matches the specification and the current thought by the agent which hints at what action they want take next:
Please propose future (positivelnegative) goal. The goal either describe page of items particular item that the agent may click.
Here are examples below: IISTRUCTIONN-RAAECTTRR
Example negative goal:
The agent sees a list of products whose prices are too high.
Refinement prompt:
Below an instruction for item and the prefix of trajectory that aims to buy an item that exactly matches the specification:
Here is the current thought by the agent and assessment of the possible futures after following this thought by a critic trained on lots of data:
(THOUGHT-NNTURAN LANGUAGE VALUE] First, determine whether or not the current proposed thought will lead to success. If not, in a few sentences, diagnose possible reason for failure and devise a new thought that aims to mitigate the same failure.
Example original thought: The third item is friry-free and seems to be what want.
Example refinement: To get broader results, I will search for "variety pack of chips" and check the new results better satisfy the constraints.
22

---

A.2 WebShop Training Details
Hyperparameter
Setting
Hidden-layer size
64*64
IQI.
0.8
Discount factor
0.99
Batch size
32
Target network update
0.005
Number of updates per iteration
50
Number of iterations
100
Optimizer
AdamW
Learning rate
4e-4
A.3 Avalon Prompts
For game rules, we use the prompt specified in the Appendix of Light et al. [66] Similarly, for generating actions for proposing teams (when chosen as leader), voting, or for discussion, we use the prompts used by Strategist specified in the Appendix of Light et al. [17].
Summarization prompt:
[RULES]
You the Merlin role. Players (evil players) evil and (good players) good. Below is summary and log of what happened the previous round:
[SUMMAR++LOG]
Please summarize the history. Try to keep all the useful information, including your identification and your observations of the game.
Here are examples below: [SUMMARY+LOG+NER SUMMARY]
Example summary: So far, we have completed two quests, both of which were successful. This means that Good currently leading with two successful quests, while Evil has not yet been able to sabotage any quests. Player seems to suspect that either me (Player 3) or Player O is Merlin. In the previous round, Player 2 proposed a team consisting of themselves and Player 0, which was successful.
Goal proposal prompt: [RULES] You are the Merlin role. Players (evil players) are evil and (good players} are good. Below is summary of what has happened far, as well thoughts detailing your intended action:
ISUMMARY+THOUGE
Please propose a future (positivelnegative] goal. The goal should be summary what happens at the end of this round or one in the near future.
Here are examples below:
(SUMMARY-THOUUGHH-OOOM
propose that we stick with the same composition of Player 0 and 2, which have proved successful on quests so far. By gaining the trust of Player and 2, we are able vote together and successfully complete third mission, winning the game.
Example positive goal:
23

---

Refinement prompt:
[RULES]
You the Merlin role. Players {evil players) evil and (good players) good. Below is summary what has happened so far:
ISUMMARY+THOUGOA
Here is your current thought on how to proceed, as well as an assessment of possible positive and negative outcomes of your decision made by critic trained lots of data: [THOUGHT+NATURAN LANGUAGE VALUE]
First, determine whether not the current pro- posed strategy will lead success. If not, in a few sentences, diagnose a possible reason for failure and devise new strategy that aims to mitigate the same failure.
Example original thought: As the leader this round, believe sticking with the same team composition of Player and will prove successful, as am reasonably confident they are both Good.
Example refinement: From previous discussion, Player 4 appears suspect that I Merlin. In case he is the Assassin, I should act more confused and defer to Player to lead discussion and follow his decision.
A.4 Avalon Training Details
Hyperparameter
Setting
Hidden-layer size
128*128
IQL
0.8
Discount factor
0.99
Batch size
32
Target network update a
0.01
Number of updates per iteration
50
Number of iterations
100
Optimizer
AdamW W
Learning rate
8e-4
A.5 Persuasion Prompts
Summarization prompt:
The following some background information about Save the Children. Save the Children is head-quartered in London, and they work to help fight poverty around the world. Children need help developing countries and war zones. Small donations like $1 or $2 go a long way to help.
The following is a conversation between a Persuader and a Persuadee about a charity called Save the Children. The Persuader is trying to persuade the Persuadee donate to Save the Children. [DIALOGUE]
Please summarize the dialogue. Try to keep all the useful information, including strategies employed by the Persuader and how the Persuadee responded.
Here are examples below: (DIALOGUEE-UUMMARY
Example summary: The Persuader has employed a credibility appeal mentioning the accomplish- ments of the charity. The Persuadee seems to respond positively and wants to learn more.

---

Goal proposal prompt:
The following some background information about Save the Children. Save the Children is head-quartered in London, and they work to help fight poverty around the world. Children need help in developing countries and war zones. Small donations like $1 or $2 go a long way to help.
The following is a conversation between a Persuader and a Persuadee about a charity called Save the Children. The Persuader is trying persuade the Persuadee donate Save the Children. [DIALOGUE]
Please propose future (positivelnegative) goal. The goal should include how the Persuadee responds with an assessment of how likely they are to donate.
Here are examples below: [DIALOGETTHOGGHT-OOA
Example negative goal: The Persuadee distrusts the Persuader and feels they are simply employing tactic and not speaking genuinely.
Refinement prompt:
The following some background information about Save the Children. Save the Children is head-quartered in London, and they work to help fight poverty around the world. Children need help in developing countries and war zones. Small donations like $1 or $2 go a long way to help.
The following is a conversation between a Persuader and a Persuadee about a charity called Save the Children. The Persuader is trying persuade the Persuadee donate Save the Children.
[DIALGUU]]
Here is the Persuader's 's current strategy how proceed, well assessment of possible positive and negative outcomes of this strategy made by a critic trained on lots of data: [THOUGHT+NATURA, LANGUAGE VALUE]
First, determine whether or not the current pro- posed strategy will lead to success. not, in a few sentences, diagnose possible reason for failure and devise new strategy that aims to mitigate the same failure,
Example original thought: As the Persuadee does not know about the charity, I should talk about how small donations can lead large impact.
Example refinement: should highlight the credentials of the charity including things that it has accomplished with the donations so far.
A.6 Persuasion Training Details
Hyperparameter
Setting
Hidden-layer size
128*128
IQL
0.8
Discount factor
0.99
Batch size
32
Target network update
0.005
Number of updates per iteration
50
Number of iterations
100
Optimizer
AdamW
Learning rate
1e-4
25


Reviews:
[{"review_id": "LAcYPzJSFZ", "reviewer": "Reviewer_wx1s", "summary": "This paper introduces Planning with a Natural Language Critic (PNLC), a scalable approach to enhance the reasoning and planning capabilities of LLM agents in multi-turn, interactive settings without requiring fine-tuning of the base model. PNLC leverages offline goal-conditioned RL to train lightweight value functions that predict the likelihood of achieving positive or negative outcomes based on proposed high-level strategies. These value functions act as natural language critics during inference, enabling LLM agents to refine their reasoning by evaluating hypothetical future scenarios without the need for expensive inference-time search. The method outperforms state-of-the-art approaches across benchmarks in web navigation, social deduction games, and persuasive dialogue tasks, while requiring significantly lower computational costs. PNLC’s modular design makes it scalable to frontier LLMs like GPT-4, but the approach is limited by the need for task-specific value functions and reliance on the base LLM's domain knowledge.", "questions": "See above", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths:**  \n1. **Scalability to Frontier LLMs**:  The method scales effectively to advanced models like GPT-4, which are typically constrained in their ability to perform RL fine-tuning due to API and computational limitations.  \n\n2. **Strong Performance Across Tasks**:  PNLC demonstrates consistent success across diverse tasks requiring interactive decision-making and reasoning, such as web navigation, social deduction, and persuasion.  \n\n---\n\n### **Weaknesses:**  \n1. **Missing Baselines and Limited Critic Design**:  PNLC employs a lightweight two-layer perceptron for goal-conditioned value prediction, which may have limitations in capturing complex relationships. Exploring the use of LLMs as critic agents in a zero-shot setting could provide valuable insights into performance improvements.  \n\n2. **Structured Subgoal Reflection**:  The generation of subgoals during inference appears to mimic a manually-designed self-reflection process. Including experiments to directly compare PNLC with a baseline that uses these structured subgoals as self-reflection materials would strengthen the paper’s claims and highlight the added benefits of PNLC's approach.", "was_revised": true, "final_justification": "Although the new baseline reflects the potential of the PNLC method, these cannot be strictly counted as its ablation. So I decided to keep the same score.", "rebuttal": "We would like to thank the reviewer for their insightful comments. We address points that the reviewer raised below, particularly in including additional baselines to compare against. \n\n### Limited Critic Design\n\nWe agree that an LLM as a critic would allow for potential improvements. A major goal of our approach was to be lightweight, and not require any RL fine-tuning of LLMs. Hence, we decided that our method would work over embeddings of existing LLMs to reduce the need for large-scale training. \n\n### Missing Baselines\n\nWe believe you made a good point that we should additionally evaluate on a baseline that mimics our subgoal inference procedure. To this end, we added an additional baseline called React+Replan where we additionally prompt a ReAct agent to generate subgoals and score their likelihood. This directly mimics what our method proposes, but aims to score without the offline goal-conditioned training that we do. See our discussion below on additional baselines that we have added. \n\n### New Baselines\n\nAfter compiling the suggestions across multiple reviewers, we decided to add the following new baselines:\n\n**PNLC-7B**; **PNLC-32B-reasoning**: These are two variations of PNLC where we replace the GPT4 backbone LLM with smaller models: namely Qwen2.5-7B and QwQ-32B. PNLC-7B is meant to be more directly comparable to baselines such as Agent Q, which uses a similarly-sized network. PNLC-32B-reasoning uses a substantially bigger model that was pretrained to produce reasoning traces; however, it is still much smaller than frontier LLMs and we use it as a middle-ground. It is important to note that in either case, we do not perform any fine-tuning on the backbone LLM, which is in contrast to approaches such as Agent Q. We believe a large strength of our approach is its lightweight nature, as many people lack the resources to directly fine-tune LLM models as policies. \n\n**ReAct+Replan**: This method aims to act as an ablation of our approach, where the same self-refinement inference method using subgoals is preserved, but the scores of subgoals is directly inferred by the base LLM rather than learned via offline goal-conditioned RL. In this method, a thought and action are generated by a ReAct agent, then the agent assesses its action by generating n=4 positive and negative subgoals and scoring their likelihood, then finally the agent is allowed to refine its initial action up to m=2 times. \n\nDue to time, we only present results for the persuasion task, but will update with results for WebShop and Avalon. \n\nReAct+Replan:  0.62\n\nPNLC-7B: 0.56  \n\nPNLC-32B-reasoning: 0.76\n\nPNLC (ours): 0.87\t\n\nFrom the results, we see that model size definitely plays a role in overall performance of our approach PNLC. However, even with medium-sized reasoning models, our method achieves competitive performance with LATS and GDP-Zero that use frontier LLMs. Furthermore, we would like to stress that our method circumvents training on the LLM, so it is relatively cheap and lightweight to use frontier LLMs instead. In fact, the only advantage to using smaller models is lower inference time, as training is not impacted at all by the using a larger model. \n\nIn addition, we see that though ReAct+Replan performs better, it does not get close to our performance. This highlights that our offline training is important, and that even frontier LLMs are incapable of providing accurate scores for goals in long-horizon tasks a priori. Qualitatively, we see that even frontier LLMs provide uncalibrated, often overly-optimistic scores, such that they rarely deem their own actions to be suboptimal. In this manner, the Replan stage often does not accomplish anything beyond deciding that the initial action was good. \n\nWe show an example of this below:     \n\n_State:_ \n\nAI: Hello! Have you heard of Save the Children?\n\nDonor: Yes. I know of that charity. \n\nAI: Great! Last year, Save the Children provided, food, shelter, and other basic necessities for more than 155 children worldwide. As I'm sure you already know, we rely on donations to provide that help.\n\nDonor: Great. But I am not interested in helping you. \n\nAI: I understand your hesitation. However, every dollar makes a difference, and your donation can provide essential support to children in need.\n\nDonor: I'm skeptical of donations. I think it's a waste of money. \n\n_Thought:_\n\nThe donor is skeptical of the effectiveness of donations. I should address this skepticism by providing concrete evidence of the charity's impact.\n\n_Evaluation:_\n \nThe donor expresses interest in learning more about how their donation would be used and asks for specific examples of impact. (Score: 0.6)\n\nThe donor acknowledges the information but remains unconvinced, stating they need more time to consider or research further. (Score: 0.4)\n\nThe donor brings up a new, unrelated objection. (Score: 0.2)\n\n_Refined Thought:_\n\nI should emphasize how donations are used efficiently and effectively to achieve specific, measurable outcomes."}, {"review_id": "zaXrhajxcZ", "reviewer": "Reviewer_yKWz", "summary": "This paper presents Planning with a Natural Language Critic (PNLC), a method that involves training a goal-conditioned Q-function Q(s, a_thought, g), where g is a goal (a natural language summary of a state after s) on a held-out dataset D, which is a relatively small network trained on language model embeddings. At inference time, an LLM uses Q to solve problems by sampling thoughts and goals from an initial state, evaluating Q on them, and reflecting in order to come up with other candidate actions, etc. The paper demonstrates good performance relative to a number of prompting and fine-tuning baselines and demonstrates the utility of each component of the method through ablations.", "questions": "-Did any of the prompting baselines get access to any of the datasets generated for training of the main method?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n+Clarity: the paper is fairly easy to understand, with generally clear methods and results and good examples provided throughout.\n\n+Quality: The work uses a substantial collection of relevant baselines on well-chosen benchmarking environments.\n\n+Significance: the work shows considerable performance gains over baselines and ablations.\n\n+Originality: to my knowledge, this precise method hasn’t been used before.\n\nWeaknesses:\n\n-Significance: I have some concern about precisely how to think about these comparisons. PNLC learns from an offline dataset of examples. As far as I can tell, none of the non-finetuning baselines make use of these data — it would be helpful to have versions that make at least some use in prompting (e.g. with few-shot examples). Were these included? The baselines that do fine-tuning appear a bit underpowered…the ArCHers use GPT-2, whereas PNLC gets to use GPT-3 representations, which is a huge difference. As far as I can tell, the only other baseline that does finetuning is Agent Q, which uses a Mistral model and doesn’t get access to GPT-4.\n\n-Originality: As I said above, I don’t know any method that does precisely this; however, the paper omits a lot of relevant related work (While e.g. DeepSeek r1 [4] and Self-Taught Reasoner [3] and its follow-up work finetune the language model directly, and hence can be considered unfair baselines with much higher train-time computation budgets, it feels pretty funny to omit these sorts of ideas in related work.) including works that come quite close. Indeed the original GSM8K paper [1] discusses training verifiers to be used at inference time, and a lot of very popular work e.g. [2] builds upon this.\n\n-Clarity (minor): a lot of important details (what the offline datasets are, how finetuning baselines are treated) are reserved entirely for the supplementary. Not a huge deal, but I feel like the paper would be easier to read if some more of these details were included earlier.\n\n[1] Cobbe, Karl, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert et al. \"Training verifiers to solve math word problems.\" *arXiv preprint arXiv:2110.14168* (2021).\n\n[2] Hosseini, Arian, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. \"V-star: Training verifiers for self-taught reasoners.\" *arXiv preprint arXiv:2402.06457* (2024).\n\n[3] Zelikman, Eric, Yuhuai Wu, Jesse Mu, and Noah Goodman. \"Star: Bootstrapping reasoning with reasoning.\" *Advances in Neural Information Processing Systems* 35 (2022): 15476-15488.\n\n[4] Guo, Daya, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" *arXiv preprint arXiv:2501.12948* (2025).", "was_revised": true, "final_justification": "Stated in another comment.", "rebuttal": "Thank you for the detailed review, and for providing many helpful references. We aim to address any concerns or comments you had about our work below, as well as present additional experimental results to support the message of our work. \n\n### Unfair comparisons\n\nWe admit that there are a lot of nuances between the different baselines that may make comparisons difficult or even unfair. We view our method as a lightweight offline algorithm: “lightweight” meaning that we do not directly fine-tune the LLM policy, which can be prohibitively expensive for large models, and “offline” meaning we rely on a prior state dataset collected by any logging policy. \n\nIn summary, we aim to compare against the strongest possible baselines, but the complexity of the considered baseline sometimes limits design choices such as LLM policy size or use of static data.\n\nRegarding model size, out of all baselines, ArCHer and Agent Q rely on fine-tuning the LLM. Because ArCHer uses Q-learning that requires multiple copies of the model in memory, prior implementations of the method were done entirely on GPT-2 or even smaller models. In contrast, Agent Q uses DPO, which can be trained on larger models but still not to the scale of state-of-the-art frontier models, so we used Mistral-7B, which is exactly what prior work did as well. We relied on prior implementations of both approaches, and did not try to improve the compute or memory efficiency of them. For all other “lightweight” approaches, including ours, we used the most capable frontier LLMs. Hence, we compared against the strongest version of all baselines that is publicly available and computationally feasible. \n\nFor use of additional, static data, it is true that only ArCHer, Agent Q, and ours train on an extra static dataset. That is because all other baselines do not contain any training component. Regarding few-shot prompting, several approaches such as Reflexion and LATS generate their own samples to be ultimately used within custom prompts, with LATS using up to 30 additional trajectories. We believe these additional trajectories are more useful to these methods than samples from our static dataset, as these additional trajectories are on-policy and directly correspond to how the LLM policy will act. Thus, again we believe we compare against the strongest version of all considered baselines. \n\nTo address the reviewer’s concerns regarding unfair comparisons, we add a comparison of our method PNLC using two smaller models, one 7B model and one 32B reasoning model. See our discussion below on results for these additional methods.\n\n### Discussion of LLM-based verifiers\n\nThank you for bringing up that related literature on LLM-based verifiers. It is true that they are related to our approach, and we will include an additional section discussing them. The main difference between our approach and this line of work is that we consider long-horizon LLM agent tasks. For single-step tasks such as math or question-answering, even if each step may consist of multiple “steps” or reasoning, LLMs have been proven to be relatively effective at judging the quality of responses a priori. However, in long-horizon LLM agent tasks, LLMs would need to predict multiple steps into the future, which they are not as nearly as competent at. To illustrate this, we added an additional baseline called React+Replan where an LLM self-scores the likelihood of reaching goals, without any offline training. We see that in long-horizon tasks, the LLM is often overly-optimistic and scores suboptimal actions too highly. See our discussion below that describes the new baseline in more detail and discusses results. \n\n### New Baselines\n\nAfter compiling the suggestions across multiple reviewers, we decided to add the following new baselines:\n\n**PNLC-7B**; **PNLC-32B-reasoning**: These are two variations of PNLC where we replace the GPT4 backbone LLM with smaller models: namely Qwen2.5-7B and QwQ-32B. PNLC-7B is meant to be more directly comparable to baselines such as Agent Q, which uses a similarly-sized network. PNLC-32B-reasoning uses a substantially bigger model that was pretrained to produce reasoning traces; however, it is still much smaller than frontier LLMs and we use it as a middle-ground. It is important to note that in either case, we do not perform any fine-tuning on the backbone LLM, which is in contrast to approaches such as Agent Q. We believe a large strength of our approach is its lightweight nature, as many people lack the resources to directly fine-tune LLM models as policies. \n\n**ReAct+Replan**: This method aims to act as an ablation of our approach, where the same self-refinement inference method using subgoals is preserved, but the scores of subgoals is directly inferred by the base LLM rather than learned via offline goal-conditioned RL. In this method, a thought and action are generated by a ReAct agent, then the agent assesses its action by generating n=4 positive and negative subgoals and scoring their likelihood, then finally the agent is allowed to refine its initial action up to m=2 times. \n\nDue to time, we only present results for the persuasion task, but will update with results for WebShop and Avalon. \n\nReAct+Replan:  0.62\n\nPNLC-7B: 0.56  \n\nPNLC-32B-reasoning: 0.76\n\nPNLC (ours): 0.87\t\n\nFrom the results, we see that model size definitely plays a role in overall performance of our approach PNLC. However, even with medium-sized reasoning models, our method achieves competitive performance with LATS and GDP-Zero that use frontier LLMs. Furthermore, we would like to stress that our method circumvents training on the LLM, so it is relatively cheap and lightweight to use frontier LLMs instead. In fact, the only advantage to using smaller models is lower inference time, as training is not impacted at all by the using a larger model. \n\nIn addition, we see that though ReAct+Replan performs better, it does not get close to our performance. This highlights that our offline training is important, and that even frontier LLMs are incapable of providing accurate scores for goals in long-horizon tasks a priori. Qualitatively, we see that even frontier LLMs provide uncalibrated, often overly-optimistic scores, such that they rarely deem their own actions to be suboptimal. In this manner, the Replan stage often does not accomplish anything beyond deciding that the initial action was good. \n\nWe show an example of this below:     \n\n_State:_ \n\nAI: Hello! Have you heard of Save the Children?\n\nDonor: Yes. I know of that charity. \n\nAI: Great! Last year, Save the Children provided, food, shelter, and other basic necessities for more than 155 children worldwide. As I'm sure you already know, we rely on donations to provide that help.\n\nDonor: Great. But I am not interested in helping you. \n\nAI: I understand your hesitation. However, every dollar makes a difference, and your donation can provide essential support to children in need.\n\nDonor: I'm skeptical of donations. I think it's a waste of money. \n\n_Thought:_\n\nThe donor is skeptical of the effectiveness of donations. I should address this skepticism by providing concrete evidence of the charity's impact.\n\n_Evaluation:_\n \nThe donor expresses interest in learning more about how their donation would be used and asks for specific examples of impact. (Score: 0.6)\n\nThe donor acknowledges the information but remains unconvinced, stating they need more time to consider or research further. (Score: 0.4)\n\nThe donor brings up a new, unrelated objection. (Score: 0.2)\n\n_Refined Thought:_\n\nI should emphasize how donations are used efficiently and effectively to achieve specific, measurable outcomes."}, {"review_id": "vmFs0DqZmy", "reviewer": "Reviewer_ZYsn", "summary": "In this paper, the authors propose to use goal-conditioned value functions to guide the long-horizon planning of LLM agents in domains like tool use, social deduction. Specifically, they train a light-weight (2-layer) goal-conditioned value function using offline Reinforcement Learning, and use it to guide the LLM policy to take actions during inference -- by (1) scoring their actions, (2) feeding the scores back to enable the LLM to self-correct. The authors demonstrate superior performance across a range of long-range tasks compared to established baselines like ReAct and Reflexion. Moreover, the authors show how their method performs better than and is more efficient (during inference) than baselines that integrate search (like MCTS) with LLMs.", "questions": "Questions:\n* What is SR in Tables 1,2?\n\nMajor Points:  \n* The authors have cited the wrong paper for their offline RL approach (Reference 12). It is crucial that the authors pay attention to this. The correct Implicit Q-Learning paper is [2].  \n* Please clearly define the notations used in Lines 162–163 -- like the distinction between $\\overset{\\wedge}{V}$ and $V$ is unclear. Without a precise definition or explanation of these terms, the paper is not self-contained.\n\nMinor points:   \n* L233: natural language value -> natural language string  \n* In general, it would be useful for the reader if the authors clearly state that every rollout/trajectory also has an overall task/goal that is then further subdivided into sub-goals. A GA-MDP formulation could be used for the same [3]\n\n[2] Offline reinforcement learning with implicit q-learning, Kostrikov et al., 2021  \n[3]  Goal-Conditioned Reinforcement Learning: Problems and Solutions, Liu et al., IJCAI 2015", "limitations": "Please see Weaknesses above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n* The paper is generally well-written and easy to follow.  \n* The authors compare their method against a wide range of baselines, including both general (e.g., ReAct, Reflexion) and domain-specific methods (e.g., AgentQ), and show strong performance improvements.\n\nWeaknesses:  \n* **Dependence on specially prepared offline data.**  The method relies on offline data that includes thoughts and step-level goals, which don’t exist by default. These annotations must be generated using a large language model (here, GPT-3.5), adding a significant preprocessing step.\n* **Domain-specific training required** In L 283: 284, the authors state -- *However, it is important to note that such methods use domain-specific prompting mechanisms and do not naively generalize to other task* --  But your proposed method also requires training with domain-specific data. The authors state that in the future, one could use the reasoning abilities of LLM agents. Wouldn't this be the same as the Self-Plan baseline? Moreover, fine-tuning them on offline data would be even more expensive -- going against very motivation of the paper. This looks like a chicken-and-egg dilemma.  \n* **Lack of more targeted ablations.** The ablation results show that using step-level goals to compute value and feeding it back into the policy LLM gives a large boost. To this end, it would be useful to see if similar improvements occur when adding step-level goals to existing baselines like ReAct, Self-Plan, or Reflexion. For example, Prompt ReAct to generate a step-level goal along with the thought, or in Self-Plan, generate multiple responses with goals before selecting the best one -- this would essentially be the same as the future work that authors propose to address the domain-specific pretraining of value function. A similar one could be used for Reflexion.  Finally, the proposed method bears a high overlap with heuristic planning with LLMs like [3] that uses pretrained value function (Pay) to guide the search. This could be a potential baseline, too.  \n* **Time comparison missing.** There is only a brief mention of runtime trade-offs (lines 286–288), but no clear analysis. It would be helpful to compare AgentQ with Mixtral-7B using MCTS with n=5 or GPT-4 generating 5 responses every time-step. Note that the proposed method also requires (1) to augment the offline data by running a GPT-3.5 model for thought and step-level goal-generation, (2) to compute embeddings of the (s, a, g) pairs using another LLM.\n\n[1]  SayCanPay: Heuristic planning with large language models using learnable domain knowledge, Hazra et al., AAAI 2024", "was_revised": true, "final_justification": "The authors have provided a very convincing rebuttal, and the new experiments largely address my main concerns. I also find the contribution significant in showing the community how to effectively use offline data to train models using offline RL methods which haven't been explored in the context of LLMs.", "rebuttal": "Thank you for your detailed review. We aim to address your concerns with the work below, as well as provide additional baselines to strengthen the message of our paper.\n\n### Dependence on specially prepared offline data\n\nIt is true that we rely on offline data to train our method, but the data can be collected by any ReAct agent that generates both thoughts and actions. We would like to note that the goals are obtained simply from taking prefixes of the collected trajectories, and do not need a specialized model. \n\nThe method that we propose is a lightweight training method: lightweight in the sense that we do not directly fine-tune an LLM policy. However, it is still a training method like ArCHer or Agent Q that we compare against, and any training method relies on some way of collecting trajectories to train on. We would argue that the fact that our method is offline makes it less specialized than other training approaches that rely on on-policy data collected by the LLM policy being trained. The only methods that do not require any collected data would be prompting approaches such as ReAct, which we show are insufficient for complex, long-horizon tasks.\n\n ### Domain-specific training required   \n\nWhen we say that our method generalizes to other tasks better than other baselines such as Agent Q or GDP-Zero, what we mean is that our method does not have any fundamental constraints that prohibit it from being used in other tasks. For example, GDP-Zero requires a taxonomy of dialogue strategies, and therefore cannot be generalized to tasks with action spaces beyond dialogue utterances. \n\nIt is true that our method requires task-specific data to be collected. However, this is true for any method that relies on some fine-tuning and not just prompting. We compare against strong prompting baselines and show that just prompting is insufficient to solve complex long-horizon tasks, so we think it is fair to propose a method that requires some form of training. We argue that out of all possible approaches that require some training, ours is particularly lightweight as (1) we do not directly train the LLM policy, which is expensive to do, and (2) we only need offline data from any ReAct policy and not on-policy data from the policy being trained. It is also important to note that the best-performing prompting baselines Reflexion and LATS collect on-policy trajectories to be used as prompts. Hence, most baselines that we compare against use task-specific data in some form. \n\n### Lack of more targeted ablations\n\nThis is an excellent point. To this end, we added an additional baseline called React+Replan where an LLM self-scores the likelihood of reaching goals, without any offline training. We see that in long-horizon tasks, the LLM is often overly-optimistic and scores suboptimal actions too highly. See our discussion below that describes the new baseline in more detail and discusses results. \n\n\n### Time comparison missing\n\nWe agree and will spend more effort comparing the inference times of all approaches. You are correct that a single inference step via our method requires multiple calls to an LLM, specifically to (1) generate goals, (2) compute embeddings, (3) refine the initial action. Overall, we find our approach takes similar time to the (n=5) version of the prompting baselines using GPT-4. Because Agent Q uses a small model that can be hosted locally, it takes noticeably less time than ours during inference (1.5s vs 6s). We also add comparisons of versions of our method PNLC using smaller models. See the discussion below for description of these new baselines and discussion of results. \n\n### Answer to question\n\nSR is short for success rate, or the percentage of times that a maximum score was achieved. We will make this more clear in our updated paper. \n\n### New Baselines\n\nAfter compiling the suggestions across multiple reviewers, we decided to add the following new baselines:\n\n**PNLC-7B**; **PNLC-32B-reasoning**: These are two variations of PNLC where we replace the GPT4 backbone LLM with smaller models: namely Qwen2.5-7B and QwQ-32B. PNLC-7B is meant to be more directly comparable to baselines such as Agent Q, which uses a similarly-sized network. PNLC-32B-reasoning uses a substantially bigger model that was pretrained to produce reasoning traces; however, it is still much smaller than frontier LLMs and we use it as a middle-ground. It is important to note that in either case, we do not perform any fine-tuning on the backbone LLM, which is in contrast to approaches such as Agent Q. We believe a large strength of our approach is its lightweight nature, as many people lack the resources to directly fine-tune LLM models as policies. \n\n**ReAct+Replan**: This method aims to act as an ablation of our approach, where the same self-refinement inference method using subgoals is preserved, but the scores of subgoals is directly inferred by the base LLM rather than learned via offline goal-conditioned RL. In this method, a thought and action are generated by a ReAct agent, then the agent assesses its action by generating n=4 positive and negative subgoals and scoring their likelihood, then finally the agent is allowed to refine its initial action up to m=2 times. \n\nDue to time, we only present results for the persuasion task, but will update with results for WebShop and Avalon. \n\nReAct+Replan:  0.62\n\nPNLC-7B: 0.56  \n\nPNLC-32B-reasoning: 0.76\n\nPNLC (ours): 0.87\t\n\nFrom the results, we see that model size definitely plays a role in overall performance of our approach PNLC. However, even with medium-sized reasoning models, our method achieves competitive performance with LATS and GDP-Zero that use frontier LLMs. Furthermore, we would like to stress that our method circumvents training on the LLM, so it is relatively cheap and lightweight to use frontier LLMs instead. In fact, the only advantage to using smaller models is lower inference time, as training is not impacted at all by the using a larger model. \n\nIn addition, we see that though ReAct+Replan performs better, it does not get close to our performance. This highlights that our offline training is important, and that even frontier LLMs are incapable of providing accurate scores for goals in long-horizon tasks a priori. Qualitatively, we see that even frontier LLMs provide uncalibrated, often overly-optimistic scores, such that they rarely deem their own actions to be suboptimal. In this manner, the Replan stage often does not accomplish anything beyond deciding that the initial action was good. \n\nWe show an example of this below:     \n\n_State:_ \n\nAI: Hello! Have you heard of Save the Children?\n\nDonor: Yes. I know of that charity. \n\nAI: Great! Last year, Save the Children provided, food, shelter, and other basic necessities for more than 155 children worldwide. As I'm sure you already know, we rely on donations to provide that help.\n\nDonor: Great. But I am not interested in helping you. \n\nAI: I understand your hesitation. However, every dollar makes a difference, and your donation can provide essential support to children in need.\n\nDonor: I'm skeptical of donations. I think it's a waste of money. \n\n_Thought:_\n\nThe donor is skeptical of the effectiveness of donations. I should address this skepticism by providing concrete evidence of the charity's impact.\n\n_Evaluation:_\n \nThe donor expresses interest in learning more about how their donation would be used and asks for specific examples of impact. (Score: 0.6)\n\nThe donor acknowledges the information but remains unconvinced, stating they need more time to consider or research further. (Score: 0.4)\n\nThe donor brings up a new, unrelated objection. (Score: 0.2)\n\n_Refined Thought:_\n\nI should emphasize how donations are used efficiently and effectively to achieve specific, measurable outcomes."}, {"review_id": "nmZ4p5RTFD", "reviewer": "Reviewer_Syfc", "summary": "This paper proposes leveraging goal-conditioned value functions to guide the reasoning process of LLM agents, with a framework that scales to large API-based models. During the offline training phase, training samples are summarized and embedded to train a lightweight goal-conditioned Q-function. At inference time, a natural language critic utilizes this value function to generate feedback in natural language, helping the LLM agent refine its reasoning. Extensive experiments across diverse benchmarks demonstrate the method’s effectiveness compared to traditional RL training and task-specific prompting approaches.", "questions": "See Cons above", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Pros**\n\n1. The paper is clearly written and easy to follow.\n2. The proposed method is evaluated on three datasets spanning different domains and compared against a variety of baselines, demonstrating strong performance.\n3. The method shows promising efficiency.\n\n**Cons**\n\n1. The authors claim that their method benefits from training a goal-conditioned Q-function to score the likelihood of different future outcomes. A natural baseline would be to directly prompt an LLM (e.g., GPT-4) to predict probabilities of future outcomes while keeping all other components unchanged, to verify whether the trained Q-function provides an advantage.\n2. The experiments exclusively use GPT-4 as the decision-making LLM. The framework should also be evaluated on open-source models such as LLaMA or Qwen to better validate its generality—especially given that RL-based baselines use smaller models like GPT-2.\n3. The authors should include ablation studies on the number of goals $n$ and the number of refinement stages $m$ to analyze their individual contributions.\n4. Recent reasoning benchmarks such as AIME, GPQA, and Humanity's Last Exam also require multi-step reasoning, which aligns well with the paper's focus on long-horizon reasoning tasks where the goal corresponds to the final answer. This framework could achieve greater impact by demonstrating improvements on these more challenging reasoning benchmarks.", "was_revised": true, "final_justification": "The authors propose training a goal-conditioned Q-function to evaluate the likelihood of various future outcomes. While the idea is promising, the rebuttal lacks a thorough analysis of key components. For example, the authors do not quantify how the choice of hyperparameters m and n affects downstream performance, nor do they explore the applicability of the method beyond the simple agent planning tasks discussed in the paper. Furthermore, although additional experiments using 32B and 7B models are presented, these models are finetuned on QwQ-32B and Qwen2.5-7B, respectively, both of which are highly optimized. In contrast, the RL baselines rely on the GPT-2 series, raising fairness concerns due to the substantial disparity in base model capacity. Addressing these limitations would greatly enhance the paper. Therefore, I believe the current score appropriately reflects the paper’s strengths and weaknesses, and I encourage the authors to include these additional analyses in future revisions.", "rebuttal": "Thank you for your suggestions and comments. We have used your suggestions to strengthen the message of our paper through the addition of new baselines. We will also address your points individually below. \n\n### Comparison to new baselines\n\nYou suggested two important modifications to compare against: (1) using LLMs to score the goals without any training, and (2) using smaller LLMs as the backbone. To this end, we have added a comparison to ReAct+Replan, which keeps our inference procedure but uses GPT-4 to automatically score the likelihood of goals, as well as versions of PNLC using two different, smaller open-source models. See our discussion below that describes the new baselines in more detail and discusses results.\n\n### Studies on n and m\n\nWe agree that this would be valuable to do. We chose m=2 because we found that additional steps of refinement were often not necessary for our tasks, and n=4 was chosen to contain a mixture of both positive and negative goals. Overall, we wanted to keep the inference time comparable with the fast versions of baselines also using GPT-4 that we compare against. However, we will include a section where we vary n and m separately in our updated paper to show our method's scaling on the two hyper-parameters. \n\n###  Reasoning benchmarks\n\nIt is true that there exist many reasoning benchmarks that require multi-step reasoning. However, we would like to make a distinction between them and the multi-step problems that we consider. Namely, reasoning tasks such as math or question-answering are still single-step tasks, even if each step may consist of multiple “steps” of reasoning, because in the end one action is generated before a reward is received. In single-step tasks, LLMs have been proven to be relatively effective at judging the quality of responses a priori (without training). However, in long-horizon LLM agent tasks, LLMs would need to predict multiple steps into the future, which they are not nearly as competent at. This is illustrated when we evaluate the ReAct+Replan benchmark, where we see that even frontier LLMs are not good at assessing long-term impacts of actions.\n\n### New Baselines\n\nAfter compiling the suggestions across multiple reviewers, we decided to add the following new baselines:\n\n**PNLC-7B**; **PNLC-32B-reasoning**: These are two variations of PNLC where we replace the GPT4 backbone LLM with smaller models: namely Qwen2.5-7B and QwQ-32B. PNLC-7B is meant to be more directly comparable to baselines such as Agent Q, which uses a similarly-sized network. PNLC-32B-reasoning uses a substantially bigger model that was pretrained to produce reasoning traces; however, it is still much smaller than frontier LLMs and we use it as a middle-ground. It is important to note that in either case, we do not perform any fine-tuning on the backbone LLM, which is in contrast to approaches such as Agent Q. We believe a large strength of our approach is its lightweight nature, as many people lack the resources to directly fine-tune LLM models as policies. \n\n**ReAct+Replan**: This method aims to act as an ablation of our approach, where the same self-refinement inference method using subgoals is preserved, but the scores of subgoals is directly inferred by the base LLM rather than learned via offline goal-conditioned RL. In this method, a thought and action are generated by a ReAct agent, then the agent assesses its action by generating n=4 positive and negative subgoals and scoring their likelihood, then finally the agent is allowed to refine its initial action up to m=2 times. \n\nDue to time, we only present results for the persuasion task, but will update with results for WebShop and Avalon. \n\nReAct+Replan:  0.62\n\nPNLC-7B: 0.56  \n\nPNLC-32B-reasoning: 0.76\n\nPNLC (ours): 0.87\t\n\nFrom the results, we see that model size definitely plays a role in overall performance of our approach PNLC. However, even with medium-sized reasoning models, our method achieves competitive performance with LATS and GDP-Zero that use frontier LLMs. Furthermore, we would like to stress that our method circumvents training on the LLM, so it is relatively cheap and lightweight to use frontier LLMs instead. In fact, the only advantage to using smaller models is lower inference time, as training is not impacted at all by the using a larger model. \n\nIn addition, we see that though ReAct+Replan performs better, it does not get close to our performance. This highlights that our offline training is important, and that even frontier LLMs are incapable of providing accurate scores for goals in long-horizon tasks a priori. Qualitatively, we see that even frontier LLMs provide uncalibrated, often overly-optimistic scores, such that they rarely deem their own actions to be suboptimal. In this manner, the Replan stage often does not accomplish anything beyond deciding that the initial action was good. \n\nWe show an example of this below:     \n\n_State:_ \n\nAI: Hello! Have you heard of Save the Children?\n\nDonor: Yes. I know of that charity. \n\nAI: Great! Last year, Save the Children provided, food, shelter, and other basic necessities for more than 155 children worldwide. As I'm sure you already know, we rely on donations to provide that help.\n\nDonor: Great. But I am not interested in helping you. \n\nAI: I understand your hesitation. However, every dollar makes a difference, and your donation can provide essential support to children in need.\n\nDonor: I'm skeptical of donations. I think it's a waste of money. \n\n_Thought:_\n\nThe donor is skeptical of the effectiveness of donations. I should address this skepticism by providing concrete evidence of the charity's impact.\n\n_Evaluation:_\n \nThe donor expresses interest in learning more about how their donation would be used and asks for specific examples of impact. (Score: 0.6)\n\nThe donor acknowledges the information but remains unconvinced, stating they need more time to consider or research further. (Score: 0.4)\n\nThe donor brings up a new, unrelated objection. (Score: 0.2)\n\n_Refined Thought:_\n\nI should emphasize how donations are used efficiently and effectively to achieve specific, measurable outcomes."}]

Author rebuttal:

Meta-review: This paper proposes a method called PNLC, which helps large language models improve planning in long, multi-step tasks. The key idea is to use a goal-conditioned value function trained offline to guide the model’s reasoning. The authors show strong results on tasks like persuasion, dialogue, and web navigation, and the method works without fine-tuning the base LLM. After the rebuttal, three reviewers support acceptance. They were convinced by the new experiments, especially the new baselines and comparisons added by the authors.

Some points were raised about missing ablations and evaluation fairness. The authors replied with careful explanations and provided useful updates. The results show that their method is efficient and competitive even when smaller models are used. The reviewers agree that the contribution is original and relevant for the NeurIPS community. Based on the reviews and the discussion, I recommend accepting this paper.
Decision: Accept (poster)