Title: Language Models can Self-Improve at State-Value Estimation for Better Search
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLMs

agents

reasoning

self-improvement

multi-step reasoning

search
Abstract: Collecting ground-truth rewards or human demonstrations for multi-step reasoning tasks is often prohibitively expensive, especially in interactive domains such as web tasks. We introduce Self-Taught Lookahead (STL), a reward-free framework that improves language model–based value functions by reasoning explicitly about state transitions. STL can be viewed as a chain-of-thought analogue of the value iteration algorithm: instead of regressing directly on numeric values, a value LLM is trained to simulate a step of lookahead in natural language—predicting the next action, resulting state, and rationale for its value. This process refines value estimates without any labeled data. The self-supervised procedure yields more accurate state-value predictions, which in turn enable lightweight search algorithms to expand fewer states while maintaining strong performance. Empirically, STL-trained value models built on moderately sized (8B-parameter) open-weight LLMs boost web agent success rates by over 39%, achieving performance comparable to proprietary models. STL also generalizes to multi-hop question answering and math puzzles. Overall, STL enables small open-source models to guide efficient search, reducing inference costs by integrating explicit reasoning with value learning.


Peer reviews:
comments: SUMMARY:
This paper proposes self-taught lookahead (STL), a self-supervised method that leverages state-transition dynamics to improve a value model capable of effectively guiding language model-controlled search without requiring ground-truth rewards or human demonstrations. The method fine-tunes a value LLM using lookahead state transitions and generated rationales, enabling lightweight yet effective search guidance across multiple tasks.

STRENGTHS AND WEAKNESSES:
Strengths:

1.Addresses the high cost of human labels in multi-step reasoning tasks, an important practical limitation in deploying LLM-based agents.

2.Evaluates STL across diverse tasks with clear baselines and ablation studies illustrating each component's contribution

Weaknesses:

1.The paper states that it uses a single lookahead step within tree search to generate self-improvement data but does not introduce how tree search itself is performed. 

2.Rationales are a key part of STL, but their structure and role remain abstract in the paper.

3.While comparisons with Reflexion, LATS, and AgentQ are present, the paper could be strengthened by explicitly discussing comparisons with recent methods, especially in 2024, highlighting differences in effectiveness and generalization.

QUESTIONS:
1.The paper does not clearly differentiate STL from existing Rewardn and Demo Free models. It seems the main change is adding rationales to trajectories as input for value prediction.

2.Consider adding a brief mention of the method’s limitations and broader impacts in the Conclusion.

3.While most equations are clear, some notations (e.g., Equation 6) could benefit from further clarification.

LIMITATIONS:
The authors discussed some limitations and broader impacts.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method called “self-taught lookahead” (STL) which can be thought of as a form of value iteration applied to language models. In STL, the model is prompted to estimate state values along with rationales for multiple successor states, and then actions with the highest value are used to form a finetuning dataset consisting of (state, action, successor state, rationale, value). The model is finetuned, and at inference time greedy search is used to select subsequent actions. Experiments are conducted on three benchmarks (WebShop, HotpotQA, and Game of 24) using a finetuned llama-3.1-8b-instruct as the value model, demonstrating STL to work as well as or better than gpt-4o as the value model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n* Significance: Training LLMs to work well in multistep, agentic settings is an important and active area of research. The demonstrated results are impressive, showing that self-improvement can enable smaller models to work as well as larger, more expensive models.  \n* Clarity: the paper is well-written with the results clearly presented. There are a number of insightful points in the paper that I think will be appreciated by others working in the space of reasoning/planning.  \n* Quality: the paper presents an impressive range of experiments, including a wide range of appropriately chosen baselines, informative ablations, and extensive analyses on efficiency and scaling. Statistical significance tests are conducted.  \n* Originality: the proposed method is a clever application of older ideas in RL to the LLMs, and in particular leverages some of the strengths of LLMs in ways that were impossible to do in the classic setting (i.e., including rationales for the value estimates).\n\nWeaknesses:\n\n* Clarity: there were a few points of confusion I had about the methods. While the appendix is already great, I think it could be further improved with a bit more detail. I discuss my particular points of confusion in more detail in the “Questions” section.  \n* Quality: there are a few methodological choices that seemed a bit strange to me, or which perhaps I didn’t fully understand. I discuss these further in the “Questions” section below.  \n* Significance: the method requires task-specific prompts for value estimation, which limits its generality.\n\nQUESTIONS:\n### Clarifications\n\n* I didn’t quite understand how the generation of rationales and values works. In Section 3.1 and Algorithm 1, it sounds as though: (1) values are first generated on their own for several actions, leveraging ground-truth transitions, and then (2) rationales are generated given the actions and outcomes. But then there is only a single prompt (e.g. Figure 9\\) for value estimation, which first estimates the rationale, and then the value. So I’m confused about what the actual order of estimation is.  \n* I don’t think I fully understand what the algorithm looks like at inference time. It would be helpful to include an additional algorithm clarifying this. Is it that several actions are drawn from the base model, then these are each evaluated by the value model, and the best one is picked?  \n* It’s not clear what the dataset sizes were for finetuning. It would be helpful to include this detail in Appendix H, along with maybe some other statistics about the dataset generation (e.g. how many tasks, how many rollouts from each task, how long those rollouts were, what was the success rate/score of the training data, etc.). Additionally, please report other hyperparameters like context size, maximum generation length, temperature, etc.\n\n### Methodological questions\n\n* I didn’t understand why the paper did not include baselines for just vanilla, out-of-the box LLMs with no search at all. The paper includes baselines with out-of-the-box models that are used in the same greedy search fashion as STL, but there does not seem to be anything which just greedily selects actions with no value estimation. I think this is also an important comparison to include with respect to the efficiency argument, since such a model will have a much smaller inference time cost. It would be great if you could add such model-free baselines with GPT-3.5-Turbo and GPT-4o for comparison.  \n* “We train value models at each position (depth) in the trajectory” (line 594\\) → this was a really surprising methodological choice to me, and seems really strange. Couldn’t you just condition the model on what depth it’s at? Also, the fact that you actually have 4x finetuned models is a big overhead (having to serve 5 models in total\\!) and needs to be mentioned in the main text when discussing computational efficiency.  \n* I found the choice to not use ground-truth rewards at all to be curious (especially given that you *do* use the ground-truth transition function). Wouldn’t you want to at least leverage these for final states? Given that you limit trajectories to be of max 5 on WebShop and then force a buy action, it seems like you should be able to include the rewards at these terminal states. The lack of grounding in real rewards might explain why multiple iterations of STL don’t seem to help on WebShop. \n* “We fine-tuned Game-of-24 value models for 10 epochs and WebShop value models for 20 epochs” (line 681\\) → this sounds like a *lot* of epochs and I’m fairly worried about overfitting. Did you do any analysis to verify that overfitting is not happening?  \n* Why did you use MCTS to generate the training data? Does it work better than BFS or just Monte Carlo rollouts? It would help to motivate this choice more, given that it is a more complicated algorithm.\n\n### Other things\n\n* While it’s great that the paper includes tests for statistical significance, it would also be good to include error bars / confidence intervals.  \n* I think there should really be some more discussion of RL-trained reasoning agents in the Related Work section, i.e. R1, o1/o3, Gemini 2.5, Claude 3.7/4, etc., given that this approach is the current zeitgeist.   \n* What’s the budget you use for MCTS?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose Self-Taught Lookahead (STL), a self-supervised method that leverages state transition dynamics to improve a value model without using any labeled data. The proposed method is used to train the Llama-3.1-8b-instruct model as a value model, which can then be used with any search algorithm at inference to improve the performance of a Language Model-based Agent. Authors conduct extensive experimentation using different value models (trained and training-free) to demonstrate the efficacy of the learned value model. The experimental analysis is conducted on three tasks: WebShop, Multiple-Choice QA (HotPotQA), and Math Puzzles (Game of 24). Experiments show that a 8B parameter learned value model can match gpt-4o performance in all the tasks. The proposed training approach is comparatively lightweight, resulting in significant experimental efficiency. \n\nThe paper is well structured and clearly written. Numerous approaches have been proposed to utilize and train LLMs as value models. To the best of my knowledge, this work presents a novel approach that demonstrates efficiency and performance gains over contemporary methods. The experimental analysis is exceptionally well done, with strong and relevant baselines. The authors provide statistically significant results, which is rare in this research space and is appreciated.\n\nOverall, this is a well thought out and well executed piece of work, and thus **I vote for a clear accept**.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The problem addressed in this work is well-positioned within the current research landscape.\n- The proposed method is comparatively lightweight and does not require human-labeled data, but still outshines the strong baselines.\n- The experimental analysis demonstrates the statistically significant performance gains while being efficient in terms of cost and search steps.\n- The paper is well-written and easy to follow.\n\n## Weaknesses\n\n- One potential weakness of this work is the optional requirement of task filters while creating the training data. While it is optional, I assume that for complex agent environments, this may prove to be challenging to build.\n- For the Reward and Demo Free setting, authors used slightly outdated models (except R1-distilled) as value models. I assume this is done to maintain budget constraints and to provide a fair comparison with existing baselines. However, it will be interesting to see the comparison between newer models, such as the Claude-3.5/3.7/4-Sonnet or Gemini-2.5-Pro, and the trained value model.\nWhile the datasets selected for this work are suitable, it would have been beneficial to see a more open-ended setting with a relatively larger and more fluid action space, such as software engineering environments.\n\nOverall, the strengths far outweigh the weaknesses, and all the weaknesses can be addressed in future work.\n\nQUESTIONS:\n## Questions\n\n- How does the branching factor apply to Greedy search? Is the next action sampled $n$ times, and the best value is selected?\n- Were there any experiments conducted with different search algorithms using the trained value model?\n- In Figure 4, what is the difference between open-source and closed-source tokens?\n- How does the size of the action space affect the performance of the proposed method? Any experiments on this topic would significantly strengthen this paper.\n- One notable result from Table 1 is that the STL with Greedy search performance is significantly lower compared to the AgentQ (trained policy model). It could be interesting to explore the interaction between these trained policy models (even though they are not reward-free) and the trained value model. Additionally, I am uncertain about the contribution of the GPT-4v value model used in the AgentQ method. Any discussion on this would be greatly appreciated.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers an algorithm to improve LLM-based value function estimates for multi-step reasoning tasks via “self-taught lookahead”: exploring state transitions via tree search, estimating the values at the new states, and performing a Bellman backup to obtain a better estimate of the value of the current states. Iterative finetuning via this approach trains a better LLM value function, which then is paired with LLM-simulated lookahead for greedy value-based action selection at inference time. The paper presents empirical results across Webshop, HotPotQA, and Game-of-24 that indicate improve task performance due to the fine-tuned value function.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: \n\nThe problem setting is valuable—LLM agents may be deployed in real-world settings where neither rewards nor human demonstrations are available, and in these settings environment transition dynamics provide a self-supervised learning signal. \n\nComparisons to other search-based methods are thorough, and indicate performance on par with/exceeding more powerful baseline LLMs. \n\nComparisons of test-time token usage and cost are informative. \n\n**Weaknesses**:\nThe non-search baselines for Webshop are not strong—looking at Figure 5 in the ExpeL paper (https://arxiv.org/abs/2308.10144), ReAct with GPT3.5 turbo achieves 35% success rate on Webshop, and Reflexion with 1 and 3 rounds improves performance to 43 and 48% respectively—while this paper’s Reflexion baseline with the same LLM (please report number of Reflexion iterations) achieves 16.4% success rate. The performance of tree search-based methods should be an upper bound on the performance of Reflexion—so I would expect these success rates to all be higher than 48%. \n\nA critical component of the problem setup is the assumption that at later/terminal states, the value function estimates provided by the LLM is “good”—so that the backup operation improves value estimates at earlier states. It seems important to include a discussion of the types of tasks where LLMs are good estimators of value at terminal states—to motivate situations in which this approach is applicable. \n\nThe discussion of the presence/absence of human demonstrations in Figure 1 feels extraneous given that, outside the IL baseline, all other discussion of problem setting is whether or not environmental rewards are present at train/test time. \n\nMissing a discussion of train-time token usage when purely discussing test-time compute usage. \n\nHow did you obtain the Human Expert row in Table 1?\n\nQUESTIONS:\nCould you please provide some context to help me understand the experimental results on Webshop--the baseline numbers for Reflexion are far worse than reported in prior work. See my prior comments for discussion of both ReAct and Reflexion numbers--which should be a lower bound for search-based success rates. This is the place where clarification could most help change my evaluation score. \n\nI would appreciate a longer discussion of Figure 3—is there an experiment you can use with a stronger value function for lookahead states that prevents the earlier decline in performance on unseen tasks? \n\nPractitioners would highly benefit from guidelines for situations in which this approach is applicable vs where it is not--could you please include a discussion of the types of reasoning tasks where LLMs provide good value estimates at terminal states? (Which seems to be the key requirement). This feels like a critical discussion to include and would also help raise my score. \n\nI would suggest including greater detail for some implementation details--number of iterations of Reflexion, source of Human performance baseline, etc.\n\nLIMITATIONS:\nSee note in weaknesses on the assumption that the value estimates from the LLM for terminal states is good—practitioners would highly benefit from guidelines for situations in which this approach is applicable vs where it is not.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}