Title: PlanU: Large Language Model Reasoning through Planning under Uncertainty
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language models; decision making; reasoning
Abstract: Large Language Models (LLMs) are increasingly being explored across a range of reasoning tasks. However, LLMs sometimes struggle with reasoning tasks under uncertainty that are relatively easy for humans, such as planning actions in stochastic environments. The adoption of LLMs for reasoning is impeded by uncertainty challenges, such as LLM uncertainty and environmental uncertainty. LLM uncertainty arises from the stochastic sampling process inherent to LLMs. Most LLM-based Decision-Making (LDM) approaches address LLM uncertainty through multiple reasoning chains or search trees. However, these approaches overlook environmental uncertainty, which leads to poor performance in environments with stochastic state transitions. 
Some recent LDM approaches deal with uncertainty by forecasting the probability of unknown variables. However, they are not designed for multi-step reasoning tasks that require interaction with the environment. To address uncertainty in LLM decision-making, we introduce PlanU, an LLM-based planning method that captures uncertainty within Monte Carlo Tree Search (MCTS). PlanU models the return of each node in the MCTS as a quantile distribution, which uses a set of quantiles to represent the return distribution. To balance exploration and exploitation during tree search, PlanU introduces an Upper Confidence Bounds with Curiosity (UCC) score which estimates the uncertainty of MCTS nodes. Through extensive experiments, we demonstrate the effectiveness of PlanU in LLM-based reasoning tasks under uncertainty.


Peer reviews:
comments: SUMMARY:
This paper proposes PlanU, an LLM-based planning framework that addresses both LLM and environmental uncertainty (e.g., stochastic state transitions). It combines quantile distribution modeling of return values and a UCC (Upper Confidence Bounds with Curiosity) exploration strategy within MCTS. The approach is evaluated on multiple benchmarks, including Blocksworld, Overcooked, and VirtualHome, using several LLM backbones. Experimental results show that PlanU consistently outperforms strong baselines.

STRENGTHS AND WEAKNESSES:
Strengths
• PlanU is the first to introduce quantile-based value estimation into LLM-driven MCTS, which is a notable step toward enabling more robust and uncertainty-aware planning in real-world LLM agents.
• Thorough testing across multiple benchmarks. The inclusion of variations such as prompt injection and prompt shuffling further strengthens the experimental credibility.

Weaknesses
• All tested backbones are in the 7–8B parameter range. Evaluating across more diverse model scales (e.g., smaller or larger LLMs) would improve the generality of the claims.
• The ablation study could be more fine-grained. For example, testing hybrid configurations such as quantile + standard UCT or mean value + UCC would better isolate the individual contributions of each module.

QUESTIONS:
•  See the weakness section.
•  line 212: a uncertainty-aware - an uncertainty-aware

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the PlanU framework, which combines the large language model (LLM) and tree search (MCTS) to find the best planning path in an uncertain environment. It uses distributed (rather than average) rewards to represent uncertainty and introduces curiosity to encourage exploration.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n1. The current mainstream exploration is indeed mainly concentrated in deterministic environments. LLM decision-making in uncertain environments is realistic and meaningful.\n\n2. The experiments in this article prove the effectiveness of PlanU, which is more effective than other MCTS and RL baselines.\n\n**Weakness**\n1. The main technical contributions of this paper largely stem from prior works, such as Monte Carlo Tree Search (MCTS), Quantile Regression DQN (QRDQN) [1], and using LLMs as action priors. The novelty of estimating novelty via the discrepancy between two neural networks also follows the idea proposed in [2].\n\n2. The experimental setup appears to be overly toy-like. Most of the failure cases are manually injected, which may not reflect the challenges of real-world environments.\n\n3. In Equation (6), the authors mention that they use the expectation operator \\( \\mathbb{E} \\) as the default \\( \\psi \\). This raises strong concerns about the necessity of learning the full distribution \\( Z(s_t, a_t) \\), as this essentially reduces to using the mean reward. In my view, this significantly undermines the novelty of the method.\n\n4. The proposed UCC score lacks theoretical justification, whereas classical UCT formulations offer well-established theoretical guarantees.\n\n5. I believe the proposed framework does not truly exploit the capability of LLMs as agents. The LLM is only used for initializing probabilities. A more powerful and effective usage would be to adopt LLMs as ReAct-style agents that can dynamically reason and decide the next action based on environmental feedback.\n\n**References:**\n\n[1] Dabney, Will, et al. \"Distributional reinforcement learning with quantile regression.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 32. No. 1. 2018.\n\n[2] Burda, Yuri, et al. \"Exploration by random network distillation.\" *arXiv preprint arXiv:1810.12894* (2018).\n\n[3] Yao, Shunyu, et al. \"ReAct: Synergizing reasoning and acting in language models.\" *International Conference on Learning Representations (ICLR)*. 2023.\n\nQUESTIONS:\n1. In the setting of this paper, the introduction of uncertainty seems to be limited to the possibility that an action may fail. However, in most cases, wouldn’t simply re-executing the same action suffice? What is the intuitive benefit of introducing a reward distribution in this context? Moreover, given the vast optimization space, it raises concerns about what kind of distribution is actually being learned.\n\n2. In Figure 5 (a), we observe that removing any single component results in almost no return improvement for the baseline. Why is this the case? Could the authors provide more explanation or insights? In Figure 5 (b) and (c), we see that the proposed method and the baseline do not seem to be clearly distinguishable, which reinforces my doubts about the validity of the experiment.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PlanU, a planning algorithm for LLM-based agents that integrates a distributional reinforcement learning approach into an MCTS framework. The core contribution is the representation of state-action values as a probability distribution rather than a scalar estimate. The authors argue that this allows the planner to more effectively handle decision-making in environments with stochastic state transitions. The method is evaluated on modified versions of some language-based planning tasks, where it is shown to outperform several baseline MCTS-based LLM planners.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__\n\n* Significance & originality: The paper addresses the important and challenging problem of enabling LLMs to plan robustly under uncertainty. The primary idea of incorporating a distributional value representation into an LLM-based planner is novel. In principle, a value distribution can carry more information than a scalar mean, potentially allowing for more sophisticated and risk-aware planning. This represents an interesting direction for research in LLM agents.\n\n__Weaknesses__\n\n* Flawed motivation and positioning: The paper's motivation is built on a questionable framing of the problem. The argument has two primary weaknesses:\n  * Incorrect claims about MCTS: The paper repeatedly asserts that standard MCTS is unsuitable for stochastic environments (e.g., Lines 104, 161). This is factually incorrect. The canonical UCT algorithm was explicitly designed for and is proven to be asymptotically optimal in stochastic settings (MDPs).\n  * Mischaracterizing a design choice as a fundamental limitation: While it may be true that the specific LLM-based MCTS baselines compared against in the paper were not designed for stochastic outcomes, this ---- in my opinion --- should be viewed as a simplifying implementation choice, likely because their original target tasks were deterministic. It is not, as this paper implies, a fundamental limitation of the MCTS framework itself when applied to LLMs.\n\n* Quality of evaluation: The choice of benchmarks and the method of introducing stochasticity feel contrived. Introducing stochasticity by, for example, having a random chance of an action failing, feels artificial and does not necessarily represent the kinds of uncertainty LLM agents will face in more naturalistic settings (e.g., ambiguous instructions, partially observable states, etc). This choice of evaluation makes it difficult to assess the true generalizability and impact of the proposed method on problems with genuine, inherent uncertainty.\n\n* The rationale for initializing the value distribution Z during the expansion phase using the policy's sampling probability is not justified. It is unclear why this is a sensible choice or how sensitive the algorithm's performance is to this specific initialization strategy.\n\nQUESTIONS:\n1. Related to the point discussed in weaknesses, canonical MCTS algorithms like UCT can handle stochastic environments. Could you please clarify why they are insufficient for the problems you are tackling, thereby requiring a distributional value function? Could you for example directly extend an UCT algorithm to LLMs and use it as a baseline?\n\n2. Could you justify the specific choice of initializing the value distribution Z based on the policy's action probability during expansion? An ablation study on this initialization choice would be helpful.\n\nLIMITATIONS:\nI would suggest the authors discuss the following points:\n\n* Generalization and scope: The evaluation is confined to planning tasks with artificially injected, simplistic stochasticity. The authors should discuss the challenges and potential performance implications of applying PlanU to domains with more complex and naturalistic uncertainty.\n\n* Failure modes: A discussion of potential failure modes would be valuable.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper introduces PlanU, an MCTS-based decision-making framework for LLM agents that explicitly handles both LLM generation uncertainty and environmental stochasticity. It models each action-node return as a quantile distribution instead of a mean value, and guides exploration with an Upper Confidence Bounds with Curiosity (UCC) score that combines distributional value estimates and a novelty bonus computed via random-network distillation on text embeddings. PlanU is evaluated on stochastic variants of Blocksworld, Overcooked, and VirtualHome.

STRENGTHS AND WEAKNESSES:
Strength:
1. The motivation is clear. 
2. The clarity is good and the paper provides intuitive figures to help understanding.

Weakness:
1. The three datasets the paper use are toy, textual and simulated. There're no real-world, long-horizon tasks with rich observations or actuation. And the model sizes are small.
2. Compute cost and token/runtime overhead haven't been discussed.
3. The paper lacks enough details about experiment setup in the main text.

QUESTIONS:
1. Can you report wall-clock time, tokens for PlanU vs RAP/ToT?
2. Can PlanU perform well on some real-world tasks?
3. Can PlanU been adapted to Web agent? 
4. I think it'll be better if the authors include PlanU's performance into figure 1.
5. Can PlanU been extended to closed-source models?
6. Can you provide a deeper comparison between PlanU and RAP since they are all MCTS-based methods?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-22