Title: Feedback-Aware MCTS for Goal-Oriented Information Seeking
Venue: NeurIPS 2025
Accepted: yes
Keywords: Information Seeking

Large Language Models

Uncertainty Reduction

Planning

Search
Abstract: Effective decision-making and problem-solving in conversational systems require the ability to identify and acquire missing information through targeted questioning. A key challenge lies in efficiently narrowing down a large space of possible outcomes by posing questions that minimize uncertainty. To address this, we introduce a novel framework that leverages Large Language Models (LLMs) to generate information-seeking questions, with Monte Carlo Tree Search (MCTS) to strategically select questions that maximize information gain, as a part of inference-time planning. Our primary contribution includes a hierarchical feedback mechanism that exploits past interaction patterns to guide future strategy. Specifically, each new problem is mapped to a cluster based on semantic similarity, and our UCT (Upper Confidence bound for Trees) formulation employs a cluster-specific bonus reward to prioritize successful question trajectories that have proven effective for similar problems in the past. Extensive empirical evaluation across medical diagnosis and technical troubleshooting domains shows that our method achieves an average of 12\% improvement in success rates and about 10x reduction in the number of LLM calls made for planning per conversation, compared to the state of the art. An additional 8\% gain in success rate is observed on average when we start with a constrained set of possibilities. Our results underscore the efficacy of feedback-aware MCTS in enhancing information-seeking in goal-oriented dialogues.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on using MCTS to improve information seeking systems in dialogue tasks. The authors proposed modifications to the MCTS algorithm for finding information-maximizing questions by prompting LLMs; and a feedback mechanism that enables learning from experience (i.e., previous runs) by updating the search tree across runs. The authors experimented with their framework in several datasets spanning medical diagnosis, trouble-shooting, and 20 questions and showed improvement compared to prior baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The proposed modifications to MCTS to maximize information gain is intuitive.\n- The authors conducted thorough experiments in multiple datasets (medical diagnosis, trouble-shooting, and 20 questions) and showed improvement performance compared to two prior baselines.\n\nWeakness:\n1. The proposed system adds significant complexity both in compute and in time. MCTS is already highly time-intensive to run, and the authors also added methods such as clustering and using embeddings of new user problems to assign bonus rewards for future tree search. However, the authors did not report statistics such as runtime in the main results (e.g., Table 1).\n2. There are many LLM based MCTS omitted in the related work section. Example includes [1-2]. Since the main contribution is a modified MCTS algorithm for the domain of question asking, the author should at least compare against one of these prior work to show that the proposed approach outperforms a \"vanilla\" LLM + MCTS implementation.\n\n\n---\n\nReferences\n\n[1] Zhou, Andy et al. “Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models.” ArXiv abs/2310.04406 (2023): n. pag.\n[2] Yu, Xiao et al. “ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning.” ArXiv abs/2410.02052 (2024): n. pag.\n\nQUESTIONS:\nQuestions:\n- How often can LLM really be prompted to generate questions that maximize information gain, as mentioned in line 150-151?\n- Why is simulation (L184-188) based on random interaction? In many previous work that applies MCTS with LLM it is quite common where the simulation process is against another LLM (by prompting or after training)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a mechanism to dwindle down a set of possible questions (combining Monte CarloTree Search with a hierarchical feedback mechanism) for LLMs during reasoning tasks. The authors devise MISQ-HF to select an optimal sequence of specific questions to reach a \"target\" state. The proposed method is validated on real-world medical datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The MCTS for question selection and cluster-based feedback mechanism are both, to my knowledge, novel and are rightfully presented as so. The use of UCT is natural to guide information-seeking questions.\n- This paper is v well written and clear in its exposition. I do suggest that the authors add a running example is added to the main text (Sec 1) to get a larger audience excited about this problem, such as a medical example to motivate why (1) an optimal sequence of direct information-seeking questions matters and (2) feedback propagation looks like intuitively.\n\nWeaknesses\n- I would have prefer a stronger grounding in existing literature. It would help to flesh out Related Work to make clear how MISQ-HF differs from baselines. In an ideal world, this could be added to Figure 1 to visually depict the sub optimality of existing methods (specifically UoT). \n- I would have liked to see examples in the main text of the how the MCTS information-seeking questions differ from UoT: I am convinced that the experimental results are sound but cannot wrap my head around how (semantically) these questions are different.\n\nQUESTIONS:\n1. While I understand this is likely out of scope, I would have loved to seen a human subject experiment that validates from a set of questions which one would a human pick, which one would an LLM pick, and which one would be optimal. This would help motivate the practical utility of MISQ-HF. \n2. How does the cost of the response to an information-seeking query fold into this setup? Some questions are more costly to answer: asking for a person their date of birth is easier (and less effort for the person) that asking a person for their current body temperature. How would this change the feedback and question selection mechanism? This consideration seems crucial for deploying this system in practice.\n3. Can you please add a pictorial depiction of closed vs open set considerations to Figure 1? It would help guide uninitiated readers.\n4. Please add a computational complexity comparison of UoT and MISQ-HF to the end of Sec 3.\n\nLIMITATIONS:\nInformation-seeking questioning is a blessing and a curse. I would have expected the authors to add significant detail (either to the conclusion or the Broader Impacts in Appendix H). An optimal set of questions may not be accepted by users themselves as (1) users may provide ancillary information or (2) users may view certain questions as coupled -- doctors might be expected to ask Questions A, B, C before jumping to the more invasive yet relevant Question D no matter how optimal omitting Question A, B, C are. I strongly encourage the authors to add such thinking (with their own expansion of course) to the manuscript. It will significantly improve that societal implications of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a generation framework for information-seeking questions.  They have applied MCTS to select questions with a strategy and introduce a hierarchical feedback mechanism to exploit past interaction pattens to guide future strategy. They provide extensive empirical evaluation across medical diagnosis and technical troubleshooting domains to show effectiveness.

STRENGTHS AND WEAKNESSES:
Strength:
1. They are utilising LLM and MTCS to generate reasonable responses for information-seeking questions. Overall, the architecture design is diligent and reasonable. 
2. The problem formulation and method description are clear and easy to follow.
3. Experimental results are effective in demonstrating their methods.
4. They provide very detailed implementation details such as code, prompts and a case study to make it trustworthy.

Weakness:
1. Engaging more baseline models to compare.
2. It would be better to incorporate more explanations into the method description sections. e.g., in Line 176, what's the fundamental motivation for the UCT, and why it can be formulated as (7)?

QUESTIONS:
1. How UCT is defined in Line 176? Can you explain more about this one?
2. in terms of the uncertainty problem, how do you evaluate it?

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-05

comments: SUMMARY:
They bolt feedback-aware MCTS onto an LLM, recycle good question paths, and end up diagnosing faster and cheaper than vanilla planners, but the trick still struggles without a fixed answer list and never punishes bad decisions.

STRENGTHS AND WEAKNESSES:
Strengths:

By folding Monte-Carlo Tree Search around an LLM, the system asks dramatically fewer questions yet lands on the right answer more often. It reuses a single search tree across cases, cashing in on past successes, and the cluster-based bonus gives it a built-in memory that nudges the planner toward historically efficient moves. Tests across medical triage, fault diagnosis, and even a 20-Questions game show double-digit accuracy gains while slashing API calls by roughly an order of magnitude, and it works with everything from Llama 3 to GPT-4o—so the gains come from the search strategy, not some hidden model magic.

Weaknesses:

The method reinforces only successful branches—there’s no penalty signal—so it keeps repeating bad ideas instead of learning from them. When the answer space is open-ended, its hit rate drops to ~28 %, showing the approach doesn’t generalize well outside neatly bounded domains.

QUESTIONS:
See Strengths And Weaknesses

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05