Title: DynaAct: Large Language Model Reasoning with Dynamic Action Spaces
Venue: NeurIPS 2025
Accepted: yes
Keywords: Sequential Decision-Making

Action Space Optimization

Submodular Function
Abstract: In modern sequential decision-making systems, the construction of an optimal candidate action space is critical to efficient inference. However, existing approaches either rely on manually defined action spaces that lack scalability or utilize unstructured spaces that render exhaustive search computationally prohibitive. In this paper, we propose a novel framework named \textsc{DynaAct} for automatically constructing a compact action space to enhance sequential reasoning in complex problem-solving scenarios. Our method first estimates a proxy for the complete action space by extracting general sketches observed in a corpus covering diverse complex reasoning problems using large language models. We then formulate a submodular function that jointly evaluates candidate actions based on their utility to the current state and their diversity, and employ a greedy algorithm to select an optimal candidate set. 
Extensive experiments on six diverse standard benchmarks demonstrate that our approach significantly improves overall performance, while maintaining efficient inference without introducing substantial latency. The implementation is available at \url{https://github.com/zhaoxlpku/DynaAct}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DynaAct, a framework for dynamically constructing compact action spaces to enhance sequential reasoning in large language models (LLMs). The framework first extracts a proxy action space from demonstration data, and defines a submodular function to balance utility and diversity. Then, it uses greedy optimization to select optimal action subsets at each reasoning step. While the proposed approach shows improvements on six benchmarks, it suffers from fundamental methodological flaws and cost-effectiveness issue that severely limit its practical value.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Dynamic action space construction is a relevant challenge in LLM reasoning, offering a middle ground between rigid manually-defined actions and unconstrained generation.\n2. The evaluations span multiple domains (general, reasoning, math) across six benchmarks, which demonstrate the proposed framework's advantage.\n3. The paper is clearly written.\n\nWeaknesses:\n1. The paper focuses on single-step math and reasoning tasks. However, such problems do not have a clearly defined notion of “step”. Modeling steps as an MDP process, as described in the preliminary section, may not be appropriate. Step-level reasoning is more suitable for agentic tasks such as WebArena [1] and ScienceWorld [2].\n2. The authors employ a small model to select high-level strategies that guide an 8B frozen model in reasoning tasks. The authors primarily compare this approach with actor-critic baselines. However, I wonder whether this architecture can outperform training-based methods. Training-based approaches can significantly reduce inference time compared to the proposed method, which requires approximately 60 seconds per problem.\n3. Generalization remains a concern. Can the strategies summarised from the current six datasets and the trained 1B critic generalize to OOD datasets such as AIME 2024/2025?\n\n[1] WebArena: A Realistic Web Environment for Building Autonomous Agents\n[2] ScienceWorld: Is your Agent Smarter than a 5th Grader?\n\nQUESTIONS:\n1. Whether the authors considered potential duplication caused by semantic similarity among the 40,822 actions used for selecting high-level strategies. Did they filter out different surface forms corresponding to the same strategy?\n2. Is the \"action\" in MCTS referred to as the whole reasoning process or a step of the process?\n3.  I would like to know more about the effect of the choice of m on the experimental results.\n4. I would like to see some examples on the value of submodular function, is the function value consistent with human intuition?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies the limitation of the current framework on MDP-style LLM reasoning on lacking a scalable and compact action space. In response, the authors propose to first mine action space from existing corpora and formulate a submodular function to evaluate the utility and diversity of the actions to form a dynamic action space. Extensive experiments on six tasks show good performance improvement over previous methods with good latency control.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. This paper is relatively well-written and easy to follow.\n\n2. The authors conduct experiments on a wide range of tasks, including both QA and math tasks.\n\n3. The analysis is well-designed, including compactness, utility, and real-time latency.\n\n\nWeakness\n\n1. Studies on other backbones can further consolidate the performance gain over baselines and show the generalization of the learned submodular function on other models, e.g., Qwen 2.5, and other-sized models, e.g., 1.5B, 70B models. It is unclear how robust the action evaluation function is with other models or if it transfers among models.\n\n2. It would be preferred if the authors could conduct qualitative analysis, e.g., examples of the actions, success and failure examples of the proposed methods, and change of action space at different steps of the optimization, etc.\n\n3. Further comparison with few-shot prompting methods is needed since the dynamic action space selection is similar to dynamic exemplar selection [1] in terms of the extra prior introduced to the frozen inference models. Currently, in F.8 (page 19), the authors experiment with few-shot baselines with embeddings from Llama 3.2 1B is not the common practice. What is the performance of newer models such as GritLM [1], Gecko [2], or Instructor [3]?\n\n[1] Learning To Retrieve Prompts for In-Context Learning: https://arxiv.org/pdf/2112.08633   \n\n[2] GritLM: https://arxiv.org/abs/2402.09906 \n\n[3] Gecko: https://arxiv.org/abs/2403.20327 \n\n[4] Instructor: https://arxiv.org/abs/2212.09741\n\nQUESTIONS:\n1.\tWill the authors open-source the observations extracted in Section 4.4 to facilitate future research?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
In this paper the authors introduce DYNAACT, a framework to construct compact action spaces dynamically to enhance reasoning in sequential decision-making tasks. Their method employs submodular optimization to balance utility and diversity in action selection, ensuring scalability and efficiency. The authors performed extensive experiments across six benchmarks (MMLU, GPQA, ARC-C, GSM8K, MATH-500) to demonstrate the strength of their approach. DYNAACT achieved 6.8% higher accuracy on MATH-500 compared to other baselines.

STRENGTHS AND WEAKNESSES:
The authors provided a rigorous methodology for dynamic action space construction using submodular functions, and through extensive experiments validated their approach across six benchmarks.

The paper is well-written, with clear explanations of the submodular optimization approach and experimental results.

The focus on enhancing reasoning in sequential decision-making using large language models is highly relevant. The paper has demonstrated the importance of well defined action spaces in MDP based reasoning. Their experimental analysis shows promise of this approach.

The dynamic action space construction is a novel research question, and the use of submodular functions to balance utility and diversity in action selection seems to be novel.

QUESTIONS:
Can this be applied to tasks with highly dynamic and unpredictable action spaces? For example in robotics?

Sub modular optimization can be computationally expensive. Do you see any scalability problems related to this for large complex domains?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
The authors develop a novel framework named DYNAACT for automatically constructing a compact action space to enhance sequential reasoning in complex problem-solving scenarios, which firstly estimates a proxy for the complete action space by extracting general sketches observed in a corpus covering diverse complex reasoning problems using large language models

STRENGTHS AND WEAKNESSES:
1. the paper is well written and easy to follow
2. the technique details are sound
3. the novelty is not enough

QUESTIONS:
The core idea is to construct high-level actions (referred to as action subsets in the paper) to group similar low-level actions — an approach that is not novel in the LLM community.

I have a few questions and suggestions regarding your method:

1. How is the number of action subsets determined in your approach? Additionally, how does this number impact the overall performance of your method?
2. In my experience, embedding-based clustering tends to be fragile and often lacks interpretability, particularly in understanding the semantics of cluster centers. Have you considered using LLM-based clustering, where the categories of low-level actions are derived via LLM reasoning?
3. Llama3 series and some benchmarks, e.g. GSM8K, MATH500, I suggest evaluating your method on AIME24 and AIME25 using the Qwen3 model series to make the experimental results more convincing and up-to-date.

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03