DynaAct: Large Language Model Reasoning with Dynamic Action Spaces
Authors: Xueliang Zhao
Wei Wu
Jian Guan
Qintong Li
Lingpeng Kong
Venue: NeurIPS 2025 poster ()
Keywords: Sequential Decision-Making
Action Space Optimization
Submodular Function
Primary area: reinforcement_learning

Abstract: In modern sequential decision-making systems, the construction of an optimal candidate action space is critical to efficient inference. However, existing approaches either rely on manually defined action spaces that lack scalability or utilize unstructured spaces that render exhaustive search computationally prohibitive. In this paper, we propose a novel framework named \textsc{DynaAct} for automatically constructing a compact action space to enhance sequential reasoning in complex problem-solving scenarios. Our method first estimates a proxy for the complete action space by extracting general sketches observed in a corpus covering diverse complex reasoning problems using large language models. We then formulate a submodular function that jointly evaluates candidate actions based on their utility to the current state and their diversity, and employ a greedy algorithm to select an optimal candidate set. 
Extensive experiments on six diverse standard benchmarks demonstrate that our approach significantly improves overall performance, while maintaining efficient inference without introducing substantial latency. The implementation is available at \url{https://github.com/zhaoxlpku/DynaAct}.

DyNAACT: Large Language Model Reasoning with Dynamic Action Spaces
Xueliang Zhao
Wei Wu
Jian Guan
Qintong Li
Lingpeng Kong*1
The University of Hong Kong
* Ant Group
{xlzhao, qttla, qtli, 1pk)ess hku. hk
(wuwei19850318, jianguanthu]@gaaill
Abstract
In modern sequential Decision-making systems, the construction of an optimal can- didate action space is critical to efficient inference. However, existing approaches either rely on manually defined action spaces that lack scalability or utilize un- structured spaces that render exhaustive search computationally prohibitive. In this paper, we propose novel framework named DYNAACT for automatically constructing compact action space to enhance sequential reasoning in complex problem-solving scenarios. Our method first estimates a proxy for the complete action space by extracting general sketches observed in corpus covering diverse complex reasoning problems using large language models. We then formulate a submodular function that jointly evaluates candidate actions based on their utility to the current state and their diversity, and employ greedy algorithm to select optimal candidate set. Extensive experiments six diverse standard bench- marks demonstrate that our approach significantly improves overall performance, while maintaining efficient inference without introducing substantial latency. The implementation is available at httpps://// thubb.b...tt......m thub. com /oom//haxx//a Ipku/Dnaactt,
1 Introduction
Recent advances in complex reasoning with Large Language Models (LLMs) AAciam et al. 2023 Jaech 2024 have established a prevalent self-improvement paradigm: given mass of problems paired with final answers, model developers first search for correct reasoning paths from a base model using test time scaling strategies [Snell al., [024], then improve it internalize these
patterns |uo et al.) 22255 through supervised tunint Guan et al., 2025] or reinforcement learning Guo al. 20255. While LLMs have exhibited remarkable reasoning capabilities, current approaches to long
long-term term reasoning in these models often suffer from fundamental limitations. On the one hand, some approaches explicitly define action space and state space and structure reasoning hierarchically, where action selection and state prediction are carried out iteratively [Hao et al.) 2023 Qi et al. 2024) along the reasoning path. However, because action spaces are often heuristically designed, the resulting actions tend to either be too specific to generalize across domains or too broad to effectively guide reasoning. On the other hand,
absence of an explicit definition for action spaces, other approaches impose specific format generation and perform reasoning in an autoregressive manner Lightman et al. 2023, Guo et al. 2025). These approaches inherently search the entire natural language space for reasoning. thereby necessitating powerful base models.
We investigate LLM reasoning within the framework of Markov Decision Process (MDP) (Hao et al.) 2023 where a reasoning trace consists of series of actions and states. Instead of focusing on
This work done during internship Ant Group. Corresponding authors.
1
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Quastiors Earth conter of the universi?
Subset
State
Thiik dbort eorler stoupfoon. Sod Crrsder post axmpiess contidity.
Shep 1: Revkww part caumpteers
State
Shep Resee post esseeption,
Pholeny's
Action of Acceas modem
Seteedikrr Furcties: F= Fax
Aaxaa moder understonding Bantakk core pricegaa
Action Space
perspactive.
④ Think about corter assumptices
ant covrptiera contrdiity.
Acaas rodem apumpt lena. exstor Cuollage athoiished prrspcttives
Good but
Step 1:
State Mus!
2 challegge catettihel perpectives.
modere conueptione,
Benkak care pereppas.
The belocermic meannl supported by
Submodulor each subset
Figure 1: Overview of the proposed method. Given the proxy action space A, the method searches for the subset A, that maximizes the submodular function, which consists of a utility term and a diversity term. The subset A, is then used for the subsequent reasoning steps.
policy learning or reward modeling, we pay special attention to action space construction, as well- defined action space is fundamental to MDP-based reasoning. Specifically, we identify two essential properties that a qualified action space should possess: (1) Scalability: it should be automatically learned from demonstration data, rather than being manually engineered, to strike a balance between generalization and utility; and (2) Compactness: it should maintain dynamically constructed, information-dens structure so that for each step desired action can be picked up from a small yet complete candidate set. The core technical challenge lies in developing principled approach that simultaneously optimizes both objectives: distilling generalizable action patterns from demonstrations while eliminating redundant candidates that would otherwise impede efficient exploration.
To achieve both scalability and compactness, we frame the problem of action space construction as a subset selection task and propose ACT, in which the action space for each reasoning step is dynamically determined by submodular function that is learned through a data-driven approach. The key idea is approximate small subset from the entire action space that achieves the optimal balance between utility and diversity, leveraging the diminishing returns property of submodular functions to ensure linear computational complexity. Our method begins by extracting general reasoning patterns from a diverse corpus of complex problems to construct the complete action space. We then define submodular function to evaluate candidate actions by jointly considering their utility to the current state and their diversity contribution. By maximizing this function via a greedy algorithm, we obtain optimal subset of actions. Consequently, reasoning follows standard Markov process: each step. candidate actions are selected via the submodular optimization, an action is then chosen according Q-function estimated via Monte Carlo tree search, and reasoning step is finally generated conditioned on the current reasoning context and the chosen action. Notably, throughout this process, only lightweight embedding model used in the submodular function requires training, while the base LLM remains frozen.
We conduct extensive experiments six benchmarks spanning general, reasoning, and math tasks. Evaluation results indicate that DYNAACT achieves significant improvements over baselines across all tasks, including MMLU, MMLU-Prs GPQA, ARC-C, GSM8K, and MATH-50K Notably, DYNAACT excels at solving complex problems, achieving a 6.8% absolute gain over the recently proposed strong model rStar MATH-500 Furthermore, extended study demonstrates that while dynamic action space construction enhances efficacy, it does not introduce significant additional latency during inference compared to the baselines.
Our contributions are fhree-fold: (1) We propose dynamic action space construction as novel research question, orthogonal to the extensive studies LLM reasoning in the community: (2) We introduce a submodular function for action space construction, which significantly improves problem-solving accuracy while maintaining reasonable inference efficiency; and (3) We empirically verify the efficacy of our method across a wide range of tasks.
2 Preliminaries
Before delving into ACT, provide some background information. We begin by formulating MDP-bbased reasoning, then describe the search strategy applied. The section finally gives brief introduction to submodular functions, which form the theoretical basis for action space construction.

---

2.1 Reasoning Framework
We formulate LLM reasoning as an MDP, defined by tuple the state space where each state 8, E S encodes the cumulative reasoning context up to step t, with the initial state derived from the input prompt. A denotes the action space with E A an action indicating the progression of the reasoning context. The transition function T : S x A - S determines how actions transform the reasoning state, while R. 8 A + R assigns rewards based on the quality of reasoning steps. The discount factor E [0, 1] balances immediate and future rewards.
Here, S represents
At each time step I, the LLM selects an action d, from a candidate set A. A, which may gener- ated automatically [Hao et al. 2023) or specified manually [Qi et al.] 2024). In our implementation. the action selection is governed by learnable value function (55 a) that estimates the expected cumulative reward:
at arg max ((s,, a), aeA:
Q
= E
* R(s+4, 0c+k) St. at
Our primary focus is on developing a principled approach to construct the candidate action set A, at each step, as this significantly impacts the efficiency and efficacy of the reasoning process in complex problem-solving scenarios.
2.2 Monte Carlo Tree Search
We employ Monte Carlo Tree Search (MCTS) for estimating Q(s a) [Silver et al. 2016).
nutshell, the estimation is achieved by simulating multiple continuations of the reasoning process from the current state after applying the candidate action. During these simulations, the outcomes of the extended reasoning traces are evaluated, and the value of an action is approximated as the average result observed across the simulations. In this way, MCTS effectively balances the exploration of less frequently visited candidate actions with the exploitation of those that have demonstrated promising progress, which is critical to ensure that the estimated value function reliably reflects the potential benefit of each action in guiding the overall reasoning process. We defer additional technical details of MCTS, including how selection, expansion, simulation, and backpropagation are performed, to Appendix
2.3 Submodular Functions
The problem of action space construction can naturally be formalized as a subset selection task, where the goal is to identify small, high-value subset from a much larger set of potential candidates. A typical approach to subset selection involves assessing the utility of each element, ensuring that every new element added contributes additional value to the overall set. To achieve this, submodular functions are often employed due to their diminishing returns property. which prioritizes the selection of elements that offer unique and informative value [Fujishige, 2005)] Kothawade et al., [022 Chen et al.) 2024]. As result, subset selection with submodular functions ensures that the marginal benefit of adding an element to smaller subset is greater than adding to a larger one, thus enhancing the compactness of the subset.
Formally, given two candidate sets X C X' and action a e X', a submodular function F(:-)) satisfies the following condition:
F(X F(XU(a):s) - F(X:s)) 2 F(X'U F(X'ss
(1)
Heading toward scalable approach for constructing compact action spaces, we take advantage of submodular functions. The problem then boils down to (1) how to define a proper submodular function; (2) how to learn the submodular function from data; and (3) how to perform action space construction with the submodular function, as will be presented in the following Section.
3

---

Algorithm Complete Pipeline of Sequential Reasoning.
Require: Input question 4, dataset D, number of groups k, candidate selection budget maximum reasoning
steps T'
1: Proxy Action Space Estimation (performed once)
2: Partition the dataset D into k groups: (Di, .2....... D
3: for i to & do
4:
Extract observation sketch
(a1, 02 Coo, > LLMQQery(D.)
5: Form the action space AA = U 0 U (01, 02,.... .),.1)
Train the embedding function e using Q-learning objective (Eq. (5) with ohservation sketches as demon- stration data
7: <- InittializeState(g))
8: for to 1 do
1s Candidate Action Selection via Greedy Algorithm
10:
X
< 0
11:
for
i = to m mo do
12:
Xa f A
13:
f- arg max.ex, F(X U (a)
X XU
A,(st) -
16:
10 Action Evaluation using MCTS */
17:
for all a (st) do
18:
((5::a) MCTS(s;, a)
19:
10 Action Selection and State Update */
20:
Or arg A((4) (8
21:
S141 < UpdateState at)
Output: {so, Qa, $1,...,
3
Method
We detail the approach to constructing A, given the current state 8,. Specifically, our method consists of three stages: estimating an approximation the complete action space as ($3.I), defining submodular function F(A,, 8t) based ($3.2) and utilizing the function to derive A, ($3.3)
Figure Soureeprooee provides an overview of the method.
3.1 Proxy Action Space Estimation
We first estimate an approximation as proxy of the complete action space, denoted as A. Specifically, we follow Wang et al. [0024a] by employing observations as candidate actions, where observations typically cues that guide the reasoning process (cf. Figure T. Given problem corpus (e.g., mathematical questions, logical puzzles, etc.), we randomly divide it into k groups and feed each group an LLM for observation collection. The division strategy ensures that each group is appropriately sized, avoiding prohibitive computational costs from the LLM. We then query the LLM extract general observation sketchs per group that can be applied broadly and focus solely on the core operations. After that, the resulting observations are collected and form the proxy action space A with duplicate items removed. The prompt for observation sketch extraction is provided in Appendix [ Notably, A can be easily scaled up by incorporating examples from broader domains or fields; one can also develop various agents by applying the method domain-specific corpora.
3.2 Submodular Function Definition
To construct optimal candidate action set that balances both utility (in terms of expected rewards) and diversity, we propose a submodular function F(Ar; 8x) for a candidate subset Ar C AA as follows:
F(Ar st) a fuun fuun (.) + faiv (At),
(2)
where [neit (Ar; 8.) measures the expected utility of the candidate actions in advancing the reasoning process, faiv (At) promotes diversity within the selected set, and a, 8 are balancing parameters. To ensure that F (Ag st) defined by Eq. 2 meets the condition given by Eq. (I) we define the utility

---

term as:
Σ exp cxp(o(s))<c(a))
(Ac; = log
where e(-) is an embedding function that maps states and actions to shared representation space. Then the diversity term is defined
faiv (At)
E min (1- (1-e(u; (l-)
ayEA.
orEA, a;za;
This formulation encourages the selection of actions that maximally distinct from each other in the embedding space, preventing redundancy the candidate set.
Lemma 1. Given the definitions of the relevance term full (A,; st) in Eq. (3) and the diversity term (At) in Eq. (4) the function F F(Ac; st) defined in Eq. (2) submodular with respect to the candidate action set A, S A.
The proof of Lemmal 1 provided in Appendix
fundamental requirement of our framework is ensuring that A, contains actions that maximize expected rewards in the reasoning process. To this end, design the embedding function e(-) to capture the effectiveness of actions in advancing the reasoning process. We formalize this requirement through Q-learning, where ((s,)"e(a approximates the Q-value- the expected future reward of executing action a in state 8t. By incorporating this formulation into the standard Q-learning update equation [Watkins and Dayan 1992 Reddy al., 2019]. derive the following optimization objective:
Σ exp (c($141)) e(d')) )) \a'EAA
[(s,, = c(st) c(a)
r log
(5)
where the training data consists state-action pairs (St, ac), with a; E A being the ground-truth action obtained from the observation sketches. The reward r is defined as r for a and r 0 for all other actions a E AA \ {a:}. }- We train the embedding model by minimizing L over all states st in the training set and all actions E A. This objective ensures that the embedding function learns to prioritize actions that contribute substantively the problem- boobemm-oolving progression.
3.3 Action Space Construction
With A and F(5-) at hand, we aim to derive Ae by selecting m elements from A that maximize F(; 8t). Formally, we construct A, by solving
As(st) = arg
max
F(X;st).
(6)
X(d.IX|=m
Owing to the submodularity of F(:-), the combinatorial optimization problem (Eq. (6) can be effec- tively approximated by greedy algorithm [Nemhauser et al. 1978) with complexity of where [A| denotes the size of A. In practice, we begin with an empty set X and iteratively add the element from AA \ X that produces the highest marginal increase in F(-:)) .) when combined with the current X. Formally, for each iteration, we define the set of remaining candidates X₃ A X and select the element such that
arg max F(X U st): xEX
The process is repeated until m elements have been selected, and the final set X is then taken as A,.
Algorithm I summarizes the complete pipeline of reasoning method, encompassing both action space construction and MCTS- reasoning path search.
Experiments
4.1 Benchmarks
We employ six standard benchmarks covering three domains: general, reasoning, and math. Specifi- cally, we use the following datasets:
5

---

(I) MMLU [Hendrycks et al. 2020] is a benchmark designed to evaluate a model's 's ability to answer a wide variety of tasks, including reading comprehension, reasoning, and problem-solving across general domains. It is widely used for assessing language model performance in broad tasks; (2)
MML.U-Pro Wang et al., 2024b) an extension of MMI.U, containing more challenging and professional-leve problems. This dataset tests the model's 's capabilitiess on more complex problems in general domains; (3) GPQA [Rein et al. 2023] focuses on evaluating reasoning and problem-solving skills, providing real-world open-domain problems that require advanced reasoning to solve; (4)
ARC-challenge (ARC-C) [Clark et al. 2018) is part of the A12 Reasoning Challenge and contains science-basee based multiple-choice questions. These questions require deep reasoning and specifically designed to challenge models in complex reasoning tasks; (5) GSM8K [Cobbe et al. [021] is a dataset consisting of grade-school level math word problems that require logical reasoning. It tests model's 'ss ability to solve elementary-level math problems; and (6) MATH-500 [Lightman et al. 2023 is a dataset containing high school-level math problems. It serves assess model's ability to handle more advanced mathematical reasoning.
4.2 Evaluation Metrics
We use exact match accuracy as the primary metric for evaluating the performance of our method. Specifically, for multiple-choice cresstio--answrrigg tasks, such as MMLU, MMLU-Pro, GPQA, and ARC-C, accuracy is calculated based on the exact match between the predicted choice and the ground-truth choice (typically denoted by letter representing the correct answer). For math problem-solving tasks, such as GSM8K and MATH-500, accuracy is calculated by comparing the predicted final answer, enclosed by \boxed(), with the ground-truth answer.
4.3 Baseline Methods
We compare the proposed method with the following baselines: (1) Zero-shot CoT: This baseline uses Llama 3.1 [Dubey et al.] 2024] with zero-shot Chain-of-Thour (CoT) prompting [Wei et al. 2022], generating reasoning paths in a single pass; (2) SC @ maj16: This method applies the self- consistency (SC) technique Waang et al.
2022]. where multiple reasoning paths are generated and the most frequent result is selected. We use the SC @ maj 16 variant, which runs 16 rollouts to increase the accuracy of reasoning; (3) RAP: The method (Hao et al.] [003]] integrates a world model and reasoning agent, balancing exploration and exploitation to efficiently find high-reward reasoning paths via MCTS. The action space is formed by automatically generated sub-question;; and (4) rStar: The method [Qi al. 2024 utilizes 5 manually defined actions as the action space. Reasoning traces are searched with MCTS rollouts, and the final trace is determined by small LLM as a discriminator.
All of these baselines use Llama-3. 3.ama-3.1-8B-Instt --Sttuttttt (Dubey et al., 2024 as the backbone, consistent with our method. Additionally, both RAP and rStar use 16 rollouts, as our method.
4.4 Implementation Details
For the proxy action space estimation ($3.1) which covers wide range of topics, including math, scientific reasoning, and more. The corpus contains a total of 24, 652 problems, which are used to form the problem set. We divide the corpus into l- = 2, 500 groups. To extract observations, we query L 2024], resulting in 40, 822 observations total. In our submodular function definition ($3.2). we set the balancing parameters and diversity. For embedding efficiency, we use Llama-3.2- 1-..--1B-Innstuce [Dubey et al. 2024 as the backbone and select the last token's 's embedding the output of the embedding function e(-). The embedding function is fine-tuned using the Q-learning objective, with a total of 83, 083 state-action pairs, and the learning rate is set to le 5. For the action space construction ($.3), we set the size of the candidate action set A, at each time step m = 5. We use Llama-3. 10ma--.1-8B--mm 1-8B-lnstruut [Duhey et al. 2024) the world model (Hao al. 2023 J, which generates reasoning steps during the MCTS process.
we use the OpennPlatypuu [Le et al.) 2023 corpus,
[Dubey et al.)
0.9 and 8 0.1 to ensure proper balance between utility
4.5
Main Results
The results of our experiments, presented in Table D. reveal several key observations: (1) Our method outperforms all bascline methods across the six evaluated benchmarks, achieving significant

---

Table Evaluation results on different benchmarks. Numbers in bold denote the best performance.
General
Reasoning
Math
Model
MMLU
MMLU-Pro
GPQA
ARC-C
GSM8K
MATH-500
Zero-shot CoT
68.87
43.45
31.82
81.06
76.12
45.40
SC@ majl6
69.66
49.36
34.34
80.63
86.66
52.00
RAP
69.46
48.70
38.89
85.41
87.79
51.60
Star
68.61
48.81
36.87
86.43
87.11
54.20
DYNAACT
70.22
51.40
39.39
88.31
89.16
61.00
improvements in general, reasoning, and math tasks. This confirms the effectiveness our method in wide range of problem-solving tasks; (2) In the math domain, observe the most notable improvements. Our method achieves 1.37% and 6.80% improvements over the baselines on GSM8K and MATH-500, respectively. The MATH-500 dataset, which requires higher level of reasoning capability, particularly benefits from the use of the submodular function. The ability to construct more compact and atility-optimin improved performance in complex mathematical reasoning tasks; and (3) While rStar performs better than RAP MATH-500, its performance in other benchmarks suffers due the limited scalability of its manually defined action space. This limits rStar's ability to effectively handle the full range of tasks, making it less scalable compared to method.
action space enables more efficient exploration, leading to
Diseussions
In addition the comprehensive evaluation across multiple benchmarks, aim dive deeper into DYNAACT to gain further insights into its underlying mechanisms. Specifically, we investigate the following research questions: (1) RQ1: How do different components affect performance? (2) RQ2:
Can DYNAACT learn to have compact action space, thereby facilitating efficiency in reasoning?
(3) RQ3: What the utility of the actions selected by DYNAACT? (4) RQ4: Does DYNAACT introduce additional latency during inference? Besides, we are also curious about (5) RQ5: How does reasoning performance vary across different levels of difficulty? and (6) RQ6: Can the submodular
function enhance diversity of actions? We leave the discussions to RQ5 and RQ6 Appendix
5.1 Ablation Study for RQ1
We exploit ARC-C and MATH-500 strike balance between difficulty and domain diver- sity, and examine four variants of DYNAACT, including: exclusion of the utility term, denoted as " util": exclusion of the diversity term, re- ferred to as " div"; submodular function without Q-learning defined by Eq. (5). denoted " q- learning", which uses -.ama-3.2-1 1----ietInsttrctt for embedding directly; and removal of the submod- ular function, represented " submodular", which generates action spaces using 1----....----I--ttt- directly.
Table 2: Ablation study.
Model
ARC-C
MATH-500
DYNAACT (full)
88.31
61.00
util
87.63
53.40
div
86.52
53.80
q-learning
87.80
55.80
submodular
85.15
52.00
The results shown in Tibicaiindicate that the full version of DYNAACT achieves the best performance across both benchmarks, underscoring the critical importance of each component in our method. Removing the utility term leads to slight decrease in performance, highlighting its contribution to overall reasoning effectiveness. Excluding the diversity term results in further performance degradation, emphasizing the need for diversity in the candidate action set. Omission of the Q- learning objective causes a noticeable drop performance, demonstrating the necessity of this learning procedure for constructing an effective submodular function. Finally, the model without the submodular function performs the worst, reinforeing the essential role of the submodular strategy in achieving effective action selection.
7

---

5.2 Compactness Study for RQ2
examine whether DYN can produce compact action spaces. Ideally, a compact action space would enhance search efficiency, leading to better reasoning performance with smaller size. compares DYNAACT with RAP where the actions are constrained be sub-questions. From the results, observe that: (1) The action spaces of RAP highly redundant. Increasing the number of rollouts yields limited performance improvements when m = or m = 10. Only when m increased to 15 do we observe noticeable performance gains with respect to the number of rollouts, though the marginal gains are still much slower compared to DYNAACT. (2) On the other hand, even with set to 5, DyNAACT still demonstrates significant performance improvements more rollouts are carried out. The advantages over RAP remain consistent across all values m, highlighting the efficacy of DYNAACT in generating compact action spaces.
RAP 2013
60
Accuracy
55
50
45
40
2
16
32
Rollout
Figure 2: A comparison of RAP and DYN with respect different action space sizes (i.e., m).
The x-axis indicates the number of rollouts, while the y-axis shows the accuracy on MATH-500.
5.3 Utility Study for RQ3
We seleet rStar the baseline since its manually designed actions are pre- sumed be overly broad, limiting their utility for reasoning. Typically, there is standard method for mea- suring "utility". Therefore, follow- ing Zhao al. [2024) we identify critical steps in problem-solving (e.g.. key insights, decisions, calculations essential for solving a problem) and examine whether the reasoning steps triggered by actions contain these crit- ical steps. The ratio of solutions containing critical steps (referred as the Critical Step Coverage) is then used as proxy metric for utility. Table 3 presents the evaluation results. We can see that DYNAACT enables more effective identification and triggering essential reasoning steps, which also explains its superiority over rStar in terms of accuracy.
Table 3: Evaluation results the utility of action spaces. "Accuracy" measured on subset of MATH-500 with Level difficulty (i.e., the most difficult subset). This subset is chosen because the "utility" of actions (i.e., if critical steps are triggered) plays more critical role in solving complex problems.
Model
Critical Step Coverage (t)
Accuracy (t)
DYNAACT
0.63
31.34
rStar
0.47
26.87
5.4 Latency Study for RQ4
To assess whether DYNAACT introduces additional latency during inference, we compare it with rStar and RAP based on relative time required to complete tasks on the MATH-500 dataset. Specifically, rel- ative time calculated as the time taken by rStar or RAP, relative to the time taken by
Table 4: Comparison of relative time and accuracy for different methods.
Method
Rel. Time (L)
Accuracy (t)
DYNAACT
1.00
61.00
rStar
0.95
54.20
RAP
1.12
51.60

---

DYNAACT. We set the action space size (denoted as m) to and the number of rollouts to 16 for all methods. Table presents the results, including accuracy on MATH-500 to examine the latency-accuracy trade-off. 'he results show that while ACT incurs slight increase in latency compared rStar, it achieves substantial improvements in accuracy. When compared to RAP, our method demonstrates lower latency, which can be attributed to the fact that the most computationally intensive operation in constructing the action space Ar is the encoding of es,)), with e(a) precomputed and cached for subsequent use. This contrasts with RAP, where sub-questions are generated in real-time, leading to higher computational overhead. Additionally, the submodular function can be computed efficiently using an approximate algorithm with linear complexity, further contributing to the reduced latency of our approach.
6 Related Work
6.1 LLM Reasoning
The exploration of LLMs' reasoning capabilities has emerged alongside studies of prompting strate-
gies, among which Wei et al. [022 demonstrated that simple prompt can elicit chain-of-thought reasoning in LLMs; Zhou et al. leveraged prompts to guide LLMs in breaking down complex problems into simpler ones; and Shinn et al. [2024] enabled LLMs to recognize their mistakes and
perform self-correction. Soon after, the research on reasoning shifted focus from prompting to data
curation (uue al.) 2024 Wen et al., [024] and learning methods [Zelikman et al. [022] [2024].
whereby the community witnessed rapid progress in tackling complex problems such as mathematics
et al., 2023 Gou et al., [023 Mitra et al. [2024] Toshniwal al. 2024), coding (Luo et al. 2024
visual comprehension [Cheng et al. 2024 Hu et al.
[2024]. and decision-making (Chen et al. 2023
Recently, the success of OpenAI ol [Jaech et 2024 and DeepSeek rl [Guo et al., 2025 has
catalyzed the rise of test-time scaling (Snell et al. 2024]. where increased computation at inference enables LLMs to engage in long-term reasoning and achieve significant improvements on Olympiad- level math AAMEE 2024), challenging coding benchmarks (Jain et al. 2024), and graduate-level QA
tasks [Rein et al. 2023 J. Our study contributes to test-time scaling research but takes an orthogonal approach to most existing efforts. Rather than focusing on data curation [Muennighoff et al. 2025
Guan et al., 2025 or reinforcement learning (uuo et al. 2025 Qi et al., 2024], we aim to construct a compact yet effective action space to facilitate MDP-based reasoning.
6.2 Submodular Optimization
Submodular optimization Fujishige 2005] has been successfully used in many applications, includ- Chen et al. 2018 2024] and computer vision [Pervez] 2017 applied submodular functions to model interpretabil- ity, framing it as a combinatorial maximization problem for efficient model explanations. Similarly, Chen al. [2018) used submodular functions for instance-wise feature selection to explain deep learning decisions. Pervez et al.] (2023) introduced conditional Poisson sampling to select key features for image and text recognition, focusing on improving recognition accuracy. More recently, Chen et al.) (2024) applied submodular subset selection to deep model attribution, enhancing interpretability by identifying critical regions and reducing misattribution. Our work applies submodular optimization to action space selection in sequential reasoning tasks. Unlike prior methods focused on feature selection and model attribution, our approach optimizes action selection for enhanced compactness and scalability, offering novel method that improves performance in complex problem-solving.
ing model interpretability Elenberg et al. 2017
al. 2023). For instance, Elenberg al.
Conclusion
We propose DYNAACT, novel approach for automatically constructing compact action spaces to enhance sequential reasoning in complex pooving-solving tasks. DYNAACT incorporates a submodular function that optimizes action selection based on both utility and diversity, leading to improved inference efficiency and performance. Extensive experiments across six standard benchmarks demonstrate that DYNAACT not only outperforms existing methods but also maintains efficient inference without introducing substantial latency. These results underscore the effectiveness and versatility of our approach in tackling a wide range of problem solving challenges.

---

Acknowledgements
We extend our gratitude to the HKU NLP group and the anonymous reviewers for their invaluable suggestions, which significantly enhanced this work. This work was supported in part by the joint research scheme of the National Natural Science Foundation of China (NSFC) and the Research Grants Council (RGC) under grant number N_ HKU714/21, and by the Ant Group Research Intern Program.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arX1v:2303.08774, 2023.
AIME-2024.
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05911 2023.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information- theoretic perspective on model interpretation. In International conference on machine learning, pages 883-892 PMLR, 2018.
Ruoyu Chen, Hua Zhang, Siyuan Liang, Jingzhi Li, and Xiaochun Cao. Less is more: Fewer interpretable region via submodular subset selection. arXiv preprint arXiv:2402.09161 2024.
Chuanqi Cheng, Jian Guan, Wei Wu, and Rui Yan. From the least to the most: Building a plug- and-play visual reasoner via data synthesis. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4941 1957,,497,44 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 803, 05457, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14 10.14 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 24x.:2207..2183, 2024.
Ethan Elenberg, Alexandros Dimakis, Moran Feldman, and Amin Karbasi. Streaming weak
submodularity: Interpreting neural networks on the fly. Advances in Neural Information Processing Systems, 30, 2017.
Satoru Fujishige. Submodular functions and optimization, Elsevier, 2005.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlic Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint NAKV::2309.14452 2023.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small lims master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04511 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992, 2023.
10

---

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Yushi Hu, Otilia Streteu, Chun-Ta Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9590-9601, 2024.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16770, 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
Suraj Kothawade, Saikat Ghosh, Sumit Shekhar, Yu Xiang, and Rishabh Iyer. Talisman: targeted ac- tive learning for object detection with rare classes and slices using submodular mutual information. In European Conference on Computer Vision, pages 1-16. Springer, 2022.
Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of 1lms. arXiv preprint arXiv:2308.077 07317, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20055 2023.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the
potential of slms in grade school math. arXiv preprint arXiv:2402.14830, 2024.
Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501. 19393, 2025.
George I. Nemhauser, Laurence A Wolsey, and Marshall L. Fisher. An analysis of approximations for maximizing submodular set functions-is Mathematical programming. 14:265 1978.
Adeel Pervez, Phillip Lippe. and Efstratios Gavves. Scalable subset sampling with neural conditional poisson networks. In The Eleventh International Conference on Learning Representations, 2023.
Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller Ilms stronger problem-solvers. arXiv preprint #rXiv:2408.06115 2024.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv: 108, 2019.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google prooff q&a benchmark. arXiv preprint arXiv: 1002,, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifrre, George Van Den Driessche, Julian Schritwiesser, Ioannis Antonoglou, Veda Panncershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):844-88, 2016.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar, Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.032. 03314, 2024.
11

---

Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstmuut-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01566) 2024.
Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves Ilm search for code generation. arXiv preprint arXiv:2409.0377 2024a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery. and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203111771 2022.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01577, 2024b.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. ffaiin-ff-thouglee prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
Jiaxin Wen, Jian Guan, Hongning Wang, Wei Wu, and Minlie Huang. Unlocking reasoning potential in large langauge models by scaling code-form planning. arXiv preprint arXiv:2409.12452 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint wtXXv:22309.12285 2023.
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03548 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15:15766115488, 2022.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024.
Xueliang Zhao, Xinting Huang, Tingchen Fu, Qintong Li, Shansan Gong, Lemao Liu, Wei Bi, and Lingpeng Kong. Bba: Bi-modal behavioral alignment for reasoning with large vision-language models. arXiv preprint arXiv:2402. 135777 2024.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, et al. Least-to-mon prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Details are discussed in the Limitations section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
Details are included in Appendix E.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details discussed in Section 4.4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
We have submitted the source code to facilitate the reproduction of our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
stw.ooddeeeuumiiiiiooooooiiy..e for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https / //nips. chips.cccpubb...... wwtplliiiggii/eeeeeeeeeee CodeSummi ssionoolicyl for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Details are discussed Section 4.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification: We do not report error bars due to the substantial computational cost associated with repeated runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are discussed Section 5.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttpp///eurrrssssc//uua///ttttt
Answer: [Yes]
Justification: Yes, the research adheres fully to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Details are discussed in the Broader Impacts section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve the release of data or models with high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Details are discussed in Section 4.4.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com/ com/datasets datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method does not rely on LLMs for any essential components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer our LLM policy httesp////eerrroooccccoororrrrrs for what should or should not be described.
19

---

A Limitations
While DYNAACT represents significant advancement in action space construction for reasoning tasks, there are several limitations that should be addressed in future work to fully realize its potential:
(I) DYNAACT demonstrates strong performance across general, reasoning. and math tasks. However, its reliance on MCTS for selecting specific actions from the constructed action space can be computa- tionally intensive, especially for large complex tasks. Alterattve test-time scaling methods, such as beam search or other exploration strategies, could be explored to further optimize the balance between exploration and exploitation, improving efficiency without sacrificing reasoning accuracy. Expanding the model's ability to scale across different search algorithms remains promising direction for future work.
(2) While DYNAACT improves reasoning, it still depends on pre-trained Llama models, which may face challenges when handling extremely large or highly specialized action spaces. Exploring methods to optimize the model's scalability, such as combining DYNAACT with stronger backbone model, could help tackle this limitation in future research.
B Broader Impacts
This work proposes a framework for dynamic action space construction to improve sequential reasoning with language models. By enabling more structured, efficient, and interpretable decision- making, our approach has the potential to benefit applications such as educational tutoring systems, automated theorem proving, and scientific problem-solving domains where reasoning efficiency and transparency are critical. The ability to distill compact, high-utility action spaces may also reduce reliance brute-force search and large-scale inference, contributing more sustainable and accessible AT systems.
At the same time, these benefits raise important concerns. Automatically constructed action spaces are learned from data and may reflect biases present in the underlying corpora, potentially reinforcing harmful patterns during reasoning. Furthermore, perform multi-step reasoning, it may be misused to generate misleading arguments or automate complex forms of manipulation. While the explicit structure introduced by framework offers interpretability benefits, future work should explore mechanisms for auditing, constraint enforcement, and safe deployment in sensitive domains.
our method enhances the ability of LLMs to
C Details of the MCTS Process
In this appendix, we provide a detailed formal description of the Monte Carlo Tree Search (MCTS) procedure used to estimate the value function Q(s, a) for candidate actions. The MCTS process comprises four stages: selection, expansion, simulation, and backpropagation. For simplicity, let the current state be denoted by and let the set of candidate actions from state be represented A(s).
Selection
Starting from the root node corresponding to the current state 8, the selection phase traverses the tree by recursively choosing actions based on a balance between exploitation and exploration. Each node in the search tree is associated with two statistics:
(s) (the number of visits to state s)
and, for each action E A(s),
N(s, a) is the number of times that action a has been selected from state 8, ((s.a) is the estimated average reward for taking action from state s.
The action a" is selected state according the Upper Confidence Bound for Trees (UCT) formula:
[n N(s,
a' a= arg max
aEA((s) a(s, +
20

---

where c is exploration constant. This selection continues recursively until node is reached that is either a terminal state fully expanded (i.e., there exists A(s) such that the corresponding child node has not yet been created).
Expansion
Upon reaching a node s that not terminal and has unexplored actions, the expansion phase selects one such action a' E A(s)) for which child node exists. A new child node s' is then created to represent the state resulting from taking action a' from 8. The statistics for the new edge (s, a') are initialized as:
N(s, 0,
Q(s,a') O.
Simulation
From the newly expanded node s', simulation (or rollout) is conducted to estimate the value the state-action pair (s, d'). The simulation proceeds by following a default policy often random or heuristic strategy- generate complete reasoning trajectory until a terminal state is reached. Let denote the reward obtained at the terminal state; this reward serves as an estimate for the value of taking action a' from state s.
Backpropagation
Once the simulation concludes with reward the backpropagation phase updates the statistics along the path from the expanded node back the root. For every state s and action a along this path, the updates are performed as follows:
+
a( + r r- Q(s,a)
N(s,a)
Additionally, the visit count for each state along the path is updated:
N(s) +- N(s)
These updates refine the estimates Q(s, a) based on the observed reward r, thereby improving the accuracy of the value function with successive simulations.
The MCTS process iterates through these steps-selection, expansion, simulation, and backpropaga- tion-until specified computational budget reached.
D Prompt for Universal Problem-Solving Sketch Extraction
Below is the prompt used for extracting universal problem-solving sketch from set of problems. This prompt designed to guide the extraction process so that the generated subgoals are broadly applicable, capture core actions, and can be flexibly applied across various problem domains.
21

---

Problen-Solving Sketch Prompt
Imagine you problem-solving expert tasked with creating universal problem-solving sketch.
You will be shown the following (number of problems) problems: (problem_text]
From these problems, extract up {n) essential subgoals that form universal sketch. The
subgoals should:
Apply broadly across different types of problems and disciplines
Use casual, everyday language from first person perspective
Avoid sequential markers like "first", "next", "then", "finally"
Focus on the core action insight needed each stage
Be as creative as possible, going beyond what you think is intuitively correct
Format your response as a numbered list, where each item expresses one subgoal. Make each subgoal self-contained so it can be applied flexibly rather than in fixed sequence.
This prompt ensures that the extracted subgoals capture the essential observations needed for con- structing the proxy action space, and that they are effective for wide range of problem-solving scenarios.
E Proof of Submodularity of F(X; St)
Proof. Let V = A. A set function f 2 7 R is submodular if for every X S Y S V and for every x E VVYY
f(XU(z)) f(x) 2 f( f())
We write
F(X;s;) a ai +
where
FX(X log Σ
exp (e(s)) e(a))
NEE
and
F2(X) = Σ mx) with mx(a) = min (1 e(a) Te(b))(b))
beXifa}
AEX
(i) Submodularity of Fi: Define
g(X) = Σ exp (e(s))) e(a))-
oEX
Note that for any x E V X.
g(X {x}) = g(X) (ep(s e(x)).
Thus, for X Y and x E V Y, we have:
1+ exp (e(s:)" e(x))
[ - Fi(X = log
g(X)
and
F - FY()) = log 1+ exp (o(s))) c(x))
g(Y)
Since X Y implies g(X) < gY),, it follows that
exp (e(s))) e(x))
exp (e(s)) e(x))
g(X)
g(Y)
22

---

and because log(1 + 2) is an increasing function for 2 2 0, we obtain
F (2}) Fi(X) 2 F U (z)) - P1().
Thus, F1 is submodular. (ii) Submodularity of P2: For any X C V and E V X, the marginal gain for P3 is given by (r X) - U V {z)) F2(X).
For each a E X, the value mx(a) = ((aa) (1 - (1=) eea)T((()) updates upon addition of æ to
mxu(=] (a) min m
e(()) }.
Similarly, for the newly added element x.
mx)(2)(0) = min - (1=)()))e(a)) aEX
Hence,
(1. XX) = Σl
min(mx(a), (ll-e(aae(x))) - mx(a)] + min (1 e(x)
aex
aEx
Let us denote, for each E X,
(x( = min(mx((a), (1 e(a) mx(a).
Then,
(z X) Σ óx(a, x) + min (1 e(a))
aex
aex
Now, consider X CCYS C V and x E V Y. For any a E X, since X C Y we have
(a) = min mx((),
min (1 (1- e(a) (1-e(a) beY e(aaa s mx(a).
Thus,
dy (a, = min(my (4), ((-e(a)"e(x))) my (a) 2 min(mx(a), - (==e(a)((=))))mm(( = 6x(
Also,
min (1 -
S aEX min (1 - o(=))
AEY
Therefore,
DF, (z |X)= = Σ x)) +
min ae
- (1-)(x)ee(a)))
xE
aeY (F 1Y) = Σ By + min (1 (1- e(x) e())))(a))
aEY
Since Y contains all elements of X and possibly additional elements with non-positive marginal increments (i.c., dy x)) < 0 for E Y ()), follows that
Σ eEX
xx) 2 Σ (a,
aeX
and
min (1 (1- e(x) e(a)) 2 min (1 - ((()))((a))
x
aEY
Moreover, the additional terms from Y X in A (x Y) further decrease the total marginal gain. Thus, we obtain
A Amp (I
X) > A (w Y).
Hence, F2 is submodular.
(iii) Combination: Since F(X: = a (X( + 8 Fx() with a 2 0, it follows that for all
X CCY C V and for all x E \
F(X
F(X: st) = a AF (x:
X) + 1 (2 X), Y) + 3 AZ Y) F(Y U (z): - F(Y;s))
2 a AF
Thus, F(X;s() is submodular.
23

---

Table 5: Evaluation results on the Level 3, Level 4, and Level 5 subsets of MATH-500. Numbers in bold denote the best performance.
Level 3
Level 4
Level 5
rStar
72.38
50.78
15.67
DYNAACT
76.19
58.59
31.34
util
68.57
52.34
17.16
q-leaming 71.43
53.13
20.90
F Additional Experimental Results and Analysis
F.1
Reasoning Performance Across Difficulty for RQ5
We evaluate the effectiveness of both the utility term and the Q-learning objective using the MATH- 500 dataset, which includes problems categorized by difficulty levels. This allows assess how each component impacts problem-solving performance across tasks of varying complexity. As shown in Ta the removal of the utility term leads to a more significant performance drop on harder problems (Level 5) compared to easier ones (Level 3). This indicates that the utility term is crucial for selecting actions that meaningfully contribute to the solution process, particularly for complex problems. In comparison, removing the Q-learning objective results in a slight performance drop. Without Q-learning, the embedding function e(.) still selects actions relevant the current state, but it fails to capture the long-ter terg-term utility of those actions. As result, while the model continues to choose relevant actions, the lack of Q-learning limits its ability to optimize the effectiveness of its actions over time.
We further compare our method with rStar, which demonstrates better performance than other baselines on MATH-500. However, due to rStar's 's manually defined action space, it faces challenges in scaling more complex problems. As shown in the results for Level problems in Table [ rStar's 's performance drops significantly to 15.67%, whereas our method achieves higher accuracy of 31 31.34%. This highlights the scalability advantage of our method.
E2 Action Diversity Analysis for RQ6
Table Evaluation results showing the impact of the diversity term.
Model
Diversity
Accuracy
Ours
0.73
31.34
- div
0.49
24.63
To evaluate the impact of the diversity term, we follow Wang et al. 22244a and calculate the diversity score of candidate action by measuring how dissimilar the actions are within the set. Specifically, the diversity score is determined by computing the ratio of dissimilar pairs of actions to the total number of possible action pairs in the set. This ratio is then averaged over all candidate action sets. We test on the Level 5 subset of the MATH-500 dataset, which consists of more complex problems that are sensitive redundancy in the selected actions. The results are shown in Table We observe that removing the diversity term results in significant drop in the diversity score, which subsequently leads to a decrease accuracy.
F.3 Additional Analysis Diversity Term in Submodular Function
To assess the sensitivity of DYNAACT to the choice the diversity term fai in Eq. we conducted additional experiments using two alternative diversity metries: Mean Pairwise Distance and Mean Cosine Distance. These alternatives were chosen to evaluate whether simpler diversity formulations could yield comparable or improved performance.
As shown in T both alternative diversity metrics lead to a performance drop compared to our original formulation. We attribute this to the fact that these metrics do not preserve the submodular property, which central to the efficiency and theoretical guarantees of our greedy subset selection algorithm.
24

---

Table 7: Performance comparison using different diversity terms.
Method
Accuracy (MATH-500)
Mean Pairwise Distance
57.80
Mean Cosine Distance
58.20
DYNAACT
61.00
E.4 Resource Consumption Analysis
While 5..4 presents relative latency comparison, we include here the raw inference time per example to offer a more complete view of resource consumption. All measurements were taken on an 8 100 GPU machine using the MATH-500 dataset, with each method evaluated under consistent rollout settings.
Table 8: Per-example raw inference time and accuracy on MATH-500 across methods.
Method
Raw Time (4)
Accuracy (t)
Zero-shot CoT
1.68s
45.40
SC@maj16
26.88s
52.00
RAP
64.51s
51.60
SSarr
54.72s
54.20
DYNAACT
57.60s
61.00
As shown in Table DYNAACT requires comparable runtime other MCTS-based baselines such as RAP and rStar, while yielding significantly higher accuracy. Although MCTS-based approaches naturally incur more latency than single-pass generation methods like Zero-shot CoT, they also enable more effective reasoning. Since our method orthogonal to the choice of search algorithm, future work could explore integration with more efficient test-time strategies (e.g., beam search sample-efficient MCTS variants) to further reduce resource consumption while preserving accuracy.
F.5 Comparison with Non-Submodular Selection Methods
To further understand the benefits of our submodular formulation, compare DYNAACT against RL-based pruning baseline. this baseline, action space truncation is performed by selecting the top candidate actions based solely on Q-value estimates, without considering submodular diversity joint utility.
Table 9: Comparison of DYNAACT with RL--based pruning baseline on the MATH-500 dataset.
Method
Raw Time (4)
Accuracy (t)
RL-based Pruning
56.89s
53.20
DYNAACT
57.60s
61.00
As shown in Table while the RL-based pruning method achieves slightly lower inference latency, significantly underperforms in accuracy compared DYNAACT. This gap highlights the limitations of greedy RL-based pruning, which may select redundant or suboptimal actions. In contrast, our submodular approach explicitly optimizes for both utility and diversity, yielding a compact yet expressive action set.
F.6 Scalability with Large Proxy Action Spaces
To assess the scalability of DYNAACT to large-scale corpora, varied the size of the proxy action space from 40, 000 to 1, 000, 000 entries, while keeping the candidate set size m fixed. All experiments were conducted the MATH-500 dataset.
As shown in Table TO the inference latency increases moderately with larger proxy spaces-rising by approximately 18 seconds when scaling from 40k to IM actions. Importantly, performance remains
25

---

Table 10: Scalability analysis of DYNAACT with varying proxy action space sizes on MATH-500.
Proxy Action Space Size
Raw Time (4)
Accuracy (1)
40k
57.60s
61.0
200k
60.52s
61.8
400k
63.99s
62.0
600k
67.93s
62.0
800k
71.47s
61,6
1M
75.84s
61.8
stable slightly improves, indicating that DYNAACT effectively handles large candidate pools without significant degradation in efficiency. This scalability is largely attributed to the caching of action embeddings and the linear-time greedy algorithm used in submodular optimization.
F.7 Empirical Study on the Uiility-Diversity Trade-off
To better understand how the balancing parameters and 8 in Eq. aafectt the trade-off between utility and diversity in the submodular function, we conducted series experiments on the MATH-500 dataset. We varied the relative weighting of the utility term fui and the diversity term fai, while keeping their sum fixed (a + 1).
B
Accuracy (MATH-500)
(0.9, 0.1)
61.00
(0.7, 0.3)
60.80
(0.5, 0.5)
55.40
(0.3, 0.7)
54.60
(0.1, 0.9)
54.80
Table 11: Effect of varying a and / on model performance.
As shown Table 11 performance is highly sensitive to the relative weighting the utility term. Accuracy drops substantially when utility and diversity given equal weight when diversity dominates. However, once the utility coefficient exceeds 0.7, the model achieves strong and stable performance. These findings suggest that while diversity contributes more robust action space, prioritizing utility is essential for effective reasoning in complex problem-solving tasks.
F88
Comparison with Few-Shot and Fine-Tuned Baselines
To assess whether the performance of DYNAACT could be attributed exemplar- based prompting or supervised adaptation, we compare it with several baselines involving few-shot prompting and fine-tuning. The few-shot baseline retrieves the top most similar examples from Open-Platypus using cosine similarity over embeddings from Llama-3.2-19 1-B and provides them as context Llama-3.1 1-8---11---sntrrcc The fine-tuned baseline directly trains corpus for epochs with learning rate of 1 x 10 For both approaches, we also evaluate their self-consistency variants (SC maj16) where 16 reasoning paths are sampled and majority voting is used for answer selection.
on the same
Table 12: Comparison of DYNAACT with few-shot and fine-tuned baselines across six benchmarks.
Method
MMLU
MMLU-Pro
GPQA
ARC-C
GSMSK
MATH-500
Zero-shot CoT
68.87
43.45
31.82
81.06
76.12
45.40
Few-shot baseline
68.94
43.23
29.80
81.83
76.65
46.60
Fine-tuned baseline
68.72
43.56
34.34
82.08
78.17
48.20
Few-shot SC@ @ maj 16
69.80
44.22
35.86
84.73
84.00
50.20
Fine-tuned mC@maj16
69.89
45.89
34.85
86.18
84.38
52,60
DYNAACT
70.22
51.40
39.39
88.31
89.16
61.00
26

---

As shown in Table 12 both few-shot prompting and fine-tuning lead to modest improvements over the zero-shot baseline. However, DYNAACT consistently outperforms all baselines across tasks, demonstrating that its gains stem from principled, dynamic action space construction rather than static exemplar use or model adaptation alone.
F.9
Case Study
To further demonstrate the effectiveness of DYNAACT, we present series of case studies in Figures each consisting "problem" and its corresponding predicted "solution". For each step the solution, the process begins with the selection of an action (chosen using ((st. a), as described in 22.1 followed by the corresponding reasoning step. These examples demonstrate how DYNAACT enhances the effectiveness of the reasoning process.
3
Problem:
Compute:
100
Solution:
Step 1: clarify the problem by identifying the key elements, constraints, and what I'm trying to find or achieve, and I rephrase it in my own words to ensure understand it correctly. We need to compute the sum of an alternating series where each term increases by 1, starting from 1 and ending at 100.
Step 2: should look for connections, patterns, relationships between the components, and consider how they interact influence each other, whether it's in mathematical, logical, contextual sense. The series alternates between addition and subtraction, which suggests a pattern where each pair of consecutive terms (one positive and one negative) could be combined to simplify the calculation.
Step I'm searching for patterns, connections, and relationships between the different parts, and thinking about how they might help me solve the problem.
We can pair the terms as follows: (1 - 2) + (3 4) + (5 - 6) ... (99- - 100). Each pair sums to -1
Step evaluate my progress, checking if my approach working and adjusting as needed ensure I'm on the right track.
There are 50 pairs in the series, each summing to -1.
Step break down complex problems into smaller, manageable parts, focusing on one aspect a time, and identifying the essential variables, assumptions, and constraints that will help me make progress, which might involve creating diagram, setting up equations, or developing a step-by-step plan. The sum of the series is the sum of the 50 pairs, which is 50 (-1).
Step 6: Complete the solution and present the final answer within
50
Figure 3: Case study: solution alternating series sum.
27

---

Problem:
The Greek army contained two types of soldiers: the upper class and the lower class soldiers. If there were a total 5 upper class soldiers, and 10 lower class soldiers in certain part of Athens, and the battle of Thermopylan lae demands a force of 4 upper class soldiers and 8 lower class soldiers, how many different battalions can be sent?
Solution:
Step 1: clarify the problem by identifying the key elements, unknowns, and goals, and try to rephrase it
my own words to ensure understand what's being asked. We need to find out how many different groups (battalions) can be formed from a mix of 5 upper class soldiers and 10 lower class soldiers, where each group must consist of 4 upper class soldiers and 8 lower class soldiers.
Step 2: look for opportunities to use analogies, metaphors, or abstractions to reframe the problem or make it more tractable, and think about how I can leverage my existing knowledge or experience to inform my solution.
This problem can approached by considering it as combinatorics problem, specifically involving combinations since the order of selection does not matter.
Step 3: consider the implications and potential consequences of my solution, and I think about how can test, validate, or refine further to ensure it's ss correct, complete, and relevant to the original problem. To form battalion, we need 4 upper class soldiers out of 5 and lower class soldiers out of 10. The number of ways to choose 4 upper class soldiers from 5 is given by the combination formula C(n, k)) TT(-- where n is the total number of items, and k is the number of items to choose. For the upper class, this becomes C(5, 4). Similarly, for the lower class, it's C(10, 8).
Step 4: consider how the problem might be transformed or re-expressed in a way that makes it more tractable can I reframe it in simpler terms, or find an analogous problem that's 's casier to solve, or identify a key insight that unlocks more straightforward path to the solution? The formula for combinations C(n,, k)
10t -T For the upper class, this is C(5, 4)
31(5 3J[5-4] 5.
For the lower class, this C(10,8) 81[10-8] 45.
Step 5: Are there any constraints, assumptions, or rules that I need to take into account when exploring solutions possibilities?
Since the selection of upper class soldiers and lower class soldiers is independent of each other, the total number of different battalions that can sent is the product of the number of ways to choose the upper class soldiers and the number of ways to choose the lower class soldiers.
Step 6: Once I think I have solution, I'll consider its implications and potential limitations. I'll ask myself questions like 'What are the potential consequences of this solution?', 'Are there any potential biases or errors in my thinking?", and 'How might this solution impact different stakcholders or scenarios?" The total number of battalions is the product of the number of ways to choose the upper class and the lower class soldiers, which is 5 45- 225.
Step 7: Look for patterns, symmetries, and analogies that can help me understand the problem more deeply and make connections to other areas of knowledge experience are there any clever ways reframe. reorganize, or reinterpret the problem that could reveal new insights or opportunities? The solution is straightforward and based on the combination formula.
Step 8: Complete the solution and present the final answer within
225
Figure 4: Case study: solution to Greek army battalion formation problem.
28

---

Problem:
A hexagon inscribed in circle:
130"
120"
120%
120
110
What is measure a, in degrees?
Solution:
Step I: need to identify the underlying structure or pattern in the problem, whether it's a mathematical relationship, a logical argument, conceptual framework. The problem involves an inscribed hexagon in circle, with specific angles labeled. The goal find the measure of angle a.
Step 2: I will decide suitable representation or model for the problem, and determine the best approach explore solution space, whether it's through calculation, simulation, or other means. To find the measure of a. 1 can use the fact that the sum of the interior angles a hexagon is 720". Additionally, since the hexagon is inscrihed in circle, the measure of each central angle formed by two adjacent vertices of the hexagon is equal to the measure of the corresponding inscribed angle.
Step 3: I evaluate my progress, checking for errors, inconsistencies, and areas for improvement, and adapt my approach needed to ensure that I' ' making progress toward a solution that is accurate, complete, and relevant.
Since the sum the interior angles of hexagon is 720%, and sum of the measures the given angles is 105 + 110" 215", the measure angle a can be found by subtracting the sum of the given angles from the total sum of the interior angles of a hexagon.
Step break down complex problems into smaller, manageable parts, focusing on one aspect a time, and identifying the essential variables, assumptions, and constraints that will help me make progress, which might involve creating diagram, setting up equations, or developing a step-by-step plan. Let's denote the measure of angle have the equation + 215 -200
Since the sum of the interior angles of hexagon is 720%,
Step 5: Are there any constraints, assumptions, or rules that I need to take into account when exploring solutions possibilities?
problem does not provide any additional constraints or assumptions beyond given information.
Step should look for potential paths forward by asking myself questions like 'What the implications this?', "How does this relate to that?", 'What would happen if I..". and "What are the boundaries or constraints this problem?" this helps me generate possihilities and explore different avenues for solving the problem.
Now, let's solve the equation + 215 720 for
Step 7: break down complex problems into smaller, manageable parts, focusing on one aspect at a time, and identifying the essential variables, assumptions, and constraints that will help me make progress, which might involve ereating diagram, setting up equations, or developing step-by-step plan.
To solve for x, we can subtract 215" from both sides of the equation: x 720" 215".
Step 8: Complete the solution and present the final answer within D
x 720" 215" 505*. However, since the measure of an angle cannot exceed 180", we need to subtract 360 from 505" to get the correct measure of angle a. Therefore, a = 505 360 = 145".
145
Figure Case study: solution to inscribed hexagon angle problem.
29


Reviews:
[{"review_id": "RnCl5m3UP5", "reviewer": "Reviewer_eEVj", "summary": "The authors develop a novel framework named DYNAACT for automatically constructing a compact action space to enhance sequential reasoning in complex problem-solving scenarios, which firstly estimates a proxy for the complete action space by extracting general sketches observed in a corpus covering diverse complex reasoning problems using large language models", "questions": "The core idea is to construct high-level actions (referred to as action subsets in the paper) to group similar low-level actions — an approach that is not novel in the LLM community.\n\nI have a few questions and suggestions regarding your method:\n\n1. How is the number of action subsets determined in your approach? Additionally, how does this number impact the overall performance of your method?\n2. In my experience, embedding-based clustering tends to be fragile and often lacks interpretability, particularly in understanding the semantics of cluster centers. Have you considered using LLM-based clustering, where the categories of low-level actions are derived via LLM reasoning?\n3. Llama3 series and some benchmarks, e.g. GSM8K, MATH500, I suggest evaluating your method on AIME24 and AIME25 using the Qwen3 model series to make the experimental results more convincing and up-to-date.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. the paper is well written and easy to follow\n2. the technique details are sound\n3. the novelty is not enough", "was_revised": true, "final_justification": "Provide new results of Qwen3 series on AIME24 and AIME25, and the feedback address most of my concerns.", "rebuttal": "We truly appreciate your considerate and insightful feedback. We are pleased that you consider our paper to be well-written and technically sound. We meticulously respond to all of your questions and suggestions in depth below. We will also make sure to incorporate your insightful suggestions into the final version of the paper.\n\n---\n\n### **Q1: Novelty of high-level actions**\n\nWe understand your concern that constructing high-level actions to group similar low-level actions has been previously explored in the literature, as noted in Section 1 of our manuscript. However, we would like to clarify that the key novelty of our work lies not in the general concept of high-level actions per se, but in the **principled construction** of a compact and scalable action space. We believe this direction — making high-level actions *scalable, compact, and generalizable* in a unified framework — addresses an under-explored and practically important challenge in the field.\n\n---\n\n### **Q2: How is the number of high-level actions determined?**\n\nI appreciate you bringing this to attention. To clarify, our framework does not involve partitioning actions into disjoint “subsets.” Rather, we dynamically construct a set of high-level actions to serve as the compact action space at each step of reasoning.\n\nIn our current implementation, we empirically fix the number of high-level actions $m$ to 5, which strikes a good balance between expressiveness and search efficiency. We also conducted ablation studies to investigate how varying $m$ impacts performance. The results are summarized below:\n\n|  | MMLU | MMLUPro | GPQA | ARC-C | GSM8K | MATH-500 |\n| --- | --- | --- | --- | --- | --- | --- |\n| m = 3 | 69.3 | 50.1 | 37.9 | 88.1 | 86.3 | 57.6 |\n| m = 5 | 70.2 | 51.4 | 39.4 | 88.3 | 89.2 | 61.0 |\n| m = 7 | 70.3 | 51.9 | 38.9 | 89.1 | 87.3 | 60.2 |\n| m = 9 | 70.8 | 50.6 | 38.4 | 88.2 | 88.9 | 59.4 |\n\nAs shown, $m=5$ achieves consistently strong performance across benchmarks, and increasing $m$ beyond this point provides diminishing or even negative returns. We will make this clearer in our paper revision.\n\n---\n\n### **Q3: Fragility of embedding-based clustering**\n\nYou raised an excellent point about the limitations of embedding-based clustering. We agree that naive clustering of embeddings can be fragile. Indeed, our own experiments (see Appendix F.5) show that using pure embedding-based methods without additional regularization leads to unstable results.\n\nTo address this, our method integrates **submodular selection as a principled approach** to ensure both coverage and diversity of action candidates. We also compared this with a variant where action categories are directly derived from LLM reasoning. Specifically, we use Llama-3.1-8B-Instruct to generate high-level actions at each step, conditioned on the trajectory so far, without applying submodular selection (**referred to as “– submodular” in Table 2**). The performance of this LLM-based variant was significantly lower due to reduced compactness and increased redundancy, which negatively impacts the MCTS exploration efficiency.\n\nTherefore, while we appreciate the benefit of pure LLM-driven clustering, our empirical findings suggest that submodular selection yields a better trade-off between semantic relevance and decision efficiency.\n\n---\n\n### **Q4: Evaluation on Qwen3 and AIME benchmarks**\n\nThank you for suggesting additional evaluation on the Qwen3 model series and AIME24/25 benchmarks. We fully agree that incorporating newer models and more challenging datasets can strengthen the empirical foundation of our work.\n\nWe would like to clarify that the Qwen3 series was released only about two weeks before the submission deadline, which limited our ability to include these models in our original experiments. Nonetheless, we have since conducted a comprehensive set of evaluations using **Qwen3** Series on **AIME 2024** and **AIME 2025**\n\nBelow are the results:\n\n### **Qwen3-1.7B**\n\n| Method | AIME 2024 | AIME 2025 |\n| --- | --- | --- |\n| Qwen3-1.7B | 13.4 | 9.8 |\n| SC@maj16 | 23.3 | 13.3 |\n| RAP | 23.3 | 16.7 |\n| rStar | 20.0 | 13.3 |\n| **DynaAct** | **30.0** | **23.3** |\n\n### **Qwen3-4B**\n\n| Method | AIME 2024 | AIME 2025 |\n| --- | --- | --- |\n| Qwen3-4B | 25.0 | 19.1 |\n| SC@maj16 | 33.3 | 20.0 |\n| RAP | 30.0 | 23.3 |\n| rStar | 30.0 | 20.0 |\n| **DynaAct** | **36.7** | **26.7** |\n\n### **Qwen3-8B**\n\n| Method | AIME 2024 | AIME 2025 |\n| --- | --- | --- |\n| Qwen3-8B | 29.1 | 20.9 |\n| SC@maj16 | 33.3 | 20.0 |\n| RAP | 30.0 | 23.3 |\n| rStar | 36.7 | 23.3 |\n| **DynaAct** | **46.7** | **30.0** |\n\n### **Qwen3-14B**\n\n| Method | AIME 2024 | AIME 2025 |\n| --- | --- | --- |\n| Qwen3-14B | 31.7 | 23.3 |\n| SC@maj16 | 36.7 | 23.3 |\n| RAP | 33.3 | 26.7 |\n| rStar | 36.7 | 20.0 |\n| **DynaAct** | **43.3** | **33.3** |\n\n### **Qwen3-32B**\n\n| Method | AIME 2024 | AIME 2025 |\n| --- | --- | --- |\n| Qwen3-32B | 31.0 | 20.2 |\n| SC@maj16 | 36.7 | 26.7 |\n| RAP | 33.3 | 23.3 |\n| rStar | 36.7 | 23.3 |\n| **DynaAct** | **46.7** | **30.0** |\n\nThese results clearly demonstrate that **DynaAct consistently improves the problem-solving capabilities of Qwen3 models**, often by large margins over the strong baselines. \n\nWe hope these additional results effectively address your concerns and further strengthen the empirical impact of our work.\n\n---\n\nWe appreciate your thorough review and valuable comments once more. Your feedback has improved our understanding of how to convey the value and merits of our work, and we sincerely value your suggestions."}, {"review_id": "nMKavjdT2N", "reviewer": "Reviewer_p9Ye", "summary": "In this paper the authors introduce DYNAACT, a framework to construct compact action spaces dynamically to enhance reasoning in sequential decision-making tasks. Their method employs submodular optimization to balance utility and diversity in action selection, ensuring scalability and efficiency. The authors performed extensive experiments across six benchmarks (MMLU, GPQA, ARC-C, GSM8K, MATH-500) to demonstrate the strength of their approach. DYNAACT achieved 6.8% higher accuracy on MATH-500 compared to other baselines.", "questions": "Can this be applied to tasks with highly dynamic and unpredictable action spaces? For example in robotics?\n\nSub modular optimization can be computationally expensive. Do you see any scalability problems related to this for large complex domains?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The authors provided a rigorous methodology for dynamic action space construction using submodular functions, and through extensive experiments validated their approach across six benchmarks.\n\nThe paper is well-written, with clear explanations of the submodular optimization approach and experimental results.\n\nThe focus on enhancing reasoning in sequential decision-making using large language models is highly relevant. The paper has demonstrated the importance of well defined action spaces in MDP based reasoning. Their experimental analysis shows promise of this approach.\n\nThe dynamic action space construction is a novel research question, and the use of submodular functions to balance utility and diversity in action selection seems to be novel.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns. Overall, I think it is a good paper.", "rebuttal": "Thank you very much for your thoughtful and constructive review. We truly appreciate your recognition of our work and your encouraging feedback on its clarity, novelty, and significance. We will also make sure to incorporate your insightful suggestions into the final version of the paper. Below, we address your specific questions in detail:\n\n---\n\n### **Q1: Can this be applied to tasks with highly dynamic and unpredictable action spaces? For example in robotics?**\n\nWe value your insightful question about the relevance of our method in fields with extremely dynamic and unpredictable action environments, like robotics. To investigate this, we carried out further experiments on **Alfworld** [1], a recognized benchmark for embodied agents. Alfworld features six categories of domestic chores that differ in complexity and unpredictability. Following the official evaluation protocol, we use success rate as the evaluation metric. We evaluated our approach using the unseen test set composed of 134 instances, averaging the results over 10 runs to minimize statistical variance.\n\nHere are the results:\n\n| Method | Pick | Look | Clean | Heat | Cool | Pick Two | Avg. |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Llama-3.1-8B-Instruct | 34.2 | 45.0 | 10.3 | 7.4 | 9.5 | 11.8 | 18.7 |\n| SC@maj16 | 47.1 | 52.2 | 24.5 | 19.6 | 21.9 | 32.4 | 31.8 |\n| RAP | 47.9 | 49.4 | 31.6 | 17.8 | 20.5 | 33.5 | 32.8 |\n| rStar | 37.5 | 42.2 | 28.7 | 23.5 | 17.1 | 35.9 | 30.1 |\n| **DynaAct (Ours)** | **59.6** | **57.8** | **42.3** | **34.3** | **29.5** | **41.2** | **43.6** |\n\nAs you can see, DynaAct substantially outperforms all baselines across all task types. This suggests that your concern is well-addressed in practice: DynaAct indeed generalizes effectively to settings with dynamic and complex action spaces, such as those found in robotics.\n\n---\n\n### **Q2: Submodular optimization can be computationally expensive. Do you see any scalability problems related to this for large complex domains?**\n\nThank you for bringing up this significant matter. We recognize the worries about the computational expense of submodular optimization. To specify, according to our profiling and quantitative analysis (refer to Table 8), **the primary source of computational burden in our framework originates from the MCTS** component instead of the submodular optimization phase. Indeed, the submodular optimization we use is executed through a greedy algorithm with linear time complexity, which proves to be highly effective in real-world applications. Additionally, as mentioned in Appendix F.6, the main factor for scalability is the magnitude of the proxy action space. In our approach, DynaAct performs effectively even with extensive proxy action spaces owing to two key strategies: (1) **Storing action embeddings,** which prevents unnecessary calculations across various evaluations; and (2) **Linear-time greedy submodular selection,** which guarantees that potential actions can be chosen efficiently without exhaustive searching. Therefore, we are confident DynaAct can manage large and intricate domains without major computational limitations because of the submodular optimization itself.\n\n---\n\nThank you again for your insightful feedback. We trust that these explanations alleviate your concerns completely and showcase the broad applicability and effectiveness of our method.\n\n---\n\n**Reference:**\n\n[1] Shridhar, M., Yuan, X., Côté, M. A., Bisk, Y., Trischler, A., & Hausknecht, M. (2020). Alfworld: Aligning text and embodied environments for interactive learning."}, {"review_id": "DySulJGstE", "reviewer": "Reviewer_ML8s", "summary": "This paper identifies the limitation of the current framework on MDP-style LLM reasoning on lacking a scalable and compact action space. In response, the authors propose to first mine action space from existing corpora and formulate a submodular function to evaluate the utility and diversity of the actions to form a dynamic action space. Extensive experiments on six tasks show good performance improvement over previous methods with good latency control.", "questions": "1.\tWill the authors open-source the observations extracted in Section 4.4 to facilitate future research?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. This paper is relatively well-written and easy to follow.\n\n2. The authors conduct experiments on a wide range of tasks, including both QA and math tasks.\n\n3. The analysis is well-designed, including compactness, utility, and real-time latency.\n\n\nWeakness\n\n1. Studies on other backbones can further consolidate the performance gain over baselines and show the generalization of the learned submodular function on other models, e.g., Qwen 2.5, and other-sized models, e.g., 1.5B, 70B models. It is unclear how robust the action evaluation function is with other models or if it transfers among models.\n\n2. It would be preferred if the authors could conduct qualitative analysis, e.g., examples of the actions, success and failure examples of the proposed methods, and change of action space at different steps of the optimization, etc.\n\n3. Further comparison with few-shot prompting methods is needed since the dynamic action space selection is similar to dynamic exemplar selection [1] in terms of the extra prior introduced to the frozen inference models. Currently, in F.8 (page 19), the authors experiment with few-shot baselines with embeddings from Llama 3.2 1B is not the common practice. What is the performance of newer models such as GritLM [1], Gecko [2], or Instructor [3]?\n\n[1] Learning To Retrieve Prompts for In-Context Learning: https://arxiv.org/pdf/2112.08633   \n\n[2] GritLM: https://arxiv.org/abs/2402.09906 \n\n[3] Gecko: https://arxiv.org/abs/2403.20327 \n\n[4] Instructor: https://arxiv.org/abs/2212.09741", "was_revised": true, "final_justification": "The reviewer acknowledges the effort of the authors, which solved most of my concerns (backbones and other baselines).", "rebuttal": "We sincerely thank you for your detailed and constructive review. We appreciate your positive comments on the writing quality, the breadth of the evaluation, and the analysis. Below, we carefully address each of your insightful concerns. We will also make sure to incorporate your insightful suggestions into the final version of the paper.\n\n---\n\n### **Q1: Generalization across model backbones and sizes**\n\nThank you for raising this important point regarding the generalizability of the proposed method. We have conducted additional experiments using Qwen-2.5 models of various sizes (1.5B, 7B, and 72B) to evaluate the robustness and transferability of the submodular action selection framework. Below are the results:\n\n### **Qwen-2.5-1.5B-Instruct**\n\n| Method | GSM8K | MATH-500 |\n| --- | --- | --- |\n| Qwen-2.5-1.5B-Instruct | 73.2 | 55.2 |\n| SC@maj16 | 82.4 | 60.4 |\n| RAP | 83.2 | 61.0 |\n| rStar | 81.6 | 62.6 |\n| **DynaAct** | **86.2** | **68.8** |\n\n### **Qwen-2.5-7B-Instruct**\n\n| Method | GSM8K | MATH-500 |\n| --- | --- | --- |\n| Qwen-2.5-7B-Instruct | 91.6 | 75.8 |\n| SC@maj16 | 93.2 | 81.4 |\n| RAP | 92.1 | 81.2 |\n| rStar | 93.4 | 81.8 |\n| **DynaAct** | **95.7** | **84.0** |\n\n### **Qwen-2.5-72B-Instruct**\n\n| Method | GSM8K | MATH-500 |\n| --- | --- | --- |\n| Qwen-2.5-72B-Instruct | 95.8 | 83.2 |\n| SC@maj16 | 95.4 | 86.6 |\n| RAP | 96.0 | 86.6 |\n| rStar | 95.5 | 85.4 |\n| **DynaAct** | **96.7** | **89.2** |\n\nThese results demonstrate that DynaAct consistently outperforms strong baselines across different model sizes. \n\n---\n\n### **Q2: Qualitative analysis of action selection process**\n\nYou bring up an important point regarding interpretability. While our primary focus has been on quantitative evaluation, we fully agree that qualitative insights are essential for understanding how and why the method behaves as it does.\n\nIn addition to the three examples in Appendix F.9, we are glad to include more comprehensive failure cases to highlight the limitations and edge behaviors of the method in the final version. Furthermore, we provide a more detailed illustration of how the action space evolves during the submodular optimization process, which helps demonstrate how DynaAct progressively refines the candidate set to balance utility and diversity.\n\nGiven the reasoning context:\n\n> Question: Is the Earth the center of the universe?\n> \n> \n> *Answer:* I clarify the problem by identifying the assumptions made in the past and restating them in my own words to better understand how they shaped the current situation. Early models placed Earth at the center, like Ptolemy’s.\n> \n\nThe change in the action space across optimization steps is shown below:\n\n|  | Action Space | Submodular Score |\n| --- | --- | --- |\n| step 1 | **action 1**: I need to identify the underlying structure or logic behind the current understanding, and examine how it fits into a broader conceptual or analytical framework; **action 2**: I look for opportunities to reinterpret the problem using analogies or alternative viewpoints, questioning traditional interpretations and seeing if other models from my experience could offer a better fit. | 0.61 |\n| step 2 | **action 1**: I need to identify the underlying structure or logic behind the current understanding, and examine how it fits into a broader conceptual or analytical framework; **action 2**: I look for opportunities to reinterpret the problem using analogies or alternative viewpoints, questioning traditional interpretations and seeing if other models from my experience could offer a better fit; **action 3** (new): I clarify the problem by re-examining its foundational principles and constraints. | 0.87 |\n| step 3 | **action 1**: I need to identify the underlying structure or logic behind the current understanding, and examine how it fits into a broader conceptual or analytical framework; **action 2**: I look for opportunities to reinterpret the problem using analogies or alternative viewpoints, questioning traditional interpretations and seeing if other models from my experience could offer a better fit; **action 3**: I clarify the problem by re-examining its foundational principles and constraints; **action 4** (new): I clarify the problem by identifying the initial assumptions that might have shaped the current understanding. | 0.89 |\n\nThis example highlights how the action set grows more diverse and focused with each step, improving the coverage of reasoning strategies relevant to the given context. We appreciate your suggestion and believe this qualitative illustration helps clarify the internal dynamics of our method and enhances its interpretability.\n\n---\n\n### **Q3: Comparison with few-shot prompting / dynamic exemplar selection**\n\nThank you for drawing this insightful connection. To better contextualize our method, we compared DynaAct with strong few-shot prompting baselines, including GritLM, Gecko, and Instructor:\n\n| Method | MMLU | MMLUPro | GPQA | ARC-C | GSM8K | MATH-500 |\n| --- | --- | --- | --- | --- | --- | --- |\n| GritLM | 68.3 | 42.8 | 30.3 | 82.9 | 79.3 | 48.4 |\n| GritLM @sc16 | 69.4 | 46.2 | 35.4 | 85.2 | 85.3 | 51.4 |\n| Gecko | 68.8 | 43.2 | 29.8 | 82.1 | 78.0 | 46.2 |\n| Gecko @sc16 | 69.8 | 46.6 | 34.3 | 83.7 | 85.0 | 50.8 |\n| Instructor | 68.8 | 44.2 | 27.8 | 81.1 | 79.9 | 49.0 |\n| Instructor @sc16 | 68.9 | 47.3 | 34.8 | 84.9 | 85.7 | 50.6 |\n| **DynaAct** | **70.2** | **51.4** | **39.4** | **88.3** | **89.2** | **61.0** |\n\nThese results show that DynaAct consistently outperforms competitive few-shot baselines across all tasks, reinforcing its effectiveness.\n\n---\n\n### **Q4: Release of observations from Section 4.4**\n\nYes, absolutely. We are committed to open-sourcing the observations described in Section 4.4 to support reproducibility and foster further research in this area. We appreciate your encouragement on this front.\n\n---\n\nOnce again, thank you for your thoughtful and constructive feedback. We believe your suggestions have helped us further validate and clarify the contributions of our work."}, {"review_id": "quFj63FTGj", "reviewer": "Reviewer_HY5o", "summary": "This paper proposes DynaAct, a framework for dynamically constructing compact action spaces to enhance sequential reasoning in large language models (LLMs). The framework first extracts a proxy action space from demonstration data, and defines a submodular function to balance utility and diversity. Then, it uses greedy optimization to select optimal action subsets at each reasoning step. While the proposed approach shows improvements on six benchmarks, it suffers from fundamental methodological flaws and cost-effectiveness issue that severely limit its practical value.", "questions": "1. Whether the authors considered potential duplication caused by semantic similarity among the 40,822 actions used for selecting high-level strategies. Did they filter out different surface forms corresponding to the same strategy?\n2. Is the \"action\" in MCTS referred to as the whole reasoning process or a step of the process?\n3.  I would like to know more about the effect of the choice of m on the experimental results.\n4. I would like to see some examples on the value of submodular function, is the function value consistent with human intuition?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Dynamic action space construction is a relevant challenge in LLM reasoning, offering a middle ground between rigid manually-defined actions and unconstrained generation.\n2. The evaluations span multiple domains (general, reasoning, math) across six benchmarks, which demonstrate the proposed framework's advantage.\n3. The paper is clearly written.\n\nWeaknesses:\n1. The paper focuses on single-step math and reasoning tasks. However, such problems do not have a clearly defined notion of “step”. Modeling steps as an MDP process, as described in the preliminary section, may not be appropriate. Step-level reasoning is more suitable for agentic tasks such as WebArena [1] and ScienceWorld [2].\n2. The authors employ a small model to select high-level strategies that guide an 8B frozen model in reasoning tasks. The authors primarily compare this approach with actor-critic baselines. However, I wonder whether this architecture can outperform training-based methods. Training-based approaches can significantly reduce inference time compared to the proposed method, which requires approximately 60 seconds per problem.\n3. Generalization remains a concern. Can the strategies summarised from the current six datasets and the trained 1B critic generalize to OOD datasets such as AIME 2024/2025?\n\n[1] WebArena: A Realistic Web Environment for Building Autonomous Agents\n[2] ScienceWorld: Is your Agent Smarter than a 5th Grader?", "was_revised": true, "final_justification": "I have raised the clarity score because the authors have clarified most of my concerns. And I keep my overall rating since it is positive.", "rebuttal": "We genuinely appreciate your considerate and thorough feedback. We thoroughly respond to all of your concerns and questions below. We will also make sure to incorporate your insightful suggestions into the final version of the paper.\n\n---\n\n### **Q1: On the applicability of step-level MDP modeling**\n\nYou raised a valid point regarding the notion of “step” in math and reasoning tasks, and the appropriateness of modeling them as MDPs. To clarify, our framework does not focus on single-step problems. In our setting, each “action” in the MCTS corresponds to **a single reasoning step** rather than the entire reasoning trace. \n\nAs you pointed out, agentic tasks such as *WebArena* and *ScienceWorld* are indeed more naturally framed as multi-step MDPs. We fully agree, and to further assess the broader applicability of our method beyond standard reasoning tasks, we applied DynaAct to both benchmarks. For evaluation, we adopt the official metrics defined by each benchmark and report the average performance over 10 independent runs to mitigate statistical variance. The results are as follows:\n\n| Model | ScienceWorld | WebArena |\n| --- | --- | --- |\n| LLaMA-3.1-8B-Instruct | 35.6 | 6.5 |\n| SC@maj16 | 43.6 | 10.0 |\n| RAP | 45.9 | 11.9 |\n| rStar | 44.2 | 10.9 |\n| **DynaAct** | **56.5** | **17.4** |\n\nOur framework exhibits a considerable performance advantage over all baseline models, showcasing its versatility even in more intricate, agent-like settings.\n\n---\n\n### **Q2: Comparison with training-based methods**\n\nWe appreciate this concern and would like to clarify that many training-based methods require significant **data curation and fine-tuning effort**, which limits their scalability and transferability across tasks. Moreover, in the context of strong instruct models (like Llama-3.1-8B-Instruct), we observed **only marginal improvements** with training-based methods.\n\nTo illustrate this, we compared our method with a training-based variant, where Llama-3.1-8B-Instruct was fine-tuned on the training sets of GSM8K and MATH-500, and evaluated on their respective test sets:\n\n| Method | GSM8K | MATH-500 |\n| --- | --- | --- |\n| Llama-3.1-8B-Instruct | 76.1 | 45.4 |\n| Training-based | 78.5 | 48.2 |\n| **DynaAct (Ours)** | **89.2** | **61.0** |\n\nFurthermore, for certain benchmarks such as GPQA and MMLU, no training data is available, making our plug-and-play design particularly valuable. \n\n---\n\n### **Q3: Generalization to OOD tasks (e.g., AIME)**\n\nThis is an excellent question that aligns with our core motivation. To evaluate this, we tested DynaAct on both AIME 2024 and AIME 2025 using **Qwen3** as the backbone (due to Llama-3.1’s low base performance on these challenging tasks). The results are:\n\n### **Qwen3-1.7B**\n\n| Method | AIME 2024 | AIME 2025 |\n| --- | --- | --- |\n| Qwen3-1.7B | 13.4 | 9.8 |\n| SC@maj16 | 23.3 | 13.3 |\n| RAP | 23.3 | 16.7 |\n| rStar | 20.0 | 13.3 |\n| **DynaAct** | **30.0** | **23.3** |\n\n### **Qwen3-4B**\n\n| Method | AIME 2024 | AIME 2025 |\n| --- | --- | --- |\n| Qwen3-4B | 25.0 | 19.1 |\n| SC@maj16 | 33.3 | 20.0 |\n| RAP | 30.0 | 23.3 |\n| rStar | 30.0 | 20.0 |\n| **DynaAct** | **36.7** | **26.7** |\n\n### **Qwen3-8B**\n\n| Method | AIME 2024 | AIME 2025 |\n| --- | --- | --- |\n| Qwen3-8B | 29.1 | 20.9 |\n| SC@maj16 | 33.3 | 20.0 |\n| RAP | 30.0 | 23.3 |\n| rStar | 36.7 | 23.3 |\n| **DynaAct** | **46.7** | **30.0** |\n\n### **Qwen3-14B**\n\n| Method | AIME 2024 | AIME 2025 |\n| --- | --- | --- |\n| Qwen3-14B | 31.7 | 23.3 |\n| SC@maj16 | 36.7 | 23.3 |\n| RAP | 33.3 | 26.7 |\n| rStar | 36.7 | 20.0 |\n| **DynaAct** | **43.3** | **33.3** |\n\n### **Qwen3-32B**\n\n| Method | AIME 2024 | AIME 2025 |\n| --- | --- | --- |\n| Qwen3-32B | 31.0 | 20.2 |\n| SC@maj16 | 36.7 | 26.7 |\n| RAP | 33.3 | 23.3 |\n| rStar | 36.7 | 23.3 |\n| **DynaAct** | **46.7** | **30.0** |\n\nThese results confirm that our method retain effectiveness on unseen, harder evaluations.\n\n---\n\n### **Q4: On semantic duplication in action space**\n\nCurrently, we only filter actions based on surface-form duplicates (i.e., exact matches). We will make this point clearer in the final version.\n\n---\n\n### **Q5: Sensitivity to the choice of m**\n\nThank you for suggesting a deeper analysis of the parameter $m$, which controls the size of the action space at each reasoning step.\n\nHere are the empirical results across six benchmarks:\n\n|  | MMLU | MMLUPro | GPQA | ARC-C | GSM8K | MATH-500 |\n| --- | --- | --- | --- | --- | --- | --- |\n| m = 3 | 69.3 | 50.1 | 37.9 | 88.1 | 86.3 | 57.6 |\n| m = 5 | 70.2 | 51.4 | 39.4 | 88.3 | 89.2 | 61.0 |\n| m = 7 | 70.3 | 51.9 | 38.9 | 89.1 | 87.3 | 60.2 |\n| m = 9 | 70.8 | 50.6 | 38.4 | 88.2 | 88.9 | 59.4 |\n\nAs shown, $m = 5$ achieves a strong tradeoff between performance and redundancy. Beyond that, marginal gains taper off or slightly regress, possibly due to an overly large candidate set introducing distractors. \n\n---\n\n### **Q6: Human intuition vs. submodular function value**\n\nThank you for your thoughtful question regarding the interpretability of our submodular objective. To demonstrate how the function behaves and whether it aligns with human intuition, we provide a concrete example below.\n\nGiven the following reasoning context:\n\n> Question: Is the Earth the center of the universe?\n> \n> \n> Answer: *I clarify the problem by identifying the assumptions made in the past and restating them in my own words to better understand how they shaped the current situation. Early models placed Earth at the center, like Ptolemy’s.*\n> \n\nWe trace the evolution of the action space during the submodular optimization process:\n\n|  | Action Space | Submodular Score |\n| --- | --- | --- |\n| step 1 | **action 1**: I need to identify the underlying structure or logic behind the current understanding, and examine how it fits into a broader conceptual or analytical framework; **action 2**: I look for opportunities to reinterpret the problem using analogies or alternative viewpoints, questioning traditional interpretations and seeing if other models from my experience could offer a better fit. | 0.61 |\n| step 2 | **action 1**: I need to identify the underlying structure or logic behind the current understanding, and examine how it fits into a broader conceptual or analytical framework; **action 2**: I look for opportunities to reinterpret the problem using analogies or alternative viewpoints, questioning traditional interpretations and seeing if other models from my experience could offer a better fit; **action 3** (new): I clarify the problem by re-examining its foundational principles and constraints. | 0.87 |\n| step 3 | **action 1**: I need to identify the underlying structure or logic behind the current understanding, and examine how it fits into a broader conceptual or analytical framework; **action 2**: I look for opportunities to reinterpret the problem using analogies or alternative viewpoints, questioning traditional interpretations and seeing if other models from my experience could offer a better fit; **action 3**: I clarify the problem by re-examining its foundational principles and constraints; **action 4** (new): I clarify the problem by identifying the initial assumptions that might have shaped the current understanding. | 0.89 |\n\nAs you can see, from **step 1 to step 2**, the submodular score increases significantly, reflecting a meaningful gain in both utility and diversity. The newly added action (Action 3) introduces a distinct angle of reasoning, thereby enriching the action space.\n\nHowever, from **step 2 to step 3**, the increase becomes marginal. This is because Action 4 shares semantic overlap with Action 2 and Action 3 (all aiming at clarification and reframing), which leads to **diminishing marginal gains** under the submodular objective. This behavior aligns well with human intuition — once a range of complementary reasoning strategies is already covered, adding similar ones contributes less value.\n\n---\n\nWe truly appreciate the time and expertise you dedicated to reviewing our work. Your comments have helped sharpen the focus and rigor of our contributions. Please don’t hesitate to let us know if further clarifications are needed."}]

Author rebuttal:

Meta-review: This paper identifies the limitation of the current framework on MDP-style LLM reasoning on lacking a scalable and compact action space. In response, the authors propose to first mine action space from existing corpora and formulate a submodular function to evaluate the utility and diversity of the actions to form a dynamic action space. Extensive experiments on six tasks show good performance improvement over previous methods with good latency control.

The paper addresses an important problem in current MDP-style reasoning methods in LLMs. The proposed method is relatively novel and backed by experiments on various backbones and benchmarks both in Q&A and maths (considering results in the paper and added during discussion period as requested by reviewers). In addition, application to tasks with highly dynamic and unpredictable action spaces was discussed during rebuttal phase and importance of using submodular selection vs embedding based methods were emphasized. The paper would become stronger by adding  more qualitative results.

I ask the authors to please include the results and important clarifications discussed during discussion period to the camera ready version of the paper.
Decision: Accept (poster)