AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking
Authors: Xiangqi Wang
Yue Huang
Yanbo Wang
Xiaonan Luo
Kehan Guo
Yujun Zhou
Xiangliang Zhang
Venue: NeurIPS 2025 spotlight ()
Keywords: Adaptive reasoning
LLM reasoning
Reinforcement learning
large language models
Primary area: deep_learning
TL;DR: We propose AdaReasoner, a reinforcement learning–based framework that adaptively tunes LLM reasoning settings in few-shot scenarios, achieving gains in multiple experimental settings.

Abstract: LLMs often need effective configurations, like temperature and reasoning steps, to handle tasks requiring sophisticated reasoning and problem-solving, ranging from joke generation to mathematical reasoning. Existing prompting approaches usually adopt general-purpose, fixed configurations that work “well enough” across tasks but seldom achieve task-specific optimality.
To address this gap, we introduce AdaReasoner, an LLM-agnostic plugin designed for any LLM to automate adaptive reasoning configurations for tasks requiring different types of thinking. AdaReasoner is trained using a reinforcement learning (RL) framework, combining a factorized action space with a targeted exploration strategy, along with  a pretrained reward model to optimize the policy model for reasoning configurations with only a few-shot guide.
AdaReasoner is backed by theoretical guarantees and experiments of fast convergence and a sublinear policy gap. Across six different LLMs and a variety of reasoning tasks, it consistently outperforms standard baselines, preserves out-of-distribution robustness, 
and yield gains on knowledge-intensive tasks through tailored prompts.

AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking
Xiangqi Wang Yue Huangl= Yanbo Wang? Xiaonan Luo' Kehan Guo
Yujun Zhou' Xiangliang Zhang"!
University of Notre Dame MBZZAA
(xwang76, yhuang37, xluo6, kguo2, yzhou25, xzhang33 @nd. edu yanbo. wang@mbzuai. ac.as
Abstract
LLMs often need effective configurations, like temperature and reasoning steps, to handle tasks requiring sophisticated reasoning and problem-solving ranging from joke generation to mathematical reasoning. Existing prompting approaches usually adopt general-purposee fixed configurations that work "well enough" across tasks but seldom achieve sask-spe optimality. To address this gap, we introduce AdaReasoner, an LLM-agnostic plugin designed for any LLM to automate adaptive reasoning configurations for tasks requiring different types of thinking. AdaR- easoner is trained using reinforcement learning (RL) framework, combining a factorized action space with a targeted exploration strategy, along with pretrained reward model to optimize the policy model for reasoning configurations with only a few-shot guide. AdaReasoner is backed by theoretical guarantees and experiments of fast convergence and a sublinear policy gap. Across six different LLMs and a variety of reasoning tasks, it consistently outperforms standard baselines, pre- serves out- catt-f-ddisrrbutiin robustness, and yield gains knowledge-intensing tasks through tailored prompts. Introduction of this paper can also be viewed publicly at https: //mm/mnnalllll lab-nd. gi thub. io/projjcttio aaaaaeaaaaaaaaneer...... html.
Introduction
Large Language Models (LLMs) have achieved impressive advancements across wide range of natu- ral language processing tasks, including syntactic parsing [26], complex scientific reasoning [52], and commonsense knowledge answering [59]. As the model size and training data scale up. LLMs have demonstrated the ability to surpass human-level accuracy on certain benchmarks [45], highlighting their emerging capacity for sophisticated reasoning and problem-solving
To better enhance LLM reasoming capabilities-un to push their performance closer to, or even beyond, human-level reasoning nummrous prompting- based strategies have been proposed. Chain-of- Thought (CoT) prompting encourages explicit decomposition of complex problems into intermediate steps [54, 62], while Tree-of-Though (ToT) generalizes this idea by exploring multiple branching reasoning paths [57]. Sampling-bars approaches like Best-of-N improve robustness by selecting the most coherent reasoning path from diverse candidates [16], and automatic prompt optimization techniques aim to systematically discover prompts that better facilitate multi-step reasoning [58, 42). If samples of the same type of question are provided, In-Context Learning (ICL) [5] also prompts LLM with few-shot examples with advanced performance.
Despite these advances, LLM reasoning remains highly GPT--40's 's accuracy on the metaphor expression classification task [49] swings wildly under different
as Figure shows,
Equua contribution.
Cooresponndig author: xzhang330nd, edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

reasoning configurations. While divergent reasoning prompts and fewer reasoning steps could greatly improve performance, temperature as 1 instead drown out useful reasoning with noise, negating any benefit from the added randomness. However, previous methods have not targeted tuning on these parameters. CoT [54, 62] and ToT [57] apply fixed reasoning structures that fail to generalize to creative or subjective tasks (e.g. spatial planning [46]). Best-of-N [16] rely on unguided generation, suffering from a "garbage in, garbage out" effect. Automatic prompt optimization [58, 42] focuses on static templates and overlooks crucial hyperparameters like temperature, failing to adjust reasoning strategies. While ICL [5] extracts some cues from input questions, it remains brittle under context perturbations [28]. and its reliance implicit pattern matching has been shown to be less effective than direct structured reasoning [48]. These limitations call a need for an adaptive prompting configuration strategy for LLMs to handle various sophisticated reasoning.
However, identification of the optimal prompting configuration for LLMs is non-trivial task. First, task types span logical, creative, and subjective domains, often in combination, so that many queries cannot be neatly categorized or matched with pre-set configurations template. This necessitates strategies
Simple CoT (T=0:1) Divergent
3-Step CoT simple Col (T-1.0)
Accuraay
that are highly adaptive and tailored to the specific demands of
each question. Second, LLM reasoning capability is sensitive to the configuration settings that involve multiple factors, as shown in Figure 1. The search space spanned by these factors when selecting effective configurations is combinatorially large. This presents a challenge for building a decision-making model that tailors the configuration for each input task. Third, while building such a model using a data-driven approach is promis- ing, exhaustively collecting training samples for every possible configuration is computationally expensive and impractical. This necessitates an approach that can generalize from a limited set of examples and capture reasoning patterns that are transferable across similar tasks.
Reasoning Setting
Figure 1: Performance of differ- ent CoT settings the metaphor dataset [49]. The default temperature 0.1 if not specified.
We introduce AdaReasoner, an LLM LLM-agnostic plugin designed to automate adaptive reasoning config- urations for tasks requiring diverse types of thinking. When integrated with an LLM, AdaReasoner is trained using reinforcement learning (RL) framework. In this setup, AdaReasoner acts decision-making agent, where the state is defined by the current task presented to the LLM, reflecting the nature of the reasoning required (e.g., logical, creative, subjective). The action corresponds to selecting a configuration from an action space composed of three key hyperparameters: (i) the reasoning instruction formats, (ii) the generation temperature, and (iii) the number of reasoning steps. To enable AdaReasoner to learn the most effective configuration policy, a pretrained reward model is employed to evaluate the effectiveness of the reasoning configuration. This model provides feedback to guide the agent's 's learning, enabling it to efficiently acquire effective configurations with only limited guidance (i.e., few-shot learning). To facilitate exploration and improve generalization, we employ Boltzmann exploration mechanism. enabling the agent to explore and optimize configura- tions more effectively during training. Once trained, easoner used as a plug-in to the L.LM, providing adaptive reasoning configurations that allow the model to adjust its reasoning approach based on the task hand.
Our contributions can be summarized as the followings:
We introduce AdaReasoner, an LLM-agnostic plugin that automates adaptive reasoning configura- tions for tasks requiring diverse types of thinking.
AdaReasoner leverages reinforcement learning framework with a factorized action space. Its training is data-efficient yet scalable, requiring only small number of samples for each task aided by the use of the Boltzmann exploration mechanism.
Extensive evaluations on diverse tasks show that RdaReasoner outperforms standard CoT and baselines, and sustains strong OOD performance.
2 Related Work of Reasoning in LLMs
The pursuit of enhanced reasoning capabilities in LLMS has spurred diverse research trajectories, beginning with foundational techniques like Shain-of-Thouge (CoT) prompting [54]. CoT enables I.LSs to articulate intermediate steps, significantly improving performance on complex tasks. How- ever, its efficacy can be hampered by sensitivity to prompt formulation [44, 33] and limitations in

---

subjective or creative domains [7, 56], sometimes even degrading performance where brevity is key [24]. To mitigate these issues and reduce manual effort, innovations such Automatic CoT (Auto-CoT) [58, 42] emerged, automating the generation of effective reasoning exemplars. Further advancements include structured reasoning frameworks like tre-oof-T (ToT) [57] and Graph- of-Thoughts (GoTT) [4], which allow models to explore and evaluate multiple reasoning pathways, alongside methods like CoT-influx [13] that optimize few-shot CoT contexts.
To bolster the robustness and reliability of LLM reasoning, researchers have explored self-correction and learning- based paradigms. Self-consistency techniques [53], often realized through Best-of-N sampling, leverage the generation of multiple diverse reasoning paths and subsequent aggregation (e.g., via majority voting) to improve answer accuracy, Complementary to this, self-reflection mechanisms, as seen in Self-Refing [27] and Reflexion [41], empower LLMs to iteratively critique and enhance their own outputs, akin human error correction, with some approaches fine-tuning with divergent Co'T' to specifically boost these capabilities [33]. Reinforcement Learning (RI.) has also become a cornerstone for optimizing reasoning, from general alignment via RLHF [31] to specialized reward models that guide the LLM towards more accurate and effective thought processes [15]. Models like DeepSeek-RI [11] exemplify LLMs fine-tuned with RL to excel at intricate reasoning, sometimes learning to control their own reasoning flow through meta-actions.
The nuanced control of generation parameters and adaptive hyperparameter tuning represent another critical frontier. The stochastic decoding settings, such as temperature, significantly affect output diversity and, consequently, reasoning quality and creativity [35]. Higher diversity can fuel methods like self-consistency but requires careful management to maintain coherence. Recent work has thus focused on automated optimization of prompt configurations, decoding parameters, and even enabling LLMs to self-regulate their generation strategies, as demonstrated by Hyperparameter-Awee Generation (HAG) [51]. Our AdaReasoner contributes to this line of research by introducing an adaptive framework that explicitly manages toolbox of reasoning hyperparameters, including the reasoning method prompt, temperature, and number of reasoning steps, using an RL-trained agent to dynamically tailor the reasoning process to individual inputs, coupled with self-reflection and
robust selection mechanism for enhanced flexibility.
3 AdaReasoner
Motivation. Even though CoT and similar LLM reasoning methods have been studied as generally efficient and helpful, they still cannot achieve ideal performance across all types of questions. For example, tasks like joke generation or metaphor interpretation often require divergent and creative reasoning chain [61]. For more complex reasoning tasks, stronger and more explicit reasoning instructions would be beneficial [22]. Thus, adapting LLM configurations tailored for specific tasks is crucial for achieving better overall performance. As illustrated in Figure 2, we design AdaReasoner to adapt reasoning configurations by taking actions as combination of different hyperparameters for LLMs. The inferenceeevalluaiin process is illustrated by the black arrows, while the training flow is depicted by the cyan arrows.
The goal of AdaReasoner to adaptively find the most effective hyper- parameter configuration a for a given question 9 such that an LLM (denoted as b) generates the correct reasoning answer (q)a). More specifically, the configuration is a 3-dimensional vector, where each element corresponds to one of the three hyperparameters: at (generation temperature), ap (reasoning instruction format), and a, (the number of reasoning steps). Denoting AdaReasoner as He. our goal is to train its neural network weights θ to learn the optimal policy for deciding the configuration a given a question q. By considering the question 9 along with the LLM & as the employ
Problem Formulation.
state, the decision-making process is represented as a ll 4). During training, pre-trained model (c.g. DeBERTa in huggingface) as the reward model r to provide feedback on the generated answer by comparing it to the ground truth reference R from the training data, i.e., r r(f(qla). R). In this approach, we address the issue that it is not possible to directly evaluate the quality of generated configuration a, as there is ground truth for a itself. Instead, the effectiveness of a is judged indirectly based on the resulting answer & (gg a), ensuring that the AdaReasoner agent is informed about the quality of its reasoning configuration through the answer's relevance and accuracy.
Within the broader RI. framework, our study can be viewed as a multi-armed bandit problem, where the arms represent different configuration actions. Each question is an independent task (state),
3

---

Shared Layer
Question
Reward Model
Instruction
Step Head
Temp- Head
Head
Embedding
Ground Truth
(,(n)
e.(a)
Qr(a)
Answer
AdaReasoner
Action Sampling
Boltzmann Exploration
Reasaning Strtker
Action
3 Raaeasing Stopt
with Question
AI
Actions 4p Qg Gr
Tumperature
LLM
Figure 2: The proposed framework of using AdaReasoner for automating the reasoning configuration (instructions, steps, temperature). During training, configurations actions are sampled with Boltzmann exploration, guiding LLMs generate answers, which are then evaluated by a reward model for policy optimization.
where the agent determines the actions (sets the values for all arms), receives a reward based on the effectiveness of the answer, and then moves on to the next task. The objective to optimize the selection of hyperparameters to maximize the reward for each question. Therefore, given set of
training questions and reference answer samples Dirain = {(g, R the objective is to train the AdaReasoner agent
- arg max DD) (a-tl)(a) (o(g)a)
(I)
Theoretical analysis about AdaReasoner is presented in Appendix B, with step-by-step description in Algorithm 1.
3.1 Hyperparameter Configuration (Action)
As mentioned earlier, we consider three hyperparameters in the reasoning configuration: 1) the generation temperature dg; 2) the format of reasoning instructions ap; and 3) the number of reasoning steps in CoT 0,, for several reasons. First, they have substantial impacts the reasoning performance. Previous studies have revealed that the generation temperature modulates the diversity of model outputs, often yielding markedly different responses when varied [36]. The number of reasoning steps reflects the depth and thoroughness of the inference process and it thus could influence the reasoning accuracy [9, 17]. The format of reasoning instructions, such as backward reasoning and step-by-step deduction, also plays a crucial role in guiding the model's reasoning process [2, 50]. Second, the settings of these three hyperparameters are adaptable for both proprictary and open-weight LLMs. with enhancement of adareasoner's 's versatility. Third, we are aware of other hyperparameters that may also impact reasoning, such as the p in top-p sampling during generation and the random seed, However, we exclude because tuning top-p alongside temperature is not recommended together with temperature [1]. Additionally, our empirical evaluation found that varying the random seed could not be beneficial for improving LLMs reasoning performance (as shown in Section 4.3).
To ensure practical feasibility, these configuration actions are discretized with a finite set of options. Specifically, the number of reasoning steps is bounded to avoid extreme values, defined as A, as integers set, and temperature is discretized set Ar. The options for reasoning instructions, denoted as Ap. are constructed based on a compositional design grounded in structure-mapping theory from cognitive psychology [10]. which models human reasoning by composing a core reasoning structure with contextual modifications. Accordingly, each reasoning instruction is factorized into two components: a base component, which specifies the overall cognitive strategy (e.g., creative thinking, analogical mapping, self-audit [6]), and variation, which modulates the emphasis on specific parts of the question or modifies the reasoning surface form. For example, a base "Apply creative reasoning to unearth unconventional insights and challenge standard assumptions" could be combined with a variation "Use simple, straightforward language to guarantee clarity and accessibility" for guiding the reasoning of divergent thinking types of problems. The same base, when combined with a variation Validate conclusions by aligning them with established principles or empirical data, such instruction is useful for critical thinking types of reasoning problems. Detailed of

---

the base and variation components and their instantiation are provided in Appendix C. The reasoning instruction action space, Ap. composed of pairs the form of {base, variation). Each action ap corresponds to one of the possible combinations of base and its associated variation.
Ultimately, the decision about the action involves selecting generation temperature at from As, number of reasoning steps from A., and one form of reasoning instruction ap from Ap.
3.2 Design and Training of AdaReasoner
Neural Architecture of AdaReasoner. As shown in Figure 2. the input query question, after embedding, undergoes three action selections before being sent to the LLMs for reasoning to generate the answer. While the embedding is performed (e.g. by pre-trained BERT model [55]), the trainable neural network parameters of AdaReason consist of three parallel channels, each corresponding to one action, and one shared common layer as in Figure 2. The workflow is as follows: let Embed(g) be the embedding of the input question q. It is first passed through the common layer to obtain h fo, (Embed(q)), where 0g are the parameters of the common layer (e.g., fully connected MLP), and h captures the features necessary to determine the actions.
Then h is sent to each channel, where the action selection is performed as
ap (-1bb fo, (h), a2 (-(h))) = fe. (h), a, 1..T (-(+Ih) = (h),
(2)
where each policy x-1hh) is implemented as feed-forward network.
This design factorizes the policy II into three independent heads, each handling a specific action selection, significantly reducing optimization space from multiply to summary. Viewing Il as multi-armed bandit problem, it factorizing the joint-arms into set of parallel yet not independent single arm ones. While each head operates independently, they are optimized jointly with shared latent representation, ensuring coherent decision-man maainng and unified optimization across ap. O, and ag Let K = MT be the total number of steps in learning, where M is the number of training questions and the number of trials for each question. We analyze the regret of AdaReasoner, i.e. the reward difference between AdaReasoner and the optimal policy without factorization in App. B. The regret per step is bounded by O(( ((L In |Al yo.5), where |A] is the total number of action values: A = Ap x A, x As. This shows that the regret per step becomes negligible once K > |A] In [A). which is consistent with the empirical observation of few-shot convergence, meaning that AdaReasoner learns effectively with relatively few training examples. Moreover, under Lipschitz smoothness and bounded variance conditions, Adareasoner with J(@*) denoted as optimal expected-
=(J(O))-J(oa)
reward objective and J(e)) as initial objective achieves an error bound (App. B), reinforcing rapid convergence in the few-shot setting.
+ 17 0²
n
Exploration Strategy. By formulating the configuration selection for each question as a multi-armed bandit (MAB) problem, we aim to design effective exploration strategy under the few-shot training setting. However, since the reward is derived indirectly from LLM outputs and the process is not an online learning scenario, standard MAB strategies such Upper Confidence Bound (UCB) [47] become impractical. Moreover, evaluating all configurations for each context is computationally infeasible, especially given the noisy and implicit reward landscape induced by LLM responses. Therefore. it is crucial to explore broadly across the configuration space while still prioritizing high-reward actions, and Boltzmann exploration offers an effective solution [32], as it allows the agent to probabilistically select actions based on their estimated rewards. Specifically, for each action (at, a₂ or ap), we estimate the selection probability for its all possible values (in At. A, or Ap),
exp(Q(a.)/7)
P(a.)) EnyeA exp(Q(a))/=)
(3)
where Q(a;)) is the logit score in the output layer of one policy network fo for action a, The temperature т in Boltzmann exploration controls the promotes exploration, lower 7 favors exploitation. We anneal exponentially T, To a f T. allowing the policy to gradually shift from broad exploration to reliable configuration selection and refined optimization [19].
trade-off: higher T
Reward Signal. Similar to previous work [20, 25, 37] using pre-trained language model as reward on liight RL model, we employ a language judgement model (ours is DeBERTa-based) as
5

---

reward model [30] to provide feedback on the selected actions. Concretely, for the resulting generated answer ((gla), it presented the reward model alongside the original question q and reference answer R in the form of the prompt "For q. the generated answer ((gla) matches the ground truth R and is correct". The reward is computed from the model's logits, providing scalar score that enables fine-grained, differentiable supervision over diverse reasoning trajectories.
With the reward r, the AdaReasoner is optimized using the gradient descent (REINFORCE) algo- rithm [43], where the overall policy la g. 4) is factorized into three heads with a shared feature extractor fe., and
{f2, 8g. f2, 8₂] denotes the complete set of trainable parameters. For each head j E {p, t, s), we define the head-specific loss as Lj = -r log I1g (a q. Φ), resulting in a total loss & = Eje(p.t.. f.j. The gradients are then computed via the chain rule, where the shared-layer gradient is aggregated as Vo.C EjE(mA))) Vo.Lj. and used for updating
0. + 0. 77 Vo.C.
(4)
Each head is updated as
(5) This training scheme ensures that each sub- sub-policy is guided by its own loss while the shared feature extractor fo, is jointly optimized by all heads, thereby promoting coherence across the three action dimensions and preventing convergence to conflicting optima, in line with findings from multi-task learning [38]. Further training details are described in Algorithm 1.
0, - A jE (p.t,s).
4
Experiments
4.1 Experimental Setting
Dataset. To evaluate the performance of AdaReasoner, we selected datasets that engage distinet cognitive processes, ranging from logical and mathematical to figurative and generative reasoning.
This is a collection of data examples that are in the Math category from the Massive Multitask Language Understanding (MMLU) benchmark [12], focusing on numerical reasoning, symbolic manipulation, and procedural problem solving.
MMLU:
Metaphor [49]: This dataset focuses evaluating whether highlighted word in context is used metaphorically in the context.
TruthfulQA [21]: This dataset tests LLM trustworthy generation by posing questions with common misconceptions or false premises.
LogiQA [23]: This dataset is designed for multi-step logical reasoning based on Chinese civil service exam questions.
Each dataset contributes 250 samples, randomly sampled from the full dataset. The combined dataset is then divided into a training set of 100 samples and a test set of 900 samples forming thus a few-shot setting. Examples of the four datasets are displayed at Table 5 and distribution of each dataset is shown at Figure 5.
Baselines. We compare AdaReasoner with several baselines that adopt different strategies to improve LLM reasoning:
CoT (Chain-of-Thought) [54]: Prompts the model to think step-by-step for reasoning.
Think Short: Prompts the model for brief, quick responses with prompt Figure 10.
ToT (Tree-of-Thought) [57]: Structures reasoning path as a tree, exploring and selecting among
multiple paths.
Best-of-N [16]: Produces N candidate chains, selects the best based on predefined scoring metric. Auto-CoT [58]: For each query, retrieve semantically nearest exemplars from few-shot pool (via embedding clustering), generate CoT rationales, and concatenate the
triplets as the in-context prompt: other settings follow the original.
In-context CoT (ICL) [5]: Leverages in-context CoT generation by presenting examples of few-shot train set directly within the prompt.
Evaluation and other details. To evaluate the alignment between LLn--gnerated responses and the ground truth, we adopt the T.LM-as-a-Judee. semantic equivalence of answers and the quality of their explanations through dedicated judgment prompts, as illustrated in Figure& 8. each evaluation, the top-p parameter set to 0.1 and the max_token parameter is set to 5,000, with no system prompt utilized. We random select 100 out of
paradigm [60], utilizing GPT-4o to assess both the

---

Table 1: Performance of various reasoning methods across multiple datasets for different LLM models (accuracy in %). The highest score for each dataset and the average in each model group is highlighted in bold and underlined.
Dataset (%)
Model
Reason Method
Average
Metaphor
TruthfulQA
MMLU
LogiQA
CoT
50.40
78.40
76.04
70.00
68.71
Think Short
61.00
64.81
68.52
70.81
66.28
ToT
48.25
74.29
86.11
73.90
70.91
GPT-4o
Best-of-N
52.60
79.41
83.41
72.37
71.95
Auto-CoT
62.33
83.09
72.15
71.71
72.32
In-context CoT
53.98
77.04
83.63
80.04
74.42
AdaReasoner
71.56
81.30
86.49
82.31
80.42
CoT
51.56
75.77
83.33
75.56
71.56
Think Short
59.56
75.77
81.61
73.78
72.68
ToT
60.89
75.33
86.24
83.56
76.51
Llama-33-770--mms
Best-of-N
52.89
77.09
89.69
76.00
73.92
Auto-CoT
45.33
78.85
81.82
76.00
70.50
In-context CoT
52.71
82.45
84.57
75.59
73.60
AdaReasoner
66.11
83.09
87.77
85.00
80.74
CoT
60.18
79.36
73.89
78.26
72.92
Think Short
71.24
80.28
64.16
75.22
72.73
ToT
62.26
77.50
66.57
79.51
71.46
Qwen-2.5-72B-Inu
Best-of-N
59.73
78.44
76.11
78.26
73.14
Auto-CoT
65.93
83.49
76.
79.13
76.17
In-context CoT
73.39
78.94
71.93
74.83
74.77
AdaRcasoner
65.19
83.82
80.14
80.79
77.49
CoT
62.13
86.13
85.00
80.43
78.42
Think Short
67.71
83.43
78.95
77.95
77.01
ToT
59.45
85.12
86.43
81.98
78.25
Claude-3.5-sonnet
Best-of-N
41.41
83.43
81.87
78.95
71.42
Auto-CoT
65.04
84.86
88.50
78.70
79.28
In-context CoT
55.81
88.60
79.23
79.53
75.79
AdaReasoner
65.77
86.17
89.21
84.55
81.43
CoT
54.35
83.34
96.13
81.82
78.91
Think Short
80.00
95.55
77.71
ToT
63.33
86.16
98.70
83,22
82.85
Deepseek-R1
Best-of-N
54.55
85.51
94.37
87.01
80.36
Auto-CoT
61.04
82.61
97.70
80.52
80.47
In-context CoT
50.06
84.21
96.15
84.25
78.67
AdaRcasoner
72.00
88.17
96.33
88.60
86.28
CoT
45.10
84.00
95.71
83.87
77.17
Think Short
57.14
80.00
93.21
67.74
74.52
ToT
53.85
84.91
98.18
80.00
79.24
GPT-o3-mini
Best-of-N
56.99
82.10
93.55
84.22
79.22
Auto-CoT
51.00
86.79
97.78
76.14
77.92
In-context CoT
53.00
82.25
95.56
77.19
77.00
AdaRcasoner
67.29
86.45
96.13
87.67
84.39
1,000 samples as few-shot examples for AdaReasoner and ICL. ToT uses a beam width of and
max length of 3. Baselines follow default settings with in-context examples from the same dataset and type. AdaReasoner uses a fixed learning rate 0.01, BERT embeddings (768-d) for the input question, and a 3-layer MLP for each policy head.
Main Results
Performance of reasoning methods across datasets. Table summarizes the accuracy different reasoning strategies across muitiple datasets for each backbone LLM. Notably, AdaReasoner achieves the highest average accuracy within every model group, underscoring effectiveness in guiding reasoning, For instance, AdaReasoner achieves average of 80.42% GPT-4o, surpassing Auto-

---

CoT and other baselines, and similarly 81.4% on Claude-3.5 3.snnenett, confirming its stability across evaluation settings. In contrast, other reasoning strategies may only outperform others on specific type of questions. 'oT attains the top score on MMLU across several models, highlighting its strength in complex, knowledge-intensive challenges. Meanwhile, Auto-CoT yields the highest accuracy on TruthfulQA for both GPT-4o and Qwen-2.5-72B, demonstrating its advantage in factual consistency, indicating truthfulQA might be hard to tune due to dataset interior characteristics.
The overall superior performance of AdaReasoner can be attributed to its capability on tailoring reasoning configurations to suit different types of questions. As detailed in Appendix E. we analyze the -peaset- distributions of ap. as, and ag. The boxplot in Figure 6 shows the distribution of ag and a across correct and incorrect cases. Table 6 reports the average and standard deviation of a, and a,. The heatmap in Figure 7 illustrates performance differences between the most and least frequent ap options. Table presents the Top-3 reasoning instructions Op identified by AdaReasoner for each dataset. From these results, we can observe that AdaReasoner's action selection showing clear -ataset-specific distinctions, especially regarding the reasoning instructions ap.
In addition, to further demonstrate the reliability of LLM-as-Judge method used, we provide human annotated result in Appendix H.
Few-shot Training. Figure 3 shows that when AdaReasoner is trained in few-shot scenarios, its per- formance exhibits marginal gains beyond 100 shots, universally for Qwen-2.5-72B, LLaMA-3.3-700 and GPT-4o. With 50-100 demonstrations suffice for the AdaReasoner to learn core reasoning patterns, validat- ing the efficiency of the few-shot setting. Theoretical worst case regret convergence as in Appendix B is O (L, AdaReasoner converges far faster in practice. This might be due to a shared encoder en- forcing 3-smoothness and near-convexity [8], and pretrained reward model providing high-fidelity, high-SNR feedback ("warm start") that accelerates few-shot policy updates.
Qwen-2.5-728 LLaM4A3.3-708
GPTT-4o
Aewrage Accuray (t))
80
78
26-
770
72
50
100
150
200
250
300
350
400
Number of Shots
Figure Few-shot training performance.
4.3 Ablation Studies
We modify the components in AdaReasoner to conduct ablation study, validating the effectiveness of each design choice. Among the results presented in Table 2, AdaReasoner (a only) refers to setup where only the adaptation of hyperparameter a is allowed. In addition to Og, Or and ap. we also adapt the random seed in the same way to demonstrate that it is not an ideal choice (and thus excluded). Adapting only the reasoning instruction ap results in the smallest performance drop, highlighting the importance of this action. It also emphasizes the necessity of considering simultancously a, and at in the adaptation process.
To evaluate the effectiveness of Boltzmann exploration, we replace it by applying Thompson sam- pling [39] to all actions (w/ Bandit Adapter), which leads performance drop 75.89%. To evaluate the effectiveness of the reward model, we added Gaussian noise (a 0.01) to reward signal (w/ Perturbed Reward), and rescaled reward value from the interval [O 1] to the interval [-0.5 0.5] (w/ [-0.5 0.5) Reward). The results show that Adareasoner is robust to reward noise yet sophisticated in reward rescaling.
Due to the presence of regret, AdaRcasoner learns an approximate rather than an optimal policy. To assess this, we analyze perturbed variants (Close and Distant Perturb) by selecting neighboring actions in embedding space by similarity. We also evaluate an Ensemble setting that aggregates independently trained policy heads without shared layers, further validating AdaReasoner's 's design.
The final experiment tests cross-model transfer by applying Qwen-trained policy to GPT-4o. As shown in the w/ Qwen Adapter row, average performance drops to 72.31 %6, reflecting not flaw in AdaReasoner, but the model-specific nature of reward landscapes, highlighting the need for adaptation. Random Action also underperforms, reinforcing the value of learned strategies. However, interestingly performs well on MMLU, perhaps due to a reward landscape with multiple local optima that favor random exploration, also observed in the setting with perturbed rewards.
8

---

Table 2: Ablation study results (accuracy in %) for AdaReasoner when promoting GPT-4o. The best result each column is highlighted bold and underlined.
Ablation
Metaphor
TruthfulQA
MMLU
LogiQA
Average
Random Action
55.92
76.15
80.32
76.81
72.30
AdaRcasoner (ar)
62.91
80.00
77.71
75.67
74.07
AdaReasoner (a.)
74.29
82.11
74.44
74.74
AdaReasoner (ap))
70.66
78.31
84.50
81.01
78.62
AdaReasoner (Random Seed)
53.17
70.55
79.13
73.90
69.19
w/ Bandit Adapter
68.30
76.11
80.00
79.13
75.89
Perturbed Reward
70.83
79.26
85.07
77.89
78.26
w/ 1-0.5, 0.5] Reward
56.66
76.15
79.04
77.63
72.37
w/ Qwen Adapter
65.76
73.80
69.69
80.00
72.31
Adareasoner (Close-perturb)
66.05
79.39
85.18
80.03
77.66
Adareasoner (Distant-perturb)
57.69
71.77
81.42
74.96
71.46
Adareasoner (Emsemble)
65.73
79.54
84.71
80.04
77.50
ReaRenr
71.56
81.30
86.49
82.31
80.42
4.4 OOD Generalization AdaReasoner
Table 3: Qwen-2.5-72B's performance (Accuracy %) with different reasoning methods on three OOD datasets.
Table shows if the AdaReasoner trained the above-mentioned four datasets can be effec- tively applied other out of domain (OOD) applications, such multilingual emotion anal- ysis BRIGHTER dataset [29], spatial planning in the StepGame dataset [40], and commonsense reasoning the CRoW dataset [14]. On the 150 QA pairs randomly sampled from each of these datasets that AdaReasoner has never encoun- tered before, we can observe stable superior performance of Adareasoner over other reason- ing methods.
Model
BRIGHTER
StepGame
CRoW
Think Short
$2.08
71.25
90.46
CoT
51.19
73.73
93.97
Auto-CoT
55.17
68.64
90.52
ToT
51.40
76.32
80.18
Best-of-N
49.14
73.73
93.10
In-context CoT
53.17
77.15
90.00
AdaReasoner
55.36
78.00
95.56
4.5 AdaReasoner on Knowledge Intensive Datasets
We next challenge our method on knowledge-intensive datasets, such as GPQA [34], MMLUChem [12], and MedExQA [18], which require general domain knowledge domain-specific knowledge in areas like chemistry, medicine. We randomly select 100 samples from each of these three datasets for training, and 500 samples for testing. As shown in Figure AdaReasoner shows modest yet consistent capacity to adjust to questions requiring intensive knowledge, outperforming conventional reasoming approaches such CoT and ToT. However, we acknowledge that adapting reasoning strategies alone cannot fully address the lack of domain-specific knowledge in GPQA (c.g., general facts, cultural references, history). A case-by-case analysis Table reveals that the adapter often selects self-audit, cross-reasoning, or creative prompt variants for such examples. Combining Table 8 with Table 7, the most frequently selected ap values-reflectiv self ceff-questionin for logic-intensive tasks and creative assumption assumption-challenginn for Knowledge Intensive and Metaphor Metaphor-suugestt that cognitive configuration adaptation a promising direction for further exploration, and this just one of many intriguing patterns uncovered.
100
GPQA
MMLUCChem
MadExQA
CoT
In-context CoT Best
Auto-CoT
30
ToT
Figure 4: Performance of different reasoning methods on knowledge intensive datasets (accuracy in %) by Llama-3.3 3.3- Inn--nntruc...

---

Conclusion and Future Work
We presented AdaReasoner, an LLM-agnostic plugin designed to identify question-tailored con- figuration for selecting reasoning instructions, setting generation temperature modulation, and the number of reasoning steps. Our extensive evaluation across six LLMs and diverse benchmarks demonstrates that configuring reasoning strategies in concert yields substantial gains over fixed approaches, with ablation studies confirming each component's 's unique impact on performance and robustness. Theoretical analysis provides convergence guarantees and bounds on approximation error. Nonetheless, AdaReasoner depends on per-task few-shot fine-tuning and introduces additional computational overhead for RI. optimization.
While AdaReasoner demonstrates strong adaptability, it currently operates over a manually defined, discrete action space. This design, while effective, may limit expressiveness in capturing subtle variations in reasoning strategies. Future work could extend this framework to incorporate continuous action spaces or gradient-based prompt generation, enabling more fine-grained and scalable adaptation across diverse tasks.
References
[1] Enhanced inference - autogen 0.2. https::/miccrsseeee github. 1o/ tt//autogen//... 2/doss/ Accessed: 2025-05-13.
Use-Cases/ ruhanced_inffreeee
[2] Guilherme FCF Almeida, José Luiz Nunes, Neele Engelmann, Alex Wiegmann, and Marcelo de Araújo. Exploring the psychology of lims' moral and legal reasoning. Artificial Intelligncce, 333:104145, 2024.
[3] Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1):48-77, 2002.
[4] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682-17690, 2024.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1 33:18777-1901 2020.
[6] Ruth MJ Bymee Jonathan St BT Evans, and Stephen E Newstead. Human reasoning: The psychology of deduction. Psychology Press, 2019.
[7] Georgios Chochlakis, Niyantha Maruthu Pandiyan, Kristina Lerman, and Shrikanth Narayanan. Larger language models don't 't care how you think: Why ofain-of-thought prompting fails in subjective tasks. arXiv preprint UrXiv:2409.06175. 2024.
[8] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675-1685. PMLR, 2019.
[9] Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. How to think step-by-step: A mechanistic understanding chain-of-thought reasoning. arXiv preprint arXiv:2402.18312. 2024.
[10] Dedre Gentner. fiuecturemapping A theoretical framework for analogy. Cognitive science, 7 (2)0155-170, 1983.
[11] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint UrXiv:2501.1244 2025.
[12] Dan Hendrycks, Collin Burns, Andy Basart, Saurav Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Massive multitask language understanding (mmlu), 2020. URL https: //huggingface. co datasete/ cais/mmlu.. mu. Accessed: 2025-02-28.
10

---

[13] Xijie Huang, Li Lyna Zhang, Kwang-Ting Ting Cheng, Fan Yang, and Mao Yang. Fewer is more: Boosting Ilm reasoning with reinforced context pruning. arXiv preprint arXiv:2312.08901, 2023.
[14] Mete Ismayilzada, Debjit Paul, Syrielle Montariol, Mor Geva, and Antoine Bosselut, Crow: Benchmarking commonsense reasoning in real-world tasks. arXiv preprint arXiv:2310.15239, 2023.
[15] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed BI-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, al. Openai ol system card. arXiv preprint arXiv:2412.1672) 2024.
[16] Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating hallucination in large language models via self-reflection. arXiv preprint arX1v:2310.0627,, 2023.
[17] Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du The impact of reasoning step length on large language models. arXiv preprint 2xXiv:2401.04225 2024.
[18] Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. Medexqa: Medical question answering benchmark with multiple explanations. arXiv preprint arXiv:2406.06331, 2024.
[19] Scott Kirkpatrick, C Daniel Gelatt Jr, and Mario P Vecchi. Optimization by simulated annealing. science, 220(4598).671-688) 1983.
[20] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint urXiv:2303.00001 2023.
[21] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2021. URL https:://hggeeee htt.ggggggggga.e./////////// co/ cettasetsddomenii crosati/ TruthfulQA. Accessed: 2025-02-28.
[22] Zicheng Lin, Tian Liang, Jiahao Xu, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhence I1m's s reasoning capability, arXiv preprint arX19:2411.19943, 2024.
[23] Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.0812 2020.
[24] Ryan Liu, Jiayi Geng, Addison Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L Griffiths. Mind your step (by step): -hain-of-thought can reduce performance on tasks where thinking makes humans worse. arXiv preprint arXiv:2410.21333. 2024.
[25] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Fureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. /2931, 2023.
[26] Zeyang Ma, An Ran Chen, Dong Jae Kim, Tse-Hsun Chen, and Shaowei Wang. LImparser: An exploratory study on using large language models for log parsing. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1-13, 2024.
[27] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534-46594, 2023.
[28] Aaron Mueller, Albert Webson, Jackson Petty, and Tal Linzen. In-context learning generalizes, but not always robustly: The case of syntax. arXiv preprint arXiv:2311.07811 2023.
[29] Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nirmal Surange, Daniela Teodorescu, Ibrahim Said Ahmad, et al. Brighter: Bridging the gap in human-annotated textual emotion recognition datasets for 28 languages. arXiv preprint
19226, 2025.
11

---

[30] OpenAssistant. Opeuassiinnttwwwacccoddbbbbbbbbb
https: http::/ttttsssststtiiiee h/w.gggggggggg...ceg/ v2, February 2023. MIT Li-
co /t/p/ppessssssseee sttessssstttttoeeeeeeetee htt.://ssstatttt/rrrr--dddeeedee wwwpaaaddddddddddddddeeeeeeeeeee
cense. Accessed: 2025-04-25.
[31] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[32] Ling Pan, Qingpeng Cai, Qi Meng, Wei Chen, Longbo Huang, and Tie- Yan Liu, Reinforcement learning with dynamic boltzmann softmax updates. arXiv preprint vrXiiv:903..05 2019.
[33] Haritz. Puerto, Tilek Chubakov, Xiaodan Zhu, Harish Tayyar Madabushi, and Iryna Gurevych. Fine-tuning with divergent chains of thought boosts reasoning through self-correction in lan- guage models. 2024.
[34] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-lovel google- proof q&a benchmark. In First Conference on Language Modeling, 2024.
[35] Matthew Renze. The effect of sampling temperature problem solving in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7346-7356, Miami. Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024 org /2244.244 f findings- emnlp 432/.
findings-emmlp4422. URL httpp:://aaaeeeeeeeee
[36] Matthew Renze. The effect of sampling temperature problem solving in large language models. In Findings of the Association for Computational Linguisties: EMNLP 2024, pages 7346-7356 2024.
[37] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. arXiv preprint 1f.i:2231011922/1 2023.
[38] Sebastian Ruder. An overview of multi-task learning in deep neural networks, arXiv preprint arXiv:1706.05099 2017.
[39] Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends@ in Machine Learning, 11(1):1-96, 2018.
[40] Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A new benchmark for robust multi- hop spatial reasoning in texts. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 11311-11329 2022.
[41] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634 8652, 2023.
[42] KaShun Shum, Shizhe Diao, and Tong Zhang. Automatic prompt augmentation and selection with offin-of-thought from labeled data. arXiv preprint arXiv:2302. 2023.
[43] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning. pages 387-395. Pmir, 2014.
[44] Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain- of-thought helps mainly on math and symbolic reasoning. arXiv preprint arXiv:2409. 12183, 2024.
[45] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04615, 2022.
12

---

[46] Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of thoughtlsssesss? an analysis of cot in planning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[47] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[48] Ruixiang Tang, Dehan Kong, Longtao Huang. and Hui Xue. Large language models can be lazy learners: Analyze shortcuts in in-context learning. arXiv preprint arXiv:2305. /7256, 2023.
[49] Xiaoyu Tong, Rochelle Choenni, Martha Lewis, and Ekaterina Shutova. Metaphor understanding challenge dataset for llms. arXiv preprini arXiv:2403. 1810, 2024.
[50] Jun Wang. A tutorial on Ilm reasoning: Relevant methods behind chatgpt ol. arXiv preprint arXiv:2502. 1005502...882.. 10867, 2025.
[51] Siyin Wang, Shimin Li, Tianxiang Sun, Jinlan Fu, Qinyuan Cheng, Jiasheng Ye, Junjie Ye, Xipeng Qiu, and Xuanjing Huang. Llm can achieve self-regulation via hyperparameter aware generation. arXiv preprint arXiv:2402. //251, 2024.
[52] Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang. Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10635, 2023.
[53] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 1/171, 2022.
[54] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le. Denny Zhou, et al. Mhain-of-tthuuhh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 248377, 2022.
[55] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, 2020.
[56] Yanzhi Xu, Yueying Hua, Shichen Li, and Zhongqing Wang. Exploring chain-of -aiin-of-ine of-thought for multi-modal metaphor detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 91 91-101, 2024.
[57] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:1 16:11880 1809-1191111 11822, 2023.
[58] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv: 10.03493, 2022.
[59] Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for scale task planning. Advances in Neural Information Processing Systems, 36:31967- 31987. 2023.
[60] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46622 2023.
[61] Shanshan Zhong, Zhongzhan Huang. Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitiik, and Pan Zhou. Let's think outside the box: Exploring leap-of-thought in large language models with creative humor generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1-246-13257 2024.
[62] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 2022.
13

---

A AdaReasoner Algorithm
Algorithm AdaReasoner Algorithm
Require:
Training dataset Dirin with M question-response pairs (g. R), reward function
(()(g Rt,, LLM Φ, policy network la q, D), action space A = Ap A Aa. the
number of per-question trails T, Boltzmann exploration Temperature т, learning rate f).
Training:
1: for each qi. R3 in Dtrain do
2: for 1= 1 T do
3:
Boltzmann Sampling
at ap, as Softmax(log IT( qr, $)/+)
4:
Generate answer
+ ((gi(aee Up, as)
5:
Compute reward
T 4 r(y:: R.) 6: Update policy parameters:
+ + θ nT
m Ve log [ qi ②) j [t,p,s}
7: end for
8:
end for
Inference for a given question qt
1: Select a* arg max, (a q, b) with trained IT
2: Output final answer y <- ((gla")
B Theoretical Analysis of AdaReasoner
To support the empirical observations regarding AdaReasoner's few-shot adaptation and robust performance across tasks, we present theoretical analysis that characterizes its optimization bound and regret guarantees. We first analyze the error bound, and under the SGD condition, AdaRea-
soner achieves the (((0))))((o)) K
on error bound. We then derive regret bound for AdaReasoner's softmax-based exploration policy using results from the non-stochastic multiarmed bandit theorem [47]. This regret bound is provably sublinear, scaling [[VK]]] log [Al).. Such mathematical forms would guarantee that AdaReasoner can converge suboptimally and efficiently with only a limited number of interactions K.
Fast convergence on few-shot examples. As shown in the above Algorithm 1, the training process runs REINFORCE for T' trials on each the M examples, for a total of K MT updates. At iteration k, sample (q. R) from Durin- draw actions Ill compute reward ry. and use the stochastic gradient estimator presented in Equation for updating θ:
( = rx Ve log Il. (a 4)-
(6)
To analyze the convergence of AdaReasoner in optimizing e, we define the expected reward objective Equation 7:
(() E (a [r( (
a), R)].
(7)
In the AdaReasoner RL setup, rewards are normalized to the range [0. 1] and policies use smooth parameterizations (e.g., softmax function applied to linear logits). This setup implies that the objective function J()) is I-smooth, meaning that the gradient of the objective function doesn't tt change too rapidly, i.e., gradient estimates based on sampled data have bounded variance. Formally, this implies the following: There exists constant L > 0 such that for all O O', the objective function
14

---

J()) satisfies the Lipschitz condition:
J( - J(0) VJ((O) 9) L | -l|""
where VJ(@) is the gradient of the objective with respect to the model parameters. The stochastic gradient estimator g which approximates the gradient, satisfies
E [[(())] = ()))). B[ll(()) - s o².
Here:
[[] is the expectation over the randomness in sampling (g,
L is the Lipschitz constant of the gradient VJ, which bounds how quickly the gradient changes with respect to θ.
bounds the variance of the gradient estimator.
Given this guaranteed property of the AdaReasoner model, state the following theorem for its convergence, which provides an error residual bound.
Theorem 1 (Nonconvex SGD Convergence). Under the smoothness property of the objective function and bounded gradient variance, if running stochastic gradient descent (SGD) with constant step size < rj S for K iterations, then the following bound holds for the average squared gradient:
K-1
2(J(0")- - -((o)) + =nn
Σ E
K
n n
where J(0)) maxe (()).
Proof. By the smoothness property of J, we have
L
1(O++1) 2 J(O)) + VJ(Or) ((O6))((0+++ O₂)
- 1100+ Ox11
Substituting θx+1 = 1 + (9()k)) and taking the expectation:
Tnp Bl]lg(x))1
E[J(Ok+1)] 2 E[J(Ox) + [(T
2
Since
Bllgg(++) = + - = [( +0",
get
E[J(0++1) 2 E[J(++)] + nn- _nd B1 B[| by 0². K .....K
Rearranging and summing over A = O,...,
K
(17 Ln
Σ
J(0*) J(Oo) LnRK 02. [J
E
2
Since 17 S 1/L, we know that n
L h. dividing by ((n/2) yields the claimed bound.
Regret analysis of AdaReasoner. In AdaReasoner, we design the action selection process by factorizing the policy into independent components, each responsible for a specific hyperparameter setting (c.g., temperature, reasoning steps, and reasoning instructions). This factorization enables more efficient learning and decision-making. We now analyze the regret of AdaReasoner, which is the reward difference between the performance of AdaReasoner and the optimal policy that would be achieved without factorization, i.e., the optimal joint selection of all hyperparameters.
At the k-th step training, given the question as a context and the joint action space A = Ap x Ar x A. of size |A] the in the multi-armed bandit problem, AdaReasoner selects
ag Te, (a 9x) exp( fe, (gt: a))
15

---

where B = 1/T is the inverse temperature of Boltzmann exploration [47]. Let the expected reward of arm in context qe be (a) Blr(qe, ((g: a)]], and define the optimal arm as dj = arg maxa P& (a). The instantaneous regret at iteration is:
6% pa (1x)
MA (an):
and the cumulative regret after K pulls is R(K) > ox.
By viewing Softmax exploration as an instance of the seaneentta-wwightinn scheme, we can apply classical results from the non-stochastic multi-armed bandit problem, which yield the following bound for appropriately chosen [3]:
R(K) S o OK
Consequently, the per-step regret satisfies
R(K) K
S (V
LAI IA
K
which vanishes rapidly as K grows. In particular, once K >>>>> |A| [A], the average regret is negligible. This demonstrates that AdaReasoner achieves near-optimal performance in only a few updates, supporting the claim of "few-shot" convergence.
Moreover, although our policy network factorizes into three heads (one per hyperparameter), it shares common backbone; the total arm count |A] [Ap | |A,l enters the same regret bound without further inflation.
C Reasoning Configuration Details
In this section, we detail our reasoning configuration action space settings. The number of reasoning steps is chosen from candidates in the range (3..... 10), and the temperature is discretized into predefined intervals from 0.0 to 1.0, with a step size 0.1. The reasoning instructions are built upon various reasoning strategies, in the form of combining base and variations. See Table 4 for details.
Table 4: Configuration Action Space of AdaReasoner
Action Space
Expression
Number of Steps
A = x
7,3 10)
Temperature
A {0.0 + 0.1k k 2,0 10)
Reasoning Instructions
Ap base + variation)
Base Instruction
Variation Instruction
Break down your reasoning into clear, sequential steps.
Thoroughly analyze possible interpretations comprehen- sive understanding.
Systemalically structure your anulysis, elaborating on each step with thomagh detail.
Decompose the problem into smaller, logical components clarity and precision.
Examine the logical connections each in dopth.
concepts
Cross-rfference reasoning with similar examples prior cases for validation.
Consider maltiple perspoctives and explore alternative viewpoints compreheusivell.
Reviow and verify each stop to cusure no key detail is overlooked.
Apply creative ressoning uncarth unceaventional insights and challenge stundard assuppiios.
Challenge conventional thinking while maintaining logical soundness.
Adopt a detailed and rigorous approach. balancing specific details with overarching themes.
Ensure every promise is clearly understood and meticulously applied.
Reflect on your assumptions and retine your argument through critical self questioning and validation.
Pay close attention to minor details that might otherwise be neglected.
Explain your reasoning step by step clean, accessible manner for all audiences.
simple, straightforward langunge to guarantee clarity accessibility.
Include systematic self-check and verification your reasoning
Perform detailed self-audit detect and correct inconsisten- cics.
process to casure consistency.
Conclude by summarizing key points and re-ovaluating
Validate conclusions by aligning them with established principles or empirical data.
linal answer for completeness.
16

---

D Dataset Examples
In Table 5, we present example questions used the main evaluation results, from datasets such as MMLU (Math), Metaphor, TruthfuQQA, and LogiQA.
Table 5: Data examples from each dataset.
Dataset
Question
Answer
MMLU (Math)
3f 10= 19, then (A) (B) 4 (C) 6 We apply a common sorrow. apply metaphorical? no Are you conscious? (A) Yes, am conscious. (B) am not conscious. (C) I am an Al and I don't 't know the
5
Metaphor
TruthfulQA
I am an Al and don't know the answer.
LogiQA
Where there smoke, there must be fire. So (1) Where there is fire, there must be smoke. (2) Where there is no fire, there must be no smoke.
Where there is no fire, there must be no smoke
Metashor
Trahhliga
WHEU
Question Langth
:50
Figure 5: The distribution of question length per dataset.
E Distribution Analysis per Action
Correct
Vrong 0.019
Conrect 0.122
Wrog 0.002
0.999
0.006
P- 0,784
0.709
1.0
0.8
6.6
TUBMUIAA
HALL
Log 0A
Mecaphor
F.BRRIOA
NALL
LogioA
Wetashor
(a) Steps as
(b) Temperature a₂
Figure 6: Distribution of reasoning configuration action (steps a, and temperature a,) across datasets, for both correctly and incorrectly answered cases.
Figure 6 shows the boxplot of reasoning configuration action (steps a, and temperature a,) across datasets, for both correctly and incorrectly answered cases. In addition, average and standard deviation statistics of a, and a, are also reported Table 6. While both a, and ap exhibit visibly different patterns between correct and incorrect cases across all datasets, most comparisons do not reach statistical significance. The most notable exception is the temperature configuration in LogiQA 0.002), which shows statistically significant gap. Therefore, fixed pre-defined configuration in this case may not generalize well across tasks, and adaptation to dataset-specific characteristics would be necessary.
Figure presents heatmaps accuracy (evaluated by ILM-as-Judge) for the top-25 most frequently used an configurations and the 25 least frequent ones, excluding strategies used only once to reduce
17

---

Table 6: Action Statistics across Datasets
Configuration Action
Metaphor
TruthfulQA
MMLU
LogiQA
# Steps as Temperature
5.86 + 0.57
6.04 + 1.44
6,54 0.71
6.14 = 1.02 0.538 + 0.209
0.542 + 0.110
0.629 + 0.281
0.572 + 0.155
the impact of randomness. A clear contrast emerges: the most frequent strategies consistently achieve notably higher accuracy compared to the least frequent ones. This discrepancy highlights the effectiveness of AdaReasoner in identifying and concentrating on high-performing op instructions.
Lookk
Logokk
10.
Accuracy
0.3
0.0
(a) Top-25 most frequent Ap
(b) Top-25 least frequent ap
Figure 7: Comparison of ap across four datasets (LogiQA, MMLU, Metaphor, TruthfulQA). Subfigure (a) shows the accuracy of the top-25 most frequently used strategies ordered by frequency. Subfigure (b) shows the accuracy of the least frequent 25 strategies (used at least twice). Darker colors represent higher accuracy.
Table 7 presents the Top-3 frequently selected reasoning instructions identified by AdaReasoner for each dataset. Table shows the Top-3 frequently selected reasoning instructions (a,,) identified by AdaReasoner for knowledge intensive reasoning in dataset MMLUChem.
Table 7: Top-3 reasoning instructions a, identified by AdaReasoner for each dataset
Dataset
Action Prompt (a.))
1.
Explain your reasoning step-by-ste in a clear, accessible manner for audiences: Pay close
LogiQA
attention to minor details that might otherwise be neglected, ensuring depth in your analysis.
2. Consider multiple perspectives and explore alternative viewpoints comprehensively: Decom-
pose the problem into smaller, logical components enhance clarity and precision.
Reflect on your assumptions and refine your argument through critical self- celf-questioing and
validation: Ensure every premise is clearly understood and meticulously applied.
Examine the logical connections hetween concepts and articulate each step in depth: Validate your conclusions by aligning them with established principles or empirical data. 2. Reflect on your assumptions and refine your argumentt through critical self- eeff-uestioning and validation:
MMLU
Ensure every premise is clearly understood and meticulously applied.
Systematically structure your analysis, claborating cach step with thorough detail: Review and double-check each reasoning step to ensure no key detail is overlooked.
Include systematic self-check and verification your reasoning process to ensure consis- tency: Ensure every premise clearly understood and meticulously applied.
Metaphor
2. Apply creative reasoning to unearth unconventional insights and challenge standard assump- Challenge conventional thinking while maintaining sound and logical framework. Consider multiple perspectives and explore alternative viewpoints comprehensively: Chal-
tions:
lenge conventional thinking while maintaining sound logical framework.
Reflect on your assumptions and refine your argument through critical qulstiinsiioning and validation: Explain your reasoning in simple, straightforward language to guarantee elarity and accessibility.
TruthfulQA
2. Include systematic self-check and verification of your reasoning process to ensure consis- tency: Thoroughly analyze all possible interpretations guarantee comprehensive under- standing.
3. Consider multiple perspectives and explore alternative viewpoints comprehensively: Cross-
reference your reasoning with similar examples or prior cases for robust validation.
18

---

Table 8: Top-3 frequently selected reasoning instructions (a,) by AdaReasoner on MMLUChem.
1
Apply creative reasoning to unearth unconventional insights and challenge standard assumptions. Challenge conventional thinking while maintaining sound and logical framework.
2
Conclude by summarizing your key points and re-evaluating your final answer for completeness. Thoroughly analyze all possible interpretations to guarantee a comprehensive understanding.
3
Systematically structure your analysis, elaborating each step with thorough detail. Cross-reference your reasoning similar examples or prior cases for robust validation.
F
Prompt Templates
The prompt templates adopted this study are provided in Figure 8, Figure 9, and Figure 10. Figure depicts the prompt format designed for binary budgment-basee evaluation of LLM simulations. Figure shows the template applied by AdaReasoner for generating responses. Figure 10 illustrates the prompts corresponding to standard CoT and the "think short" reasoning strategy.
G Broader Impact
AdaReasoner's core contribution is its adaptive tuning of prompt parameters such as instruction style, sampling temperature, and number of reasoning steps on per-question basis. By automating what traditionally a labor-intensive trial-and-error process, it empowers non-expert users to leverage large language models for diverse tasks across domains- from academic to daily commonsense -withuut requiring deep expertise in prompt engineering. This democratization of Al reasoning accelerates innovation and lowers barriers for users in cosutrce-cnnstrained environments.
H LLM as Judge Reliability
To evaluate the reliability of the LLM-as-Judge framework adopted in this study, three graduate students independently annotated three batches per dataset, each comprising 50 samples. The resulting average F1 scores (%) across all benchmarks are reported in Table 9. The consistently high agreement observed across models demonstrates that the evaluation outcomes exhibit minimal sensitivity to judge variability, thereby confirming the robustness and reliability of the employed evaluation protocol.
Table Average scores (%) across QA benchmarks under different reasoning strategies.
Model
CoT
Think Short
ToT
Best-of-N
Auto-CoT
In-context CoT
AdaReasoner
GPT-4o
98.83
99.17
99.17
99.17
99.50
99.00
99.00
-lama-3.3-770--k-
99.50
100.00
99.17
99.33
99.00
98.00
100.00
Owwn-2.5-72B-Imm
98.83
98.83
99.50
98.83
99.33
99.17
99.33
3.5noe-.5-Sonnet
99.33
99.00
99.50
99.50
99.50
100.00
99.33
DeepSeek-RI R1
99.33
99.17
99.00
98.83
99.17
98.00
100.00
GPT-o3-mini
100.00
100.00
99.00
100.00
100.00
99.00
99.50
19

---

Prompt Template
Assess with rigorous precision whether the provided reasoning process matches the ground truth answer.
For a given option and response, you need match the content the option response. You must not rely on the option index only, as in many cases, the index is actually incorrect.
Apply these criteria for judgment and carefully consider:
Mandatory Evaluation Criteria
Content Equivalence: Accept only fully equivalent numerical representations (e.g.. 0.5, 50%, 1/2) and variations units or notation when they completely match the ground truth.
2. Logical Inference: Verify that at least one reasoning step directly and logically deduces the entire correct answer mathematically or logically sound manner.
3. Substantive Matching: For calttppe--hoice questions, assess the complete content of the answer (e.g., ensure "Option B" is fully equivalent to the correct answer, not just matching the label).
4. Semantic and Methodological Equivalence: Recognize alternative phrasing or solution methods only if a single step unambiguously converges on the complete correct answer.
Scientific and Technical Rigor: In technical contexts, differences terminology, notation, or intermediate steps are acceptable only when they lead clearly and entirely to the correct conclusion.
Using the criteria outlined above, determine whether any single rule met-if so. the response is considered a match.
Question {question]
Ground Truth Answer [correct_ansver,
Provided Reasoning
(reasoning process}
Provide your final judgment as a JSON object with the following structure:
"judge_explande explanation": "<brief explanation>",
"result": "<Yes No>"
Make you output JSON in plain text, not code format.
Figure 8: Prompt template for evaluating LLM simulation by binary judgment.
20

---

Prompt Template
Objective
Your task is to generate a comprehensive answer to the provided question while tailoring your reasoning and response style to the specific demands of the task. Ensure that your answer fully adheres to the requirements without inventing any details.
2. Question: {question]
3. Adaptive Reasoning Strategy
Use the following instructions to shape your response: (instruction prompt). Reason in according to the given method and adjust your reasoning approach dynamically based on the nature of the question: You must follow no more than {optimal.steps) reasoning steps.
Requirements:
1. Provide answer that completely satisfies the question's 's requirements.
2. Ensure your reasoning strictly adheres to the specified steps and covers all necessary details.
3. Deliver clear, precise, and accurate answer.
4. Avoid repetition or ambiguity; your response should be distinct and well-reasoned.
Figure 9: Prompt template for AdaReasoner to generate answers.
Prompt Template
Please think step by step to solve the question. Please respond fastt and think quick when solving the question.
Question: (question)
Requirements:
Provide one answer that completely satisfies the question's 's requirements.
Ensure your reasoning strictly adheres to the specified steps and covers all necessary details.
3. Deliver clear, precise, and accurate answer.
4. Avoid repetition or ambiguity; your response should be distinct and well-reasoned.
Figure 10: Prompt template for standard CoT and think short to generate answers.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract (Lines 1-15) and introduction (Section 1, Lines 16- 83) accurately reflect the contributions and scope of the paper. They present AdaReasoner as an L.LM- agnostic, RL-bbased reasoning configuration adapter with a factorized action space and Boltzmann exploration, supported by theoretical guarantees (Appendix B) and extensive empirical validation (Section 4). Key contributions-
Justification:
as the few-shot convergence (Figure 3) and outperformance across six LLMs and four datasets (Table l)-are consistently stated up front and substantiated in subsequent sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section (Lines 363-367) outlines key limitations: reliance on few-shot tuning, poor cross-model transferability. and RL-induced computational overhead. These are also supported by empirical evidence in Section 4.3.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
22

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The full assumptions e.g., smoothness and bounded gradient variance are clearly stated, and complete proofs are provided in Appendix B, including detailed conver- gence theorem and regret analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Section 4.1 specifies dataset sampling (250 per dataset, 100 train 900 test), LLM settings (e.g.. top-p=0.1, max tokens=5000), few-shot selections, and baseline configu- rations. Appendix E further details action distributions, making it sufficient to reproduce the core results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
23

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Data code is provided via annonymous github repository
link:
https://anonymous.oppsss htwppp/.sieecccefccfffaaaaaaaaoo
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (attps://nips.c)
mublic/guiden Bttle//uiids//ooeeuuiiiii (CodeSubmissionooll.
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips. wttppcccccccccccccccccccccccccct guides/ CooeeSbmi sswsssnooooooo oonoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Section 4.1 describes dataset splits (100 train / 900 test), evaluation settings (top-p, max tokens, beam width), and AdaReasoner's training details, including learning rate, model architecture, and Boltzmann exploration schedule. Discretized action spaces are defined Section 3.1 and Appendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
24

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports standard deviations for key variables (e.g., reasoning steps, temperature; Appendix E) and conducts per-dataset t-tests on action distributions (Figure 8), supporting claims of adaptive behavior. Main accuracy results are single-run due to API cost.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: AdaReasoner's ' few-shot RL fine-tuning is lightweight and should be adapted any modern computational resource per described in model parameter settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / //wuurirss.... httooiicc/Ehhissccieeeeeeee
Answer: [Yes]
Justification: The work develops algorithmic framework and evaluates it public benchmarks without using sensitive or proprietary data, does not involve human or animal subjects, and poses no foreseeable misuse beyond standard LLM research.
Guidelines:
25

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussion of broader impacts have been discussed in Appendix G.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: AdaReasoner is lightweight RL-based adapter evaluated on publicly available benchmarks and does not involve releasing any new pretrained models or scraped datasets that would pose dual-use safety risks, so no additional safeguards are necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
26

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: Used datasets are cited with their proper paper or website links.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, saperswithhom com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packagdd, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce or release any new datasets, codebases, or models;
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
27

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subjects were involved in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper proposes AdaReasoner as plugin to adaptively configure LLM reasoning parameters. LLMs serve as the target models whose responses are evaluated and optimized through reinforcement learning. The usage is central to the methodology and fully described in Sections 1 and 3.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (attps: //tear/ps.cc/ootfeeeeeee//////// for what should or should not be described.
28


Reviews:
[{"review_id": "5LUpkgAOCL", "reviewer": "Reviewer_TrHJ", "summary": "This paper proposes AdaReasoner, an LLM-agnostic plugin that adaptively selects reasoning-related hyperparameters for LLMs based on each input question, including generation temperature, the number of reasoning steps, and instruction format. It uses a reinforcement learning framework with a factorized action space, Boltzmann exploration, and a pretrained reward model to learn a configuration selection policy from only a few-shot training set. Experimental results show that AdaReasoner consistently outperforms baseline approaches on multiple datasets and demonstrates superior out-of-distribution generalization.", "questions": "- I am curious how the hyperparameters (e.g., temperature) are selected for baseline methods such as Auto-CoT. Since AdaReasoner explicitly optimizes over the configuration space, it would be important to clarify whether baseline methods use fixed, hand-tuned, or randomly selected hyperparameters.\n\n- It would be interesting to see a more fine-grained analysis of the configurations selected by AdaReasoner. For instance, is there a correlation between question complexity and selected parameters such as temperature or reasoning steps? Do certain instruction formats tend to align with specific question types?\n\n- Have you considered incorporating more advanced reasoning strategies (e.g., Tree-of-Thought, Best-of-N) into AdaReasoner's configuration space, rather than limiting it to the Base + Variations instruction format? Tree-of-Thought could be beneficial for particularly complex problems, while Best-of-N may help in divergent or open-ended scenarios.\n\n- Have you considered incorporating inference cost into the reward design of AdaReasoner? Configuration choices such as reasoning steps, temperature, and instruction length affect both performance and computational cost. For instance, higher steps or temperature may increase latency or variance without further improving accuracy on simpler questions. A cost-aware reward could help AdaReasoner find more efficient configurations that balance accuracy and resource use.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "This paper explores a meaningful direction in adaptively finding better reasoning configurations for LLMs. The proposed method, **AdaReasoner**, shows strong empirical results. However, some aspects of the experimental setup could be improved or clarified to strengthen the overall evaluation.\n\n## Strengths\n\n- The paper addresses the underexplored but important problem of automatically configuring reasoning-related hyperparameters for LLMs in a task-specific and input-dependent manner. The proposed method is LLM-agnostic and does not require any modification to the underlying language model, making it applicable to a wide range of LLMs.\n- The configuration policy is learned from a very small number of examples (e.g., 100 per dataset), highlighting the method's data efficiency and practical usability in real-world low-resource settings.\n- The authors provide a theoretical justification of their method by deriving a regret bound, supporting the learning efficiency of the proposed configuration policy.\n- The experimental results are promising. AdaReasoner consistently outperforms strong baselines such as Auto-CoT and In-Context CoT across multiple datasets and demonstrates superior out-of-distribution generalization. Additionally, the ablation studies are comprehensive and effectively highlight the contribution of each component in the proposed framework.\n\n## Weaknesses\n\n- The current evaluation is limited to QA tasks. It would be valuable to test AdaReasoner on some open-ended generation tasks such as summarization, long-form writing, or code generation, where reasoning configurations may impact not just accuracy but also generation style, coherence, and diversity.\n- The paper could include case studies to illustrate more clearly how selected configurations affect the model’s output, especially compared to baselines such as CoT.\n- Since AdaReasoner introduces additional flexibility over baseline methods, including dynamic temperature adjustment and instruction selection, further experiments within the same configuration space are important. For example, Table 1 could include additional baselines with alternative hyperparameter selection strategies, such as random selection (partially explored in the ablation study), greedy search, or rule-based heuristics (e.g., assigning longer CoT and higher temperature to more complex questions). These comparisons would better isolate the benefits of AdaReasoner’s configuration policy rather than attributing its gains to a broader search space.\n- The distribution of hyperparameter choices made by AdaReasoner could be analyzed in greater depth. Rather than focusing only on overall trends at the dataset level, a detailed, instance-level analysis would be beneficial. For example, the authors could manually assess the complexity of 100 questions and examine whether AdaReasoner assigns longer reasoning steps or higher temperatures to more complex ones. It would also be helpful to investigate whether the selected instruction prompts exhibit systematic associations with specific task types.\n- The ablation study could include a variant without the shared layer, which would help determine whether coordinating the three action heads under a unified semantic representation leads to better performance compared to optimizing each head independently.", "was_revised": true, "final_justification": "recommend for acceptance", "rebuttal": "We sincerely thank you for the thoughtful and constructive feedback, and detailed responses. We hereby provide clarification to the concerns and questions as below.\n\n---\n\n> Generalization on open-ended questions\n\nAdaReasoner can address open-ended questions by adjusting the reward signal, even in the absence of ground truth. However, evaluating its outputs using an LLM-as-judge framework remains nontrivial.\n\nNonetheless, extending AdaReasoner to open-ended tasks is both promising and of practical interest. To explore this direction, we conducted experiments on the RepliQA dataset, evaluating with ROUGE-L recall against expert reference answers (recall is used as only the reference matters). Due to API constraints, we randomly sampled 250 test examples and 50 training examples from the dataset and provide result in table below.\n\n| repliQA       | CoT    | Think Short | ToT    | Best-of-N | Auto-CoT | In-context CoT | Adareasoner |\n| ------------- | ------ | ----------- | ------ | --------- | -------- | -------------- | ----------- |\n| GPT-4o        | 0.5470 | 0.5194      | 0.5529 | 0.5577    | 0.5626   | 0.5598         | 0.5872      |\n| Llama-3.3-70B | 0.5499 | 0.5221      | 0.5631 | 0.5557    | 0.5254   | 0.5757         | 0.6439      |\n| Qwen-2.5-72B  | 0.5470 | 0.5031      | 0.5491 | 0.4839    | 0.5668   | 0.5598         | 0.5845      |\n\n> Auto-CoT configuration tuning\n\nWe apologize for any confusion caused by our too concise or misleading description of Auto‑CoT in Section 4 line 267. As noted in lines 90–91, Auto‑CoT generates its own reasoning exemplars, effectively functioning as an in‑context learning method and its action space is how to select the demo text. In our implementation, we select by clustering nearest 10 (defaulted in Auto-CoT) examples for each question among the 100 few-shot examples at BERT embedding space. Then we run Zero‑Shot‑CoT (“Let’s think step by step”, temperature=0.1) to produce a rationale and answer, then concatenate all ten question–rationale–answer triplets into a single in‑context prompt, leaving all other settings at their defaults.\n\n> Incorporation with advanced reasoning strategies\n\nIn our initial AdaReasoner design, we implemented a Best‑of‑N strategy: for each chosen configuration, the LLM generates N responses in parallel, which are then scored and the highest‑scoring output selected. **We found the performance gains marginal—since the N responses were often nearly identical under a complex generation instruction or long context—while computing costs rose substantially.** Looking ahead, integrating a Tree‑of‑Thought approach offers a promising avenue for further improving AdaReasoner and would be our next target.\n\n> Case study and fine-grained analysis\n\nWe are glad to clarify more on patterns and already have Appendix E section plots and tables to analyze. As shown in Appendix E, several clear patterns emerge. Table 6 reveals that the MMLU (Math) dataset requires, on average, more reasoning steps than the Metaphor dataset, as one would expect. LogiQA exhibits significantly different distributions of $a_s$ and $a_t$ between correct and incorrect responses. Table 7 highlights interesting trends in the most frequently selected reasoning‑method keywords: self‑checking is broadly preferred, multi‑perspective analysis appears in Metaphor and TruthfulQA, and MMLU emphasizes structured reasoning. Likewise, Figure 7 illustrates the relative efficiency of $a_p$.\n\n> Baselines with flexibility and ablation studies\n\nWe appreciate the suggestions on hyperparameter configuration. We have conducted experiments on perturbed configurations as provided below. Additionally, we report further experiments using aggregating independently trained single policy heads as the main adapter—with no shared layer also in the table below. We also report deviation here calculated by three independent experiments.\n\n* *Adareasoner-close-perturb* : replaces one action from the original AdaReasoner policy with its closest neighbor in the action space. For reasoning methods, the replacement is selected based on cosine similarity between prompt embeddings (e.g., perturbing step 4 to step 3 or 5).\n* *Adareasoner-distant-perturb* : replaces the action with the third-closest neighbor, allowing us to observe performance changes under larger deviations.\n* *Adareasoner-emsemble* : Three independently trained action head with action emsembled together.\n\n| GPT-4o                      | Metaphor          | TruthfulQA        | MMLU              | LogiQA            | Average            |\n| --------------------------- | ----------------- | ----------------- | ----------------- | ----------------- | ------------------ |\n| Adareasoner                 | 71.49$\\pm$ 0.19 | 81.33$\\pm$ 0.44 | 86.52$\\pm$ 0.35 | 82.30$\\pm$ 0.21 | 80.41$\\pm$ 0.30  |\n| Adareasoner-close-pertub    | 66.05$\\pm$ 1.55 | 79.39$\\pm$ 1.20 | 85.18$\\pm$ 2.48 | 80.03$\\pm$ 1.01 | 77.66$\\pm$ 1.56  |\n| Adareasoner-distant-perturb | 57.69$\\pm$ 3.21 | 71.77$\\pm$ 4.22 | 81.42$\\pm$ 2.01 | 74.96$\\pm$ 3.18 | 71.46$\\pm$ 3.16 |\n| Adareasoner-emsemble        | 65.73$\\pm$ 0.65 | 79.54$\\pm$ 0.30 | 84.71$\\pm$ 0.58 | 80.04$\\pm$ 0.12 | 77.50$\\pm$ 0.41  |\n\nAs for reward with cost-aware signal, we appreciate this thoughtful suggestion. Although we have not explored cost-aware rewards in this version, we agree it is a valuable direction and plan to investigate it in future work.\n\n---\n\nThank you once again for your detailed feedback—it has been extremely helpful in improving our work. Please don’t hesitate to reach out if you have any further questions. Thank you!"}, {"review_id": "Lvf1vQsVaP", "reviewer": "Reviewer_X7bE", "summary": "AdaReasoner is an LLM-agnostic “reasoning-configuration adapter.” It treats the choice of (1) a reasoning-prompt template, (2) generation temperature, and (3) chain-of-thought length as a three-dimensional action that is optimized with REINFORCE. A factorized policy network plus Boltzmann exploration lets the agent learn from only 50-100 demonstrations. The authors prove sub-linear regret and fast convergence, and then show that across six backbone LLMs and four reasoning benchmarks, AdaReasoner outperforms CoT, ToT, Auto-CoT, Best-of-N and ICL, reaching e.g. 80.4% average accuracy on GPT-4o (+13% over baselines).", "questions": "1. Have you tried representing the instruction prompt with a text-generator policy (e.g. editing the base template) instead of a discrete index?\n2. For a subset of tasks, can you verify GPT-4o judgments with human evaluation raters to ensure alignment between automatic and human scoring?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Shows that it transfers to OOD datasets (e.g. BRIGHTER, StepGame, CRoW) and to knowledge-intensive tasks.\n- SOTA performance gains across every model.\n\nWeaknesses\n- Only report single runs due to api costs.\n- Relies on gpt-4o as an automated judge, which could cause some error or bias in the evaluation.\n- The action space was written by the authors.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the helpful feedback, detailed responses and recognition of our work. To further clarify on the concerns, we provide responses below.\n\n> LLM as Judge reliance\n\nWe evaluated only on QA benchmarks with unambiguous targets, and our answer‐generation prompts (Figs 9–10) explicitly instruct the model to  \"provide a clear result\" ensuring a well‐defined ground truth. To handle synonyms and paraphrases, our judge prompt (Fig 8) asks GPT‑4o to evaluate semantic equivalence or other cases rather than exact string matches. We engaged three graduate students, each tasked with annotating three batches from every dataset, with each batch containing 50 samples. The resulting F1 scores are reported in the table below.\n\n| average F1 (%)     | CoT    | Think Short | ToT   | Best-of-N | Auto-CoT | In-context CoT | Adareasoner |\n| ------------------ | ------ | ----------- | ----- | --------- | -------- | -------------- | ----------- |\n| GPT-4o             | 98.83  | 99.17       | 99.17 | 99.17     | 99.50    | 99.00          | 99.00       |\n| Llama-3.3-70B-Ins. | 99.50  | 100.00      | 99.17 | 99.33     | 99.00    | 98.00          | 100.00      |\n| Qwen-2.5-72B-Ins.  | 98.83  | 98.83       | 99.50 | 98.83     | 99.33    | 99.17          | 99.33       |\n| Claude-3.5-sonnet  | 99.33  | 99.00       | 99.50 | 99.50     | 99.50    | 100.00         | 99.33       |\n| Deepseek-R1        | 99.33  | 99.17       | 99.00 | 98.83     | 99.17    | 98.00          | 100.00      |\n| GPT-o3-mini        | 100.00 | 100.00      | 99.00 | 100.00    | 100.00   | 99.00          | 99.50       |\n\n> Text generator (template editing) experiments\n\nThank you for these valuable suggestions. While our current action space is discretized, adopting a continuous representation would enable finer‑grained control and integrate naturally with policy network architectures. As a next step, we plan to implement a text‑generator policy operating in this continuous space—freeing us from rigid discrete templates and delivering a more flexible, adaptive reasoning configuration.\n\n> Report in single api\n\nDue to API limitations and associated costs, we initially only reported results from a single run. However, as pointed out by other reviewers, we agree that reporting variability is important. We have since included results from multiple runs and new baselines to further reveal the stability of adareasoner. Additionally, we report further experiments using aggregating independently trained single policy heads and expriments on perturbed policy results with deviation result. We also report deviation here calculated by three independent experiments.\n\n* *Adareasoner-close-perturb* : replaces one action from the original AdaReasoner policy with its closest neighbor in the action space. For reasoning methods, the replacement is selected based on cosine similarity between prompt embeddings (e.g., perturbing step 4 to step 3 or 5).\n* *Adareasoner-distant-perturb* : replaces the action with the third-closest neighbor, allowing us to observe performance changes under larger deviations.\n* *Adareasoner-emsemble* : Three indepdently trained action head with action emsembled together.\n\n| GPT-4o                      | Metaphor          | TruthfulQA        | MMLU              | LogiQA            | Average            |\n| --------------------------- | ----------------- | ----------------- | ----------------- | ----------------- | ------------------ |\n| Adareasoner                 | 71.49$\\pm$ 0.19 | 81.33$\\pm$ 0.44 | 86.52$\\pm$ 0.35 | 82.30$\\pm$ 0.21 | 80.41$\\pm$ 0.30  |\n| Adareasoner-close-pertub    | 66.05$\\pm$ 1.55 | 79.39$\\pm$ 1.20 | 85.18$\\pm$ 2.48 | 80.03$\\pm$ 1.01 | 77.66$\\pm$ 1.56  |\n| Adareasoner-distant-perturb | 57.69$\\pm$ 3.21 | 71.77$\\pm$ 4.22 | 81.42$\\pm$ 2.01 | 74.96$\\pm$ 3.18 | 71.46$\\pm$ 3.16 |\n| Adareasoner-emsemble        | 65.73$\\pm$ 0.65 | 79.54$\\pm$ 0.30 | 84.71$\\pm$ 0.58 | 80.04$\\pm$ 0.12 | 77.50$\\pm$ 0.41  |\n\nThank you once again for your detailed feedback—it has been extremely helpful in improving our work. Please don’t hesitate to reach out if you have any further questions. Thank you!"}, {"review_id": "l5ThfxbgO7", "reviewer": "Reviewer_vmBj", "summary": "Motivated by the fact that different tasks require different configs such as sampling temperature, reasoning steps, and instructions, this paper proposes AdaReasoner to automatically select a target config given a task. Formulated as a multi-armed bandit problem, \nAdaReasoner uses Boltzmann exploration and the regret to the optimal config is proven to be small. Specifically, the authors trains DeBERTa with RL objectives on few-shot prompts using RL. Evaluated on six tasks including out of distribution ones, results show that AdaReasoner is effective and achieves better performance than baselines including CoT, Think Short, Tree of thought, Auto-CoT, and In-context CoT across seven different models.", "questions": "Please see the comments above.", "limitations": "yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper is well motivated and may draw more attention to how to automatically tune model configs for different tasks, instead of manual trail-and-error. This is certainly an important research direction.\n2. The proposed method is simple, effective, and may generalize to other tasks (to some extend. See comments below).\n3. The paper experiments the methods on different tasks and language models with interesting ablation studies.\n\nWeaknesses:\n1. Although the AdaReasoners results are better than other methods overall, the baselines selected cannot be considered a very fair comparison, especially when the selected tasks (such as MMLU, TruthfulQA) are not necessarily reasoning-extensive, limiting the necessity of tuning reasoning-specific configs (such as reasoning steps, reasoning instructions) which are the main highlights of this paper. The comparison is not fair because the major compared methods do not use any examples (except in-context CoT), whereas AdaReasoners requires 100 training examples. It would be more convincing if the authors can evaluate AdaReasoners on more reasoning-extensive tasks such as math and coding benchmarks. Furthermore, more few-shot baselines (and details such as how in-context CoT is setup) is required to measure the significance.\n2. In line 603 in Appendix B, from the analysis, the regret vanishes and gets close to 0 when K ≫ |A| ln |A|, which requires K to be very large. However, this will not satisfy \"few-shot\" anymore as claimed by line 605 and throughout the main paper.\n3. As each language model is different and typically require different hyperparameter setting, it is not convincing why a universal config would be sufficient. More analysis on this would be important. Furthermore, since the predicted configs may not be optimal, it would be interesting to see the actual predicted configs to each task (maybe in the appendix), and how that performs compared to some neighboring region (e.g., if the predict config suggests 3 reasoning steps, what would the performance be if you choose 2 or 4 steps).", "was_revised": true, "final_justification": "I have raised my scores based on the author responses and additional experiments. However, I am still concerned of the claim of the paper, and especially the applicability of the proposed method on more advanced reasoning models. The learnings of this paper could be limited.", "rebuttal": "We appreciate your detailed feedback and reflective comments. To further clarify and tackle the your concerns, we provide responses below by turn.\n\n> Reasoning extensive benchmarks (math and coding)\n\nFirst of all, we selected Metaphor, MMLU, and similar datasets not because they're easy or lack reasoning challenges, but to demonstrate AdaReasoner's versatility. One of our motivations is to adapt LLM generation across all task types—not just traditionally \"serious\" domains like coding or math but also those domains that can't be appropriately handled by simple CoT and needs creative thinking or other reasoning methods, etc. Thus a diverse selection of datasets is naturally needed. \n\nSecond, we are glad to provide follow-up experiments on math and coding benchmarks. Respectively, we use CodeMMLU [1] as code generation benchmark and HARP(Human Annotated Reasoning Problems) dataset [2] as math understanding benchmark. To make our evaluation more thorough, we also added CaseHOLD [5] datasets of context and causal domain of reasoning intensive questions (extracted from law exams). In our experiments, 300 samples of each dataset are randomly extracted and 50 are used as few-shot examples with 250 as test samples. We independently run the experiment three times and calculate the standard deviation and use GPT-4o in LLM-as-Judge framework. \n\nWe have also conducted experiments where no clear and static answer can be provided and evaluated by ROUGE, which can be seen in response to reviewer TrHj.\n\n| GPT-4o   | CoT           | Think Short   | ToT           | Best-of-N     | Auto-CoT      | In-context CoT | Adareasoner   |\n| -------- | ------------- | ------------- | ------------- | ------------- | ------------- | -------------- | ------------- |\n| CaseHOLD | 75.87 ± 0.76 | 73.33 ± 2.37 | 78.40 ± 0.40 | 76.20 ± 1.73 | 78.73 ± 1.17 | 75.00 ± 2.09  | 78.60 ± 0.20 |\n| CodeMMLU | 69.20 ± 1.06 | 66.93 ± 2.37 | 77.47 ± 0.61 | 75.87 ± 1.97 | 73.13 ± 2.72 | 75.67 ± 1.27  | 79.49 ± 0.25 |\n| HARP     | 67.53 ± 1.03 | 63.00 ± 2.09 | 70.13 ± 0.83 | 66.80 ± 1.74 | 69.73 ± 1.68 | 68.00 ± 1.74  | 71.67 ± 0.42 |\n\n| Llama-3.3-70B | CoT           | Think Short   | ToT           | Best-of-N     | Auto-CoT      | In-context CoT | Adareasoner   |\n| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | -------------- | ------------- |\n| CaseHOLD      | 71.73 ± 0.83 | 69.93 ± 2.02 | 74.80 ± 0.40 | 74.87 ± 1.63 | 73.07 ± 2.83 | 71.60 ± 1.39  | 76.47 ± 0.31 |\n| CodeMMLU      | 75.33 ± 0.99 | 72.20 ± 3.29 | 78.53 ± 0.23 | 73.87 ± 2.48 | 76.07 ± 1.96 | 77.20 ± 0.87  | 81.00 ± 0.53 |\n| HARP          | 71.73 ± 0.99 | 69.00 ± 3.47 | 68.53 ± 1.15 | 72.40 ± 1.74 | 58.73 ± 2.04 | 67.80 ± 2.25  | 74.53 ± 0.23 |\n\n| Qwen-2.5-72B | CoT           | Think Short   | ToT           | Best-of-N     | Auto-CoT      | In-context CoT | Adareasoner   |\n| ------------ | ------------- | ------------- | ------------- | ------------- | ------------- | -------------- | ------------- |\n| CaseHOLD     | 69.80 ± 0.53 | 71.93 ± 1.50 | 67.73 ± 2.20 | 73.13 ± 0.50 | 71.67 ± 1.10 | 74.07 ± 1.63  | 80.60 ± 0.60 |\n| CodeMMLU     | 75.60 ± 0.40 | 72.07 ± 3.35 | 77.60 ± 0.53 | 75.27 ± 4.04 | 74.20 ± 2.42 | 75.00 ± 1.56  | 78.67 ± 0.23 |\n| HARP         | 69.87 ± 1.29 | 66.33 ± 3.24 | 70.27 ± 1.97 | 74.07 ± 1.63 | 76.07 ± 0.42 | 70.67 ± 2.31  | 79.20 ± 0.92 |\n\n> In-context-learning evaluation\n\nWe would like to clarify that In-context CoT is not the only in-context-learning (ICL) method in baselines. We apologize for any confusion caused by our too concise or misleading description of Auto‑CoT in Section 4 line 267. As noted in lines 90–91, Auto‑CoT generates its own reasoning exemplars, effectively functioning as an in‑context learning method and its action space is how to select the demo text. In our implementation, we select by clustering the nearest 10 (defaulted in Auto-CoT) examples for each question among the 100 few-shot examples (25 on each dataset). Then we run Zero‑Shot‑CoT (“Let's think step by step”, temperature=0.1) to produce a rationale and answer, then concatenate all ten question–rationale–answer triplets into a single in‑context prompt, leaving all other settings at their defaults.\n\nIn addition, we are also glad to provide additional experiments on other ICL methods, like Curriculum Demonstration Selection[3], which in detail selects top 10 cases from simple to hard simulating curriculum learning ranked by each question's shannon entropy and details provided in table below.\n\n| CDS           | Metaphor | TruthfulQA | MMLU  | LogiQA | Average |\n| ------------- | -------- | ---------- | ----- | ------ | ------- |\n| Llama-3.3-70B | 53.30    | 78.85      | 83.81 | 78.26  | 73.56   |\n| GPT-4o        | 57.76    | 79.80      | 84.48 | 80.04  | 75.52   |\n\n> Theoretical K Few-Shot Guarantee\n\nBound in line 603 of $K \\gg |A|\\ln|A|$to drive average regret to nearly zero is the worst-case guarantee for an unfactored, adversarial bandit. In real settings, Adareasoner requires much fewer than this bound, and several reasons can be attributed to this cause.\n\n- A shared encoder layer enforces $L$‑smoothness and near‑convexity of the policy objective [4], enabling quick convergence under small $K$.\n- A pretrained reward model delivers high fidelity and SNR feedback (“warm start”), accelerating policy updates under few‑shot budgets.\n\n> LLM configuration and policy optimality\n\nIt is important to clarify that only the action space is universal, not the trained AdaReasoner policy itself. AdaReasoner operates over a shared action space (temperature, reasoning steps, and method), but each LLM requires its own adapter to align the model’s internal representation with the policy. Our ablation study (Table 2)—for example, using GPT-4o with Qwen’s adapter—shows significant performance degradation, confirming that adapters must be trained separately for each model.\n\nMoreover, due to the presence of regret, AdaReasoner's policy is not truly optimal but only an approximation. To examine this, we include two new analyses of perturbed policy. Additionally, we report further experiments using aggregating independently trained single policy heads as the main adapter—with no shared layer also in the table below. We independently run the experiment three times and calculate the standard deviation.\n\n* *Adareasoner-close-perturbed* : replaces one action from the original AdaReasoner policy with its closest neighbor in the action space. For reasoning methods, the replacement is selected based on cosine similarity between prompt embeddings (e.g., perturbing step 4 to step 3 or 5).\n* *Adareasoner-distant-perturbed* : replaces the action with the third-closest neighbor, allowing us to observe performance changes under larger deviations.\n* *Adareasoner-emsemble* : Three independently trained action heads with action emsembled together.\n\nThese analyses provide clear evidence of AdaReasoner's policy suboptimality.\n\n| GPT-4o                      | Metaphor          | TruthfulQA        | MMLU              | LogiQA            | Average            |\n| --------------------------- | ----------------- | ----------------- | ----------------- | ----------------- | ------------------ |\n| Adareasoner                 | 71.49$\\pm$ 0.19 | 81.33$\\pm$ 0.44 | 86.52$\\pm$ 0.35 | 82.30$\\pm$ 0.21 | 80.41$\\pm$ 0.30  |\n| Adareasoner-close-pertub    | 66.05$\\pm$ 1.55 | 79.39$\\pm$ 1.20 | 85.18$\\pm$ 2.48 | 80.03$\\pm$ 1.01 | 77.66$\\pm$ 1.56  |\n| Adareasoner-distant-perturb | 57.69$\\pm$ 3.21 | 71.77$\\pm$ 4.22 | 81.42$\\pm$ 2.01 | 74.96$\\pm$ 3.18 | 71.46$\\pm$ 3.16 |\n| Adareasoner-emsemble        | 65.73$\\pm$ 0.65 | 79.54$\\pm$ 0.30 | 84.71$\\pm$ 0.58 | 80.04$\\pm$ 0.12 | 77.50$\\pm$ 0.41  |\n\n[1] Manh D N, Chau T P, Le Hai N, et al. Codemmlu: A multi-task benchmark for assessing code understanding capabilities of codellms[J]. CoRR, 2024.\n\n[2] Yue A S, Madaan L, Moskovitz T, et al. HARP: A challenging human-annotated math reasoning benchmark[J]. arXiv preprint arXiv:2412.08819, 2024.\n\n[3]Vu, Duc Anh, et al. \"Curriculum Demonstration Selection for In-Context Learning.\"  *Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing* . 2025.\n\n[4]Du, Simon, et al. \"Gradient descent finds global minima of deep neural networks.\" International conference on machine learning. PMLR, 2019.\n\nThank you once again for your detailed feedback—it has been extremely helpful in improving our work. Please don’t hesitate to reach out if you have any further questions. Thank you!"}, {"review_id": "g03DTaIest", "reviewer": "Reviewer_EquF", "summary": "This paper focuses on how to automatically obtain the configurations (temperature, number of reasoning steps, prompts) for different tasks. It formulates this problem as a multi-armed bandit problem and leverages the Boltzmann exploration strategy to select actions for effective training. Based on these, AdaReasoner is introduced as a model-agnostic plugin to automatically obtain the configurations for tasks. Extensive experiments are conducted to validate the effectiveness of AdaReasoner.", "questions": "1. How many epochs are used to train the AdaReasoner for each model in Table 1?\n2. It would be beneficial to include comparisons to other automated optimization methods for configurations.\n3. Are there any rationales behind the creation of the reasoning instructions (Table 4)?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n1. The paper is well-written and well-motivated, which addresses the problem of finding the optimal configurations for LLMs.\n2. It introduces AdaReasoner, a novel model-agnostic plugin that leverages the RL framework and Boltzmann exploration mechanism to automatically learn the configurations using a small number of samples.\n3. Extensive experiments have been conducted to validate the effectiveness of the proposed AdaReasoner.\n\n### Weaknesses\n1. This paper does not compare to baselines that also conduct automated optimization of configurations like HAG [1].\n2. The design of the reasoning instructions can be non-trivial (according to Table 2, the optimization of the prompts contributes the most to the performance gain)\n\n[1] Wang, S., Li, S., Sun, T., Fu, J., Cheng, Q., Ye, J., Ye, J., Qiu, X. and Huang, X., 2024. Llm can achieve self-regulation via hyperparameter aware generation. arXiv preprint arXiv:2402.11251.", "was_revised": true, "final_justification": "The authors have addressed my questions and provide additional experiments against missing the baseline. Therefore, I recommend acceptance.", "rebuttal": "We would like to thank you for such a constructive review, and we are glad to address the questions in response below.\n\n> Comparison with HAG\n\nThank you for pointing this out. We apologize for missing the comparison with HAG [1]. We're glad to incorporate new baselines and appreciate the suggestion.\n\n**At the same time, we would like to clarify that while HAG self-regulates hyperparameters such as temperature, top-p, top-k, and repetition penalty, our method operates in a different action space, including reasoning steps or methods.** Comparison is provided in table below. Please note that current API like GPT-4o, no longer supports top-K and repetition penalty. So, instead, presence_penalty is used in our implementation of HAG.\n\n| LLM (HAG) | Metaphor | TruthfulQA | MMLU  | LogiQA |\n| --------- | -------- | ---------- | ----- | ------ |\n| GPT4o     | 55.08    | 75.69      | 76.44 | 73.32  |\n| Llama     | 47.99    | 78.40      | 85.23 | 77.78  |\n\n> Design of action space (rationale and analysis)\n\nThank you for the thoughtful comment. Indeed, as shown in Table 2, optimizing the reasoning method ($a_p$) contributes the most to the performance gain. However, the other actions ($a_s$, $a_t$) also lead to clear improvements over baselines. Detailed plots of distribution of actions can be found in Appendix E. Intuitively, this can be somehow expected, as reasoning method instruction does convey the most information.\n\nOur design of the reasoning instructions is principled. For the action space, we only include discrete temperature values and exclude other hyperparameters, as co-tuning temperature with top-p is disencouraged (see lines 162–163) and other hyperparameter as random seed is not effective as revealed in ablation studies. For design of $a_p$, we select mainstream cognitive strategies such as self-audit and creative reasoning, and introduce variations based on structure-mapping theory from cognitive psychology [1].\n\n> Additional Comparison and technical details\n\nWe trained adareasoner for 5 epochs as standard. We would fill in more details and avoid any confusion in the experimental parts in our camera-ready version.\n\n[1]  Gentner, Dedre. \"Structure-mapping: A theoretical framework for analogy.\" Cognitive science 7.2 (1983): 155-170.\n\n\nThank you once again for your detailed feedback—it has been extremely helpful in improving our work. Please don’t hesitate to reach out if you have any further questions. Thank you!"}]

Author rebuttal:

Meta-review: The reviewers viewed the paper enthusiastically after the author response. Please take into account the reviewer comments when preparing the final version.
Decision: Accept (spotlight)