Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Authors: Qinglin Zhu
Runcong Zhao
Hanqi Yan
Yulan He
Yudong Chen
Lin Gui
Venue: ICML 2025 spotlightposter (2025)
Keywords: Large Language Models
Reasoning
Embedding Perturbation
Bayesian Optimisation
Primary area: deep_learning->large_language_models
TL;DR: We propose an embedding-based search framework that optimises the first token's embedding through controlled perturbation and Bayesian refinement to enhance reasoning accuracy and coherence in large language models with minimal computation.

Abstract: Large Language Models (LLMs) struggle with complex reasoning due to limited diversity and inefficient search. We propose Soft Reasoning, an embedding-based search framework that optimises the embedding of the first token to guide generation. It combines (1) embedding perturbation for controlled exploration and (2) Bayesian optimisation to refine embeddings via a verifier-guided objective, balancing exploration and exploitation. This approach improves reasoning accuracy and coherence while avoiding reliance on heuristic search. Experiments demonstrate superior correctness with minimal computation, making it a scalable, model-agnostic solution.

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Qinglin Zhu
Runcong Zhao
Hanqi Yan
Yulan He
Yudong Chen Lin Gui
Abstract
correct answer by following language- based instructions.
Large Language Models (LLMs) struggle with complex reasoning due to limited diversity and inefficient search. We propose Soft Reasoning, an embedding- - search framework that opti- mises the embedding of the first token to guide generation. combines (I) embedding perturba- tion for controlled exploration and (2) Bayesian optimisation refine emheddings via verifier- guided objective, balancing exploration and ex- ploitation. This approach improves reasoning ac- curacy and coherence while avoiding reliance on heuristic search. Experiments demonstrate supe- rior correctness with minimal computation, mak- ing it a scalable, model-un model-agnostic solution.
Despite these efforts, two key challenges remain: (1) En- hancing generation diversity typically relies on increasing the temperature parameter, which flattens the token distri- bution. This, however, does not necessarily result in better coverage of the correct answer, increasing lww-probability token likelihood indiscriminately may introduce noise rather than meaningful exploration (Holtzman et al., 2020). (2) Existing planning and search methods such as sampling multiple reasoning paths rely heavily on heuristic strate- gies, guided by prompts (Hao et al., 2023; Qi et al., 2025b). However, these approaches do not directly adjust for the model's internal representations, thereby making the search process inefficient and highly dependent on surface-level prompt variations. This often leads to a "wild-goose chase", where search remains constrained by randomness and indi- rect heuristics rather than systematic optimisation.
1. Introduction
To address these challenges, we propose Soft Reasoning, a novel approach using controlled embedding exploration: (1) By injecting a Gaussian embedding into the decoding of the first answer token, can adjust the distribution of low- probability tokens in more controlled manner than uniform temperature tuning, leading to more flexible generation. (2) Treating the LLM as a black box verifier, we apply Bayesian optimisation (Frazier, 2018) on the injected embedding to maximise verification-based reward. This allows us to use observerd rewards to directly guide the exploration in the embedding space. As result, Soft Reasoning improves performance without strong verifier- even when both generation and verification originate from the same model.
Large language models (LL.Ms) have demonstrated remark- able potential in various reasoning tasks, particularly on rel- atively simple and common benchmarks (Huang & Chang, 2023; Xu et al., 2025). Despite this, they still face signif- icant limitations in complex tasks (Lightman et al., 2024: Wang et al., 2023a), which often require deeper levels of thought, and answers generated solely based maximum likelihood are frequently incorrect. To increase the proba- bility that the correct answer is included among generated candidates, many existing approaches aim to enhance gener- ation diversity through multiple sampling (Lightman et al., 2024). A common mechanism for achieving such diver- sity is temperature scaling, which adjusts the randomness of token selection (Brown al., 2024). Complementary to this, planning- based methods, such as thain-of-thought reasoning (Wei et al., 2022; Wang et al., 2023a) or tree- structured search (Yao et al., 2023), attempt to locate the
As illustrated in Figure 1, Soft Reasoning leverages injected vectors to change the distribution the next generated to- ken, rather than simply flattening the output probability curve. In this injection-based generation process, decoding is performed via greedy search, ensuring that each injected vector corresponds to a unique generated sequence. This guarantees both controllability and repeatability. The effect of each injected vector can then be evaluated using reward function that accounts for both correctness and coherence of the generated sequence. Next, in a sequential way, we identify promising directions for further exploration based on all observed injection-reward reward pairs by utilising Bayesian
'Equal contribution 'King's College London, UK "The Alan Turing Institute, UK UUniversity of Warwick, UK. Correspon- dence to: Yudong Chen <yudong.chen warwick.ac.uks. Lin Gui <lin.I.gui kkk.ac.kk>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Moinstream Approsch
Tuning the tampessture flatting curve, the candicates doesn't change
Question
Sampling and Iterativelyy decoding the answer
Darrell and Allen's ages ratio 7:11. toe
now is 1622, salculate Allen's age 10 years from now.
DareuU
Allen
their
LLMg
Saft Resesoning
Adjusting the distribution by njecting different controlllale Goussian embecding
Sampling herativaly decodding the arrower
Decoding the Answer
82
E
Updaringg according to Bayasian optimisation untill converge
Danell
Allen
thei
ago
rato
resuit
OITO
Mln
Jtel
r300
boult
Distribttin the first token
rst
Daneli
Allen
ther
Figure 1. Comparison of Mainstream and Proposed Approaches.
optimisation. Again, thanks the one-to-one mapping be- tween initial vectors and generation outcomes, this process enables an effective search for optimal vectors that enhance generation quality. A notable advantage of this approach is that it operates without requiring access to the internal parameters of the LLM, allowing for efficient, low-resource control over generation behaviour.
cleus) introduce randomness but struggle to balance qual- ity and diversity (Fan et al., 2018; Holtzman al., 2020). High-temperatte settings can lead to incoherent outputs (Minh et al., 2025), and adaptive methods like min-p sam- pling (Minh et al., 2025) require careful tuning. Debiasing- Diversifying Decoding (D3) mitigates amplification bias but increases computational cost (Bao et al., 2024). Crucially, most methods overlook the impact of initial token selection, which significantly influences reasoning outcomes (Wang & Zhou, 2024). Our approach addresses this by perturbing initial token embeddings with Gaussian noise, reshaping the probability distribution to improve exploration while maintaining quality and efficiency.
Our contributions can be summarised as follows:
We propose reasoning method, Soft Reasoning, which combines embedding perturbation and Bayesian opti- misation to better control low-pre cow-pobabbility token selec- tion, enabling more flexible and diverse generation than temperature tuning.
Instead of language- based instruction or heuristic search, our method directly optimises the embedding of the first generated token to control the direction of thinking and exploration in LLM, effectively reducing the searching and reasoning complexity, and improving efficiency and accuracy.
2.2. Efficient Exploration of Solution Spaces
Efficient solution space exploration is crucial for enhancing LLM reasoning while maintaining practical computational costs. Increasing generated samples improves coverage (Brown al., 2024) but is computationally prohibitive. Op- timising test-time compute allocation is more effective than scaling model size (Snell al., 2025), though it requires task-specific strategies. Mutual reasoning frameworks lever- aging self-play and MCTS (Qi et al., 2025b; Yan et al., 2024), as well as Tree of Thoughts (ToT) (Yao et al., 2023), explore multiple reasoning paths but incur high computa- tional overhead. Thought Space Explorer (TSE) (Zhang & Liu, 2024) enhances reasoning breadth but additional cost. Soft Reasoning refines these approaches by integrat- ing controlled initial-token embedding perturbations with strategic search algorithm inspired by MCTS and mutual reasoning. By introducing exploration early through embed- ding perturbation and guiding search via a verifier, we im- prove efficiency without excessive computational overhead, striking balance between exploration and exploitation to optimise reasoning performance.
Soft Reasoning is able to control and optimise the rea- soning without accessing model parameters or requir- ing additional verifier, allowing seamless integration into different mainstream LLMs. Experiments across variety of I.LMs and reasoning tasks demonstrate improved correctness and efficiency of Soft Reasoning over traditional decoding.
2. Related Work
2.1. Decoding Strategies and Diversity
Recent advances in LLM decoding aim to enhance diver- sity for tasks requiring creativity and exploration. Tradi- tional methods such as greedy and beam search often pro- duce repetitive outputs (Holtzman et al., 2020; Welleck et al., 2019), while sampling-based approaches (top-k; nu-
2

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
3. Preliminary: Temperature Scaling
where ey represents independent random perturbations drawn from a standard normal distribution, and o is sealing factor controlling the magnitude of the perturbation. This formulation allows us to sample from the local vicinity of the original embedding 2, exploring variations around the initial token representation.
A common approach for generating diverse outputs is tem- perature scaling, which controls the randomness the token generation process by modifying the softmax distribution over the model's output logits. For given temperature > 0, the probability of selecting token w(() at time step is given by:
For each perturbed embedding 24, we introduce a corre- sponding special token mapped 24 and add it to the vo- cabulary. This special token is then used as the first token for generating an answer. Since we use greedy decoding, this I, fully determines the entire output sequence, i.c. the (2) (3)
(ep(l,,ato//)) P(w((t) | 11000)1;077) En exp(Lr,e/T))
where ()lit-1) represents the sequence of tokens (w(t),.. ,..., wl(-1)) generated from the first token up to the (t-1)th token, l,, denotes the logit at time corresponding to token w, and F controls the sharpness of the distribution. This scaling flattens the distribution but preserves the rel- ative ranking of token probabilities. When t is low, the results concentrate on few -robbbbbbbbiltyy tokens, lead- ing to overly deterministic generations with limited diversity. When r is high, the model may sample low-probability to- kens, leading to incoherent outputs.
remaining tokens w!? w ...., wi
(L) are then determinis- tically generated in sequential manner:
wi = argmax P(w 24, Wi .q.0)
(2it-1)
(1:L) to be the complete output based
We denote y/4 == wi on the initial perturbed embedding 2;. We then repeat this process & times to generate different answers: 91....: ykk Since each output y; is fully determined by x, embedding perturbation effectively serves sampling mechanism over the entire answer space.
While this approach increases diversity, it lacks control, blindly flattening token probabilities; adaptability, as it ig- nores verifier feedback; and efficiency, often requiring mul- tiple samples or retraining (Joy et al., 2023; Xie et al., 2024). These limitations make it ineffective for structured reason- ing tasks that demand precise and efficient exploration.
3.2. Exploring the Embedding Space
Randomly sampling points with infinite computational re- sources could theoretically approximate the optimal solu- tion, but this approach is highly inefficient, especially given the computational expense of sampling with an LLM. In- stead, we adopt Bayesian optimisation, which consists of two key components: an objective function and an acqui- sition function that determine where sample next. We use Expected Improvement (EI) as our acquisition func- tion, which offers closed-fome solution (Frazier, 2018) with negligible computational cost, making it significantly more efficient by comparison. El effectively balances explo- ration (searching uncertain regions) and exploitation (refin- ing promising areas), selecting the point with the highest El at each iteration to guide the optimisation process toward convergence.
Generating accurate answers in complex tasks requires both exploring reasoning paths and verifying for their correct- ness. To achieve this, we propose a two-step framework as shown in Figure 2: (1) Embedding perturbation applies a Gaussian adjustment to the first-token embedding for con- trolled modifications beyond uniform tuning, (2) Bayesian optimisation refines the perturbed embedding to maximise a verifier- guided reward, improving reasoning path selection.
3.1. Embedding Perturbation
Given generative model go and natural language question prompt g, the first token (1()) is generated using greedy de- coding, which selects the token with the highest probability from the model's predicted distribution:
Optimisation Objective. To evaluate the objective func- tion with i sampled perturbed embeddings, we consider the sequence æ1:k = (xI...., I.)) where each E IR The corresponding answers are then generated as described above: f/..= = (v,, ya Comparing and refining mul- tiple generated answers has been shown to improve per- formance (Miao al., 2024). Additionally, since LLMs are primarily trained for text generation rather than explicit judgment, prompting them to regenerate and compare out- puts can yield better results (Zhang et al., 2024). Building on these insights, we propose a verifier-guide approach, where the model evaluates a batch of candidate answers and produces a refined output yo = V()xs). The correctness of
w argmnax P(w O),
where P(- q; f)) represents the probability distribution over the possible tokens predicted by the model ga. with input q.
Let E R represent the embedding of the token (1), This embedding serves as a prior, representing a "correct starting point" the latent space. To explore the neighbourhood of this embedding, we define a set of perturbed embeddings i; for l,...,K. ls as follows:
I3 = 2 + oE,,
& &~ ((, T):
3

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Question prompt
LLMg
Perturhed Embeddings
Diverse Outputs Y1sk
Xisk
w(2)
X1
w
Y1
Tveritter(1t)
X2
Y2
Verifier
Tverttter (12)
x1
Embe dding of w(()
X1
2
dei
Xk
00
yk
Towiifor (tb)
X2
f(x1a)
T Fcoherence (v
Tverifier 1
foo
Bayesian credible intervals for f(x)
Where sample next
++=1
x1
* Xi++
Loop until convergence
f(x) fx(ea)
Estimate of fx)
EIGe)
The point maximise BI(x)
Figure 2. Overview of Saft Reasoning. Starting with natural language question prompt, the model generates initial token embeddings (L),, which, due to greedy decoding, determine the entire output. These embeddings are perturbed to create candidate embeddings risk. leading to outputs yı. through greedy search, which are then evaluated for coherence and verifier feedback. Bayesian optimisation framework updates its estimation of the space based on this feedback and selects the next sampling point that maximises the expected improvement, balancing exploration and exploitation to refine the search for high-quality outputs.
an answer y is then assessed as binary indicator (0 or 1), based on its alignment with the verifier's final output. The verifier is the same model as the generator employed.
Bayesian Optimisation. Our goal is to maximise f()), as defined in (1), over the embedding space RRP. To optimise this black-box function, Bayesian optimisation uses a prior distribution on the domain to represent our beliefs about the behavior of the function and iteratively updates this prior using newly acquired data. Specifically, we model the prior joint distribution multivariate Gaussian distribution:
The embedding space may not be uniform, implying that perturbations in different directions can lead to uneven se- mantic shifts (Li al., 2023; Park et al., 2024). In some dimensions, even small perturbations can significantly alter meaning, potentially disrupting grammar context consis- tency and leading to incoherent outputs. To address this issue, we introduce the coherence term to prune low-quality generations, ensuring that only outputs with desirable se- mantic and syntactic properties are retained. To evaluate the quality of generated output y. we define an objective function (xr) that balances correctness and fluency:
f(11m) ~ (No(2mm)) Eo xim))),
where no (0(.)))) is the prior mean vector, and Eo(Ii:n T1:n) is the prior covariance matrix.
After observing f(11.) we aim to infer the value of f fx) at new point .r. Using Bayes' rule (Rasmussen & Williams, 2006), we update the posterior distribution of fz) condi- tioned these observed values:
f(z) Everifier()) (v) + Teoherence (y);
where:
f(x) | [(TII)) ((vu))))()))
Verifier Score (I'eritter): This is binary indicator provided by the verifier, reflecting the correctness of y:
Here, (x( and oz represent the posterior mean and variance, respectively. A detailed discussion on the choice of the prior distribution and the computation of the posterior distribution provided in Appendix A.2.
Tverifier (y) 1 [y.=
Coherence (Fcoherence): This term evaluates the fluency of the generated sequence based on token probabilities:
A naive way find the maximiser at this stage would be to select among the previously evaluated points 2 Th the one with the highest observed function value. Let fī := maxm<k f f(rm) denote this value. If we were to sample another point x E IR and observe f(x) then the value of the best observed point would either be f() (if f(z) 2 f2) or S (if fx) < ft). The improvement the value of the
Teoherencs (3)
log P ())
where ()) is the probability of generating token (e))) from the LLM's entire vocabulary.

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
best observed point could be expressed as [f(x)) -fll+ max(f( f 0)-
To address this, we leverage a dimension reduction approach
based on random embeddings (Wang et al., 2016; Nayebi et al., 2019). Specifically, if a function f : R - I has an effective dimension de I D, then with high probabil- ity, there exists a lower-dimensional representation g(u) == f (Au), where A is random projection matrix. This allows optimisation to be performed in a lower-dimensional space F instead of the original IR. Using this approach, we iter- atively optimise the function in the reduced space and map solutions back to the original space. Theoretical foundations and implementation details are provided in Appendix A.5.
While we would ideally choose 2 to maximise this improve- ment, f(z) is unknown until after the evaluation. Instead, we select z: that maximises the expected improvement under the posterior distribution, defined as
FI, (1) := Fx fx]
-
(3)
where E& denotes the expectation taken with respect to the posterior distribution (2). Using integration by parts, we can write EI (3) in closed- expression:
EIg (1) = [mx(x) - 1²] + + (x))
(.(x)) fh
4. Experiments
(x
(x(2)) -f
We benchmark Soft Reasoning against strong baselines and conduct ablation studies.
-lux(x) - ff 20
(x)
where o and denote the probability density function and the cumulative distribution function of the standard normal distribution, respectively.
4.1. Experimental Setup
Datasets and Models. We conduct experiments using three LLMs: Llama-3. L 1-8B-Instrucct (Meta, 2024). Qwen2- 7B- Instructt,, Qwen2-70B 70B-IIntuttt (Yang al., 2024), and Mistral-8B- (Jiang et al., 2023). The models are eval- uated on four benchmark datasets, including three complex mathematical tasks (GSM8K (Cobbe et al., 2021). GSM- Hard (Gao et al., 2023), SVAMP (Patel et al., 2021)), and one commonsense reasoning task StrategyQA (Geva et al., 2021). For the Qwen2-70B-In model, we additionally eval- uate its performance the AIME-2024 benchmark.
Our next sampling point, Ja+1 E RRD is the maximiser of EL. We then iteratively update the posterior distribution and the EI function. Details on how we select x to maximise Elg(2) (z) can be found in Appendix A.3. Convergence is con- sidered achieved when the change in the objective function between consecutive iterations satisfies Ifee fa-1l < e, where e is predefined threshold. Additionally, the algo- rithm terminates after maximum of K iterations if conver- gence has not been reached.
In defining fz), we assume an ideal verifier with perfect accuracy, meaning it provides an error-free assessment of correctness. However, in practice, the verifier's s accuracy is less than 1, introducing uncertainty into its evaluations. To address this noise in Bayesian optimisation, we use adaptive version of the El acquisition function that explic- itly incorporates observation uncertainty. This adaptation dynamically adjusts the exploration rate based on uncer- tainty, ensuring a higher probability of convergence while balancing exploration and exploitation (Vakili et al., 2021; Tran-The al., 2022). Theoretical foundations and imple- mentation details are provided in Appendix A.4.
Baselines. Our baselines include: (1) CoT Prompting, which includes zero-shot CoT (Kojima al., 2022) and few-shot CoT (Wei al., 2022); (2) Self-Consistency (SC) Decoding (Wang et al., 2023c), which involves sampling answers at various temperatures T E {0.4, 0.6, 0.8} and se- lecting the final answer through majority voting; (3) FIRE (Chen et al., 2025), which adjusts the decoding process by setting the temperature of the first token to 30 to enhance diversity, while subsequent tokens are generated using the standard temperature setting; (4) CoT-Deecding (Wang & Zhou, 2024), which generates k answers by sampling the top-k tokens from the probability distribution of the first to- ken. Each of these top-k tokens is used as the starting point for decoding the remainder of the answer; and (5) RAP (Hao et al., 2023), which uses Monte Carlo Tree Search to explore reasoning paths strategically, balaneing exploration and ex- ploitation to find solutions efficiently. Note that RAP re- quires problem decomposition via examples; hence only report its performance in the few-shot setting. Additionally, we compare Soft Reasoning with recent controlled genera- tion approaches, including Trainable Prefix Scorers (Mudgal et al., 2024) and Constrained Fine-tuning (Qi et al., 2025a), with further details provided Appendix B.2.
Dimension Reduction. One shortcoming of using tradi- tional Bayesian optimisation methods for identifying the point with maximum EI (Mockus, 1975; Hvarfner et al., 2024) is that they perform poorly when the search space ex- ceeds 20-30 dimensions due to the curse of dimensionality (Kandasamy et al., 2015; Letham et al., 2020; Wang al., 2023b). In nigh-dimensional spaces, surrogate models re- quire an exponentially larger number of points to accurately estimate the maximum of the EI function, making optimi- sation highly inefficient. With the dimension of embedding vectors for LLMs typically ranging from 768 to 8192 or more, traditional methods are impractical in our setting.
5

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Table 1. Performances of different reasoning methods on Accuracy (%) across all benchmarks.
GSM8K
GSM-Han Mard
SVAMP
StralegyQA
Model
Method
Zero Shot
Few Shot
Zero Shot
Shot
Zero Shot
Few Shot
Shot
Few Shot
CoT
53.0gn.o
77.4g0.0
14.0a0
28.020.0
61.020.0
83.0g00
58.5.0.0
68.520.0
SCIr 0.4)
73.0a1a
80.4+1.
25.7+a4
31.8215
79.1
87. 87.1
64.740.7
71.640s
SC = 0.6)
73.6125
80.611a
24.511.1
31.2413
16.1
87.7a12
59.9220
71.3
SC(r = 0.5)
65.0120
81.111a
21.8113
30.8109
69.6120
87.4112
54,4126
72.7
LLaMA3-8B-II
FIRE
73.8
79.6
25.2g3.
25.7aa.
81.5
87.6120
63.0ga7
72.
CoT Decodding
73.9+1.
80.3
24.8
30.3
83.241a
88.2+1.0
64.641.6
73.3
RAP
80.7
32.7
87.9
73.4
Soft Reaxoming
79.41.2
84.3g14
282.1.
35.7410
88.2113
9022,0
67.2
75.610.
wo Fverller
76.8
82.0g0.
26.3
34.820.3
86.741a
89,5
66.2226
74.3
w/o Fechnence
77.4221
83.4
27.9+1.
35.3+1.5
84.6424
40.
66.041.3
75.0+1.5
Col
64.52u0
82.5100
40.0100
55.5100
43.5100
86.0100
63.0g00
70.0g00
SC = 0.4)
81.2
85.721.5
47.5+14
55.4207
72.3g2.
90.3+12
67.1
71.1
SC(r 0.6)
80.211.
85.4199
46,2119
53.4106
77.3
90.4106
67.5
69.1
SC(r 0.8) FIRE
80.0gu
85.1
47.3213
55.4
78.6221
90.6g12
67.0gn.
70.
Qwen2 7B Ins
81.0g18
83.021.3
45.1
51.0+1.
76.3
90.6102
67.640.
68.
CrT-Decoding
82.0g2.
84.542.1
46.7+2.3
$2.1
78.6+16
89.7
65.9+1.
69.5+221
RAP
86.2+1.2
56.240.
90,8+1.1
71.3
Soft Reazoning
88.6
90.0;14
$3.7
$8.7
83.4:24
92.2
68.1
10.3
w/o Fvatllie
87.0+10
89.7+1.8
51.2+2.1
58.3+1.0
73.5+4.0
90.5+1.8
66.040s
68.3+1.6
w/o Techerencs
87.322
89.2+1.s
52,0+18
60,0+1.
76.7+14
90.7
66,540u
69,5
CoT
42.0ga.0
$4.0200
14.5a00
24.020.
52.0go.
72.0g00
62.0g0.
69.0g0.
SCir 0.4)
$2 22.
58.3+1.5
19.541.
26. 141.
67.4428
77.8
63.941.
72.6+1.2
SC 0.6)
55. las.
57.4110
20.7
25.3
69.7+16
78.4+20
64.241.0
71.7
SC 0.8) FIRE
50.2:..6
57.7428
19.1
26.621.1
68.3109
77.6
64.9
12.
Mistral-7B-Im
47.2+2.0
56. $6.143.2
18.1
26.3+14
67.1
78.4+12
64.2+10
71.0,22
CoT-Decoding
47.3
58.2+2.8
16.62a7
27.4+16
69.4
78.6+14
63.541.
72.7+2.
RAP
58.611.
27.6,11
79.4411
72.4413
Soft Reaxoning
614125
62.7
25.8118
32.5
72.2122
82.1.1
66.111
72.8115
w/o Furtlier
59.5
59.7
24.8108
30.8221
69.5
79.8
64.8100
71.8214
w/o Fecerenc
61.2
60.3+a.
25.5+33
29.5
70.2+1.
80.0g10
65.5
72.0g3.
Setup & Hyperparameters. Experiments are conducted in zero-shot and few-shot settings, with prompts including I, 2, 4, and 8 exemplars for few-shot settings. To reduce vari- ance, each configuration is repeated five times with different random seeds. We report the mean and standard deviation of accuracy across all runs. The convergence threshold is set to 0.01.
Coverage Analysis. For each method, we calculate the probability of covering the correct answer in at least one of the generated answers. Our approach consistently achieves the highest coverage across all models and datasets. For instance, on GSM8K with LLaMA3-8B- in the zero-shot setting, our method attains 91.8% coverage, outperforming FIRE (84.5% and CoT-Decoding (85.3%). Detailed cover- age probabilities for all models and datasets can be found in Table 14 in Appendix B.8. These results demonstrate that our controlled exploration strategy effectively enhances the likelihood of generating correct answers, highlighting its robustness over traditional methods.
4.2. Experimental Results
Overall Performance. Table 1 presents the accuracy of Soft Reasoning compared to baselines across four bench- marks and three LLMs under zero-shot and few-shot (8-shot) settings. The full table and the results for the Qwen2-70B- Ins model can be found in Tables 13 and 12, respectively, in Appendix B.S. Our approach consistently outperforms the besf-performing baseline across different models, especially in the zero-shot setting (average improvement of 5% GSM8K and 3% on GSM-Hard). Similar gains appear in the few-shot setting, where our method achieves the highest accuracy on most tasks and model variants. While effective, SC requires extensive hyperparameter tuning (e.g. varying temperature values) for each individual model and dataset to achieve optimal performance. contrast, our more system- atic search method improves solution quality consistently without the need for separate tuning in each scenario.
Effect of Exploration with Embedding Perturbations and Bayesian Optimisation. A natural question to con- sider is why adding noise to embeddings leads to more di- verse answer generation than temperature tuning. We follow Naik et al. (2024) to investigate this from the perspective of neuron activations in the Transformer's MLP layers. As shown in Figure 3, applying our method increases the acti- vation rate of neurons by roughly 3 3-4% in nearly all layers relative to the Self-Consistency (SC) baseline, suggesting that our perturbations stochastically trigger more diverse neural pathways.
To probe whether a specific subset of "critical neurons" may be responsible for correct reasoning, we identify neurons
6

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
portion of test examples that terminate after the n iteration for each dataset in both zero-shot and few-shot settings, reported in Table 2. With maximum of 4 iterations, no search exceeds the fourth iteration, and only small fraction require iteration 4. This rapid termination suggests that the El-driven sampling strategy quickly identifies promising re- gions of the embedding space most queries, minimising the need for further rounds of exploration.
Corne::
0.40
SC Wion
Activation Rate
0.35
0.30
0.25
0
5 10 15 Layer
20
25
30
Accuracy
06
Inktiaiiation
Fitteration
Ixeration
1.00
Figure 3. MI.P layer activation rates across Transformer layers for the first five tokens of generated answers, sampled 200 times. The curves compare method (Soft Reasoning) and the Self- Consistency (SC) baseline, further separated into correct and in- correct answers.
0.0
0.45
Intillsation
Correlation
Matrix
0.00
Activation Rate
0.35
0.50
0.30
0.25
Figure 5. Visualisation of the correlation matrix evolution during Bayesian optimisation across iterations.
10
15
20
Layer
Figure Activation rates of critical MLP neurons across Trans- former layers during initialisation and iterations.
Table 2. Proportion (%) of test examples that terminate at the n iteration for each dataset and setting using LLaMA.
in single sample whose activations exhibit the strongest correlation with correctness. In Figure 4, we track the ac- tivation rate these critical neurons across our Bayesian optimisation iterations. We observe steady increase, partic- ularly in layers 15-30, suggesting that our iterative sampling and verification increasingly activates these key pathways.
Shot
Iteration
GSM8K
GSM-HHard
SVAMP
StrategyQA
65.042.
70.3
64.8
58,8
30.1
24.413.0
28.9118
31.1
Zero
4.1+1.6
4.921.
5.4+1.
8.821.
0.810.8
0.4
0.9
1.410.5
76.8122
77.412.2
79.712.
66.7
20.740.
20.6+2.
19.122.
26.613.
2.5
1.8.0.6
1.221.0
6.1
Together, these findings support the hypothesis that em- bedding perturbation and controlled exploration approach not only diversifies generation but also systematically uncov- ers and reinforces the neuron activations crucial for deriving correct answers. For more detailed experimental procedures, please refer Appendix B.3.
0.0g0.0
0.220.3
0.0.0.0
0.6
Table 3. Comparison of performance and token usage hetween our method and RAP across benchmarks.
Category
Methol
GSMSK
GSM-Hlard
SVAMP
SinttggQAA
Convergence Bayesian Optimisation. Another ques- tion regarding our search algorithm is how quickly and reliably it converges. To investigate this, track two key metrics across our Bayesian optimisation iterations: (1) The evolution of the correlation matrix among the sampled em- bedding points. As shown in Figure 5, the correlation matrix becomes more structured over iterations, showing higher correlations among top-perti perforrff candidates. (2) The pro-
Resalt
Reasoning
RAP
80.7
32.7
$7.0
75-41
Input Taken Count
RAP
25710.84 Sk
35152.14
15058.54
17426 2k
Soft Reasoning
1457.18 1k
1847.88
1172.86
1180.88
Ourput Token Count
Soft Reazoning
RAP
211.98
35L.14
262.54
602.54
241.24
162.48
274.16
155.48
RAP
Time (min)
Soft Roasoning
23.2
29.4
18.4
17.4
Efficiency and Performance Analysis. Table presents a comparison of our approach with RAP in both performance

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
and efficiency. Our method achieves better accuracy on all tasks while drastically reducing computational overhead. Specifically, our input token consumption averages only 6. 6.19% of RAP's, our output token usage is 63.28% of RAP's, and our inference time is 14.3% of RAP's. These results highlight that our method not only improves accuracy but also substantially reduces token usage, thereby delivering superior overall efficiency. Additional analysis of inference time and memory usage is provided in Appendix B.5.
tasks, making it difficult to guarantee good performance in unseen settings. In contrast, El performs robustly without requiring task-specific tuning.
To investigate how the number of optimised tokens affects performance, conducted addi- tional experiments where we optimised embeddings for the first à tokens (instead of just the first token).
Optimisation Scope
Table 5. Accuracy (%) when optimising embeddings for different numbers of initial tokens using LLaMa.
4.3. Ablation studies
Objective Function. To evaluate the importance of each reward component, we removed either the verifier score (w/o Everifier) or the coherence term (w/o )ccherence) from our method. Table 1 compare these ablated variants with our full approach. In all tasks and model configurations, omitting either term degrades both accuracy and coverage, indicating that both components are vital. The verifier score clearly helps filter out incorrect or spurious solutions, while the coherence penalty ensures each output remains semantically consistent, particularly for complex multi-step reasoning. Indeed, both correctness-guided verification and semantic coherence play essential roles in navigating the solution space effectively.
Shot
#token (le)
GSMSK
GSM-Hard
SVAMP
StrategyQA
(ours)
79.4
28.2
88.2g1.
67.2-0.7
2
75.0g0.
24.8
83.020.5
68.5_0.5
Zero
69.7
22.2
85.5
66.3_0.
10
61.0ga.1
17.2+1.6
82.820.
67.3_1.9
20
52.2g2.3
19.0
74.3
67.8_2.5
(ours)
84.3
35.7.1.0
90.220.6
75.6
2
83.3
34.7+1..
90.2
74.2-1.3
Few
5
81.2.2.1
29 29.8
88 38.32
71.7=1.2
10
73.7+2.
23.840.3
86.820.8
71.7=1.6
20
62.0gu.
18.241.9
81.5
68.7_1.1
As shown in Tab 5, the performance generally degrades as k increases, especially beyond 5 tokens. This suggests that naively extending to multiple tokens can introduce instabil- ity or overfitting. We also compared with RAP (one of our baselines), a sice-search-bee sreaech-based method that operates at the sequence level rather than token-by-token toh.----ooooeehhuuhhhhhhh it shares similar ideas with token-wise search. While RAP achieves strong performance, it incurs substantially higher cost and still underperforms our approach. Developing multi-token optimisation strategy that can achieve both high accuracy and cnst-efffectveeess would require deeper investigation and extensive experimentation.
Why Choose EI? There are various acquisition functions for Bayesian Optimization (BO), such as the UCB score, Probability of Improvement (PI), and GP-UCB. While PI and GP-UCB are viable alternatives Expected Improve- ment (EI), the cumulative regret bound for GP-UCB matches that of EI (Shahriari et al., 2015). In contrast, PI considers only the probability of improvement and ignores its magni- tude, making it less theoretically grounded and more prone to premature exploitation (Srinivas et al., 2010).
0.85
GSMBK
0.38
GaMdHard
Table 4. Performance comparison between different acquisition functions across datasets and settings using LLaMa.
0.54
0.36
Shot
Method
GSM8K
GSM-Hard
SVAMP
StrategyQA
0.34
2
0 0.32
El (ours)
1..4=1.2
28.2:1.8
88.2+1.3
67.2+0.7
Pl
74.6+1.5
28.0=1.5
85.3+1.0
66.9*1.8
Zero
UCB [i=1
76.7+1.3
27.7=1.6
86.0#1.5
66.7+1.5
UCB 8=2
77.9=1.9
27.8=0.8
15.0=1.0
66.8+2.3
UCB A=5
7..6#1.9
27.7=0.8
85.320.8
66.7=1.0
50
0.26
45
50
Dimenrsion
Dimmesion StrategyOOA
El (ouns)
84.3+1.4
35.7+1.0
90.2+0.6
75.6#0.8
SWAMP
0.70
Pl
1..22
35.2=0.8
89,5+2.2
74.3+0.4
Few
UCB A8=1
83.340.3
34.7+1.2
88.0+0.0
74.7+1.5
UCB /0=2
83.321.6
36.2:1.5
88.7+1.4
75.721.9
0.74
UCB 8= 81. 8+0.9
34.2+2.5
89.7+0.8
75.0=0.7
0.72
0.68
As shown in Table 4, the experiments show that PI consis- tently underperforms compared to El, as expected from the theoretical discussion above. For GP-UCB, its performance is sensitive to the choice of the exploration parameter and is, in most settings, worse than EL. We also note that the op- timal parameter choice for GP-UCB varies across different
0.85
0.66
0.64
0.84
Dimmension
Dimension
Figure 6. Accuracy (solid lines) and standard deviation (shaded areas) across reduced dimensions.
8

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Space Dimensionality. We evaluate our Bayesian optimisation approach under var- ious reduced dimensions before mapping back to the full embedding space (Figure 6). Across all four tasks and both zero- and few-shot settings, performance tends to improve up to d = 50. Although increasing d to 60 sometimes yields a small additional gain, the differences are minor, and d = 50 consistently achieves near-best or best results. To further evaluate the robustness of random projection, we conducted additional stability experiments, which are presented in Appendix B.4.
Impact of
Table 7. Binary classification accuracy (%) of different verifier strategies. each determining whether a generated answer is correct. Single strategies judge or generate in isolation per answer, while Multi strategies consider multiple candidate solutions together.
Verifier
GSM8K
GSM-Hard
SVAMP
Strategy
Single-Judge
75.9
60.9
82.7
63.9
Multi-Judge
80.4
46.8
87.5
67.3
Single Generate
78.0
40.7
82.7
71.7
Multi- Celti-Geeeretee (ours)
87.6
78.2
93.4
78,9
Table 7 reports the binary classification accuracies for each verifier. Multi-Generate yields the highest verification ac- curacy on all datasets. This indicates that leveraging the model's generative capabilities leads to more reliable cor- rectness assessment.
Impact of Special Token Placement. We compare three ways of inserting the perturbed special token into the prompt: at the beginning (Först), somewhere in the middle (Middle), or as an appended token (Last). Table shows for both zero- shot and few-shot settings, placing the special token at the end of the prompt (Last) generally yields higher accuracy and better coverage. One possible explanation is that placing the special token last ensures minimal disruption to the original semantics of the prompt, while still allowing Soft Reasoning to alter the initial token embedding and induce sufficiently diverse generation pathways. Based on this observation, we adopt the Last placement strategy in all subsequent experiments.
Table 8, Final accuracy (%) achieved by using different verifier strategies in overall framework using LLaMa.
Shot
Verifier
GSM8K
GSM-Hard
SVAMP
StrategyQA
Single-Judge
76.3
28 28.1
83.0
63.0g1.7
Multi-Judee
77.4 77.422a
20.5a22
80.5+18
67. 67.1
Zero
Single-Generate
76. 76.6211
27.6
84.3100
66-4200
Mulii Generate
79.4
28.2+1.
88 88.2g1a
67.2
Single-Judge
82-4218
35.0214
89.6 69.6214
72.0a1.
Multi-Judge
82 82.5
36.1+2
90.1
73.2+12
Few
Single-Genent Generate
82.421.
34.5214
80.741.2
74.7+12
Mulii Generate
84.3.14
35.7a10
90.2g0e
75.6g08
Table 6. Comparison of accuracy and coverage across different special token placements using LLaMa.
We compare final solution accuracy in Table 8. While sin- gle verifier judge or generate answers individually, multi- candidate generation leads to the highest end-to-end per- formance. Multi-Generat outperforms alternatives in both zero-shot and few-shot settings, harnessing the model's gen- erative capacity more effectively than judgment- based veri- fiers. Notably, even substituting simpler verifiers keeps our framework competitive with strong baselines, underscoring the robustness and efficacy of generation-based verifica- tion. Details on how the number of sampled embeddings k influences performance are provided in Appendix B.7.
Type
Shot
Heration
GSM8K
GSM-Hard
SVAMP
StrategyQA
First
77.7=26
25.5a13
65.0za9
07,0e1.
Zoto
Middlc
78.3g3.
27.3-13
84.0aa
67.8
Result
Last (ours)
79.4
28.2.1
88.2.1.
07.2
First
82.1
29.0
89 29.2,0a
Fow
Middie
82. 82.3
32.3:11
89.7 89.7
74.023.
(ours)
g3.3
90.2.0o .00
15.6
First
85.8
$1.2
98.0
92.7
Zero
Middle
80.5
12.72n8
93.3
93. 1a.s
Last (ours)
91.8
37.0
93.8
93.711.3
Coverage
First
92.0gam
40.8,11
94.7.1a
93.416.
Few
Middle
92.221.0
41.7g95
94.5go5
90. 1g.
(ouns)
92.2.a.
_9.8
95.8 1122
93.3
Verifier Comparison: Judgement vs. Generation. In- spired by recent work suggesting that LLMs can be more adept at generating correct outputs than critiquing existing ones (Miao et al., 2024; Zhang et al., 2024), we explore four verifier strategies: Single-Judge, which evaluates each candidate independently; Single-Generac which regener- ates a purportedly correct answer for each candidate: Multi- Judge, which scores multiple candidates collectively; and Gentri-Geeeette,, which produces new solution from multi- ple candidates, labeling any matching candidate as correct. Given its consistently strong performance across settings, we select Maltti-eeeerteeaa as the default verifier in our exper- iments. The detailed defimitions of these prompt templates are provided in Appendix B.6.
5. Conclusions and Future Directions
We introduce an embedding mabeedding-based optimisation framework that enhances LLM reasoming by refining the first-token embedding. By integrating controlled perturbations with Bayesian optimisation, Soft Reasoning improves accuracy, is model-agnostic, and remains computationally efficient. Our approach relies on a verifier that may provide unreliable feedback, impacting optimisation. It also operates at the token level, which poses challenges for interpreting how perturbations influence reasoning. Future work will focus on improving verifier reliability, extending optimisation beyond the first token, and enhancing interpretability to better understand perturbation effects on reasoning.
9

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Impact Statement
Dai, D., Dong, L.,, Hao, Y., Sui, Z., Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Pro- ceedings the
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10113886/1122222 acl-Iong.581. URL https: http///aaaannttoooooooooooo org/ 2022. acl aalalllllloo waaecllloog.... 5811
Acknowledgments
This work was supported in part by the UK Engineer- ing and Physical Sciences Research Council (EPSRC) through a Turing AI Fellowship (grant no. EP/V020579/1, EP/V020579/2) and a New Horizons grant (grant no. EP/X019063/1), and KCL's Impact Acceleration Account (grant no. EP/X525571/1). A PhD studentship from the Chinese Scholarship Council funds Qinglin Zhu. The au- thors also acknowledge the use of the King's 's Computational Research, Engincering, and Technology Environment (CRE- ATE) at King's College London. We would also like thank Dr. Han Zhang and Professor Tao Gui for their valuable suggestions during the rebuttal period.
Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neu- ral story generation. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp- 889-898, Melbourne, Aus- tralia, July 2018. Association for Computational Lin- guistics. doi: 10.18653/ / //aclanthology wtt.:y...og....o.. 1082.
URL https:
Frazier, P. I. A tutorial on bayesian optimiza- tion, 2018. URL https: /tt::://dd.... http:/do...rggg//// wrg./10.85500/ arXiv. 1807. 02811. arXiv preprint.
References
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: program-aide lan- guage models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org. 2023.
Bao, K., Zhang, J., Zhang, Y., Huo, X., Chen, C., and Feng, F. Decoding matters: Addressing ampli- fication bias and homogeneity issue in recommenda- tions for large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods Natu- ral Language Processing, pp. 10540-10552, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10.18655/w1/2022.ccmmm.... 589. URL https: wwlaaaaaaantog...rrrroaoo..a emnlp- monlp-mmin... 589/.
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, Did Aristotle Use a Laptop? A Question An- swering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Lin- guistics ( (TACL), 2021.
Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., and Hu, Z. Reasoning with language model is planning with world model. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 8154-8173, Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10.18653//122023.mmlpp 507. URL https:://///m http:s//aaaaatttlhlaeae wttp:/llooooroooooo/e////// emnlp-main, 507/.
Brown, B., Juravsky. J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL https: https://arxiv. /awp/v/r/vvvv//////////e./..aav org/ hb5s//b/222/ abs//40// 2407 21787.
Chen, W., Zhang, Z., Liu, G., Zheng, R., Shi, W., Dun, C., Wu, 7., Jin, X., and Yan, L.. Flaming-hov initiation with regular execution sampling for large language models. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Findings of the Association for Computational Linguistics: NAACI 2025, pp. 7118-7127, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN URL https:/ /aa/aaaaaanttoooooo org/ 2025. find ngs naacl. .966.
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference Learning Representations, 2020. URL https: htttps://operre//// hwwp//wppew/eeepp/oo/peppeoe.eep wtwpeeviwwwwwwweforreeeeeee id=rygGOyrFvi:
Huang. J. and Chang, K. C.-C. Towards reasoning in large language models: A survey. In Rogers, A., Boyd-Graber, J., and Okazaki. N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 1049-1066 Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 10. 101118861110000 12201/2...22.2/-s..... findings-acl.
Cobbe, K., Kosaraju, V. Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
10

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
67. URL https::/acllanthollrry..
Meta. Introducing meta Ilama3: The most capable openly available IIm date, 2024. URL https: / mett.. com/blog wolllo//ettte meeta 11ama-3a.. 3/.
findings-acl.66 67.
Hvarfner, C., Hellsten, E. O., and Nardi, L. Vanilla bayesian optimization performs great in high dimensions. In Pro- ceedings of the 41 International Conference on Machine Learning (ICML), volume 235, Vienna, Austria, 2024. PMLR.
Miao, N., Teh, W., and Rainforth, T. Selfcheck: Using Ilms to zero-shot check their own step-by-step reason- ing. In Proceedings of the International Conference on Learning Represeniations (ICLR), 2024.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C.,,
Minh, N. N., Baker, A., Neo, C.,, Roush, A. G., Kirsch, A., and Shwartz-Ziv, R. Turning up the heat: Min-p sampling for creative and coherent LLM outputs. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /tppnrevieieeeeeeeeeeeeeeeeeeeee net forum? id-FBkpCyujts.
Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel. G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint N.K::2300006825, 2023.
Joy. T., Pinto, F., Lim, S.-N., Torr, P. H. S., and Dokania, P. K. Sample-dependent adaptive temperature scaling for improved calibration. Proceedings of the Thirty- Seventh AAAI Conference on Artificial Intelligence (AAAI- 23), 2023.
Mockus, J. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference, pp. 400-404. Springer, 1975.
Kandasamy, K., Schneider, J., and Póczos, B. High di- mensional bayesian optimisation and bandits via additive models. In Proceedings of the International Conference on Machine Learning, pp. 195-304. PMLR, 2015.
Mudgal, S., Lee, J., Ganapathy, H.,, Li, Y., Wang. T., Huang, Y., Chen, Z., Cheng, H.-T., Collins, M., Strohman, T., Chen, J., Beutel, A., and Beirami, A. Controlled decod- ing from language models. In Proceedings of the 41 In- ternational Conference on Machine Learning, ICML'24. JMLR.org, 2024.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 2022.
Naik, R., Chandrasckaran, V., Yuksekgonul, M., Palangi. H., and Nushi, B. Diversity of thought improves reasoning abilities of llms. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Letham, B., Calandra, R., Rai, A., and Bakshy, E. Re-examining linear embeddings for high-dimensional bayesian optimization. Advances in Neural Information Processing Systems, 33:1546 15554-6588.,,,111111111111114118 2020.
Nayebi, A., Munteanu, A., and Poloczek, M. A frame- work for bayesian optimization in embedded subspaces. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97, pp. 4762-4761. PMLR, 2019.
Li, L., Ren, K., Shao, Y., Wang. P., and Qiu, X. Per- turbscore: Connecting discrete and continuous pertur- bations in nlp. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pp. 6638-6648, Singa- pore, December 2023. Association for Computational Linguistics. doi: 1...1666//11/2223.nnnnng.....
Park, J. H., Lee, G., Park, S., and Cho, S. I. Not All Classes Stand on Same Embeddings: Cal- ibrating a Semantic Distance with Metric Tensor. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. URL tetttps://openreee id id=2HUznVsBBp. Last modified: 13 Nov 2024.
442. URL https://p// / /aallannii wwclanthooogy...ooooooooo org/ findings-ener emnlp- .emelp..444. 442/.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2024. URL https: http::::///ppeeeeeieeeee http:/iwwwwweet/ounnneeeeeee id=v810pN6F01.
net/ forum?
Patel, A., Bhattamishra, S., and Goyal, N. Are NLP models really able to solve simple math word prob- lems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 2080-2094, Online, June 2021. Association for Compu- tational Linguistics. doi: 10.18653/v1/2011 www.cl-maalt 168. URL https: /ttps:///aaatttthhc org/ 2021. naacl-main in 168.
Meng, K., Bau, D., Andonian, A. J., and Belinkov, Y. Lo- cating and editing factual associations in GPT. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https:////// httpp::/ htt::///peeeeeiieeeee httpr/iwwww.eee..ee.eeeeeeee norum? id= 6WA5

---

Soft Reasoning:
Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. Safety alignment should be made more than just few tokens deep. In The Thirteenth International Conference on Learning Representations, 2025a. URL https:/ htt///perrvvvveee eww. www.iiiwneeeeeee net/ forum? id Ld=6MgggPPDDE.
doi: 10.18653///20033ac/.oon.g 147. URL https: / //aclanthology. wwg/
//eelaatthollgyy.rrr//22.....a. wtt.:22..cllooogg.. 147.
Wang. X. and Zhou, D. Chain-of-thought reasoning without prompting. In The Thirt--iggtt Annual Con- ference on Neural Information Processing Systems, 2024. URL https: /oppereveeeeeeweeeeeeeeeeeeeeeee net forum? 7S0BOJP.
Qi. Z., MA, M., Xu, J., Zhang. L. L., Yang, F., and Yang, M. Mutual reasoning makes smaller LLMs stronger problem-solver. In The Thirteenth International Confer- ence on Learning Representations, 2025b. URL https: / net / forum? 6add66
Wang, X., Jin, Y., Schmitt, S., and Olhofer, M. Recent advances in bayesian optimization. ACM Computing Surveys, 55(13s):1-36, 2023b. doi: 10. 10.1145/3582078.
Wang. X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2023c. URL https: / /oppnreveeeeeeeeeeeeeeeeeeereeee net/ forum?? i id=1PLINTMMrw. Poster presentation.
Rasmussen, C. E. and Williams, C. K. I. Gaussian Processes for Machine Learning. MIT Press, 2006.
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and De Freitas, N. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEF, 104 (1):148-175) 2015.
Wang, Z., Hutter, F., Zoghi, M., Matheson, D.,, and Freitas, N. D. Bayesian optimization in billion dimensions via random embeddings. Journal of Artificial Intelligence Research, 55(1):361-387, 2016.
Snell, C. V., Lee. J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://///...... wwwpeerreiew.nee/foorroeeeeeee Ad=4FWAwZtd2m
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 14824-24833 2022.
Srinivas, N., Krause, A., Kakade, S., and Seeger. M. Gaus- sian process optimization in the bandit setting: no regret and experimental design. In Proceedings of the 27th In- ternational Conference on International Conference on Machine Learning, ICML' 10, pp. 1015-1022, Madison, WI, USA, 2010. Omnipress. ISBN 9781 1781605589077.
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., and Weston, J. Neural text generation with unlikelihood training, 2019. URL https://arxiss org/ /abg//////// 1908. 04319.
Tran-The H., Gupta, S., Rana, S., and Venkatesh, S. Regret bounds for expected improvement algorithms in gaussian process bandit optimization. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 151, Valencia, Spain, 2022. PMLR.
Xie, J., Chen, A. S., Lee, Y., Mitchell, E., and Finn, C. Calibrating language models with adaptive temperature scaling. In Proceedings of the 2024 Conference Empir- ical Methods in Natural Language Processing (EMNLP), pp. 18128-1888 18138. Association for Computational Lin- guistics, 2024.
Vakili, S., Khezeli, K., and Picheny, V. On information gain and regret bounds in gaussian process bandits. In Banerjee, A. and Fukumizu, K. (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 130, pp. 82-90. PMLR, April 2021.
Xu, F., Hao, Q. Zong, Z., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., Ouyang, T., Meng. F., Shao, C., Yan, Y., Yang. Q., Song, Y., Ren, S., Hu, X., Li, Y., Feng. J., Gao, C., and Li, Y. Towards large reasoning models: A sur- vey of reinforeed reasoning with large language models. arXiv preprint ar.iv:2501..0666000 2025. URL https: / / //doi. org 0. 0888850 2501 2250100660 .96886.
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.- W., and Lim, E.-P. Plan-and-solve prompting: Improv- ing zero-shot chain- Mfain-of-thouu thouuuhh reasoning by large lan- guage models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 6lst Annual Meet- ing of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pp. 2609-2634, 2634, Toronto, Canada, July 2023a. Association for Computational Linguistics.
Yan, H., Zhu, Q., Wang, X., Gui, L., and He, Y. Mir- ror: Multiple-perspective self-reflection method for knowledge-rich reasoning. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.). Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Vol- ume Long Papers), pp. 1086-7103, Bangkok, Thailand,
12

---

Soft Reasoning:
Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
August 2024. Association for Computational Linguis-
A. Methodology Supplement
tics. doi: 10. 10.18655/w1/20044accllooggg
https:
A.1. Technical Details on Optimisation Objective
//aclanthology wwtph/ooooooo.og/22/////// wwg. acl woong.822))
Although we simply sum the two components, they natu- rally operate in a progressive, tie-breaking manner. If two candidate outputs differ in their verifier scores, the one with the higher rverifier value immediately results in higher fxx). Only when the verifier scores are identical does Teoherence serve to break the tie, making the process effectively hierar- chical, even in this additive form.
Yang, A., Yang, B., Hui, B., Zheng. B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J. Dang, K. Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P. Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu. X.. Fan, Y., Yao, Y., Zhang, Y., Wan, Y. Chu, Y., Liu, Y. Cui, 7.,, Zhang. 7.,, Guo, Z., and Fan, Z. Qwen2 technical report, 2024. URL https:/ / / /arxiv..vv.. www./bsg////oooooooo/..o/... 2/t.//207..1110 10671.
A.2. Technical Details on Bayesian Optimisation
In constructing the prior distribution, for finite collection of points Vi.., the prior joint distribution on them is:
f(rim) (N(o((1mm) Eo Eo(Ii:n Fim)),
Yao, S., Yu, D., Zhao, J.. Shafran, I., Griffiths, L., Cao, Y., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https: /ttp:///ppeeeeeee////eee / forum? d= 5xc ecx01h.
where [(Inn) = [f(I), (an))" #o( (co(=))..... Mo(xn)] and the covariance matrix is:
En(
2o( 21)
Eo(In: xn)
Zhang. J. and Liu, K. Thought Space Explorer: Navigating and Expanding Thought Space for Large Language Model Reasoning. In 2024 IEEE International Con- ference on Big Data (BigData), pp. 8259-8251, Los Alamitos, CA, USA, December 2024. IEEE Computer So- ciety. doi: 10.1 (0010099BBgDD0002222222111113813 https: / doi http://.....ccccooooooroocce.... 10. 1109 (wt.1/99//ggatttae 22242233..2222222 2024. 1082563..
We set mo(x)) O, indicating no additional preference for function values at the prior stage. For the covariance matrix Eo(r;, xz)) we use the Gaussian kernel with bandwidth E:
URL
Eo(wi, = k(a,,, mr)) = exp (spp-l11
=exp
||, - 2211? 2F2
The kernel is chosen such that when z; and 2y are close, the kernel value k(x,, 23)) approaches 1, indicating strong correlation, and when they are far apart, the kernel value approaches 0, reflecting weak correlation.
Zhang, W.. Shen, Y., Wu, L., Peng. Q., Wang. J., Zhuang, Y., and Lu, W. Self-contrast: Better reflection through inconsistent solving perspectives. In Ku, L.-W., Mar- tins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3602-3622, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10. 10.18665//1//2044ac--oog
After observing (3 aim infer the value of f(x new point x. Using Bayes' rule (Rasmussen & Williams, 2006), we update the posterior distribution of f(x condi- tioned on these observed values:
197. URL https: http///aaaaatlll......//o org/2024
f(x) I ((rss) N(yn(o)
ael Del-long.13 197/.
where the posterior mean and variance are given by
P& (x) = Eo(a, xisk Do (f (o(x)a)) + po(x),
01((x) = Eo(x, x) Eo(F1s,
[oo xisk (1. Eo(L
A.3. Maximising Expected Improvement
To select the point x that maximises the expected improve- ment (I), we adopt a sampling-basee approach. Specif- ically, we randomly sample large number of candidate
13

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
points from standard normal distribution. In our exper- iments, we generate 5000 points 1:222..---500000 where each point z; I2 is sampled as:
This sublinear bound ensures that, with high probability, the regret grows at slower rate than the number of iterations, thereby guaranteeing the convergence of the optimisation process.
34 ~ 1), i= 1=2,,...,,,00 ,0000.
We set d = 0.1, ensuring a 90% probability of convergence while balancing exploration and exploitation. smaller d (e.g.. 0.01) strengthens theoretical guarantees but increases exploration, slowing convergence. A larger 8 (c.g.. 0.2) favours exploitation, accelerating convergence but weaken- ing guarantees. Our choice provides stability and efficiency without excessive exploration.
For each sampled point, we compute the expected improve- ment EIg (z) using (3). After evaluating the expected im- provement for all sampled points, we select the point with the maximum (2r) as the next candidate for evaluation. This method provides an efficient and practical way to ap- proximate the global maximum of (x) within the sam- pling budget.
A.S. Addressing the Curse of Dimensionality in Bayesian Optimisation
A.4. Adaptive Expected Improvement
Traditional Bayesian optimisation struggles in high- dimensional spaces due to the curse of dimensionality, mak- ing it impractical for embedding vectors used in LLMs. To address this challenge, we leverage the following result from high-dimensional optimisation (Wang et al., 2016; Nayebi et al., 2019), which allows to perform optimisation in lower-dimensional space and to subsequently map points back to the original space.
In defining f(z), we assume an ideal verifier with perfect accuracy, meaning it provides an error-free assessment of correctness. However, in practice, the verifier's ss accuracy is less than 1, introducing uncertainty into its evaluations. This results in noisy observations, where the observed score og deviates from the true function value f(xx). We model this noise as:
0 f(xk) +
Tik (O,
Theorem A.2. Let D be the dimension of the embedding vectors. A function f : RBD + IR is defined have an effective dimensionality de, with de S D, the following condition is satisfied: 3 a subspace E of dimension de, such that VEg E E C RRD and Iι E E C RR, where E is the orthogonal complement of E, the function satisfies: f(zg + +x1) fIg). In other words, de is the smallest dimension that retains all variability of f. Now, for d Z de consider a random matrix A E RR. Dxd with independent (O, 1) entries. Then,
where À is objective noise constant, determined by the inherent noise level. To address this noise in Bayesian opti- misation, we use an adaptive version of the El acquisition function that explicitly accounts for the uncertainty in ob- servations:
(a(x)) - fh wxox(x) [+(x)) fz
Elg (x) = [px(x) - -E] + (z)o
- -lux(z)= - fkl whoN(1) (x)
Vz E RP, Bu E R such that f(x) f(A)).
where we = -7> + 1+ ln(1/) is the adaptvee scal- ing factor, and information gain term "/k is defined as:
This result implies that for any optimiser x² E RR,", there exists a corresponding point u" E TRd such that f(x*) f(Au)). Therefore, instead of performing optimisation in the nigh-dimensional space, we can optimise the function g(u) := f (Au) in the nnver-dimensional space.
The max f((01)))((((x)))
1 log de + X X (1))))
2
[(((11:); (Ixak) represents the mutual information be- tween the function values and the noisy observations, and 5 is hyperparameter in (0, 1), controlling the balance be- tween exploration and exploitation. This adaptation is in- spired by the following observation (Vakili et al., 2021: Tran-The al., 2022):
Suppose initial sampled points and their evaluations are denoted by the set Us
((ri,g())),,....... (ux, ((x))) }. where each u, E I2 has independent standard normal en- tries. Similarly, we initialise E R Dxd as a random matrix with independent (0, 1) entries. We then find the next point to sample Ux+1 E IR by optimising the acquisition function:
Theorem A.I. For any choice of 8 (0, 1), with probability at least 8, the cumulative regret Ry satisfies
4h+1 argmax Ely(u [ux,,
wERa
T
R EiS - = O(rvTT). km
where EIg(- (4.) represents the Expected Improvement conditioned on the current dataset Uy with the objective
14

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
function being g(-) on IRd Then, we augment the dataset
superior results on GSM8K, GSM-Hard, and SVAMP, par- ticularly in the few-shot setting. In a fairer comparison (without any training), our method consistently outperforms the prefix scorer across all tasks.
Uk+1 == U4 U {(u++1, (AAu+1)))
and iterate.
B.3. Additional Details on Neuron Activation Analysis
A.6. Setting the Convergence Threshold
B.3.1. OVERALL ACTIVATION RATE
The convergence threshold determines when the algorithm stops iterating. While smaller thresholds generally lead to more precise results, they can also increase computational costs due to additional iterations. We set 0.01 in our experiments, as this value strikes a balance between conver- gence quality and computational efficiency.
We analyse the neuron activations in the GLU-based MLP layers of the LLaMA model (Naik et al., 2024). Specifically, the hidden representation in the i-th layer is computed as: h' = (act. (act.fn(h'w;) @ kwW3) W
(4)
where @ denotes element- multiplication, and act fn(.) is a non-linear activation function. We consider the j-th neuron inside the i-th FFN layer activated if its activation value [act_fn/ (h' WW) exceeds zero
B. More Experimental Details
B.1. Experiment Settings
We evaluate all methods on four benchmark datasets (GSM8K, GSM-Hard, SVAMP, and StrategyQA) using 200 randomly sampled test examples per dataset, with a max- imum output length of 300 tokens. For SC, FIRE, and CoT-Decoding, we follow baseline settings with a tempera- ture of 0.8 and sample 5 outputs per example for majority voting. For RAP, we adopt the original paper's settings (Hao et al., 2023), using Monte Carlo Tree Search with 4 actions, confidence threshold of 8, depth limit of 5, and 10 search iterations.
We compare with our Bayesian-opimmistton--oedd perturbation method. For a single question (i.e., same prompt), we generate 200 samples using LLaMA and record neuron activations for each of the first 5 output tokens. We then visualise and compare the average activation rates between SC and our approach in Figure 3.
(SC) (Wang et al., 2023c)
B.3.2. KEY NEURON IDENTIFICATION AND VERIFICATION
We identify key neurons by analysing activation rates in SC-generated samples. We first separate these samples into correct and incorrect categories. For each neuron j in layer i, we calculate the activation difference:
B.2. Comparison with Controlled Generation Methods
We additionally compare our method against two recent controlled generation approaches: Trainable Prefix Scor- ers (Mudgal et al., 2024) and Constrained fine-tuning (Qi
Aij aoooreee ((r.j) S (a..)),
et al., 2025a).
where aig denotes the activation value of the j-th neuron in the i-th layer. We first select the top 25% of neurons with the highest activation frequency, then rank these by Ai.j. and select the top 20% with the largest positive values, resulting in 5% of all neurons as "key neurons."
The prefix scorer method uses trainable scorers to guide de- coding, while constrained fine-tuning proposes a fine-tuning objective aimed improving robustness against adversarial prompts. Both require additional training, making direct comparison with our training-free free method less straightfor- ward. For a fair comparison, we implemented baselines without extra training where possible.
This statistical approach to identifying critical neurons is grounded in prior research demonstrating that it is possible to trace information flow within transformers and isolate neurons with causal influence on model predictions. Such studies have used targeted interventions like activation re- placement or ablation to validate neuron importance (Dai et al., 2022; Meng et al., 2022).
Method
Triiiin
Shot
GSMBK
GSM-Hfard
SVAMP
QAategyQA
Constmised Fine Fine-tuing
((LoRA)
78.3
13,6a3
83.5.n1
81.3
Profix Sooeer
Zeno 75.216a
26 16.
836,us
65.2 65.211a
Reconiing
79.4,3.
182
882.13
67.21007
Prefis Sester
X
81.2414
336.14
88.5.1
72.4pii
Soft Resonning
x
Few 14.3
36.7.
90.2.cc
75.61Gs
Table 9. Accuracy (%) comparison with trainable prefix scorers and constrained fine-tuning methods. Our method requires no additional training.
To validate the functional significance of the identified key neurons, we conducted targeted masking experiments fol- lowing this line of work. Masking the critical neurons iden- tified for each input led to a significant drop in accuracy from 62. 14% to 13.27%. As a control, we randomly masked an equivalent number of neurons under identical settings, re- sulting in a considerably higher average accuracy of 41. 1.89%
As shown, while constrained fine-tuning achieves the best performance on StrategyQA with additional training, our requiring any extra training-achieves achieves
15

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
The substantial gap (41.89% + 13.27%) confirms that the identified neurons play essential role in the model's s rea- soning process, beyond what would be expected by chance.
Inarmenione Act. pag. MB)
Intemeeiatns
KV Cache
KV Cade
Dutsct
Methal
Time (mas)
Act. (peak, X4B)
(pns, MB5)
(peak, MB]
RAP
Nearening
Rouoving
1896
B.4. Random Projection Stability
SVAMP
Sa Reueeiig
RAP
142.52
18.41
1464,3
932.4
2080.5
1393.
394.5
1858
494.0
216.
Stutegy
Sal Rousning
KAP
148:73
17:44
1833.5
748.0
1935.
9524
241
1180
375.0
264.0
While dimensionality reduction alleviates the curse of di- mensionality, it may lead to information loss. To examine whether the negative impacts of dimensionality reduction via random projection are controllable, we conducted addi- tional validation experiments.
Table 10. Comparison of inference time and memory usage be- tween RAP and our method.
12.30% of RAP's while also consuming significantly less memory, further validating its computational efficiency ad- vantage.
We tested the stability of random projection by running simulations using 50 different random projection matrices. For each run, applied Bayesian optimisation under the same settings and recorded the final performance. The ex- periments were conducted on the GSM8K dataset using the LLLMM338B-Imm model. The results are summarised in Figure 7.
We also report inference time (minutes) across all baselines:
Method
GSM8K
GSM-Hard
SVAMP StrategyQA
SC(r=0.4)
26.58
33,91
20.54
20.04
SC(r=0.6)
26.12
34.46
21.76
19.87
SC(7=0.8)
27.28
34.86
21.16
20.80
FIRE
26.70
32.26
21.59
20.17
0.8500
Max
Co'I-Decoding
26.56
32.55
21.53
20.60
RAP
184.52
234.14
142.52
149.73
Soft Reasoning 23.15
28.42
18.41
17.44
Q3
0.8450
Table I1. Inference time (minutes) comparison across baselines.
Accuracy
0.8425
Modian
Our method consistently achieves the lowest inference time across all tasks, further demonstrating its efficiency beyond token-level savings.
0.8375
B.6. Prompts for Verifiers
0.8350
Min
We provide four prompt templates corresponding to the verifier strategies introduced in Section 4.3. These templates illustrate how each verifier approach is instantiated:
Figure 7. Distribution of final accuracy across 50 random projec- tion matrices. Performance remains stable, indicating that random projection does not introduce significant variance.
Single-Judge. This prompt asks the model to evaluate the correctness of a single final answer. The user provides a question, along with final answer, and the verifier must decide whether that answer is correct.
The results indicate that performance remains stable across different random projections, with minimal variance be- tween runs. This demonstrates that although some degree of information loss is inevitable, it does not introduce signifi- cant instability into the optimisation process. The chosen dimension (d = 50) offers a good balance between perfor- mance and computational efficiency.
This prompt provides multiple candidate answers and asks the verifier to assess their correctness collectively. The user includes each candidate's 's reasoning, and the verifier classifies which answers are correct.
Multi-Judge.
B.5. Computational Efficiency
The prompt demonstrates scenario where the verifier itself is prompted to re-generate the cor- rect solution for one candidate. If the newly generated solution matches the candidate's ' answer, the candidate is deemed correct.
Single-Generate.
We supplement the token count statistics with comparisons between our method and RAP on inference time and mem- ory usage. To evaluate the latter, focus the two vari- able components: (1) KV cache, and (2) intermediate activa- tions, since model weights remain constant across methods. Using vLLM's block-based memory tracking, we report both average and peak usage, sampled at I-second intervals.
Multi-Generate (Ours). This prompt processes all can- didate answers together and generates a new solution believes be correct. Any candidate matching this newly
The results show that our method's 's inference time only
16

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
generated solution is labeled correct. This leverages the model's generative capacity more thoroughly than pure clas- sification, yielding better verification accuracy in practice.
B.7. Effect of the Number of Samples k.
GSMBK
GSM-Hard
0.93
0.50
0.90
0.45
0.84
0.81
Racu: Coverogg
Recu Coveage
0.78
Sample Size (k)
Sample Size (k)
StrategyOA
0.96
0.95
0.90
0.93
0.84
0.90
0.80
Coverage
Sample Size (k)
Sample Size (k)
Figure 8. Accuracy (solid lines) and standard deviation (shaded areas) across different sample sizes kt.
We investigate how the number of sampled embeddings & in each iteration influences final performance. Figure shows that increases from 3 to 5, the final accuracy and cover- age steadily rise for all evaluated tasks, although gains tend to plateau or fluctuate slightly after & = 3. This upward trend suggests that moderate increase in & promotes better exploration of potentially correct solutions in the embedding space. Based on these observations, we adopt = as our default setting, balancing solution diversity with computa- tional cost.
B.8. Additional Results
Tables and present the full experimental results for all methods and configurations across the four benchmarks and three LLMs (LLaMA3-SB-Instruc) and Mistral-7B -nnstrueB These tables extend the summary reported in the main text, showing detailed accuracy. cover- age, and standard deviations for zero-shot to 8-shot setups. They provide a comprehensive view of how each baseline and our approach perform under various hyperparameter and prompt configurations.
Swen2-7B-Instruue
17

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Prompt for Single-Judge
Based on the given question and the previous answers, please provide your judgment on the correctness of the final answer.
Question:
Jessie currently weighs 9 kilograms. After she started to go jogging every day, she lost 62 kilograms first week and kilograms in the second week. How much did she weigh before starting to jog?
Answer:
140 211. So, Jessie weighed 211 kilograms after weeks of jogging. Since she weighed kilograms initially, she weighed 211 9 202 kilograms before starting to jog. Answer: 202
62
Correct:
Question:
{User )uestion}
Answer: (Previous answers)
Correct:
Prompt for Multi-Judge
Based on the given question and the previous answers, please provide your judgment on the correctness of the final answer.
Question:
Jack is stranded on desert island. He wants some salt to scason his fish. He collects liters of scawater an bucket. the water is 20% salt, how many ml of salt will Jack get when all the water evaporates?
Your previous
Thought: 1250 ml water evaporates, leaving
of Answer:
1. Thought: The total amount of water is 2 liters 2000 ml. The amount of salt 20% of 2000 ml 0.20 x 2000 ml
{00.20 x 2000 = 440) ml. Answer: 400
Thought: 20% of liters is 2
{(2 x 20/100 0.4))0.4 liters. Since there 1000 ml in 1 liter, 0.4 liters is
0.4 x 1000 = (00.4 1000 400)) 400 ml. Answer: 400
20% x liter = ((20 x 0.1 0.2;)0.2 liters of salt. Since Jack has 2 liters of seawater, he will get 0.2 2 = ((0.2 x 2 = 0.4))0.4 liters of salt. Since there are 1000 ml in liter, Jack
Thought: liter scawater is 20% salt. So, liter of scawater
will get 0.4 1000 (00.4 x 1000 (4000 ml of salt. Answer: 400
Thought: 20% of 2 liters is
((2 20/100 /0000 0.4))0.4 liters. There 1000 ml in 1 liter, 0.4 liters
x 1000 (40.4 1000 4003)
400 400
Answer: 400
Correct:
2, 3,
Question:
[User Question}
Your previous answers: (Previous answers)
Correct:
18

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Prompt Single-Generane
Based the given question and the previous answers, please provide your analysis and final answer, starting the final answer with "Answer:"
Question:
Jack is stranded on desert island. He wants some salt to season his fish. He collects liters of seawater in old bucket. If the water 20% salt, how many ml of salt will Jack get when all the water evaporates?
Your previous answers: 1250 ml of water evaporates, leaving 1000 ml of salt. Answer: 1000
Analysis:
Let's think step by step. Jack has liters of scawater, and 20% of salt. liters 2000 ml, so the amount is 20% 2000 ml 0.20 x 2000 400 ml of salt.
Answer:
Question:
{User Question}
Your previous answers: (Previous answers)
Analysis:
19

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Prompt Multi-Generate (ours)
Based on the given question and the previous answers, please provide your analysis and final answer, starting the final answer with "Answer:"
Question:
Artemis is making tea for party. She knows her mom drinks an 8-ounce of tea and uses one ounce of tea. She will use this same ratio for the party. The party has 12 people there and each of them wants 6-ounce cup of tea. How many ounces of tea does she need?
Your previous answers:
0. Thought: 8 ounces of tea for cup, so I ounce of tea for ş of cup. For 12 people, she needs 12 x 9 ounces of tea. Answer: 9
Thought: ounces of is needed each person. Since there are 12 people,
ounces of are needed. Since
each ounce tea used for cup. 72 ounces tea will make 72 cups of tea. Answer: 72
2. Thought: 6 ounces of tea is 6
& 4 of an 8-ounce cup. For 12 people, she needs 12 x 8 = 9 ounces of tea. Answer:
4
3. Thought: 12 x 6 72 ounces of tea needed. Since each ounce of tea is used for cup, Artemis needs 72 ounces of tea. Answer: 72
Thought: ounces tea used for cup. ounces tea, she will use of the amount tea. 12 people,
will need 12 9 ounces of Answer: 9
Analysis:
Let's think step by step. Artemis uses ounce of tea for an 8 8-ounce cup, so for 6-ounce cup, she will use of an ounce of tea. For 12 people, she needs 12 x 9 ounces of tea.
Answer:
9
Question:
{User Question}
Your previous answers: (Previous answers)
Analysis:
Table 12, Performances of different reasoning methods on Accuracy (%) across all benchmarks using Qwen2277BB-nstruett
AIME-2024
GSM8K
Gard
SVAMP
StrategyQA
Method
Zem Shot
Few Shot
Zero Shot
Few Shot
Zero Shot
Few Shot
Zero Shot
Few Shot
Zero Shot
Few Shot
COT
3.3
91.0
91.0
51.5
65.0
93.0
92.0
79.0
90.0
SC(r 0.4)
3,3
3.3
93.340.6
91.821.0
62.340.
68.7
93.2+0.3
93.820.3
78.2
89.6
SC(T = 0.6)
2.2+3.3
2.243.8
93.7
92.2=1.0
62.740.3
68.2=1.4
93.8+0.6
93.1
78.8
90.0=1.3
SCIr = 0.8)
3.311.9
2.241.9
94.0
93.5
62.812.0
68.3
93.740.3
93.7
78.4
88.8223
FIRE
2.21199
3.3433.
91.410.
92.510.4
60.3
65.7
93.5
93.5
78.241.9
89.5209
CoT-Decoding
2.2
2.241.9
93.64199
93.5
61.0423
66.0
94.2
93.8
78.8
89.0+1.
RAP
4.443.4
93.52004
69.1
93.7+2.1
90.4
Soft Reasoning
6.712.7
11.1.17
94.310.3
94.821.3
63.3108
72.2
94.0
94.2
79.6
89.2
20

---

Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration
Table 13, Performances of different rcasoning methods on Accuracy (%) across all benchmarks.
Method
Shor
Shot
LLaMA3-BB RB- Instruct
Shoc
Shor
Shot
S
Qwen2 7B- Iistuctttt
Shor
Shot
Shoe
Shot
Mistral 7B-Inttrcct
Shor
Shot
Shot
GSMMK
COT
53.0
73.0200
75.5
79.0na
77.4gn.
64.5
70.0.0.
66.5.0.
81.5100
82.5.a.
42.0po8
43.5
53.5
54.500
54.0a.
SC 0.4
73.0z1.
75.4424
75.7a14
50.721.1
80.4g1.4
8122.08
82.4g1a
83.7a0.
86.92u
85.7c15
52.92na
53.5-1.5
59.9223
60.4117
58.3
SC
73.
73.4
724
80.6-11
502g1a
84.521
84.2
863309
854ga9
55.1
56.5-20
596607
60.311
S74
SC 0.8)
65.0g2.
74.5
66.8g17
5
81.1g1
50.0g0
82.7ga0
82.6
85.4
85.1g16
502gn4
51.4
564497
59.820
57.224
FIRE
a3
a
764420
7844aa
79.6.50
81,0
81.0
73.6,a
82.5
83.0.sa
47.2
52.
53.
60,6
56.1iaa
CoT-Doodding
76721
786
80.3
83.0g2
81.6
76.222
85.7
84.5gx1
47.3ga0
56.423
57.3
57.6
58
RAP
77
80.4
80.7
84.7.00
85.7g1.0
87.4.09
86.2...
57.6
57.1
59.8.16
586115
Soft Romsnnng
29.4
79.4,0
83.0g5
84.3,14
$8
89,8,9
90.0214
61.2.56
61.2.09
61.4119
GSM--Haad
COT
14.0.c.0
22.5a00
24.5
2605
28.0 28.0
400.ao
39.0.0.
48.0
53.0
55.5.a.0
14:5an0
22.0....
21.5
23.5ano
24.0za.
SCG - 0.4)
25.7
25.3404
28.6212
32.2
31.8218
47.5
48.6-114
55.7gL
55.241a
19:5214
26.5-1
26.921
27.3419
26 16.
SC 0.6)
24.5
25.7
28.43.4
32.0
31.2a1.
462at.
50.1a29
54.2
56.0111
$3448.
20.7a1
25.7.6.
27.8
31.0a25
25.31.
SC 0.8)
21.8
24.8119
28.2a1r
30.6112
30.8.09
47.3
47.8a27
S1.8.Ls
55.7a13
55.4sa
19.1=28
26.8-1
26.61.1
29.8a21
26.6
PIRE
25.2a1.
24.1a1a
27.0gn8
27.9
25.2az1
45.1
43.4220
52.9
49.022a
51.0218
18.1ar
28.8-18
27.0uu
25.52u8
3614
Doco-noodiig
a
28.0ga4
30.321.
46.7ga
44.1g1a
53.9
50.121
52.1g10
16.6077
26.6.30
28.0g07
26.0
A7
RAP
26.7110
31/411a
3244111
32.7
$3.2
S5.
15.1
56.2 55.9
27.
28.0
28.4117
27.6113
Soft Romsoning
28....
30.2
35.3
33,2
35.7.1.0
537711
$7.4
57.5
57.4112
58.7.0.
258:1a
29.1.1.2
32.3
30.0.12
32.3
SVAMP
COT
61.0g6.
81.0100
83.5g
0400
83.0.00
43.5
83.0200
84.0ga.
85.5200
86 86.0ga.
52.0g00
65.0.00
660200
72.0ga
SC - 0.4)
g9.
85.8
86.5.7
87.3
87.1g10
72.3
90.2g1.
89.400
90.310a
90.3g1.
67 67.4
7..5.1.
73.7110
765515
SC = 0.6)
76.1.10
862200
86.8
96.
87.7a12
17.3..a
90.6109
90.20..
91.4ana
90
697714
75.8...
75.60a
7588114
T84:20
SC 0.8)
69.6.20
863322
86.210
4
874-12
786-2.
90.3gur
90.1...
90.8407
90.6-1.2
25.1
76.911
776
FIRE
81.5
36.614
86.aas
86.1g14
87.6250
76.32z
89.9214
89.7gas
89.3
90.6
22.7-1a
76.921
778211
78Ag1a
CoT CoT-Decoding
83.2
87.8410
87.5
87.5
88.2-1.
78.6214
90.3:00
90.0g..
90.3411
89.7
69.4
77.8-1
77.7
76.9225
78.6214
RAP
78.4
86.8410
87.9
90.8.ur
91.2
90. 1u
90.8
0.0
0.0
78.411
194411
Noft Roasoning
28.2...
89.2
19.
00.2-.0.
K34.24
92.3.0.
92.4
90.8
02.2.0.5
72.2
79.0-p.
784
80.0214
82.1
Strniegy
COT
58.5
63.0100
68.0ga0
67.5
68.5
63.0ga4
54.5.00
63.5gs.
66.01no
70.0=aa
62.0gon
62.5_no
63.5
68.5
69,0ga
SC = 0.40
68.1124
68.6
69.2
71.6
67.1
67.1
66.5
69.
g1.
63.9
$7.6
63.9
70.2
72.6
SC = (.6)
59.9220
69.
682aas
70.742a
71.3a15
67.5
66.5217
67.42.
67.7a14
69. 69.1.1.2
64.2a1
5877-1
64.8.0.7
69.6112
71.7203
SC 0.8)
54.4.0
68.0g0
67.9gm
70.41a
72.7.1.
67.0.14
68.0,19
68.3g1.
67.7
70.1
64.114
59.9_12
665
69.914
12.1
FIRE
63.0.37
70.8a14
71.8a13
70:212
72.8.15
67.6.0.
68.4g08
68.4
67.3a12
68.1
642214
66.5.1.
682s1
72.6221
OoT-Decoding
64.621
71.1
1144s0
70.5220
73.3218
65.9cta
68.342
6877
67.5415
69.5-11
63.521a
68.2-04
68
70.6118
72772a
RAP
71.
20.6111
71.5414
73A.1.1
67.5
68.2g1
(8.5413
71.3
68.5-11
70.607
72.1414
724
Noft Rossonngg
67.2.07
71.010
72.3.1
738
75.6.00
68.1g13
09.6110
689
70.311
66.1g1
70.1
71.2g1
727730
728,15
Table 14. Coverage rates of correct answers across different models (%) on all benchmarks.
LLaMA3-8B-Innan Instrrctt Shot
Qwen2-77 7--- 7B-Instuct
Mistral Mistral-7B-Innte 7B-Intrrut
Method
Shot
Shot
Shoo
Shot D
Shot
Shot
Shot D
Shot
Shot 4
Shot
GSMSK
SC(+=0.4)
79.8
69.2
35.7
89.5
91.0.15
23.1
88.9217
91.4g1.
93.5
94.5
73.2gam
69.3_1
35,5
74.91
73.4
SC(+=0.6) ())
78.4
66.61a6
72.6.1.
90.61.3
91.1
91.2
9244
93.8107
91.2
73.0
22.7
762
76.5
73.
SC(r=0)8
71.02..;
62.
66.8a1.
404411
98.2.1.
92.9ala
88.611
91.621.1
93.aana
93.8
71.2a1a
20.1-2
74.3
74.5a28
73.2.1.4
PIRE
84.5
83.9.0
888320
4
88.2.06
868
78.1
84.2g1a
90611
908..0
63.1
70.4-45
709
76.7ax
73.9.3.
CoT-
85.3
18.
90.6i3.1
90.2.0.
90.5g0.
88.42.
81.2a20
85.5
92.0gna
91.7uas
63.2a24
12.6...7
75.5
74.3a23
76.61.2
RAP
87.6
W9.1
89.5-n
38.3
92.4
91.4
92.3-a.
12.4-a.1
74.5a1.
258
75.611
Sofi Reasoming
91.8
91.
9244
92.6114
92.2.nx
95.9.m
96.820
96.4..
96.821
96.62a.
8544..
79.0
82.3
81.9214
82.5
GSM-HHand
SC(r=0.4)
27.
31.7a0a
38.6219
41.6211
43.42n
62.6g
53.4g17
62.4u.
62.6217
64.3
31.1
37.5.1
40.9
39.9
38.2215
SC(+=0.6)
28.2
33.0g0
38.0214
39.9
43.0_11
63,3
56.22nn
62.4g
65.3
65.8_1.
30.211
37.3_60
38.511
41.3
19.
SC(+=0.8) B)
25.1
32.3a24
37.8a1.
39.3
42.2a1.
61.8
$1.9219
61.1ai.
63.8
65.121.
31.7214
38.2-1.
39.2
40.9218
37.0m
FIRE
32.02..
33.1
38.311
3
40.6g90
54.4
49.3
56.7230
56.829
60.611
2622n5
37,4-11
39
39.1
CoT-Decoding
33.1
13.6119
39.8
42.3
42.2.15
55.1
19.3,00
$6.611
5744ns
61.5
27.7
36.9_1.
40.3
38.5
384107
RAP
33.4g14
40.231.5
41
41.9
43.9.1.5
53.7a18
60.2
62.1
64.2.1.
37.6
40.6.1.1
40.3.1.7
38A.14
Soft Reasoning
37.0-
37.0
464415
47.3azi
49.8-10
69.
67.8-u
70.7=L.
71lena
40.7213
43.3-12
44.41a
45.9a11
45.2
SVAMP
SC(+-004)
84.6
91. 31,8
91.622.
92.8404
92.92ma
91.721.
94.320e
93.9g0
93.72ma
94.0=a.
61 911.
85.2213
24.5
86.311
87.720.4
SC(r=0.6)
22.1
92.3g00
92.7a1.
93.0an.
93.8un.
92.3g1.3
94.4and
94.6.o.5
94.3ans
94.0u1.0
67.5
85.7
86.3.1.2
87.9.14
88.9.14
SC(+=0.8)
73.8-22
91.7
93.3ea
93.5
94.62a7
92.5et.
94.5207
94.5
91.1an
94,0=a0
72.1
86.5
87.1et.
88.3210
88.2.1.
PIRE
89.9
93.7203
95.5200
92.41a
93.821
21.5
93.9211
94.7210
94.518
95.1
(2.2
87.9218
89.
90.211
CoT Decoding
90.8_e.
93.5g1
94.2.34
93.8
93.9g1a
91
94.4.65
94.2
94.6110
93.4
67.0gna
89.5_66
$8.7g13
88.9217
90.411
RAP
93.6411
93.8aa
9344
94.1211
91.3
94.72a0
91.7412
94.8-ar
88.7-12
89.7
89.7413
90.721
Soft Reasoning
93.M
95.5go
45.5ga.
95.1
M5.M
97.0.07
96.Man
95.600.3
96.0.1
97.8u35
91.2
90.5
91.020.5
Smaiogy24
SC(r=0144)
85 5
M4.7g1
84 D
75.7.10
84.7.0a
K5.7
K5.0a17 7
26.7
738
70.5
74.9
84.4gon
45
SC(==0.6)
81.1
89.2
86.3ga
88.5
83.2
85.1
86443.
85.9
87.5
75.0gna
73.6_1a
76.911
86.1
$8.0
SC(re0) B)
86.6222
88.9227
87.1
lana
89.6.12
848.17
84.9.2
85.2a12
8822115
760014
74.1-00
77.2a1
878811
89 8
FIRE
91.22u.
92.6410
90.1221
89. 89.1
98.6211
84.4.L
87.0217
884413
89.4
74.
81.021.
796614
86.4217
CecoDng
92.4.c4
98.4a11
87.4114
89.1
90.6.1.6
84.7
87.3a18
86.9
86.81n
88.9-24
75.62na
82.2.2.
79.7a14
87.3a14
890
RAP
90.5x12
89.62144
89.3
91.1a1
88.6
87.6
88.6x12
88.7e1.
81.6-1.
79221.
86.6412
89.311
Soft Reasoning
93.7 21.4
94.011
93.5
90.9
93.3.1a
884.0.
89.0.os
89.8...
91.610
90441.0
81.2,13
84.1.07
824,14
87.3
902.1
21


Reviews:
[{"review_id": "EBcF3qAbl9", "reviewer": "Reviewer_M1nu", "summary": "To address the challenges of insufficient diversity and low search efficiency in large language models (LLMs) for complex reasoning tasks, this paper introduces a novel responsive sampling strategy. By applying Gaussian perturbations to the embedding of the first token generated by the LLM, using the correctness and coherence of the output as the objective function, and leveraging Bayesian optimization to iteratively search for optimal embedding points, this approach avoids the blindness of traditional temperature tuning and the inefficiency of heuristic search. Experimental results demonstrate significant accuracy improvements on three mathematical reasoning datasets and one commonsense reasoning dataset. Additionally, the study validated that Bayesian iteration enhances neuron activation rates in the MLP layers of LLMs, providing neurophysiological evidence for the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Verifier Setup**\n\nThanks for the comments. Regarding the questions about details of the verifier, **NO** separate or stronger LLM is used for verification (Line 31-33, right column); the same model as the generator is employed (i.e. LLaMA3-8B-Ins, Qwen2-7B-Ins, and Mistral-7B-Ins). \n\n- The verifier score $r_{verifier}(y) = \\mathbb{1}_{y_v = y}$ is a binary indicator, where $ y_v$ is the model's regenerated answer based on the current question and previous responses. Specifically,  $y_v$ is obtained by prompting the same LLM with a verification query:\n\n  > *\"Based on the given question and the previous answers, please provide your analysis and final answer.\"*\n\n  The exact prompts used are listed in Appendix B.3. \n- We also conducted ablation studies (see *Verifier Comparison: Judgment vs. Generation*, Line 427, left column) to compare different verifier strategies and assess their impact on performance.\n\nThis setup ensures a fair comparison, as no external or more capable model is used.  It also highlights one of the main contributions of this work: our BO framework's ability to enhance reasoning capabilities using a single unified LLM without additional verifiers. Details on the verifier will be added to our paper.\n\n**2. Dimension Reduction**\n> While dimensionality reduction alleviates the curse of dimensionality, random projection may lose critical information. Further validation is needed to confirm that the negative impacts of dimensionality reduction are controllable.\n\nWe agree that dimensionality reduction may lead to some information loss. However, it offers a practical tradeoff: retaining more information in high-dimensional spaces often results in poor sample efficiency for Bayesian optimisation, making it harder to find good configurations under a limited evaluation budget. As shown in Figure 6, we  experimented with several projection dimensions and found that our current setting (d=50) provides a good balance between optimisation performance and computational cost. \n\nTo assess the stability of random projection, we ran simulations using 50 different random projection matrices. The box chart below shows the distribution of results:\n\n```\n0.850 ┤   ┐      ◀ Max\n      │   │\n0.846 ┤ ┌─┘─┐    ◀ Q3\n      │ │   │\n0.842 ┤ │ ─ │    ◀ Median\n      │ │   │\n0.838 ┤ └─┐─┘    ◀ Q1\n      │   │\n0.835 ┤   ┘      ◀ Min\n```\n\nThe performance remains stable across multiple runs, indicating that random projection does not introduce significant variance or instability.\n\n**3. Critical neurons**\n\n> The definition of critical neurons relies on statistical methods; citing related work would help justify this choice.\n\nWe appreciate the reviewer’s concern. Prior works have shown that it is possible to trace information flow within transformers and identify neurons with causal influence on model predictions by applying targeted interventions, such as activation replacement or ablation [1, 2].\n\nFollowing this line of work, we **mask the critical neurons** identified for each input. This leads to a **significant drop** in accuracy to **13.27%** (from **62.14%** before masking). For comparison, we randomly masked the same number of neurons and repeated the experiment under identical settings, resulting in an average accuracy of **41.89%**. This substantial gap (**41.89%→13.27%**) demonstrates that the identified neurons are indeed functionally important, beyond what would be expected by chance.\n\n[1] Damai Dai et al., 2022. Knowledge Neurons in Pretrained Transformers. ACL\n\n[2] Kevin Meng et al., 2022. Locating and Editing Factual Associations in GPT. NeurIPS\n\n**4. Computational efficiency**\n\n> Regarding computational efficiency, readers may expect comparisons of time and space costs rather than solely token count statistics.\n\nWe thank the reviewer for the suggestion. We supplement the token count statistics with comparisons between our method and RAP on inference time and memory usage. To evaluate the latter, we focus on the two variable components:\n(1) KV cache, and (2) intermediate activations, since model weights remain constant across methods. Using vLLM’s block-based memory tracking, we report both average and peak usage, sampled at 1-second intervals.\n\n|Dataset|Method|Time(min)|Intermediate Activations(avg,MB)|Intermediate Activations(peak,MB)|KV Cache(avg,MB)|KV Cache(peak,MB)|\n|-|-|-|-|-|-|-|\n|GSM8K|RAP|184.52|1628.7|1874.4|252.5|568.0|\n|GSM8K|Ours|**23.15**|**1137.2**|**1178.1**|**176.5**|**312.0**|\n|GSM-Hard|RAP|234.14|1985.4|2354.9|426.5|574.0|\n|GSM-Hard|Ours|**28.42**|**881.2**|**1096.2**|**254.6**|**336.0**|\n|SVAMP|RAP|142.52|1464.8|2089.5|384.5|494.0|\n|SVAMP|Ours|**18.41**|**932.4**|**1393.2**|**185.8**|**296.0**|\n|StrategyQA|RAP|149.73|1833.5|1935.9|241.4|376.0|\n|StrategyQA|Ours|**17.44**|**748.0**|**932.4**|**118.0**|**264.0**|\n\nThe results show that our method's inference time is only 12.30% of RAP's while also consuming significantly less memory, further validating its computational efficiency advantage.", "claims_and_evidence": "The embedding optimization-based framework proposed in the paper significantly enhances the accuracy and efficiency of LLMs in complex reasoning tasks through Gaussian perturbation and Bayesian optimization. Experimental results demonstrate that it outperforms mainstream baseline methods on multiple benchmark datasets. Ablation studies validate the necessity of both the verifier and coherence terms, while neuron activation analysis reveals its mechanism of enhancing reasoning through diverse neural pathways and further proves the effectiveness of Bayesian optimization. A minor limitation is that the description of the verifier-guided approach remains somewhat ambiguous, particularly regarding how it produces a refined output y_v in the experimental section.", "methods_and_evaluation_criteria": "The paper proposes a responsive sampling method to address the challenges of insufficient answer diversity and low search efficiency in large language models (LLMs) for complex reasoning tasks. It ensures generation diversity through Gaussian embedding perturbation of the first token and greedy sampling, explores the embedding space via iterative Bayesian optimization to ensure accurate and coherent answers, and employs dimensionality reduction techniques to effectively tackle the high computational costs of high-dimensional embedding spaces. The method is compared with three strong baseline methods (CoT, Self-Consistency, FIRE) on three mathematical reasoning datasets and one commonsense reasoning dataset, validating its effectiveness in complex reasoning tasks.", "theoretical_claims": "Yes, the theoretical foundation of the paper incorporates the Expected Improvement (EI) from Bayesian optimization theory and random projection methods for dimensionality reduction. These concepts are provided with specific and clear explanations in both the main text and the appendices.", "experimental_designs_or_analyses": "In terms of dataset selection, recent research indicates that reasoning capabilities demonstrated in mathematical problems can generalize to other tasks. The paper also conducts experiments on more general commonsense reasoning tasks, validating the generalization ability of the proposed method. This dataset selection is therefore reasonable. For model selection, the paper employs Llama-3.1-8B-Instruct、Qwen2-7B-Instruct, and Mistral-8B Instruct as backbone models, avoiding dependence on specific architectures. Regarding baseline selection, the study includes CoT Prompting、Self-Consistency Decoding、FIRE、CoT-Decoding、multi-path generation and RAP (Monte Carlo Tree Search), covering both mainstream and state-of-the-art methods. The experimental section first evaluates the overall performance of the method on complex reasoning tasks. Under zero-shot and few-shot settings, it compares the proposed approach with strong mainstream baselines on three mathematical reasoning datasets and one commonsense reasoning dataset. Significant accuracy improvements demonstrate that controlled embedding exploration outperforms existing baselines in accuracy, diversity, and efficiency. The paper then conducts interpretability analysis by comparing neuron activation rates in MLP layers across different iterations, illustrating the effectiveness of Bayesian iteration. Additionally, ablation studies validate the contributions of the objective function and dimensionality reduction techniques.", "relation_to_broader_scientific_literature": ".", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths: \n1. The main experimental results demonstrate significant improvements compared to baselines. \n2. The methodology section is logically structured, guiding readers through the process of exploring the embedding space, Bayesian optimization, the required objective function and iterative procedures, and dimensionality reduction to mitigate inference costs. The appendices provide corresponding and detailed supplementary explanations. \n3. The experimental design is rigorous, with sufficient motivation and extensive experimental validation for each component of the method. \nWeaknesses: \n1. While dimensionality reduction alleviates the curse of dimensionality, random projection may lose critical information. Further validation is needed to confirm that the negative impacts of dimensionality reduction are controllable. \n2. Critical neurons play a pivotal role in the experiments, but their definition and identification rely solely on statistical-based approaches. The statistical significance of this method lacks theoretical justification, and it would be beneficial to supplement it with relevant prior work as a feasibility rationale. \n3. Regarding computational efficiency, readers may expect comparisons of time and space costs rather than solely token count statistics. \n4. It would be desirable to include additional details on the verifier’s experimental setup and implementation.", "other_comments_or_suggestions": "No", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dLwpXevUGa", "reviewer": "Reviewer_8Nqh", "summary": "This paper introduces a novel embedding-based search framework to enhance complex reasoning in Large Language Models (LLMs). It perturbs the embedding of the first token with Gaussian noise and optimizes this perturbation via Bayesian optimization (BO), guided by a verifier model. Experiments on multiple challenging datasets (GSM8K, GSM-Hard, SVAMP, StrategyQA) across three models (LLaMA, Qwen, Mistral) demonstrate consistent and significant accuracy improvements over existing strong baselines, especially in zero-shot settings. The approach achieves these gains efficiently, converging in few iterations without requiring access to model internals. Overall, the method offers a scalable and effective way to systematically improve LLM reasoning capabilities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Verifier Reliability**\n\nThanks for the comment. You are right in saying that incorporating more reliable, domain-specific verifiers could further improve performance. Our framework is flexible enough to integrate such tools, and we agree that doing so would likely yield gains in tasks where those verifiers are applicable [1].\nThat said, our current focus is on improving reasoning in scenarios where such external tools are not available or applicable — for instance, in common-sense or social reasoning, where exact computation isn’t useful. By relying on internal LLM verifier signals, we aim to explore the model's capacity to self-reflect and reason without additional guidance  (Line 31-33, right column).\n\n[1] Large Language Models for Mathematical Reasoning: Progresses and Challenges. ACL\n\n**2. Scaling to Larger Models**\n\nThanks for the suggestion. Note that our method involves modifications within the model’s parameter space, and due to the closed-source nature of GPT-4, it is not possible to test on it. Nonetheless, the point is well-taken and we have conducted additional experiments using Qwen2-72B-Instruct, a significantly larger model (72B) than those used in the main paper (7B and 8B), across both zero-shot and few-shot settings:\n\n|Method|AIME-2024(ZS)|AIME-2024(FS)|GSM8K(ZS)|GSM8K(FS)|GSM-Hard(ZS)|GSM-Hard(FS)|SVAMP(ZS)|SVAMP(FS)|StrategyQA(ZS)|StrategyQA(FS)|\n|-|-|-|-|-|-|-|-|-|-|-|\n|COT|0|3.3|91.0|91.0|51.5|65.0|93.0|92.0|79.0|90.0|\n|SC(τ=0.4)|3.3±2.7|3.3±3.3|93.3±0.6|91.8±1.0|62.3±0.6|68.7±1.5|93.2±0.3|93.8±0.3|78.2±0.8|89.6±1.4|\n|SC(τ=0.6)|2.2±3.3|2.2±3.8|93.7±0.3|92.2±1.0|62.7±0.3|68.2±1.4|93.8±0.6|93.1±0.5|78.8±1.6|90.0±1.3|\n|SC(τ=0.8)|3.3±1.9|2.2±1.9|94.0±0.5|93.5±0.5|62.8±2.0|68.3±0.3|93.7±0.3|93.7±0.3|78.4±2.5|88.8±2.3|\n|FIRE|2.2±1.9|3.3±3.8|91.4±0.8|92.5±0.4|60.3±0.6|65.7±2.0|93.5±0.9|93.5±0.5|78.2±1.9|89.5±0.9|\n|CoT-Decoding|2.2±1.9|2.2±1.9|93.6±1.9|93.5±1.1|61.0±2.3|66.0±1.7|**94.2**±1.2|93.8±1.5|78.8±1.6|89.0±1.5|\n|RAP|-|4.4±3.4|-|93.5±0.4|-|69.1±5.2|-|93.7±5.0|-|**90.4**±5.0|\n|Ours|**6.7**±2.7|**11.1**±1.7|**94.3**±0.3|**94.8**±1.3|**63.3**±0.6|**72.2**±0.6|94.0±1.0|**94.2**±1.3|**79.6**±0.3|89.2±1.2|\n\nWhile we still see some improvements, the gains are indeed smaller on easier benchmarks like GSM8K and SVAMP, as large models can already achieve very high accuracy. \nTo further stress-test our method, we evaluated it on the more challenging `AIME-2024 dataset`. Here, we observe that our approach continues to yield significant gains even at the 72B scale, showing its robustness and scalability to harder tasks and more capable models. We will add these results to clarify the method’s applicability beyond smaller-scale settings.\n\n\n**3. Optimisation Scope**\n\nTo investigate this, we conducted additional experiments where we optimised embeddings for the first k tokens (instead of just the first): \n\n|#token(k)|GSM8K(ZS)|GSM8K(FS)|GSM-Hard(ZS)|GSM-Hard(FS)|SVAMP(ZS)|SVAMP(FS)|StrategyQA(ZS)|StrategyQA(FS)|\n|-|-|-|-|-|-|-|-|-|\n|1(ours)|**79.4**±1.2|**84.3**±1.4|**28.2**±1.8|**35.7**±1.0|**88.2**±1.3|**90.2**±0.6|**67.2**±0.7|**75.6**±0.8|\n|2|75.0±1.8|83.3±1.3|24.8±0.6|34.7±1.0|83.0±0.5|90.2±1.4|68.5±0.5|74.2±1.3|\n|5|69.7±3.5|81.2±2.1|22.2±1.0|29.8±0.8|85.5±0.3|88.3±0.3|66.3±0.6|71.7±1.2|\n|10|61.0±3.1|73.7±2.3|17.2±1.6|23.8±0.3|82.8±0.8|86.8±0.8|67.3±1.9|71.7±1.6|\n|20|52.2±2.3|62.0±2.6|19.0±1.3|18.2±1.9|74.3±0.6|81.5±2.0|67.8±2.5|68.7±1.1|\n\nThe performance generally degrades as k increases, especially beyond 5 tokens. This suggests that naively extending to multiple tokens can introduce instability or overfitting. \nWe also compare with RAP (one of our baselines), a tree-search-based method that operates at the sequence level rather than token-by-token—though it shares similar ideas with token-by-token search. While RAP achieves strong performance, it incurs substantially higher cost and still underperforms our approach. \nDeveloping a multi-token optimisation strategy that can achieve both high accuracy and cost-effectiveness would require deeper investigation and extensive experimentation. It remains an interesting direction for future work and would warrant a separate paper.\n\n**4. Coherence Evaluation**\n\nWe appreciate the reviewer’s observation. To address this, we evaluated coherence using two metrics: perplexity and a coherence score rated by DeepSeek-R1-Distill-Llama-70B, prompted to rate responses from 1 (poor) to 5 (excellent). We tested 800 samples (200 from each of four datasets) using LLaMA3-8B-Ins as the base model: \n\n|Type|SC(τ=0.4)|SC(τ=0.6)|SC(τ=0.8)|FIRE|CoT-Decoding|Ours|\n|-|-|-|-|-|-|-|\n|Perplexity ↓|7.22|8.51|9.98|6.87|6.73|**6.53**|\n|Coherence Score ↑|3.93|3.88|3.73|3.70|3.80|**4.08**|\n\nWe would like to mention that coherence is not the main objective of our method, but serves as a filtering criterion to discard low-quality completions and support answer correctness.\n\n**5. Related Works** \n\nWe thank the reviewer for the suggestions. We will discuss them in the final version.", "claims_and_evidence": "The claims made regarding improved accuracy, diversity of solutions, and computational efficiency are convincingly supported by thorough empirical evaluation. Results consistently favor the proposed method over baselines like Chain-of-Thought, self-consistency, and FIRE. One minor weakness is the claim about improving coherence, which is indirectly inferred from correctness but not explicitly measured.", "methods_and_evaluation_criteria": "The proposed methods (embedding perturbation and Bayesian optimization) are clearly justified and suitable for the problem of enhancing reasoning in LLMs. Evaluation criteria (accuracy, coverage), datasets (GSM8K, GSM-Hard, SVAMP, StrategyQA), and baselines used (CoT, FIRE, RAP, etc.) are appropriate and widely accepted for benchmarking LLM reasoning improvements.", "theoretical_claims": "The theoretical underpinnings (Bayesian optimization, embedding perturbation, dimension reduction) are sound and appropriately applied, though no fundamentally new theoretical insights are developed. The key theoretical novelty lies in the innovative combination of embedding-space optimization and BO for controlling LLM generation trajectories.", "experimental_designs_or_analyses": "The experimental design is robust, sound, and fair. Results were validated with multiple seeds and clearly show low variance, enhancing reliability. Ablation studies (token placement, verifier strategies, embedding dimensionality) further validate design choices. A minor limitation is evaluating on subsets (200 samples per dataset), though this is mitigated by randomness and multiple seeds.", "relation_to_broader_scientific_literature": "The paper advances beyond standard methods like chain-of-thought prompting and heuristic decoding strategies (self-consistency, FIRE, RAP) by proposing a principled embedding-space search strategy. Unlike discrete-token sampling methods, it leverages continuous embedding perturbation and Bayesian optimization, significantly improving reasoning accuracy. It also aligns with recent trends exploring LLMs' abilities for self-verification and iterative refinement.", "essential_references_not_discussed": "The paper should discuss foundational work on **diverse decoding**, particularly \"Diverse Beam Search\" [1] (Vijayakumar et al., 2016), which explicitly balances diversity and quality in generation. Additionally, \"Self-Refine\" [2] (Madaan et al., 2023) is a relevant **iterative self-improvement** method, which aligns conceptually with the idea of model-guided refinement.\n\n[1] Vijayakumar, A. K., Cogswell, M., Selvaraju, R. R., Sun, Q., Lee, S., Crandall, D., Batra, D. (2016). Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1).\n\n[2] Madaan, A., Yazdanbakhsh, A., Tandon, N., Gupta, P., Alon, U., Yang, Y., Yazdanbakhsh, A. (2023). Self-Refine: Iterative Refinement with Self-Feedback for Large Language Models. arXiv preprint arXiv:2303.17651.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- Strong empirical results demonstrating significant accuracy improvements.\n- Computational efficiency and model-agnostic nature make it widely applicable.\n- Clear and rigorous presentation of methods, analyses, and experiments.\n\n**Weaknesses:**\n\n- Dependence on potentially imperfect internal LLM verifier signals.\n- Perturbation limited to the first token; may not correct deeper reasoning errors.\n- Experiments restricted to smaller-scale models and specific reasoning tasks, raising questions about scalability to larger models or broader domains.", "other_comments_or_suggestions": "1. **Verifier Reliability:** Could you clarify whether incorporating external or domain-specific verifiers (e.g., exact math calculators) is straightforward within your framework, and whether you expect substantial performance gains from a more reliable verifier?\n2. **Scaling to Larger Models:** Have you performed initial tests on larger models (e.g., GPT-4 scale)? Would you anticipate similar performance improvements, or could diminishing returns become significant at larger scales?\n3. **Optimization Scope:** Have preliminary experiments been conducted on optimizing embeddings for multiple tokens rather than just the first token, and would you expect significant gains from such extensions?", "questions_for_authors": "1. **Verifier Reliability:** Could you clarify whether incorporating external or domain-specific verifiers (e.g., exact math calculators) is straightforward within your framework, and whether you expect substantial performance gains from a more reliable verifier?\n2. **Scaling to Larger Models:** Have you performed initial tests on larger models (e.g., GPT-4 scale)? Would you anticipate similar performance improvements, or could diminishing returns become significant at larger scales?\n3. **Optimization Scope:** Have preliminary experiments been conducted on optimizing embeddings for multiple tokens rather than just the first token, and would you expect significant gains from such extensions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F39KBWBiSf", "reviewer": "Reviewer_hpZB", "summary": "This work proposes a novel way of exploring the search space of LLM responses via perturbing the given input embedding in the generated sequence. In particular, authors design an online learning scheme that uses bayesian optimization to adjust parameters of the noise so that the generated outcome (after adjustment) leads to better reward or score. Authors conduct thorough comparison with strong baselines to show improvements coming out of from better search exploration in contrast to naive sampling with temperature.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> In my opinion this work could get even more impact and recognition if authors might show the effectiveness of this approach in terms of sample efficiency when its used during training with preference optimization or reward-based training.\n\nThank you for this insightful and forward-looking suggestion. We agree that extending our embedding-based exploration method to training—particularly within preference optimisation frameworks—to enhance sample efficiency could be a promising direction.\n\nSpecifically, our Bayesian optimisation approach to embedding perturbations can be integrated into preference optimisation as an efficient and targeted data generation mechanism. A key bottleneck in current reward-based training frameworks lies in the use of standard generation methods (e.g., temperature sampling), which often produce outputs that, while differing in perplexity, appear similarly plausible—making it difficult for reward models or human annotators to assign differentiated scores. Our proposed method could indeed offer an alternative approach. Instead of relying on random sampling or temperature-based decoding to generate candidate answers, our method explores promising regions in the embedding space, producing diverse and high-quality outputs with fewer trials. During training, we hope that these outputs can be evaluated using human preferences or reward models to construct preference pairs or ordered sequence with more significant distinct scores, which is able to provide more efficient supervision signals for optimising the model.\n\nIf it is possible, by having those more distinctive generated samples and preference-aligned candidates in each iteration, the training process can access richer supervision without increasing the number of queries or annotations. This could lead to better utilisation of limited feedback and help the model learn more effectively from each batch of generated candidates. Additionally, one of the main properties of our proposed framework is that it is model-agnostic and lightweight. It can be seamlessly integrated into standard preference optimisation pipelines without modifying model architectures or requiring access to gradients.\n\nWe appreciate the suggestion and view this integration as a promising approach for generating more preference-aligned data during training. We plan to explore its impact on training efficiency and reasoning performance in future work.", "claims_and_evidence": "- Simple token-level distribution adjustment (increasing the temperature) leads to higher chances of sampling hallucinated or degenerate outputs. The proposed method instead is steering the sequence in a desirable way according to a given reward/score model (or self-rewarding score)\n- The idea of using random projections to reduce the dimension of the embedding to optimize is well motivated by highlighted issues of bayesian optimization in high-dimensional space", "methods_and_evaluation_criteria": "Proposed method effectively improves sample efficiency of the search within the search space when an outcome based reward or the score is available. Authors showed on real world examples how this allows us to find a better prediction or outcome compared to more usual best-of-n methods that iterate over independent samples from the model.", "theoretical_claims": "Methodology described in this work is clearly written, although I am not an expert in Bayesian optimization, in particular.", "experimental_designs_or_analyses": "I read experimental results and analysis: the choice of baselines makes sense. The only experiment which might be very useful here is to verify how useful this approach might be in RL training where we need to sample useful trajectories from an LLM. This method could be a promising direction for training-time sampling strategies.", "relation_to_broader_scientific_literature": "The idea itself does not have any groundbreaking components in it, but its a smart application of known methods to come up with a very useful framework for guided generation with optimization during inference. Related work provides meaningful connection to other guided generation algorithms.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "In my opinion this work could get even more impact and recognition if authors might show the effectiveness of this approach in terms of sample efficiency when its used during training with preference optimization or reward-based training.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1vqet7UgbA", "reviewer": "Reviewer_MeVe", "summary": "This paper proposed a Bayesian Optimization based approach to improve the test time performance of pre-trained LLMs. The authors propose to sample from an initially Gaussian distribution, where the perturbation vector is added to the distribution of the first token in the answer to control answer generation. In order to improve the answer, the authors proposed coherence reward and verifier reward as the objective for BO. Expected Improvement (EI) has been applied as the acquisition function.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Verifier Setup**\n\nRegarding the questions about details of the verifier, **NO** separate or stronger LLM is used for verification (Line 31-33, right column); the same model as the generator is employed (i.e. LLaMA3-8B-Ins, Qwen2-7B-Ins, and Mistral-7B-Ins). \n\n- The verifier score $r_{verifier}(y) = \\mathbb{1}_{y_v = y}$ is a binary indicator, where $ y_v$ is the model's regenerated answer based on the current question and previous responses. Specifically,  $y_v$ is obtained by prompting the same LLM with a verification query:\n\n  > *Based on the given question and the previous answers, please provide your analysis and final answer.*\n\n  The exact prompts used are listed in Appendix B.3. \n- We also conducted ablation studies (see *Verifier Comparison: Judgment vs. Generation*, Line 427, left column) to compare different verifier strategies and assess their impact on performance.\n\nThis setup ensures a fair comparison, as no external or more capable model is used.  It also highlights one of the main contributions of this work: our BO framework's ability to enhance reasoning capabilities using a single unified LLM without additional verifiers. We will add these details on the setup of the verifier to our paper in the next revision.\n\n**2. Related Works**\n\n> Lack of discussions and comparisons to other controlled generation works [1,2]\n\nThanks, we will add the comparison in the revision. Our goal is to propose an **efficient reasoning framework that does not require additional or stronger verifiers, nor task-specific model fine-tuning**. Instead, we leverage verification from the model itself to enhance the reasoning ability of LLMs. \n\nThe methods in [1,2] serve different purposes: [1] uses trainable prefix scorers to guide decoding, while [2] proposes a fine-tuning objective aimed at improving robustness against adversarial prompts. Both approaches require additional training, making direct comparison with our training-free method less straightforward. \n\nWe implemented a baseline based on [1] using a blockwise best-of-8 decoding (block size=16), where the model scores its own without any fine-tuning. To improve the scorer's quality, we applied self-consistency. For [2], we performed token-wise constrained fine-tuning on LLaMA3-8B-Ins model using LoRA (5 epochs, learning rate 2e-5, rank 16). \n\n|Method|Training|Shot|GSM8K|GSM-Hard|SVAMP|StrategyQA|\n|-|-|-|-|-|-|-|\n|Constrained Fine-tuning[2]|✅(lora)|-|78.3±0.7|13.6±0.5|83.5±0.7|**81.3**±0.8|\n|prefix scorer[1]|❌|Zero|75.2±0.9|26.1±1.3|83.6±0.9|65.2±1.3|\n|ours|❌|Zero|79.4±1.2|28.2±1.8|88.2±1.3|67.2±0.7|\n|prefix scorer[1]|❌|Few|81.2±1.6|33.6±1.4|88.5±1.2|72.4±1.1|\n|ours|❌|Few|**84.3**±1.4|**35.7**±1.0|**90.2**±0.6|75.6±0.8|\n\nAs shown, while [2] achieves the best performance on the StrategyQA task with additional training, our method—without requiring any extra training—achieves superior results on the other three tasks, particularly in few-shot. In a fairer comparison (i.e. without training), our method consistently outperforms the prefix scorer [1] across all tasks.\n\n[1] Controlled Decoding from Language Models. ICML\n\n[2] Safety alignment should be made more than just a few tokens deep.\n\n**3. Why choose EI?**\n> There exist multiple acquisition functions for BO, such as UCB score, etc., is there any theoratical/empirical results that support the choice of EI?\n\nIndeed, PI and GP-UCB can also be used as the acquisition function. The cumulative regret for GP-UCB has the same rate (Theorem A.1) as EI [3]. Unlike EI or GP-UCB, PI only considers the probability of improvement, without accounting for its magnitude. It is thus less theoretically grounded and more prone to premature exploitation [4].\n\n|Method|GSM8K(ZS)|GSM8K(FS)|GSM-Hard(ZS)|GSM-Hard(FS)|SVAMP(ZS)|SVAMP(FS)|StrategyQA(ZS)|StrategyQA(FS)|\n|-|-|-|-|-|-|-|-|-|\n|ours(EI)|**79.4**±1.2|**84.3**±1.4|**28.2**±1.8|35.7±1.0|**88.2**±1.3|**90.2**±0.6|**67.2**±0.7|75.6±0.8|\n|PI|74.6±1.5|82.1±1.2|28.0±1.5|35.2±0.8|85.3±1.0|89.5±2.2|66.9±1.8|74.3±0.4|\n|UCB β=1|76.7±1.3|83.3±0.3|27.7±1.6|34.7±1.2|86.0±1.5|88.0±0.0|66.7±1.5|74.7±1.5|\n|UCB β=2|77.9±1.9|83.3±1.6|27.8±0.8|**36.2**±1.5|85.0±1.0|88.7±1.4|66.8±2.3|**75.7**±1.9|\n|UCB β=5|75.6±1.9|81.8±0.9|27.7±0.8|34.2±2.5|85.3±0.8|89.7±0.8|66.7±1.0|75.0±0.7|\n\n\nThe experiments show that PI consistently underperforms compared to EI, as expected from the theoretical discussion above. For GP-UCB, its performance is sensitive to the choice of the exploration parameter and is, in most settings, worse than EI. We would also like to mention that the optimal parameter choice for GP-UCB varies across different tasks, making it difficult to guarantee good performance in unseen settings. In contrast, EI performs robustly without requiring task-specific tuning.\n\n[3] Taking the human out of the loop: A review of Bayesian optimization. Proc. IEEE\n\n[4] Gaussian process optimization in the bandit setting: No regret and experimental design. ICML", "claims_and_evidence": "Strength:\n- The proposed algorithm improves the performance of three different base LLMs, and has demonstrated better performance than other perturbation/sampling algorithms.\n- BO algorithm converges as shown in Figure 5.\n\nWeaknesses:\n- Lack of discussions and comparisons to other controlled generation works [1,2]\n\n[1] Mudgal, Sidharth, et al. \"Controlled Decoding from Language Models.\" International Conference on Machine Learning. PMLR, 2024.\n[2] Qi, Xiangyu, et al. \"Safety alignment should be made more than just a few tokens deep.\" arXiv preprint arXiv:2406.05946 (2024).", "methods_and_evaluation_criteria": "There is one tricky part that makes me confused about the method:\n- How is the verifier score calculated? What is the verifier? Is it another LLM?\n\nIn L189, there lacks of explanation about $y_v$. I think the choice of verifier matters a lot on the performance and has a large influence on the impact of this algorithm. If the verifier is another more performant LLM, I'd like to see the performance of that LLM. It is very hard to judge the contribution of this paper without such information and this also raises questions about whether the comparsion is fair.\n\nAnother question regarding algorithm is that:\n- Why choose EI?\n\nThere exist multiple acquisition functions for BO, such as UCB score, etc., is there any theoratical/empirical results that support the choice of EI?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Most experimental designs are fair, but I'd like to know more details about the choice and design of the verifier.", "relation_to_broader_scientific_literature": "People working on autoregressive model might also be interested in this approach.", "essential_references_not_discussed": "[1] Mudgal, Sidharth, et al. \"Controlled Decoding from Language Models.\" International Conference on Machine Learning. PMLR, 2024.\n[2] Qi, Xiangyu, et al. \"Safety alignment should be made more than just a few tokens deep.\" arXiv preprint arXiv:2406.05946 (2024).", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rmnw0PyXMO", "reviewer": "Reviewer_W5Wg", "summary": "The authors proposed a new LLM inference sampling framework that searches for different reasoning paths by controling a Gaussian embedding that is inserted to the sequence. They proposed embedding perturbation for controlling the sampling in the continuous space, and come up with Bayesian optimisation to guide the sampling via a verifier-guided objective. Comprehensive comparisons with other sampling strategies and ablation studies are provided for proving the effectiveness and efficiency of the framework.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.Why Embedding Search Outperforms Discrete Sampling**\n\n> A theoretical provement of \"searching for a start token in the continuous embedding space\" being better than \"sampling every token in the discrete token space\" can further improve the soundness of the paper.\n\nThank you for the suggestion. Searching for a start token in the continuous embedding space is preferable to sampling every token in the discrete token space from the theoretical, interpretability and the cost perspectives:\n\n- **Optimal solution to the objective.** The set of the embeddings of all tokens forms a discrete subset of our continuous space. Importantly, this means that the optimal solution (maximiser) to our objective function may **not** lie within this discrete subset.\n- **Role of the soft token.** As the soft token in our framework is not intended to approximate a specific discrete token, but rather to serve as a functional control token, influencing the overall response. Optimising in the continuous space allows exploration of representations that may lie between or beyond discrete tokens, enabling smoother control and better performance than fixed-token selection.\n- **High computational cost.** With the vocabulary size |V|>30k, sampling every token in the discrete token space requires generating the output for each token x and computing the corresponding f(x), making it computationally expensive at O(|V|).\n\n\n**2. Computational efficiency**\n\n> The authors only provide a comparison with RAP in terms of token usage, but I believe comparisons with other methods in Table 1 and comparison on inference time cost can be added to further prove the efficiency of this method.\n\nAs the reviewer suggested, we additionally report inference time(min) cost across all baselines:\n\n|Method|GSM8K|GSM-Hard|SVAMP|StrategyQA|\n|-|-|-|-|-|\n|SC(τ=0.4)|26.58|33.91|20.54|20.04|\n|SC(τ=0.6)|26.12|34.46|21.76|19.87|\n|SC(τ=0.8)|27.28|34.86|21.16|20.80|\n|FIRE|26.70|32.26|21.59|20.17|\n|CoT-Decoding|26.56|32.55|21.53|20.60|\n|RAP|184.52|234.14|142.52|149.73|\n|Ours|**23.15**|**28.42**|**18.41**|**17.44**|\n\n\nOur method consistently achieves the lowest inference time across all tasks, further demonstrating its efficiency beyond token-level savings.\n\n\n**3. Neuron activation**\n\n> It is intriguing that the authors have found that applying this approach can increase the activation rate of neurons by 3–4%, how do you understand this phenomenon, and have you tried to analyze why this occurs?\n\nThank you for the insightful question. We believe this phenomenon occurs because our perturbation strategy encourages broader exploration within the neuron space, thereby increasing the likelihood of activating previously dormant or marginal neurons across the architecture. In our setting, we define a neuron as `activated` if its activation value is greater than zero. As our perturbations produce more diverse token embeddings, the resulting activations tend to be more widely distributed across different neurons as well, thus increasing the overall number of activated neurons.\n\nHowever, we would like to stress that the overall increase in activation rate does not necessarily translate to improved final answer accuracy. Rather, the key contributing factor is the ability to activate more neurons identified as critical for the current problem instance [1]. The increased activation rate simply raises the likelihood of activating these critical neurons. The BO process then further optimises the soft embedding to consistently target them, as illustrated in Figure 4 of our paper.\n\nTo further illustrate this point and to analyse the functional importance of these neurons, we have conducted an ablation study and found that **masking critical neurons** reduced accuracy from **62.14%** to **13.27%**, whereas random masking only dropped it to **41.89%**. This confirms that the activated neurons are non-random and impactful, consistent with prior findings on causal neurons [2].\n\n[1] Andy Zou et al., 2025. Representation Engineering: A Top-Down Approach to AI Transparency. \n\n[2] Kevin Meng et al., 2022. Locating and Editing Factual Associations in GPT. NeurIPS\n\n\n\n**4.Interpretation of the Soft Token**\n\n> Have you tried to interpret the sampled first token in your framework?\n\nWe appreciate the reviewer’s suggestion. We conducted nearest-neighbor analysis to interpret the optimised soft token, but found it to be a clear outlier in the embedding space. Its Euclidean distance from all vocabulary tokens are several orders of magnitude beyond normal variation (e.g., z-score > 3000), suggesting that it does not correspond meaningfully to any real token. This is expected, as the soft token in our framework is not intended to approximate a specific discrete token, but rather to serve as a functional control token, influencing the overall response.", "claims_and_evidence": "Most claims are well supported by ablation studies or prior works, but not with theoretical provement. \n\nA theoretical provement of \"searching for a start token in the continuous embedding space\" being better than \"sampling every token in the discrete token space\" can further improve the soundness of the paper.", "methods_and_evaluation_criteria": "The method and the experiments are reasonable. Although controlling reasoning path via only one token embedding is a bit counter-intuitive, there are similar attempts in prior works like prompt tuning and other decoding strategies like FIRE and CoT-Decoding.", "theoretical_claims": "Most equations are seemingly correct to me.", "experimental_designs_or_analyses": "The experiments are comprehensive and convincing. The main experiment includes 3 LLMs and 4 datasets, and there are detailed ablation studies to verify the effectiveness of each design. Experiments are repeated 5 times for reducing variance.\nBut for experiments about efficiency, I think there is still room for improvement. The authors only provide a comparison with RAP in terms of token usage, but I believe comparisons with other methods in Table 1 and comparison on inference time cost can be added to further prove the efficiency of this method.", "relation_to_broader_scientific_literature": "The paper's approach is an exploration in the field of LLM's decoding strategies. It can be regarded as an extension to prior first-token based strategies like FIRE and CoT-Decoding, which is mentioned in the paper.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Please refer to the above sections for Strengths and refer to \"Claims and Evidence\" and \"Experimental Designs Or Analyses\" sections for Weaknesses and Suggestions.", "other_comments_or_suggestions": "Please refer to \"Claims and Evidence\" and \"Experimental Designs Or Analyses\" sections for Weaknesses and Suggestions.", "questions_for_authors": "1. It is intriguing that the authors have found that applying this approach can increase the activation rate of neurons by 3–4%, how do you understand this phenomenon, and have you tried to analyze why this occurs?\n\n2. To my knowledge, there are methods trying to interpret \"soft\" tokens in the embedding space, e.g., finding their nearest neighbors (discrete tokens of LLMs) in the embedding space, have you tried to interpret the sampled first token in your framework?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper addresses a timely topic, employs a sound methodology, presents experiments that effectively support the claims, and is clearly written. The AC carefully checked the paper and agree with the reviewers.

Overall, this submission is strongly recommended for acceptance.
Decision: Accept (spotlight poster)