$\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space
Authors: Peihao Wang
Ruisi Cai
Zhen Wang
Hongyuan Mei
qiang liu
Pan Li
Zhangyang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Reasoning
Test-Time Training;Textual Optimization
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose $\nabla$-reasoner, an iterative decoding approach with policy refinement by test-time gradient descent on latent textual representations to improve LLM reasoning.

Abstract: Scaling inference-time compute for Large Language Models (LLMs) has unlocked unprecedented reasoning capabilities.
However, existing inference-time scaling methods typically rely on inefficient and suboptimal discrete search algorithms or trial-and-error prompting to improve the online policy. In this paper, we propose $\nabla$-Reasoner, an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop to refine the policy on the fly. Our core component, Differentiable Textual Optimization (DTO), leverages gradient signals from both the LLM’s likelihood and a reward model to refine textual representations. $\nabla$-Reasoner further incorporates rejection sampling and acceleration design to robustify and speed up decoding. Theoretically, we show that performing inference-time gradient descent in the sample space to maximize reward is dual to aligning an LLM policy via KL-regularized reinforcement learning. Empirically, $\nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark, while reducing number of model calls by approximately 10-40% compared to strong baselines. Overall, our work introduces a paradigm shift from zeroth-order search to first-order optimization at test time, offering a cost-effective path to amplify LLM reasoning.

Published as a conference paper at ICLR 2026
V R-REASONE:: LLM REASONING VIA TEST-TIME GRADIENT DESCENT IN LATENT SPACE
Peihao Wang'; Ruisi Cai", Zhen Wang', Hongyuan Mei', Qiang Liu', Pan Lit, Zhangyang Wang'
'The University of Texas at Austin, PUC San Diego, *TTIC, *Georgia Tech (peihaowang, ruisi cai, lqiang, at let.aaaawgeeeteeeee. edu, 2hw0850ucsd.co edu
hongyuanettic. edu, http://gghhhaau.cccmm//rraaaoaaa
edu.
ABSTRACT
Scaling inference-time compute for Large Language Models (LLMs) has unlocked unprecedented reasoning capabilities. However, existing inference-time scaling methods typically rely inefficient and suboptimal discrete search algorithms or trial-and-error prompting to improve the online policy. In this paper, we pro- pose V-Reasoner, an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop to refine the policy on the fly. Our core component, Differentiable Textual Optimization (DTO), leverages gradient signals from both the LLM's likelihood and reward model to refine textual representations. V-Reasoner further incorporates rejection sampling and acceleration design robustify and speed up decoding. Theoretically, we show that performing inference-time gradient descent in the sample space to maximize reward is dual to aligning an LLM policy via KL-regularized reinforcement learn- ing. Empirically, V-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark, while reducing number of model calls by approximately 10-40% compared to strong baselines. Overall, our work introduces a paradigm shift from zeroth-order search to first-order optimization test time, offering cost-effective path to amplify LLM reasoning.
INTRODUCTION
Large Language Models (LLMs)
have unlocked remarkable reasoning capabilities (Radford et al. enabling machines to tackle challenges considered exclusive to human cognition, such as solving complex mathematical problems (Cobbe et al.) 2021 Lewkowyez
2018 2019 Brown et al., 2020).
et al. 2022 Uesato al., 2022 Lee et al., 2023 Yang et al. 2024b) and executing long-horizon
planning (Liu et al. 2023a Valmeekam et al. 2023 Song et al. (023). Such capabilities arise
through large-scale pre-training on massive datasets, followed by careful post-training alignment
(Wei et al., 2022a Ouyang et al. 2022 Guo et al., (025). A prevailing observation has indicated that scaling both model size and training data leads continual improvements in LLM reasoning ability
(Kaplan et al. 2020 Hoffmann et al.) 2022).
Nevertheless, recent empirical findings increasingly suggest that scaling inference-time computation can be also crucial and perhaps more cost-effective than expanding pretraining to further enhance reasoning and problem-solving abilities (Snell al.) (024). Chain-of- -fhhughh (CoT) (Wei al.) 2022b) demonstrates that prompting LLMs at the test time to generate longer sequences with intermediate reasoning steps significantly improves their reasoning accuracy. Built on CoT, Wang al. (2022) further scales the inference compute by sampling multiple reasoning chains and selecting the most consistent one, leading to enhanced performance. More recently, inference-
scaling
has been augmented with reward models to refine reasoning quality. Notable approaches such as Tree-of-Thought (ToT) (Yao et al., 2024) and Massoning-as-Plamiin
(RAP) (Hao et al., 2023) cast LLM reasoning deciston-making problem and employ strategic sampling algorithms estimate the reward-to-go, thereby refining the sequential prediction policy at each decoding step. Underlying these approaches are extensive prompting-base based search procedures that traverse the sequence space,
"Equal contribution.

---

Published as a conference paper at ICLR 2026
with the LLM serving guiding heuristic. However, such approaches often struggle to adequately explore the sample space and thus become sensitive to sparse and noisy reward signals as reasoning chains grow longer and the search space expands exponentially. Consequently, their performance tends to saturate even when inference-time computation is substantially increased.
While existing methods fall into zeroth-order algo- rithms that rely solely on reward values, we note that first-order methods, providing directional guidance for optimization, can be even more effective in search- ing for optimal solutions, overcoming the sparsity of the reward landscape (see an intuitive comparison in Fig. D. In fact, gradient information is readily available during the L.LM reasoning process, as both the LLM and reward function can be differentiable. In this paper, we introduce V-Reasoner, novel rea- soning algorithm that applies inference-time gradient descent in the sample space to refine the outputs of a base policy prior to next-token prediction. The over- all pipeline follows an iterative decoding process. At each step, the language model first generates a full completion together with its per-token logits, serving as the initial rollout. The core component, termed Dif- ferentiable Textual Optimization (DTO), then refines these token logits via gradient descent.
Ocinnl
V
orgene ViLL
Figure I: LLM reasoning can be formulated as maximization problem over the reward landscape. (left) Traditional inference-time scaling methods are zeroth-order methods, sampling numerous candidate responses and evaluating them to identify higher-quality solutions. (right) This paper introduces first-order method for inference-time scaling, where reward gradients are directly leveraged to guide the search process toward highly re- warding responses.
DTO formulates the reasoning process as a continuous optimization problem over the reward land- scape, directly leveraging gradients to refine textual representations. Specifically, DTO applies gradient descent to optimize the initial logit vectors from the base policy under an objective that combines the reward function with the sequence-level log-likelihood estimated by the language model. To enable end-to-end differentiability, we employ the parameters into one-hot token vectors (Bengio et al. 2013). In this formulation, the reward function provides directional signals that guide tokens toward high-reward regions, while the ligliikl term regularizes the tuned sequences remain fluent and consistent with the pre-trained LLM distribution (Hoang et al. 2017: Qin et al. 2022 Kumar et al. (022).
estimator to map logit
After refining the logits with DTO, V-Reasoner treats the optimized logits of the first token as an improved policy and samples the next-token prediction from this updated distribution. We further integrate V-Reasoner with rejection sampling, which accepts the token drawn from the refined policy only if it can yield continuation with higher reward; otherwise, the method reverts to the initial choice. Through iteratively interleaving decoding and refinement, V-Reasoner scales inference-time reasoning by allocating additional computation to improve the LLM policy via gradient-basse based updates on the output space, efficiently backended by the parallel execution of transformer models. To further increase decoding throughput, we introduce a set of acceleration strategies that selectively skip tokens unlikely to benefit from DTO and reuse rollouts shared among decoding steps.
Theoretically, we show that DTO enables bidirectional gradient propagation along the sequence, facilitating global modifications that are crucial for effective reasoning (Bachmann & Nagarajan| 2024] Hao et al.) 2023). Furthermore, we establish a close connection between DTO and RL algorithms (Schulman et al. 2017 Ouyang et al. 2022 Guo et al., 2025). We prove that sampling from an optimized LLM trained with RL is equivalent directly drawing samples from the reference LLM and subsequently refining them through the gradient flow induced by DTO. This insight provides a new theoretical perspective for test-time approaches for reasoning.
Empirically, ReRasone significantly enhances the mathematical reasoning capabilities by 10-40% across multiple models and benchmarks. It consistently outperforms strong inference-time baselines such as Best-of-N and RAP (Hao et al. 2023), while achieving accuracy on par with more costly training-based methods (e.g., GRPO). We further show that V-Reasoner scales compute more effectively: by leveraging parallelized execution of attention, it can utilize more compute per model forward pass. Henceforth. when comparing with sampling papplinggollyy methods (e.g.. BoN), V-Reasoner achieves superior results while reducing the number of model calls by up to 40.2%.
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES
In this section, we formulate LLM reasoning as a decision-making problem, introducing the necessary notations and common approaches to address this problem along the way.
Notations. Let == {6, R i [[]]]} be the vocabulary set, where d, is the i-th canonical basis and [V] is the vocabulary size. We denote sequence over this vocabulary as x = [ [a|]|] where æ E V/ is the 1-th token for every E [|=]]] and |x] represents the length the sequence. We also denote &g as the subsequence up to and including the i-th token, expressed as [01,-.. [...
The space of all sequences with finite length is given by V* = Ulen V.d
Language Models and Reward Models. In this paper, we focus on autoregressive language models (Radford et al.) 2018 2019 Brown al. 2020), denoted TLLM V - [0, 1]. The language model can characterize the conditional probability of a question-answer pair. Given a pair of questions and answers æe y E V", the model estimates their likelihood by the following factorization:
) E [0, 1]11 as the categorical distribution of ys given the prefix y and æ. In addition, we define reward model as the function r: 1/" - R. r (z) evaluates the correctness of the response y for question æ. In this work, mainly focus on outcome reward (Cobbe et al.] (02I), which is often sequence classifier, offering an overall score for the entire response sequence. Our proposed method can also generalize to process reward (Lightman al. 2023).
TLLM TLLM(y)z) 11." TLLM (g:lyyyssss U,i-,, v). We also denote Cat (FLLM (- +))
Reasoning as Decision Making. Reasoning with LLMs can be framed as search algorithm that aims identify a high-rewarding response: arg min
(z)lz)) Due the combinatorial nature of this optimization, directly finding the optimal solution is intractable, as the search space grows exponentially with the sequence length, i... | In autoregressive decoding, this challenge reduces to making sequential decisions for the next token, each of which must ultimately contribute to minimizing - R() This decision process is often formalized via a Bellman equation:
"t FLLM (1 x) arg max Eggel Eygelr P Clu 1.w) [r (ryssi-, Vi, y24+ [z)],
(I)
B.EV
where LLLM is refined version of the original policy TLLM- The expected reward-to-go in Eq. iis also known as the Q-function. The recursive structure of this formulation implies that greedy decoding is not globally optimal, and identifying the optimal next-token prediction inherently requires
look-ahead rollouts and backtracking (aao et al. 2024
Hao et al. 2023 Besta et al. 2024).
Existing Approaches. Current techniques tackling L.LM reasoning via decision making can be broadly categorized into training-time time and inference-time methods. Training-time time approaches include supervised tine-tuning (SFT) as well as model-free policy optimization techniques, such as (Shhuman al. (2017); Guo et al. (2025) ; Rafailov al. (2024). Our focus is on inference-time methods, which aim to improve the decoding process of an LLM without additional training. These methods are typically model-based and value-based seeking to refine existing policy by directly solving the Bellman equation. For example, Best-of-N (BoN) (Stiennon et al. 2020) tackles Eq. bbyy sampling N independent full trajectories from a base policy "..... y
LLLM (-|az), and selecting the one with the highest reward y arg max (x)) ..... (N)) (ly More structured approaches, such as Tree-of- Thoughts (ToT) (Yao et al.] (024) and Reasoning as- aas-Plannin (RAP) (Hao et 2023)). explore the solution space and approximate Q-functions stochastically on the fly through rollouts and recursive evaluation.
3 REASONING WITH GRADIENT-DRIVE DECODING
Overview. In this section, we introduce V-Reasoner, a novel reasoning algorithm that scales inference-time computation by performing gradient descent the sample space to refine the outputs of a base policy. The overall pipeline, as illustrated in Fig. 2 is structured as an iterative decoding process. Given a prefix æ, the model first generates an initial response (") V-Reasoner then represents the generated sequence through its per-token pre-softmax logits z x0)) and optimizes these logits via gradient descent to maximize the sequence-level reward r(y) az) (Sec. 3.1). After
3

---

Published as a conference paper at ICLR 2026
Algorithm 1 R-Rassoner: Decoding with DTO
Algorithm 2 Diffcrentiable Textual Optimiza- tion (DTO)
Require: Prompt æ, language model TLLM, re- ward model r. stop criteria StopCriteria()..
Require: Prefix æ. initial logits 2, language model TLLM, reward model r, and the num- ber of training steps T.
1: repeat
2: y. z= TLLM (-|az)
3: 2 t DTO(x, 2, (LM, r).
1: (1) <
4:
y1 ~ softmax(21/T) (2I/T)
2: for w-r-1l 1, 1, ,T do
5:
ū y then
3: for every i 1,... y do
6:
y. ~ T TLLM (- [11. x)
4: j + arg max 43
7:
ir > r(y x)) then
5: y; + oj. + softmax (2/
8:
x + concat [x. y1
9:
else
CoopGradssoftmaxx /T)
10:
- concat x., y1]
6:
end for
end if
7: Lull = -- E. log TLLM (1 2:)
12: else
8:
Creward = -r
t concat|aa [a, 31l
9:
C=Cnlh -rrreardd
14:
end if
10: 2(+11 f x(1) - VV
15:
until StopCriteria(z))
11: end for
16:
return
12: return 2 (T)
Figure 2: Basic implementation of V-Reasoner. V-Reasoner an iterative decoding algorithm driven by DTO. At each decoding step, DTO applies gradient descent on the logits initialized from the base model to optimize a reward-informed loss to refine the policy. The updated policy is then combined with rejection sampling, leading to high-reward responses. The pseudocode for the full implementation with acceleration techniques (Sec. 3.3 is deferred to Appendix
optimization, V-Reasoner resamples the first token of the generated sequence using the fine-tuned logits Σ1. the resampled token differs from the original, the subsequent tokens are regenerated. and this candidate token is accepted only if its yielded response achieves higher reward under r r--(æ) The procedure then proceeds to the next token by incorporating the first generated token into the prefix, and repeating this optimization-an -ad---n--d----ingg loop. V-Reasoner scales inference- time reasoning by allocating additional computation to optimize the policy's outputs via iterative gradient descent. To further improve efficiency, we propose series of system co-design strategies that selectively skip tokens unlikely to benefit from optimization and reuse model outputs and KV caches to accelerate decoding (Sec. 3.3
3.1 DIFFERENTABLE TEXTUAL OPTIMIZATION
The core step of our algorithm is leveraging gradient information to refine an initial response generated by the base policy. Existing reward- guided decoding methods (Wei et al. (0226] Wang al. 2022 Yao et al.] 2024 Hao et al. 2023) can be regarded as zeroth-order approaches, as they rely solely on reward values. However, reward feedback is often sparse, and searching for improved solutions based only scalar reward values can be sample-inefficien. particularly when the base policy is weak. We note that most reward models are inherently differentiable, as they are typically implemented with transfommer-bsee sequence classifiers (Stiennon et al., [2020] Ouyang et al., 2022) Dong et al.) (024). This opens the door to exploiting not just reward values but also reward gradients, which provide richer directional information to guide samples toward high-reward regions. Motivated by this, we reformulate the search problem in Eq. aas a gradient- based differentiable optimization. We term this approach Differentiable Textual Optimization (DTO), which differentiates reward over token space for progressive response improvement.
Objective. Our overall goal is to refine given sequence of tokens (9) as to maximize the reward. However, directly maximizing r x)) risks reward hacking (Pan et al. 2022). as the optimization trajectory of y may drift away from the distribution under which r(y xx) well well-calibrated typically the prior distribution induced by TLLM- To mitigate this, we constrain y to remain within the high-density region of LLLM- Concretely, regularize the log-likelihood of y. thereby penalizing
4

---

Published as a conference paper at ICLR 2026
deviations from the distribution represented by the language model. The resulting objective function to be minimized is given by:
( (= : -r(y)a))) log TLLM (L)M(())))
where À >> 0 is a hyper-parameter balance the reward value and the regularization term. Intuitively, Eq. 2 seeks a response that not only achieves a high reward but also maintains fluency and faithfulness in natural language (Kumar et al. 2021] Qin al., [022] Yuan et al. 2025). To estimate the log-lik likelihhood log TLLM (aly), we decompose it sequentially from left to right, which results in
the next-token prediction loss: log TLLM (y x)) -' y, log Cat (TLLM (V)))-1))))
Parameterization. The token space of is discrete where gradients cannot directly oper- ate. Therefore, we propose to parameterize the tokens via the underlying logit vectors used to sample them. At the initialization stage, we use the -LM--eeerated logits to initialize 20))
| During optimization, we use Gumbel-softmax straight-through trick to parameterize /T)) (Bengio et al. 2013
+ softmax(z, (1) /r) StopGrad(softmax/ (x(xx()))))))))))
y arg max jelv =!)
Jang et al., 2016), where ò; denotes the i-th canonical basis and > 0 is the temperature coef- ficient. By this means, gradient descent can be equivalently performed on the space of 21)) as:
((+1) = = - (z2())))
As will demonstrate in Sec. 4 the gradient of L propagates information bidirectionally. Preceding tokens act as a regularizer on successive tokens, enforcing consistency with the autoregressive generation process, while trailing tokens propagate outcome-level reward signals and full-sequence alignment back to earlier tokens through attention. This implements closed-loop control of earlier predictions influencing subsequent decoding steps, thereby capturing the key recursive structure of reasoning characterized in [q.[] Furthermore, in Sec. 4 we establish a close connection between DTO, which optimizes directly in the sample space, and policy-optimization (e.g.. PPO) that operate in the parameter space. We show that DTO provably shifts the drawn samples toward the reward- maximizing distribution induced by the original policy.
3.2 ITERATIVE DECODING WITH DTO
In this section, we elaborate on the detailed iterative generation process with DTO integrated for policy improvement. Akin to autoregressive decoding, V-Reasoner generates the full response token by token. The sampling of each token consists of the following two steps:
Policy Improvement via DTO. Starting from prefix æ E V*, we let the LLM TLLM generate continuation sequence ( along with its pre-softmax logits (0), We then apply the DTO algorithm to optimize 20)) for I' steps, yielding refined logits ž. The logits corresponding to the first token are treated as the improved policy for predicting the immediate next token, intentionally adjusted to yield higher reward when used to generate the continuing responses. Accordingly, we resample the next token from this updated policy: ÿ1 softmax(2, /T)
Rejection Sampling. Once
next-token candidate ÿ1 is obtained, we first compare it with the initial prediction yı. If y yı, no effective policy update occurs, and we proceed directly to additional rollout conditioned on
the next token for policy refinement. If ÿ1 # yı, we perform yn as the next token, yielding a new response y. Both y and ÿ are then evaluated under the reward function, and the token that yields full response with the higher reward is retained.
Test- Time Scaling, We scale computation in V-Reasoner along two axes: (1) increasing the number of gradient update steps used by DTO to refine the policy, and (2) performing rejection sampling among rollouts yielded by the original and updated policy. Comparatively, allocating additional compute to gradient- based optimization not only more effecttive in incorporating reward signals into the policy, but also more efficient than purely autoregressive decoding. This efficiency arises because computing the full-sequence gradient V leverages the parallel execution of transformers: single gradient step propagates updates across all tokens within one model call, whereas standard autoregression generates only a single token per model call. As we will show in Sec. sampling from the policy refined by DTO yields significantly higher chance of reward improvement.
5

---

Published as a conference paper at ICLR 2026
3.3 ACCELERATING V- --REASONER
The naive implementation of V-Ressoner is inefficient due to two primary bottlenecks: (1) decoding each token requires full optimization procedure, where each step involves backpropagation through two large models; and (2) generating a single token requires an additional full rollout. In this section, we demonstrate that V-Reasoner is amenable to several strategies that significantly accelerate both optimization and generation, while adaptively allocating compute to the tokens that matter most.
Gradient Caching. The gradient backpropagated the logits z can be decomposed via the chain rule as: VzL= d2 ac By By wherein the term UC dominates the computational cost, since it requires a by full forward and backward pass through both the language model and the reward model. However, we observe that y - the onc-hot vectors indicating the maximal entries in the soft logits z changes infrequently as optimization proceeds. Exploiting this property, we cache the gradient By once computed, and reuse it until the maximal entries of flip. In our implementation, we retain the ac for every i E [[v|] whenever y is updated, and otherwise recover cached gradient gi = By.
ac
efficiently using the surrogate loss Leache = - y, 9: to recover the saved gradients (g: Ne vll whenever y remains unchanged from the previous iteration. See Algorithm 4 in Appendix B
v
Rollout Trajectory Reusing. We further note that the rollout strategy can be improved to reduce
unnecessary computation and better leverage the KV cache. In the naive implementation (Sec. 3.2 Algorithm V-Reasoner generates a sequential rollout to optimize the next-token prediction policy for every decoding step. However, the rollout trajectory, including both tokens and logits, continuing from the previously accepted token can be directly reused as the rollout for the subsequent token. In Algorithm
[]] we skip the rollout at the beginning of each step and reuse y>2 and #>2 as the rollout for the next token if the resampled token ÿ is rejected; otherwise, we continue with ÿ and ž for the next step. We also limit the total number of rollouts as N Nmaz- Once the total number of rollouts exceeds the N Nmaz. we terminate the iterative policy refinement and generate the remaining tokens using standard autoregressive decoding. See more details in Algorithm in Appendix
Confidence- and Gradient-Guided Token Selection. Running DTO to optimize the policy at every decoding step can result in redundant computation. We observe that token logits with either high confidence (see Appendix C.3) or small gradients are unlikely to be modified under DTO. To address this, we introduce two selection criteria, entropy-based and gradient- based to determine which tokens should undergo policy refinement. Specifically, we define two hyperparameters, cent and Egrod- DTO is applied only when the entropy of the token logits satisfies H (21) > cent and
the gradient magnitude exceeds |/ 21 Cll > ggad. where (.) denotes the entropy of a categorical distribution. We refer readers to Algorithm in Appendix B for more details.
THEORETICAL ANALYSIS
Interpretation of Gradient Updates. We analyze the gradient of L to reveal how DTO updates the response. The derivatives terms of the 1-th token OL /xx under loss Eq. 2 can be decomposed
as follows:
ly a log Cat (TLLM( x
ac
ar (x)
log Cat (FLLMI
Oy:
By:
dy
8prefie
8 Sroward
8yont
We defer the derivation to Appendix C.I The first term, 6pecixx updates the token x; based on its preceding context, aligning the next-token policy with the autoregressive prediction probabilities produced by the language model. The second term, &pustiix propagates information from subsequent tokens through the attention mechanism, encouraging global consistency with respect to its future context. Finally, the reward gradient greward provides a sequence-level signal, transmitting information from later tokens to y, via attention. As highlighted by /achhann & Nagarajan (2024). the order of generation plays a crucial role in complex reasoning or algorithmic tasks. Pure teft-tt-rgght generation can fall short of error accumalation, making it insufficient for intricate logical reasoning processes. An ideal decoding method for reasoning should allow for iterative refinement of the reasoning chain in both forward and backward directions Yao al. 2024
Hao et al. 2023).

---

Published as a conference paper at ICLR 2026
niference-Time Gradient Descent is "Deamortized" PPO.
We theoretically establish the connec-
tion between the test-time textual optimization and parametric RL-based training. RLHF (Schulman
et al. 2017 Ouyang et al. 2022 and RLVR (Guo et al., 2025; Shao et al., 2024 have been demon-
strated to be particularly effective for mathematical reasoning tasks (Wang et al. 2023a Zhao et al.
2023
Dong al. 2024 (Shao al. 2024). The primary objective of RI. is to fine-tune a pre-trained
LLM using RL algorithms, ensuring that its responses to specific prompts maximize a given reward function. Among various RL algorithms, KL-regularized approaches, such as Proximal Policy Opti- mization (PPO) (Schulman et al., 2017). have been widely adopted in practice. In this section, we uncover the hidden connection between PPO and our proposed DTO. Formally, let p : V - R represent an LLM to be aligned with the reward function r, initialized from the pre-trained policy TLL.M. PPO optimizes for p by minimizing the following functional objective defined over the space of distributions:
(p) == - --y--pAr(yr) + DRL (P)ITEEM,
(3)
where the first expectation term estimates the expected reward, while in the second term, the KI.- divergence regularizes the distributional discrepancy between p and TLLM- Assuming LLMS' input domain can be extended to the ambient space beyond discrete vocabularies, then we can show the relation between (stochastic) gradient flow of Eq. and functional solution to PPO:
Theorem 4.1. Suppose {0' denotes the Wasserstein gradient flow minimizing Eq. in the distribution space with boundary conditions pO = -LLMM and p² =p" = arg min, LPpo Then we can draw samples from p² by first initializing x. ~ TLLM and simulating trajectory he20 de' dt
following the stochastic gradient flow of Eq. are Brownian motions.
V[(x')++++ +/2,, where {E, E (N, 1)2
Theorem is proved in Appendix
Theorem 4.1 shows that instead of optimizing the entire policy to satisfy the reward function w.r.t. Eq. there exists a trajectory driven by the gradients of Eq. 2 the sample space that can directly generate samples from the optimal distribution minimizing Eq. 3 Pre-training scaling and test-time scaling can be unified and interpreted through
Theorem 4.1 as two complementary forms of statistical inference: parametric and non-parametric (particle-based inference (Liu & Wang, [016)[Chen
Chen et al.) 2018). The pre-training stage corresponds parametric inference: global parameter is optimized minimize the overall loss across dataset, amortizing the cost of individual samples into a shared parameter. Increasing the size of this parameter space enhances the model's representational capacity, thereby reducing the average cost per sample. In contrast, test-time scaling via DTO is analogous to non non-parametric inference, which performs optimization in the sample space, treating each sample as an independent "particle" that minimizes its own cost. This allows for fine-grad fine-grained adaptation at the individual sample level. The Wasserstein gradient flow provides a mathematical framework to describe the relationship between the dynamics of measures (global distributions) and individual samples, thereby bridging the conceptual gap between pre-training scaling and test- test-time scaling.
EXPERIMENTS
5.1 RESULTS ON MATH REASONING
Experiment Details. Tab. T compares our test-time method, V-Reasoner, against a variety of baselines. We benchmark its performance against other test-time approaches, including greedy decoding, Self-Consistency (SC) (Xie et al., 2024), Best-of-N (BoN) (Stiennon et al. 2020). tree- search based methods: Tree-of-Thought (TTT) (Yao et al. 2024) and Reasoning (RAP) (Hao et al. 2023), and the iterative refinement approach: TPO (Li al. 2025). Additionally, we include training-bass based methods such as Supervised Fine- Tuning (SFT) and GRPO (Guo et al.
Planning
Fine-Tuning
2025) for comprchensive comparison. We evaluate two model families, Qwen-2.5-main
Yang
et al. 2024a) and 1lama-3.1 (Grattafoori et al. (204)), on four representative mathematical reasoning
benchmarks: MATH-500 (Hendrycks et al.[[021] AIME24, AIME25, and AMC (Li et al. 2024). We leverage reward models from the Skywork- V2 family (Liu et al. 2025):: for Skywork-V2-00 V2-Qwen-4B for Qwen-based models and -kyyorr--V--Llame 8B for Llama family models. For BoN and SC, we let N 8 to match N Nmaz = used in our methods. For TPO, we set the number of samples per step as Naamples 2 and the number of refinement steps Nrefine 2. For ToT and RAP, adopt the default hyperparameters in Hao et al. (2024) to yield meaningful results. Please refer to Appendix D for more experimental details.
7

---

Published as a conference paper at ICLR 2026
Table 1: Accuracy (%) on math reasoning datasets compared with baseline methods, including both test-time and training-time approaches. We skip results on AIME datasets for Llama-3. 1lama-3.1-85 as it is incapable of generating reasonable performance. We mark the best performer in bold and the runner-up with underline. Our method outperforms all test-time baselines and even achieves performance on par with the training-basee methods (SFT and GRPO), respectively.
Models
Methods
MATH-500
AMC
AIME24
AIME25
Greedy decoding
43,8
33.0
6.7
6.7
SC
Xic al. 2024
(N 8)
69.8
49.4
22.5
20.0
BoN
Sheenoo
2020
(N 8)
70.2
50.1
22.5
13.3
ToT
Yao
2024
57.8
42.4
6.7
10.0
Qwen-2.5-7B
Hao al.
2023
68.6
50.1
18.3
14.2
SFT
Ouyang et al
2022
65.8
36.4
6.3
11.7
GRPO Guoo al.
2025
70.8
52.8
20.8
16.7
V-Reasoner mamox
71.0
23.3
15.0
Greedy decoding
71.2
43.0
5.3
7.5
SC Xie al. 2024
(N
8)
76.6
55.5
25.0
22.5
BoN Strennon
2020
(N 8)
77.8
55.9
22.5
18.3
Qwen-2.5-7B-Instte
ToT
2024
75.4
48.2
20.0
18.3
Hao el al. 2023
12.5
TPO
Li ct al 2025
77.6
55.9
6.7
11.1
V V-Reasoner (N
8)
80.4
56.8
26.6
20.0
Greedy decoding
19.3
SC Xie et al. 2024
(N
8)
54.8
25.7
BoN Stiennon ct al.
2020
(N = 8)
52.2
26.
ToT
Yao et al 2024
50.2
25.6
Llama-3.1-8P In--rust
RAP Hao et 2023
55.4
25.8
SFT Ouyang et al.
2022
46.6
V P-Reesoner (N
8)
55.8
28.9
Performance Comparison. Our method shows superior performance across all models and bench- marks test-time methods. We even reach comparable performance with training- based methods. Specifically, with the Qwen-2.5-7B base model, our approach achieves the highest scores among test-time methods MATH-500 (71.0%) and AIME24 (23.3%), and remains highly competitive with the training-based GRPO method (trained with 35k examples). The advantage is even more pronounced with the instruction-tunel Qwwen2255-BBIInstruu and
where our method again leads all other approaches on all benchmarks. Notably, on (wwen-2.5-7B. B-Inttuutt our method scores 80.4% on MATH-500 and 56.8% AMC, and Llama-3.1-88 1-1---Instructt our method achieves 55.8% on MATH-500 and 28.9% on AMC. In the meantime, we also provide an example to show how -eResonerr works in real- weal-world scenarios in Appendix D.4
5.2 COST COMPARISON
We note that V-Reasoner can exhibit an efficiency advantage by utilizing compute more effectively. This stems from V-Reasoner's ability leverage parallelized attention execution compute gradients and update the decoding-time policy on the fly, which often leads to a small practical runtime. To
12005
20000
8032
4032
6000
4000
2300
sc
2002
Bis OV
2000
Fell 1M
2000
oers fhc
eata the =
cura 18x22
10006000
AIMES
Saboue
Bacave
Figure 3: A comparison of computational cost, measured by the number model calls. Our method reduces costs by up 40.2% compared baselines.
8

---

Published as a conference paper at ICLR 2026
capture this advantage, propose using the number of model calls surrogate theoretical efficiency metric. This metric counts both a single recurrent computation and a parallel forward pass model call. The choice is motivated by the practical observation that, under ideal attention parallelization, a single model call regardless of whether it is executed recurrently or in parallel incur comparable wall-clock cost. Thus, it serves unifying metric that reflects algorithmic complexity under idealized system-level and hardware optimizations, while mitigating discrepancies arising from implementation details (e.g., compatibility with serving engines).
Fig. compares the computational cost of our method with several baselines. For fair comparison, we set the number of generated samples for all approaches (N for SC and BoN, and Nmar for our V-Reasoner). Our method delivers superior performance at significantly lower cost than SC and BoN. For instruction-tun -unded models, reduces number of model calls by up to 40.2%, while for base models, it outperforms all baselines using only about 90% of this metric. The reason for the reduced cost is twofold: (1) with confidence- and gradient-guie guided selection, rollouts usually start from the middle of the sequence, instead of from the beginning BoN and SC do; (2) the optimization cost with gradient caching remains lightweight while our DTO enables efficient parallelizable execution of transformers and revision of tokens. These results imply that V-Reasoner has stronger prospects for achieving inference efficiency. Unlike mial-and-error aan-error-based based test-time scaling methods (e.g.. BoN), which repeatedly resample outputs without guidance, our method updates the decoding policy in targeted and strategic fashion.
5.3
TIST-TIME SCALING LAW
Qwen2.5-Math-78
0.57
0.52
0.56
0.50
0.28
0.48
0.54
0.46
0.53
0.44
0.22
0.51
Ours
Ours
Ours
0.50
BoN
SC
SC
SC
2000
3000 4000
5000
6000
3000
4000
5000
6000
2000
3000 4000
5000
Cost (#Calls)
Cost (#Calls)
Cast (#Calls)
Figure 4: Test-time scaling curves comparing our method with BoN and SC. We change the number of samples N for BoN and SC and number of rollouts N for method. The results show V-Reasoner achieves superior performance with reduced cost across multiple models.
We present our test-time scaling curves in Fig. comparing our method against Best-of-N (BoN) and Self-Consistence (SC). The figure plots accuracy against computational cost, showing how each method's performance scales as more resources are used. As is evident across all models, our method's curve consistently lies above the baselines. This indicates that for any given computational budget (number of calls), our approach achieves a higher accuracy. These results demonstrate that V-Reasoner offers a superior trade-off between performance and computational cost, establishing more efficient frontier compared to these sample-heavy techniques.
5.4 ALGORITHM ANALYSIS
Dependencies on Reward Models. Our approach relies on the gradient signal from the reward model to optimize policy at test time. To study the dependency the
Table 2: Ablation study on reward model choice.
Model
Skywork-Qwen3--8f
MATH-500
80.4
80.8 (+0.4)
quality of reward models, we further eval-
AMC
56.8
57. (+0.3)
uate our approach Instruct paired with the larger wwwork--Reward. W2--V22-Q---- 12-Qwen3-88 reward model. We note that our original choice Skywork-Rewarde 1et-ward-22-Qwee3---- smaller and weaker reward model according to
Owen2.5-Math-72.

---

Published as a conference paper at ICLR 2026
the RewardBench (Malik al. 2025 According to the Tab, the performance gap between the 4B and 8B variants remains consistently below 1 point across both MATH-500 and AMC. This indicates that using a smaller reward model does not lead to significant performance degradation compared with the larger, stronger version. This justifies our original choice (in Tab. T and further suggests smaller reward models may be preferable for improving efficiency.
Table 3: Analysis of rejection rate (%) in rejection sampling. We N = for the BoN baseline and also set Naa = 8 for our V-Reasoner. The theoretical rejection rate of the baseline is 66.0%.
Rejection Rate Analysis. As described in Sec. 3.2 and A Algorithm V-Reasoner first ap- plies DTQ to directly optimize the policy in the logit space, and then compares continuation y generated from a resampled token with the origi- nal rollout sequence y. The token sampled from the optimized policy is adopted only if it yields continuation with higher reward. Henceforth, it becomes essential to quantify the acceptance rate of tokens drawn from the optimized policy in order to justify the effectiveness of DTO.
Model
Baseline
V-Reasoner
Qwen-2.5
65.9
32.8
Qwen-2.5-Inntrue
66.5
28.9
3-asaa3-Instruc 66.9
40.1
To this end, we measure the rejection rate, defined as the percentage of candidates produced by DTO that are rejected for failing to improve the reward. For comparison, also evaluate this metric on a baseline that performs rejection sampling without DTO, which is equivalent to BoN. Theoretically, performing rejection sampling N times over identical distribution yields rejection rate of 1 (- 1/k) that converges to one as N - oc. For N 8, the expected rejection rate is approximately 66.0%. We report our measured rejection rate in Tab. 3 We report the empirical rejection rates in Tab. 3 The results show that rejection sampling without DTO closely matches the theoretical prediction, while rejection sampling with DTO significantly reduces the rejection rate (by up to 30%). This confirms that DTO is effective in improving the next-token policy, producing tokens that lead to continuations with higher rewards.
6 CONCLUSION AND LIMITATIONS
We presented --Reasoner, an inference-time reasoning framework that introduces Differentiable Textual Optimization (DTO) to refine token logits via gradient-based optimization. By combining gradient signals from both the LLM likelihood and a reward model, V-Reasoner enables more effec- tive policy improvement than zeroth-order search methods, while incorporating rejection sampling and speedup techniques to boost effectiveness and efficiency. Theoretically, we show that aligning with reward function is equivalent to gradient- hased optimization the sample space. V-Reasoner delivers substantial performance gains over base models while consistently reducing computation cost, illustrating a sharper and more efficient scaling paradigm for LLM reasoning.
I Limitations. In line with previous observations Yuue et al.), the performance of V-Reasoner appears to remain bounded by the capabilities the underlying base model and reward model, particularly under limited computation. The base and reward models required to share the same vocabulary to allow for end-to-end logit optimization. Moreover, integrating V-Reasoner into efficient LLM serving pipelines requires more careful system co-design to incorporate test-time gradient descent.
ACKNOWLEDGMENTS
Authors thank Shibo Hao, Junbo Li, and Sina Alemohammad for helpful discussions. This work was supported in part by NSF Awards 2145346 (CAREER) and the NSF AI Institute for Foundations of Machine Learning (IFML). It was also supported by computing support on the Vista GPU Cluster through the Center for Generative AI (CGAI) and the Texas Advanced Computing Center (TACC) at The University of Texas at Austin. Peihao Wang is in part supported by Google PhD Fellowship in Machine Learning and ML Foundations. Ruisi Cai is in part supported by NVIDIA Graduate Fellowship. Zhen Wang is supported by OpenAl Research Grant and Gordon and Betty Moore Foundation Postdoctoral Fellowship.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research primarily concentrates on developing inference algorithms enhance reasoning in large language models (LLMs), with a particular emphasis on mathematical reasoning. It relies solely on extant LLMs and does not involve training, fine-tuning model weights, or creating new LLMs. As result, the work does not raise any novel domain-speciff ethical considerations or societal impacts beyond those already well-documented in relation to large-scale language models more broadly.
REPRODUCIBILITY STATEMENT
We include pseudocode plus a detailed version in both the main text and Appendix B Complete derivations and proof's are provided in Appendix C Additionally, Appendix [ contains the full list of hyperparameters, datasets, and model checkpoints required reproduce our experimental results.
THE USE OF LARGE LANGUAGE MODELS
Large language models are used solely for sentence-level proofreading. All research ideation and paper writing were originally carried out by the authors.
REFERENCES
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. arXiv preprint 2XXv:2443300666 2024.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3333, 2013.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi. Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17682- 17690, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leameer. Advances in neural information processing systems, 33:1877- 2020. 2020.
Changyou Chen, Ruiyi Zhang, Wenlin Wang, Bai Li, and Liqun Chen. unified particle-optmmizatin framework for scalable bayesian sampling. arXiv preprint arXiv: 1659, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1411 2021.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.0777 07863, 2024.
Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Learning iterative reasoning through energy minimization. In International Conference Machine Learning, pp. 5570-5582. PMLR, 2022.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05177 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et The llama herd of models. arXiv preprint 24X 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smymis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprint 2.X1:25066004788 2025.

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12990 2025.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 230.. 2023.
Shibo Hao, Yi Gu, Haotian Luo. Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, al. Llm reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. arXiv preprint arXiv:2404.05221, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint ar arXiv:2103.0333 2021.
Cong Duy Vu Hoang. Gholamreza Haffari, and Trevor Cohn. Towards decoding as continuous optimization in neural machine translation. arXiv preprint 1rXivv1701.028555 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
James Huang. Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. Deal: Decoding-time alignment for large language models. arXiv preprint arXiv:2402.06144 2024.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, arXiv preprint 161 00 144, 2016.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint vrXiv:2001.08666 2020.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment reward-guided search. arXiv preprint arXiv:240..01694, 2024.
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yulia Tsvetkov. Controlled text generation as continuous optimization with multiple constraints. Advances in Neural Information Processing Systems, 34: 14522-14554 2021.
Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. tradient-based constrained sampling from language models. arXiv preprint arXiv: 125588 2022.
Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint 2XXi:2307.0038.. 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for crrameter-efficient prompt 2021.
tuning. arXiv preprint
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances Neural Information Processing Systems, 35:3843-3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. cwm http:///rrpppttooooooaaaatttteee wtw.iiaadddaattt..deeeeee 2024.
[https://hugging:: co /A co//AL-HO//MummnaaMMttt--o--0 (tttp::// (atpaaaaaaaaaacaaataaaaaoteeoote wttptttwww..coopoooettttuaaaaaa.
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 101.00/90. 2021.
12

---

Published as a conference paper at ICLR 2026
Yafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li, and Yu Cheng. Test-time preference optimization: On-the-fly alignment via iterative textual feedback. arXiv preprint arXiv:2501.12895, 2025.
Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning. arXiv preprint arXiv:2309.07124, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprini arXiv:2305.20000, 2023.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprini arXiv:2304. 11477, 2023a.
Chris Yuhao Liu, Liang Zeng. Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. Skywork-reward Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.00 0335,, 2025.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances neural information processing systems, 29, 2016.
Xin Liu, Muhammad Khalifa, and Lu Wang. Bolt: Fast energy-basee controlled text generation with tunable biases. arXiv preprint arXiv:2305. /2018, 2023b.
Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Ha- jishirzi, and Nathan Lambert. Rewardbencch Advancing reward model evaluation. https 2025.
Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions fokker- planck equations through gradient-log-dent estimation. Entropy, 22(8):802, 2020.
Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Berg- BergKKirkpatrick. Mix and match: Learning- free controllable text generationusing energy language models. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 401-415, 2022.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-anss answering with human feedback. arXiv preprint 2122:22112.0933222000
Bernt Øksendal. Stochastic differential equations. Springer, 2003.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint (eXXi:2201.03544: 2022.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495. 2023.
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538 110889951111111 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
13

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.00 03300, 2024.
Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. Toward human readable prompt tuning: Kubrick's the shining is good movie, and good prompt too? arXiv preprint arXiv:22 10539, 2022.
Taylor Shin, Yasaman Razeghi, Robert I. I Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15990 159800 2020.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 11m test-time compute optimally can be more effective than scaling model parameters. arXiv preprint (rXiv:2448.033114 2024.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Feotshot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2998-3009, 2023.
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-basee feedback. arXiv preprint arXiv:2211.14275. 2022.
Karthik Valmeekam, Sarath Sreedharan, Matthew Marquez, Alberto Olmo, and Subbarao Kambham- pati. On the planning abilities of large language models (a critical investigation with a proposed benchmark). arXiv preprint arxiv:2302..67000 2023.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: A label-free free step-by st-ppp verifier for Ilms in mathematical reasoning. arXiv preprint ar arXiv:2312.08933 2023a.
Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P Xing, and Zhiting Hu. Promptagent: Strategic planning with language models enables expert-level prompt optimization. arXiv preprint arXiv:2310. 16427, 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery. and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2033.1117,, 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206. 200.022200....8, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. th-ouut prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022b.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made casy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36, 2024.
Haoyi Wu, Zhihao Teng, and Kewei Tu. Parallel continuous chain-of of-thought with jacobi iteration.
arXiv preprint 2506 18582, 2025.
14

---

Published as a conference paper at ICLR 2026
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Xie. Sell-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36, 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning. 2024.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 120.4099.12222 12122, 2024a.
Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with retrieval- -aggeeeed language models. Advances in Neural Information Processing Systems, 36, 2024b.
Shunyu Yao, Dian Yu, Jeffiey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yige Yuan, Teng Xiao, Li Yunfan, Bingbing Xu, Shuchang Tao, Yunqi Qiu, Huawei Shen, and Xueqi Cheng. Inference-time alignment in continuous space. arXiv preprint 2.X.::5555..008.1 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model?, 2025. URL https://arxiv. org/abs/2504.13857
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text. arXiv preprint 24Xi:24406.07466 2024.
Boyi Zeng, Shixiang Song, Siyuan Huang, Yixuan Wang, He Li, Ziwei He, Xinbing Wang, Zhiyu Li, and Zhouhan Lin. Pretraining language models to ponder in continuous space. arXiv preprint arXiv:2505.20674. 2025.
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. arXiv preprint arXiv:2310.11523, 2023.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910 2022.
Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Reasoning by superposition: A theoretical perspective on chain of continuous thought. arXiv preprint wrXiv:2505.12544 2025.
15

---

Published as a conference paper at ICLR 2026
A OTHER RELATED WORK
Scaling LLM Reasoning at Inference Time. The paradigm of scaling inference-time compute has emerged as a powerful strategy to amplify the reasoning capabilities of LLMs. Chain-of-thougee (CoT) prompting Wei et al. (2022b) pioneered this direction by eliciting multi-step reasoning explicitly, Subsequent works scale inference-time compute by sampling more reasoning chains or exploring larger reasoning spaces. For instance, Best-of-N (BoN) (Stiennon et al.) 2020 Nakano et al. 202
and Self-consistency (SC-CoT) Wang et al. (2022) repeatedly sample multiple chains and select the best one using heuristics such as predefined rewards or the consistency. More advanced approaches
have introduced various search algorithms to efficiently explore the reasoning space Yao et al. (2024);
Besta et al. (2024); : Hao et al. (2023) 2024). Tree-of-thought (ToT) Yao et al.] (2024) formulater
reasoning as a tree search problem, where the LLM generates thoughts while heuristic rewards
guide tree traversal. RAP Hao et al. (023 2024) employs Monte Carlo Tree Search (MCTS) to
strategically balance exploration and exploitation in the search space. Recent work by [nell et al. (2024) further formalizes the empirical benefits of test-time compute scaling. In contrast to these sampling and search-based methods, our work proposes gradient- based optimization to traverse the reward landscape more efficiently, bypassing the trial-and-error inefficiencies of sampling approaches. Notably, our method is orthogonal to recent RL-trained methods for test-time scaling, such as OpenAT's ol and DeepSeek's 's RI - our method can be directly applied to these models to refine their long CoT inference, though we leave this exploration to future work.
Inference-time Constrained Decoding. Constrained decoding is traditional problem to study in text generation, with popular applications including controllable text generation and preference alignment. Recently, dnference-time alignment has been heavily studied to align LLMs with human values as alignment rewards Li et al. 2023); Huang et al. (2024); Khanov et al. 2024). Most of them
formulate the problem reward- -errardggidedd search process. More relevantly, controllabe text generation
has leveraged energy- -aasdd models (EBMs) (Kumar et al. 2021] Qin et al. 2022] Kumar et al.,
2022. Mireshghallah al. 2022 Liu al. 2023b Yuan al. 2025 to relax discrete sequences
into continuous spaces, which enables gradient-based optimization (e.g., Langevin dynamics) to steer generation toward objectives defined by arbitrary energy functions. While sharing similar reward-constrained decoding principles and gradient-based methodologies, end-to-end sequence optimization is notoriously unstable, often producing broken sequences or failing to converge, which limits its applicability to reasoning tasks. In contrast, our approach introduces an efficient and novel gradient-based framework integrated with the iterative decoding, tailored for reasoning tasks.
Prompt Optimization. A distinct but related line of research focuses on optimizing prompts rather than decoding sequences. Established principles from the above sections apply here as well. Search-based methods Zhou et al. (2022): Wang et al.] (2023b): [Pryzant et al. (2023): Yuksekgonul et al. (2024) iteratively refine the prompts through automated tndd--and-error, while gradient-based approaches include directly optimizing soft prompts Li & Liang (002T); Lester et al. 202T). or
searching for discrete prompts via gradients Shin et al. (2200) Shi et al. (2022); Wen al. (2024). Although these techniques share conceptual similarities with our method, esp. the gradient- optimization methods, they operate the input (prompt) space rather than the output (reasoning chain) space. Our work focuses on optimizing reasoning trajectories directly, complementing rather than competing with prompt optimization methods future work could explore synergies between the two paradigms.
Continuous Latent Space Reasoning. Another line of research explores reasoning in a continuous latent space, bypassing discrete token-level operations. These approaches typically perform iterative refinement on the model's hidden states to solve reasoning problems. For example, some methods frame reasoning as an energy minimization process (Du et al., (222) or a fixed-poine iteration problem within the latent space, which can be parallelized for efficiency (Wu et al. (025). Others propose increasing test-time compute by applying recurrent updates to latent representations, effectively deepening the model's computation on-the-fly (Geiping et al. (025). This paradigm has also been supported by specialized pre-training objectives that encourage models to "ponder" in continu- ous space (Zeng et al. 2025) and has received theoretical analysis under the lens of continuous (Zhu et al. 2025). While conceptually related in their use of iterative refinement, these methods are fundamentally different from ours. They operate within the LLM's latent space, modifying internal hidden representations. In contrast, our work, V-Reasoner, performs optimization
16

---

Published as conference paper at ICLR 2026
Algorithm 3 V-Reasoner: Decoding with DTO
Require: Prompt a, language model #LLM, reward model stop criteria StopCriteria, and the maximal number of rollouts Nmaz.
1: y, TLLM(((a))
2: Nr +-1
3: repeat
4:
if ((1) 2 cent and |V L11 2 Egrad then { --DOO(0, TLLM,
> Token selection (Sec. Policy refinement with DTO (Sec.
3.3
5:
6:
end if
7:
Yi
softmaxx21/7)
8:
if
ÿ1 + then
9:
y, 2 TLLM
10:
N, t N,+1
y: y1 (+) > r(yla) then
Rejection sampling (Sec.
3.2
12:
<- concat (z. V1
13:
レ y. x
Rollout reusing (Sec.
3.3
14:
continue
15:
end end if
16:
17:
x concat x. 3/
18:
y V22, 222
Rollout reusing (Sec. D Early stop (Sec.
3.3
19:
if N, 2 Maa then concat y] break
3.3
21:
end
23:
until StopCriteria(z)
24:
return
directly in the output space. We manipulate the token logits, continuous relaxation of the discrete vocabulary, guided by reward gradients. This direct textual optimization allows us to refine the reasoning chain itself at inference time without altering the base model's internal forward pass requiring any specialized training, uniquely positioning our method as post-hoc, gradient- based search algorithm the sequence space.
B IMPLEMENTATION
B. PSEUDOCODE
In Algorithms D [ndd]] we present basic implementation for V-Reasoner, Below we give a detailed
pseudocode for a full version with all acceleration techniques integrated (Sec. (.3). In Alg. 3 we list the complete version of iterative decoding with confidence- and gradient-guided token selection, rollout reusing, and early stop techniques. Alg. 4 we present the full DTTO algorithm with gradient caching. We note that these techniques significantly accelerate the decoding speed of V-Reasoner, as demonstrated in Sec.
B.2 GENERALIZATION TO PROGRESS REWARD.
The reward function can take different forms: it may provide outcome reward (Cobbe et al. 2021 offering an overall score for the entire response sequence, or a process reward (Lightman et al. 2023 which assesses individual intermediate steps and assigns series of scores accordingly. Thus, beyond a single reward defined over the whole sequence, we denote the total reward as the sum of rewards
obtained from different subsequences: R(y x) = E!' r(yer/z). In the case of outcome reward, the reward is only assigned at the end of the response sequence, meaning r(y<)/z) = 0 if l Conversely, when using process reward, rewards are assigned incrementally, with r(y<ılæ) only if y is an end token of thought (Xiong et al. (024).. Our framework can scamlessly incorporate progress reward by replacing r(y]a with this generalized version R(y az).
17

---

Published as a conference paper at ICLR 2026
Algorithm 4 Differentiable Textual Optimization (DTO)
Require: Prefix initial logits language model ALLM. reward model r, and the number of training steps
1: y < None
2: g1 92.92." ........ + None
3: while T do
4:
for every i = 1, y do
5:
jr t arg ja 2/
6:
(1)
6z- + softmax(2,
(0) /T) StopGrad(soltmax(e"" /T)
(x)
7:
end for
8:
if * y then
ly(t)) Lill - Σ ri=l logTLIM(y) 1;
9:
10:
Cremard r((y())))
11:
L = Lill ACreward-
Eq.
12:
+-+ Dc for every i [
13:
9: oy.
Gradient caching (Sec. 3.3
14:
else
C Σl |
Di=1 91 yi
(t)
Surrogate loss with cached gradient (Sec. 3.3
16:
end if
(1++
(1) V/
18:
1 L 1.
end while
20:
return z
C DEFERRED THEORY
C.I GRADIENT DERIVATION OF C
We derive the gradients of Eq. 2 summarized as the following proposition. We consider the generalized reward function which is written as a summation over rewards defined over different subsequences (Sec. B.2
Proposition C.1. The gradient of loss function C(y) = - r()), log TLLM (g)=)) takes the form of @C(y) oy
Sprefiz + Spostfix + Xoreward where:
6prefix =
log Cat
(4)
lul
Olog Cat (TELM( (-1: y.
8 postfic
(5)
Oy:
i=1+1
Or(ys:
Breward
(6)
0y:
Proof. The proof is done by elementary derivative calculation. First of all, we write down the expanded expression of the loss function:
v log T LLLM
((y)
(3
(x)
(7)
r (x)
Σ
Vi. log FoLLM(ee
(8)
[
18

---

Published as a conference paper at ICLR 2026
For specific token index 1 [|y|], decompose the loss into five components:
y
((y) -
r (yss
())
Σ
Us. log NogTLL((eolMs-11
(9)
1=1
i=1 uEl
4/1
42
Σ vE[[V]
yi log (e
Σ vE[[VI
Yi. log www.gualleeee..sss....... (10)
where 001 = 0 and .ы. Ty.
am = 0 because they do not involve yı. 1l2 only depends on y through the term Vt, while II, only depends y via log (e. (e.1)))111. Φ). Next, compute the gradients for 22 Il2, and Il3, respectively.
wl
242
Or(ys:lz)
8 Breward
(11)
Oy:
Oy:
All
6 prefiix
|- log
(12)
oy:
log Cat (TIIM(
(13)
l
Ølla
log TLLA y5: y1
8 fost
(14)
Oyt
Oy
t=++1 VE
i Bog Cat
y1.
(15)
Oy:
1=111
as desired.
Remark C.2. Our proposed DTO fundamentally differs from previous works that utilize gradients for controlled generation (Qin et al. 2022 Kumar et al. |02 2022 Mireshghallah al. 2022 Liu et al. 2023b), where fpsrtfis is often detached from the computational graph, and only prior context used guide subsequent token prediction.
C.2 PROOF OF THEOREM
Suppose {p'Te2o denotes the Wasserstein gradient flow minimizing Eq. 3 in the distribution space with boundury conditions 0² -ILLM and 1Dh p² = =arg min, (Pro(p). Then we can draw samples from p² by first initializing x TLLMM and simulating trajectory {ar fe following the stochustic gradient flow Eq. dt V
Theorem C.3 (Restatement of Theorem 4.1p.
don
V2a, where {4 E (N) are Brownian motions.
Proof. First of all, we derive the Wasserstein gradient flow for pt on W2(R xN) under the functional cost LpPO (Chen al. 2018):
O.p² = -
O,p' + +v (VV=
8LPPO
0,
(16)
5p'
where &Cpro denotes the first variation of LPPO terms of Pr. and the V is the divergence operator. Ap' We derive the first variation as below:
8Lppo
5
[
+ log p'(x)
->r((x)) -xr(x)
o² (2))d
(17)
8p
5pt
TLLM (
->r(xx) + log ('()) -
(log))looTTLLL -logTLLMM(2
(18)
19

---

Published as a conference paper at ICLR 2026
The gradient of &Cp Apt
can be expressed as:
8LPPO
Vx
V (-Ar(x)) log pxx) log TLLM (x))
(19)
Now we substitute the above equations into Eq. 16 and find the following partial differential equation of p²:
Oep' + Vx [p'V (-Ar(x)) + log (x) (LLM
(20)
Lz N
O.p' + Ve (- -
+
(V log p² (x)) 0
(21)
Szyy
1,1=1
0,p' V. [p' (-XV Vn log TLLM (x)) Ap (z) 0,
(22)
where A means the Laplacian operator Lij 0² ow? we obtain velocity field for particles x' E LaNN da
By Fokker-Plank equation (Maoutsa et al.
2020
dt x=-Vv(((x)) - V
(x) + V2E, = vx((x)) + v2E,,
(23)
where (2:120 denotes Brownian motion (Øksendal 2003
Conversely, if x follows the above dynamics starting from the initial distribution xf pp, then the density function p² of æ follows the time evolution in Eq. 16 (see Maoutsa et al. (2020)) Since we assume the limiting condition p² + p", we conclude that x p in distribution 0c.
C.3 JUSTIFICATION OF CONFIDENCE BASED TOKEN SELECTION
We re- re-parameterize the policy categorical distribution through a softmax function ensure differentiability. In Sec. 3.3] we mention that we can skip optimizing tokens whose logits is over- confident gradient descent is unlikely significantly change its resultant distribution. We provide theoretical evidence for this argument.
Without loss generality, we consider the scenario where we only optimize a single token æ E I within a vocabulary V. We initialize logit vector E RR,, then apply the softmax transformation to obtain the corresponding categorical distribution:
exp(2;) MM exp(2,)²
2;
E [[V]]]
(24)
The loss function defined over a: and by chain rule, we derive the gradient w.r.t z, for [[V]]
ac ØL dx (diag(æ)) 20:0 1), OC
OL
Ox x;
x. T
(25)
0z. dx 0z;
dx.
dx
where the fact that the Jacobian matrix of softmax function is diag(æ) xx The derivation above indicates that the gradient magnitude for the i-th logit is proportional to its corresponding post-softmax probability. When æ; is small the initialization, then its underlying representation z; cannot be updated effectively. This limitation underscores the necessity of skipping tuning with high confidence throughout the decoding stage.
D MORE ON EXPERIMENT
D.1 EXPERIMENT DETAILS
We used temperature of 0.5 and top-p of 0.95, and set the maximum generation length to 1024 for AMC and MATH-500, and 3072 for AIME, for all baselines and our methods. We report the average performance across 4 independent rums smaller datasets such as AMC, AIME-24, and AIME-25. For SFT experiments, we randomly sampled a 10k subset from the Open-thoughts dataset (Guha al. 2025 For GRPO experiments, adopted random 35k subset from the Numina-Math
20

---

Published as a conference paper at ICLR 2026
dataset (Li et al. 2024 Exclusively for V-Reasoner,
Cent 0.25, Cyrad 8, learning rate to (Liu
0.01, and number of iterations to 20 in all experiments. We use Skywork- Reward-V2-Qwent al. 2025 the reward model for Qwen family and Skywork-Rewand -errr--eaad---laaaeeee 12-Lllama33.-881 (Liu et al. 2025 as the reward model for Llama model. Below, we list a more comprehensive summary of the experiment setups.
Table Summary of Experimental Settings.
Setting
Value
Generation Hyperparameters
Temperature
0.5
Top-p
0,95
Max Generation Length (AMC, MATH-500)
1024
Max Generation Length (AIME)
3072
Evaluation
Independent Runs (MATH-500)
1
Independent Runs (AMC, AIMB-24, AIME-25)
4
Training Data Subsets
SFT
random sample from Open-thoughts Guha 2025
GRPO
35k random sample from Numina math ct al. 2024
V-Reasoner Hyperparameters
Fent
0.25
Cgrad
8
Learning Rate
0.01
Number of Iterations
20
Optimizer
Adam-W
I.R Scheduler
Cosine
Min LR
0.001
Reward Models
For Qwen Family
Skywork-Rewand-Ve -22cwaand-22-Qwen3--BB (Liu et al. 2025
For Llama Model
Skywork-Rewart -ww---ewwaad--eeeee V2-Llamaa-3.1-88 (Liu el al. 2025
D.2 EFFECTIVENESS OF ACCELERATION TECHNIQUES
In Sec. 3.3 we proposed three major techniques to improve the decoding efficiency of --Reasoner. In this section, quantify the contribution of each technique to the overall speedup. Instead of ablating them in rumning time (since removing any one of these components causes the algorithm to in prohibitive amount of time, often more than several hours per sample), monitor how each technique reduces the cost of the specific stage to which it is applied. As a result, we find our gradient-caching mechanism bypasses more than 63.8% of the model calls in parallel forms required for gradient acquisition, and trajectory reuse eliminates more than 74.1% 1% of autoregressive model calls. Both techniques exploit the sparsity token updates. The token ooken-selection strategy further complements these components by skipping the full optimization procedure when appropriate; we observe that it effectively avoids 89.2% of token-optie ootemiitimi steps across the sequence.
D.3 WALL-CLOCK TIME BENCHMARKING UNDER IDEALIZED SETTINGS
In this section, we measure and compare the wall-clock running time of our method with strong baseline, BoN. We consider an ideal- ized deployment setting where dynamic serving frameworks (e.g.. vL..M) and batching are fully utilized, so that all prompts are processed in parallel. Under this assumption, both genera- tion and optimization requests can be executed as batched operations. To estimate the running time in this setting, first record execution traces of the reasoning process for both BoN and V-Reasoner. The trace includes statistics such as the number of generated tokens per request, the number of gradient computations, and the number of tokens involved in optimization. We then group
Table 5: Comparison of wall-clock execution time for 83 prompts on the AMC dataset.
Method
Gen.
Optim.
Total
FLOPs
BoN (N
136.1
136.1
9.54
10
V-Reasoner
106.2
45.9
152.1
2.46 10
21

---

Published as a conference paper at ICLR 2026
and replay these traces to simulate parallel generation and optimization. The recorded statisties used to reconstruct the execution process. The simulation sends generation requests to a vLLM server and performs gradient computation over batched sequences. We measure the corresponding wall-clock running time together with the computational FLOPs. The experiment was conducted model running 8x 140GB NVIDIA
the AMC dataset using the H200 GPUs. The results are demonstrated in Tab, [5] As shown in our benchmarks, for batch of 83 prompts, the BoN baseline (NV = requires 136.1 seconds of total execution time. In comparison, our approach completes the generation phase in 106.2 seconds, with an additional 45.9 seconds of overhead for the iterative optimization process (calculated as the total optimization time distributed across 8 GPUs). This results in total wall-clock time of 152.1 seconds. We further analyze the computational efficiency by comparing the total FLOPPs. While V-Reasoner performs significantly more operations than the BoN baseline (2.46 x 10a7 vs. 9.54 x 10t5 FLOPs), the total wall-clock time remains comparable. This is because hardware utilization (TFLOPS/s) during the DTO phase where attention is executed in parallel is significantly higher than during the auto-regressive phase, where attention must be computed sequentially.
D.4
A SIMPLIFIED EXAMPLE ON GSM8K
The problem asks:
Josh buys a house for $80, 000 and then puts in $50, 000 in repairs. This increased the value of the house by 150%. How much profit did he make?
Greedy Decoding Baseline (Failure Analysis)
The original output reads:
Josh buys a house for $80, 000 and then put.s in $50, 000 in repairs. The value of the house increased by 1508, the new value of the house is 80, 000 1.5 = $120, 000. Josh sold the house for 120, 000, he made profit of 120, 000 80, 000 50, 000 -10, 000. The answer is -10, 000.
This greedy decoding baseline fails by interpreting "increased by" simple multiplication (80, 000 1.5), resulting in:
Incorrect New Value $80, 000 x 1.5 S120, 000
and an incorrect profit of:
Incorrect Profit = $120, 000 $80, 000 $50, 000 = $$00 000
DTO- Token Revision (Correction)
Our method corrects this error through DTTObbased refinement.
Initial Input and Refinement Round (Operator Correction)
Initial Output Segment: "..., so the new value of the house is 80, 000 x 1.5 = $120, 000."
The algorithm performs refinement at the multiplication token "x".
Through gradient descent, the model updates the multiplication to- ken "x" to the addition symbol "+", revising the logical structure to: ...., so the new value of the house is 80, 000 +..."
Refinement Round 2 (Calculation Correction)
Subsequently, when calculating the increment, the model initially outputs 80, 000 x 1.5 $1 20, 000. The calculation is incorrect because the model ignores the preceding 80, 000++
The algorithm flags the token 120 (representing $120, 000) due to a low score.
Gradient optimization identifies 200 as the top replacement, correcting the calculation for the new value:
Correct New Value $80, 000 + ($80, 000 1.5) $80, 000 $120, 000 $200, 000
22

---

Published as a conference paper at ICLR 2026
Final Correct Derivation
The model completes the string to form "$200, 000", allowing it to derive the correct profit:
Correct Profit $200, 000 $80, 000 $50, 000 $70, 000
Conclusion
In conclusion, after two rounds of iterative refinement, the final output becomes:
Josh buys house for $80, 000 and then puts in $50, 000 in repairs. The value of the house increased by 1508, so the new value of the house is 80, 000 80, 1.5 $200, 000. Josh sold the house for 200, 000, so he made a profit of 200,000 80, - 50, 000 70, 000. The answer is 70, 000.
23


Reviews:
[{"review_id": "d9nlDgQJCI", "reviewer": "Reviewer_sgwy", "summary": "This paper introduces ∇-Reasoner, an inference-time reasoning framework that applies gradient-based optimization to refine LLM outputs during decoding. The core innovation is Differentiable Textual Optimization (DTO), which optimizes token logits via gradient descent using signals from both the LLM's likelihood and a reward model. The approach combines an objective that balances reward maximization with log-likelihood regularization (Eq. 2), employs the straight-through estimator for differentiability, and integrates rejection sampling to accept only improved tokens. To enhance efficiency, the authors introduce gradient caching, rollout trajectory reusing, and confidence/gradient-guided token selection. Theoretically, they establish that sampling from an RL-optimized LLM is equivalent to drawing from the reference LLM and refining via DTO's gradient flow (Theorem 4.1). Evaluated on mathematical reasoning benchmarks (MATH-500, AMC, AIME), ∇-Reasoner achieves over 20% accuracy improvement while reducing computation by approximately 40% compared to Best-of-N and Self-Consistency baselines, demonstrating superior test-time scaling efficiency.", "questions": "- Gradient quality with straight-through estimator: The straight-through trick provides biased gradients. Have you investigated whether this bias significantly impacts convergence, or explored alternative differentiable relaxations (e.g., Gumbel-softmax)?\n- Reward model dependency: How does performance degrade with reward models of varying quality? Have you experimented with different reward model architectures or training strategies? Could process rewards provide better gradients than outcome rewards?\n- Optimization landscape: Can you characterize the reward landscape's properties (smoothness, multimodality, local optima)? Does DTO sometimes get stuck in local optima, and if so, how might this be addressed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n- The shift from zeroth-order search to first-order gradient-based optimization for test-time reasoning is conceptually appealing and well-motivated by Figure 1\n- The theoretical connection between DTO and PPO via Wasserstein gradient flow (Theorem 4.1) provides an elegant unification of parametric and non-parametric inference perspectives\n- The gradient decomposition (δ_prefix, δ_postfix, δ_reward) offers clear intuition about how DTO enables bidirectional information flow along sequences\n\nQuality:\n- The experimental evaluation spans multiple model families (Qwen-2.5, Llama-3.1) and benchmarks with consistent improvements\n- The ablation study in Table 2 effectively demonstrates DTO's impact on rejection rates (reducing from 66% to ~30-40%)\n- The cost-accuracy trade-off analysis (Figure 3, Figure 4) convincingly shows efficiency gains over sampling-based baselines\n- The comparison includes both test-time methods (SC, BoN, ToT, RAP) and training-based methods (SFT, GRPO), providing comprehensive context\n\nClarity:\n- The motivation is clearly articulated—gradient information provides richer directional guidance than scalar rewards\n- Figure 2 and Algorithms 1-2 effectively communicate the overall framework\n- The progression from basic formulation to acceleration techniques is logical and well-structured\n- The gradient decomposition in Section 4 provides valuable interpretability\n\nSignificance:\n- Addresses the fundamental limitation of sparse reward signals in test-time scaling\n- The 40% cost reduction while maintaining or improving performance has practical value\n- The theoretical framework connecting test-time and training-time optimization could inspire future work\n- The method appears general enough to work with different reward models and base LLMs", "weaknesses": "Experimental Analysis:\n- While the cost comparison uses \"number of calls,\" actual wall-clock time could differ due to backward passes. Could you provide runtime measurements to complement the theoretical cost analysis?\n- The comparison with RAP and ToT might not be entirely fair if those methods weren't given comparable computational budgets. Could you ensure all baselines use similar total compute?\n- Table 1 shows ∇-Reasoner sometimes underperforms training-based GRPO (e.g., Qwen-2.5-7B on AMC: 51.5% vs 52.8%). This suggests potential limitations—could you characterize when training-time methods remain superior?\n\nImplementation Details:\n- The method requires modifying the decoding loop and maintaining gradients through generation, which could complicate integration into existing serving infrastructure. Have you explored compatibility with standard inference optimization (e.g., KV caching, speculative decoding)?\n- The acceleration techniques are crucial for practicality but add complexity. It would be valuable to quantify each technique's individual contribution to overall speedup\n- For the Llama-3.1 experiments, you note inability to evaluate on AIME due to model incapability. This suggests potential brittleness—how does the method degrade with weaker base models?\n\nComparison Fairness:\n- The SFT baseline uses only 10K examples while GRPO uses 35K—this asymmetry makes it difficult to assess whether ∇-Reasoner truly matches training-based methods or is being compared against undertrained baselines\n- For the \"comparable performance with training-based methods\" claim, some results show ∇-Reasoner trailing GRPO (e.g., AIME25 on Qwen-2.5-7B: 15.0% vs 16.7%)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hrBonXjMB0", "reviewer": "Reviewer_gxvJ", "summary": "This paper focues on test-time scaling to improve LLM's performance for reasoning task. Commonly, we can use a reward model to choose the best sampling from a pool of candidates (BoN). Such a method can be considered as zeroth-order search. In this paper, the authors propose a new iterative generation framework, ∇-Reasoner, that applies differentiable optimization over token logits into the decoding loop to redfine the policy with the signal from reward model. The Differentiable Textual Optimization (DTO) can be considered as a first-order optimization, used at test time. It utilizes the gradient signals from the LLM's likelyhood and reward model to refine the textual representations. \n\nThe proposed method is theoretically justified, and empirically shows significantly better accuracy and efficiency than various strong baselines.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, with a clear motivation. \n2. The proposed method, DTO, is interesting and novel, offering a new way based on gradient optimization to test-time scaling. And DTO is supported with theoretical justification.\n3. Extensive experiments show a significant accuracy and efficiency improvement.", "weaknesses": "1. Lack of reward models. In Appendix D, I notice you apply different reward models for differetn policy models. May I ask why? Could you offer an ablation study for the choice of different reward models for the same policy model.\n2. The improvement statement in the abstract is overstated.  The 20% accuracy improvement and 40% less computation are for different baselines, incuring some confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JIUlXg9sim", "reviewer": "Reviewer_wy2L", "summary": "This paper investigates expanding computation during inference. Existing methods tend to saturate in long-chain reasoning due to the influence of sparse and noisy rewards. The authors propose performing gradient descent on token logits in the sample space during decoding. Theoretically, DTO supports bidirectional gradient propagation along the sequence and establishes an equivalence to reinforcement learning–based training. Empirically, it outperforms baselines across multiple benchmarks, achieving comparable or superior reasoning accuracy at a lower computational cost.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The method demonstrates outstanding performance across multiple benchmarks. \n2.The authors made practical considerations, enabling the proposed method to integrate well with existing LLM inference acceleration infrastructures.\n3.The authors derived gradients over discrete text and attempted to provide theoretical guarantees for DTO", "weaknesses": "1. The authors introduced a hyperparameter in the objective function but did not conduct experiments to analyze its impact. \n2.The proposed method heavily relies on the performance of the reward model, yet only one reward model was used in the experiments. We hope to see results under multiple reward models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ziO0xytuHj", "reviewer": "Reviewer_qoEv", "summary": "This paper introduces $\\nabla$-Reasoner, a test-time differentiable reasoning framework for large language models (LLMs). It proposed Differentiable Textual Optimization (DTO) that performs gradient descent on logits. The paper provides some empirical experiments and theoretical analysis.", "questions": "1. What do you mean in \"it reduces costs by up to 40.2%\" in Sec 5.1's cost analysis? How did you get the figure? Which baseline are you comparing with? Is it about wall-clock latency or FLOPs or something else? This claim is not supported by any concrete experiments in the paper.\n\n2. As mentioned above, why do you use different RMs for different policy LMs? Would the performance degrade when the quality of RM is poor or the output of policy model is OOD for the RM?\n\n3. Is the overhead significant especially when RMs are large or output sequence is long?\n\n4. Can your theory be more aligned with the actual methodology? \n\n5. Could you provide some case studies, that is, concrete examples on what your framework is doing on some certain questions and models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The idea sounds interesting. It is interesting to perform training on logits during test time since it has some challenges. The authors also proposed methods to reduce the heavy computational overhead, showing their awareness of practical utility.", "weaknesses": "My major concern is that **the experiment setting is considerably vague, incomplete and flawed** in many aspects.\n\n1. **Lack of efficiency analysis**. This is a strong concern I have on this paper. Although the authors proposed methods to reduce the heavy overhead, the detailed analysis and experiments are missing. In Sec 5.1, “Efficiency” is measured solely by model calls, ignoring **wall-clock latency, FLOPs, and GPU memory**, which are the core components of efficiency. Intuitively it is very likely that the overhead would be significant since getting gradients would require at least 2x GPU memory, and the latency is likely substantial since it requires performing gradient descent back of forth. In addition, it is important to show whether the efficiency is strongly dependent on the choice of policy models and reward models, and the output sequence length. Lack of such analysis clearly undermines the soundness of the paper.\n\n2. **Unfair comparison with test-time methods**. It is unfair to compare your methods with vanilla SC and BoN using both 8 rollouts. With tools like vLLM library, vanilla SC and BoN can be done flashly in one go, which is much more efficient than your proposed framework if the number of rollouts are the same. I suggest the authors align the budgets (wall-clock latency) with these methods to ensure fairness.\n\n3. **Training-based baselines are not convincing**. In practice, we often apply SFT then RL, not to perform them separately. It's unfair to compare SFT and GRPO separately since they should be incorporated as a whole framework. The paper also did not demonstrate how the budget of the training-based baselines aligns with $\\nabla$-reasoner.\n\n4. **No comparison with other test-time training / inference-time adaptation baselines**. This is a serious issue since there are already so many test-time training methods, including but not limited to [1][2]. **None of these are compared with $\\nabla$-reasoner.**\n\n5. **Limited domain coverage**. All datasets are about math reasoning. It is unclear how the method performs on other domains like coding, general reasoning, open-ended QA, helpfulness and safety.\n\n6. **Missing analysis of dependence on reward model quality**. The proposed method’s effectiveness appears highly sensitive to the quality and compatibility of the reward model (RM) used for gradient guidance. Appendix D shows that different RMs are applied for different policy LLMs. Why do you do that? Can your framework still work well when the quality of RM is not that good or the output of policy model is OOD for the RM? I suggest incorporating detailed experiments on the effect of applying different RMs across different tasks to show that whether your method is sensitive with the choice of RM or not.\n\n7. **Missing details for reproducibility**. The paper incorporates many experimental details in Appendix D, which is good. However some details are still unclear, including hardware (which is crucial for efficiency analysis) and evaluation methods (rule-based evaluation or strict string matching?). The authors also did not provide their code.\n\n\nIn addition to the experiment settings, I also have concern on the theoretical analysis. Theorem 4.1 is not sound enough since there is a **clear gap between the theorem and actual methodology**. The Wasserstein gradient-flow theorem assumes continuous time, continuous space, infinitesimal steps, and Gaussian diffusion noise. However, actual DTO uses finite discrete steps and deterministic updates. Hence, the theory is **intuitive but not rigorous**, that could act as a motivational study but not a rigorous proof of the equivalence.\n\n[1] Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback  (ICML 2025)\n\n[2] Learning to Reason from Feedback at Test-Time  (ACL 2025 main)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)