Rethinking Thinking Tokens: LLMs as Improvement Operators
Authors: name: Lovish Madaan, name: Aniket Didolkar, name: Suchin Gururangan, name: John Quan, name: Ruan Silva, name: Ruslan Salakhutdinov, name: Manzil Zaheer, name: Sanjeev Arora, name: Anirudh Goyal

Reasoning training incentivizes LLMs to produce long chains of thought (long
CoT), which among other things, allows them to explore solution strategies with
self-checking. This results in higher accuracy, but inflates context length,
token/compute cost, and answer latency. We ask: Can current models leverage
their metacognition to provide other combinations on this Pareto frontier,
e.g., better accuracy with lower context length and/or latency? Abstractly, we
view the model as an improvement operator on its own "thoughts" with a
continuum of possible strategies. We identify an interesting inference family
Parallel-Distill-Refine (PDR), which performs the following: (i) generate
diverse drafts in parallel; (ii) distill them into a bounded, textual
workspace; and (iii) refine conditioned on this workspace, producing an output
that seeds the next round. Importantly, context length (hence compute cost) is
controllable via degree of parallelism, and is no longer conflated with the
total number of generated tokens. We report PDR instantiations of current
models that give better accuracy than long CoT while incurring lower latency.
Setting degree of parallelism to 1 yields an interesting subcase, Sequential
Refinement (SR) (iteratively improve a single candidate answer) which provides
performance superior to long CoT. Success of such model orchestrations raises
the question whether further training could shift the Pareto frontier. To this
end, we train an 8B thinking model with Reinforcement Learning (RL) to make it
consistent with PDR as the inference method. On math tasks with verifiable
answers, iterative pipelines surpass single-pass baselines at matched
sequential budgets, with PDR delivering the largest gains (e.g., +11% on AIME
2024 and +9% on AIME 2025).