ThoughtTerminator: Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Models
Authors: Xiao Pu
Michael Saxon
Wenyue Hua
William Yang Wang
Venue: COLM 2025 (2025)
Keywords: reasoning model
overthinking
decoding
tool use
evaluation
benchmark
TL;DR: We evaluate the relationship between problem difficulty and token cost in reasoning models, benchmark how efficiently different models allocate tokens, and introduce a simple training-free decoding method to reduce overthinking, Thought Terminator.

Abstract: Reasoning models have demonstrated impressive performance on difficult tasks that traditional language models struggle at. However, many are plagued with the problem of overthinking---generating large amounts of unnecessary tokens which don't improve accuracy on a question. We introduce approximate measures of problem-level difficulty and demonstrate that a clear relationship between problem difficulty and optimal token spend exists, and evaluate how well calibrated a variety of reasoning models are in terms of efficiently allocating the optimal token count. We find that in general, reasoning models are poorly calibrated, particularly on easy problems. To evaluate calibration on easy questions we introduce DUMB500, a dataset of extremely easy math, reasoning, code, and task problems, and jointly evaluate reasoning model on these simple examples and extremely difficult examples from existing frontier benchmarks on the same task domain. Finally, we introduce ThoughtTerminator, a training-free black box decoding technique that significantly improves reasoning model calibration.

Published as a conference paper at COLM 2025
THOUGHTTERMINATON Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Models
Sophia Xiao Puc+ Michael Saxon W.
Wenyue Hua William Yang Wang
CUniversity of California, Santa Barbara
W "University of Washington Microsooft
Contact: xiao_ puwucsb.
xiao_pu@ucsb. edu, mssaxon@uw.edd edu
Abstract
Reasoning models have demonstrated impressive performance on diffi- cult tasks that traditional language models struggle at. However, many are plagued with the problem of overthinking overthinking-ggneraad generating large amounts of unnecessary tokens which don't improve accuracy on a question. We introduce approximate measures of problem-level difficulty and demon- strate that a clear relationship between problem difficulty and optimal token spend exists, and evaluate how well calibrated variety of reasoning models are in terms of efficiently allocating the optimal token count. We find that in general, reasoning models are poorly calibrated, particularly on easy problems. To evaluate calibration on easy questions we introduce DUMB500, dataset of extremely easy math, reasoning, code, and task problems, and jointly evaluate reasoning model on these simple examples and extremely difficult examples from existing frontier benchmarks on BRRUGHTTERMINAAA
the same task domain. Finally, we introduce a training-free free black box decoding technique that significantly improves reasoning model calibration.
1 Introduction
Investment in improving the capabilities language models has recently turned from data- and train-time-scaling to inference-scaling,
training so-called reasoning models to expend more runtime compute generating chains of thought (Wei et al., 2022), debate (Liang et al., 2023), and self-corrections (Pan et al., 2024) in order to more robustly and correctly answer queries (Wu et al., 2024). This work appears to show a direct, positive relationship between inference spend and "reasoning task" performance (Jaech et al., 2024).
Under the inference scaling paradigm, controlling costs is critical. Unfortunately, open reasoning models such as DeepSeek (DeepSeek-Al et al., 2025) and QwQ (Qwen, 2025) have demonstrated a tendency to expend unnecessary inference tokens after the answer has already could be generated, problem referred as ozerthinking (Chen et al., 2024).
We need to precisely define overthinking in order to mitigate it. Chen et al. (2024) define overthinking as the amount of times the model repeats the correct answer its intermediate reasoning chain. From this definition, they used supervised fine-tuning and direct preference optimization to train reasoning models prefer to select the shortest answer. Similar work applied knowledge distillation from non-reasoning models to blend concise answering abilities with the superior performance of reasoning models (Yang et al., 2025). However, both of these methods require retraining, process that may be costly or have unintended consequences on performance.
Training free methods which seek to manage overthinking include selective invocation of chain-of-thee oftihgof-thought on tasks where it has known benefit (Sprague et al., 2024), early stopping of reasoning chains using probe-based confidence final answer tokens (Fu et al., 2024), or simply eliciting reasoning model-like behavior from non-reasoning models using continuing
* "Co-first contributions.
All work completed the University of California, Santa Barbara.

---

Published as a conference paper at COLM 2025
MATH500
ZEBRA
12000
15000
12500
10000
Avg True Token
Takens
Tahern
Min True Token
6000
Max True Token (Rt)
Avg True Token (Fit)
5000
3000
Token (Fit)
2000
Difficulty
Diffculty
Difficutty
Figure 1: Question-level difficulty (g, vs average token spend (M( for Deepseek-ri - (model M) on three reasoning datasets. Difficulty scores are scaled by 10 for readability, We observe clear relationship between question difficulty and token spend distribution.
phrases like "wait...", which can be halted any time (Muennighoff al., 2025). Limitations of these methods include requiring external knowledge of task type, white-box access to the base model, or the use non-reasoning models for precise control (Yu al., 2025).
In this work we analyze the token spend & difficulty calibration of reasoning models. Starting from the supposition that more difficult problems require more thought, we first characterize this difficulty-cost relationship in variety of open reasoning models across three reasoning datasets datasets-MATTH5N MATH500 (Lightman et al., 2023), GPQA (Rein et al., 2023), and ZebraLogic (Lin et al., 2024) allowing to introduce difficulty-calibratee measure of overthinking.
As these three existing datasets only allow us to assess overthinking on hard problems, we introduce DUMB500, dataset of "ass"" queries to explore overthinking on easy inputs. Finally, we introduce FROUGHTTERMIN.CC ATRRINNATOR, a tool-based, training-fre black box decoding strategy to mitigate overthinking using difficulty-caline callbrtt conditioning. We show that THOUGHT TERMINATMNNN is simple and effective way to control overthinking in reasoning models without requiring any access to gradients or training'.
2 Difficulty Calibration in Reasoning Models
This work is concerned with how optimally a reasoning model M, given question pair (q,a), allocates its answer token spend (m(()) with respect the pair's difficulty a).
Given that increased inference scale leads to higher performance across variety of reasoning tasks, we hypothesize that the difficulty of a question correlates with its optimal token spend. We characterize question's model-specific difficulty D a) as a model's incorrect answer rate over n samples of that question q against gold answer a.
Dm(g.a) = p(a + a *a~ ~ [I(((q) #a)
(1)
H
We then compute model-agnostic difficulty estimate D of q the expected difficulty (g) a)] over a set of models M. By assessing difficulty across a diverse collection of mod- els, this measure provides rough operational notion of difficulty that is both reproducible and relevant for analyzing inference efficiency of current LLMs.
a(g,a) =[(((g.)) R
Σ E1(M(()) #a) MiEM
(2)
M n
Each answer à; incidentally sampled from M in response to question is associated with its own token spend 5 (2,). Is there a relationship between the difficulty of each question and the token spend that naturally occurs?
1 Data and code available gith wwwp//pphaa///uMMSS.S..
2

---

Published as a conference paper at COLM 2025
Model
Local overthinking Oen +
Global overthinking Og
Non-reasoning language models
Qwen2-78-Instruct
291
219
542
354
708
473
L1ama-3.1-88 1-8-Inrrrutt
1971
1755
genma-2-2b-it
152
gemma-2-9b-in -2-9b-it
131
161
gemma-2-27b-15
178
187
deepseek- Srepseek-11m-7bchhae
155
90
Reasoning language models
QwQ-32B-rreview
2923
3698
QwQ-32B
13662
11248
5730
4262
SeepSeek-R1-isttll--
4232
5755
3881
4001
Table 1: Local and global overthinking scores (rounded to integers).
We assess the difficulty D and token spend S.M using reasoning and non-reasoning models from the DeepSeek (DeepSeek-A) al., 2025), Qwen (Yang et al., 2024; Qwen, 2025), Gemma (Mesnard et al., 2024), and LLaMA (Dubey et al., 2024) families for all questions in MATH500 (Lightman et al., 2023), GPQA (Rein et al., 2023), and ZebraLogic (Lin et al., 2024).
Figure 1 contains scatter plots of DM and S for each answer from DeepSeek-R1-78 for three datasets. We observe that-similar to the dataset & model-wise relationships between performance and token spend documented in prior work (Muennighoff et al., 2025) there also exists clear relationship between question-level difficulty and average token spend.
Additionally, we note considerable variance in the token spend between answer samples for each question. These reasoning models exhibit considerable inconsistency in their efficiency between samples. This leads to two natural questions:
1. How well-calibrated reasoning models in consistently realizing their optimal token spend per-question?
2. Is it possible to improve the calibration of reasoning models in their token spend?
2.1 Quantifying Overthinking
We formalize observational overthinking, the failure in consistency reasoning model has at realizing the minimum possible token spend per question.
The observed minimum spend of question is the shortest reasoning chain in set of model's correct answers. We measure observational overthinking in terms the difference between model's typical token spend and this observed minimum. For questions sampled from dataset D, the global overthinking score Og of a model is the mean difference between the length of each reasoning chain and the global observed minimum spend for each question.
Ox(M)= Σ (B[S(a M(g)] min (S(a M:lg)))
M;EM
RED
The local envelope overthinking score Oenv is the mean difference between the maximum and minimum spends for each question for each model.
Oenu (MM) = Σ (max[S(a> M(4)] min MIq)))
(4)
9ED
Table 1 presents the calibration scores for the full set of LLama, Qwen, Gemma, and DeepSeek models evaluated the three datasets. These calibration scores represent
3

---

Published as a conference paper at COLM 2025
Chat grading: rubrics
Code grading: test cases
Earth Sun?"
"How long does for the
Bancking
Strings
Tuples
Loogs
Li
Write code that returns
Rubric (general):
Knowteigg
if-E1se
Variables
Fibonacci number.
Accuracy, Conciseness...
Random
Test cases:
Rubric (knowledge Qs)
enswer precise?
Daily Chaty Chatt
code
Math
N(O) =
N(1)
N(G) 5
Answer: year
Printing
N(9) 34
</>
Commoraanto
chat
Social
Math grading: answer key
Greatings
DUMB500
hamphraanng
Task grading: rubrics
Sefcriietive
task
Mexcepos
" "wrtte email to Alice
"What 17?"
egen
Additiin
math
Daily Life
Anology
Einall
confirming 3pm call"
Answer: 22
bateracctinn
Creebo soare
Work
Rubric (genoral): same as
tha
Division
Matrsticttion
Frections
Geometyy
Comperhha
Words
Rubric (specific): is the email to Allce?
chat
Evalusted by exact match ettermpt followed by LM judge
Squares
Logic
is the meeting at 3pm?
Figure 2: DUMB500 dataset composition and grading method. The dataset contains four subsets, CHAT, CODE, TASK & MATH, which each graded with subset-specific methods. MATH are graded with traditional answer pairs. CHAT and TASK are graded using combi- nation of LM-judged rubrics and where appropriate, answers. CODE outputs are generated as test case coverage.
expected quantities of tokens wasted, they averages in excess of minimum spend values. Lower is better. As expected, the reasoning models with propensity to overthink have considerably higher overthinking scores than the non-reasoning models.
One weakness of our overthinking evaluation so far is that we have very few questions that have low difficulty but high overthinking tendency. As reasoning models are most useful for challenging frontier tasks, easy question benchmarks for reasoning models don't 't exist. However, to get full picture of the overthinking problem, must also be evaluated simple prompts. In the next section we introduce a resource, DUMB500, to do this.
3 Extending Overthinking Evaluation with DUMB500
While it is common knowledge that reasoning models tend to overthink on simple queries (Chen et al., 2024), no resource has been proposed systematically evaluate this tendency on simple, straightforward questions.
The DUMB500 dataset specifically designed evaluate models on simple questions that humans can answer effortlessly. The goal is not to challenge models with intricate logic but rather to assess their fundamental ability to recognize simplicity and provide concise, correct responses. To the best of our knowledge, DUMB500 is the first dataset explicitly focused on extremely simple (and sometimes deliberately naive) questions. DUMB500 consists of 500 manually curated questions spanning four domains:
Mathematics (Math): Basic arithmetic, comparisons, geometric properties, and logical reasoning.
Conversational Interaction (Chat): Casual dialogue, self-reflection, common knowl- edge, and basic object interactions.
Programming & Computing (Code): Fundamental coding concepts, including variables, loops, conditionals, and data structures.
Task Execution (Task): Simple natural language processing tasks such as paraphras- ing, translation, and basic writing.
4

---

Published as a conference paper at COLM 2025
dataset dumb
50
math500
gPqa
Questions
30
4
8
Difficulty
Figure Joint difficulty distribution of the four datasets we evaluate in this work. Difficulty scores are scaled by 10 for readability. DUMB500 helps balance the distribution of question difficulty in our analysis, enabling fuller assessment of reasoning model overthinking.
Each question is designed to be trivial for humans, requiring minimal cognitive effort, while still serving as litmus test for overthinking in reasoning models. The dataset allows us to evaluate models based on two key dimensions:
Accuracy: Can the model correctly answer simple questions?
Efficiency: Does the model avoid unnecessary elaboration?
We manually crafted 500 simple questions across diverse range of common knowledge, arithmetic, and practical queries. The full list of question classes with their descriptions listed in subsection A.1. Figure shows the distribution of question types in DUMB500 as well as sample questions and answers.
3.1 Evaluation techniques for DUMB500
In addition to the extremely simple MATH questions presented in DUMB500 which are evalu- ated using simple accuracy methods also introduced CHAT, CODE, and TASK questions, which require more sophisticated evalua- tion. They are evaluated as follows:
methods-iideniccllyy identically MATH500, GPQA, and ZebraLogic-
CODE questions include set of test cases for the program described in the prompt. A python-based autograder checks that the requirements are met.
CHAT questions belong to of seven subtasks (eg., greetings, acknowledgements All chat answers are evaluated according to a set of generic requirements, such as appropriateness and conciseness. Depending on the subtask, specific requirements such precision and accuracy are checked. When accuracy assessment is required, an answer is also provided.
TASK questions generally include instructions for the assistant to produce some kind of writing or answer some work-related question. In additino to using the same generic requirements as CHAT, TASK questions have one or more question-specifi requirements which check that the implicit instructions in the prompt are followed (See Figure 2). The CHAT and TASK requirements are checked using an LM (gpt-4o-mini) judge?.
3.2 From Dumb to Hard Questions
We evaluate the same set of models as in Table on DUMB500 and analyze their accuracy and token spend across different subsets. Figure depicts the distribution of questionwise difficulty scores across the MATH subset of DUMB500, MATH500, GPQA, and Zebral Zebral assessed using those models. This confirms that DUMT fills in gap in our analysis, adding a considerable quantity of easy questions with which to analyze overthinking.
M judge performance discussed in subsubsection A.2.4

---

Published as a conference paper at COLM 2025
Math
Code
0.8
Avg. Score
Avg. Score
0.6
Model
0.4
Qwen2-7B-instruut
QwQ-32B-Preview
0.2
QwQ-32B
0.0
0.0
Uama-3.2 1orma-3.--1B-nsseie
2.11Innnstt
102
10
104
10a
10?
10
Uama-3.2 1orma-3.233--a5 3.2-3Bt-nttt
Tokens
Tokens
Uamma33 1amm--.11-60 3-1--331---8-ese
3.1-8B-Instruct
Task
Chat
gemma-2-2b-tt
gemma-22-b-it
pemma---227--2
Avg. Score
deepseek- -i-----l---b-hht
0.3
DeepSeek-RD Bai-sekkRR--istill-O- R1-Distill-Qwen-1..
Avg.
0.2
DeepSeek-RD RL--Distil--Lama--8
0.1
0.0
0.0
10a
10a
10
10a
10
Tokens
Tokens
Figure Relationship between average token spend Sp (Tokens) and average score for the evaluated models on each subset of DUMB500.
Figure 4 shows the relationship between model-level accuracy and token spend for the tested models. As expected, on these simple math questions there is no positive relationship between token spend and accuracy, as these questions are extremely easy. For the other do- mains, we observe a negative correlation between token spend and evaluation requirement pass rate (labeled accuracy).
4 THOUGHTTERMINATOR
Reasoning models often express inference scaling in natural language through tokens expressing uncertainty, like "wait..." " or "let me check this..." (Muennighoff et al., 2025) Thus, overthinking often manifests superfluously after the correct answer has already been found.
tendency overuse these extending expressions
From this insight, hypothesize that simple text-augmentation methods be used to counteract this tendency, reminding the model of how long its output has been, and how soon it should come to answer. of interrupt messages at a fixed token interval which are inserted into the autoregressive stream, alerting the model of how many tokens has spent and how many remain.
BROUGHTTEEMMNGES
realizes this series
Sometimes, these timing messages and reminders alone are sufficient to get the model to provide its answer in concise manner. If answer isn't provided before the end of the time limit, terminating prompt and constrained decoding forces the model to output a final answer.
Figure 5 shows an example of a base reasoning model and one using THOUGHTTERMINS TOR answering question. ThOUGHTTERMINATONN operates on a reasoning chain in three stages: scheduling, running, and terminating.
Scheduling. Given an input question THOUGHT TERMINATOR needs an estimate of how many tokens necessary to produce correct answer in order to set its interrupt rate and termination time.
complications in consistently extracting the CHAT and TASK answer snippets across the diverse output formats employed by different models, a problem that can some- times be worsened by longer context, particularly in LM judging, Appendix Table 5 demonstrates that length effects on scoring consistency are probably negligible- answers from early, late, combined segments of the model output, the within-model scores remain consistent.
?While encountered
we attempt to extract

---

Published as a conference paper at COLM 2025
students attend meeting. There is muitiple of girls attending, and more girls than boys. How many boys are at the meeting?
Vanilla reasoning model
Thought Terminator
Okay, so I have this problem here about a math students. number girls is multiple of 13, and there are more girks...
think only need around tokens solve.
mosting
(many a multiple of G can be 13, 26, 39, etc., but since the total
(ater)
600 tokens remaining. I'll be back.
given
means
Airight, let's try figure out this problam. There are 35 students in totel the math club meeting. know that the number girls multiple 13, and there are more girls then boys...
number of students 35, number girls...
wart, also
condition there
are more girls than boys, which transiates to G B. Since G B- 35, can rewrite this C
Jinny (oter)
100 tokens remaining. I'll be back.
Then, the number boys would be 35 35-3- 26- 9. Let check this makes
Therefore, the number of boys at the meeting
Just double check, let me add them up: 26 girls 9 boys 35 students total. Yop, that adds up. Also, 26 indeed a multiple of 13, and is greater than 9, so all. the given conditions satistied. But wait, think...
"Final Answer number of boys at the mocting is 9. In summary, we have determinnd...
Time is up. Answer NOW
.i.gsss than boys, need determine the number of boys. Thus, the number of boys at the meeting is M(boxee(9]))
There are 9 boys at the meeting.
Correct in 645 tokens.
Correct 1358 tokens.
uses a reasoning model's (calibrated) estimate of the difficulty of a problem to set its intervention, periodically interrupting the reasoning model's output to remind it of the amount of remaining tokens. Once the token allotment has been used, it forces the model to provide an answer with constrained decoding.
Figure& 5: THOUGHTTERMING
Under our difficulty-calibratee token budget hypothesis, we assume that the number of required tokens can be estimated based on the difficulty of the question. In deployment, THOUGHTTER TERMINTTMNNNN ATOR is used in the tool-use paradigm, where running model makes its own estimate of the difficulty of an input question and then invokes it.
We experiment with both trained difficulty estimator and zero-shot one (gpt-4o) to produce token spend estimates for each problem to characterize performance this setting, To train difficulty estimator, we divide the training set questions into 10 balanced bins based on their difficulty scores. We then finetune Llama model perform difficulty estimation (training details provided in Appendix A.3.1).
To convert the predicted difficulty level into an appropriate number of answer tokens, we compute the averaged length of minimal successful answers for each difficulty level in the training set.
Running. Once the deadline has been set in scheduling, the base reasoning model's generation process runs. Every и min (250, deadline steps an interrupt message* is inserted into the token stream, notifying the model of how many tokens have been used and how many remain.
At each interrupt, THOUGHTTERMINATOR performs a regex check for the expected (and specified in the prompt) final answer format. If an answer is detected, the reasoning chain is immediately terminated and the answer is returned.
Terminating. If a final answer hasn't been produced by the deadline, termination message is shown to the model, and then final output is immediately generated with constrained decoding using the same answer-finding regex.
"Fxmmple interrupt message, termination message, and prompt provided in Appendix A.3.
7

---

Published as a conference paper at COLM 2025
MATH500
GPQA
1.0
1.0
0.0
0.8
Poss@1
0.6
Pass@10
0.6
0.4
0.4
Legend
0.2
DeepSeek-R1 Ri-----ii--- Qwen-1.5B
1i-DDiiil--Qwen-1..5
0.0
DeepSeek see-elllk------- R1-Distill Qwen-7B
DeepSeck R1 Distill-llmma
Tokens
Tokens
QwQ-32B
ZEBRA
DUMB500
QwQ-32B-Prees
1.0
1.0
*
0.8
0.8
Base
Pass@10
Pass@10
Base-32k
0.6
0.6
Terminator
0.4
0.4
1.B x вot 10F 10"
10a
Tokens
Tokens
Figure 6: Comparison of the relationship between Pass@10 and token spend for the evalu- ated reasoning models in the "Base" setting and with THOUGHTTERMINATH
Model
Base
Thought Terminator
Local Oenv |
Global Og |
Accuracy
Local Oenv |
Global Og +
Accuracy 1
QwQ-328-Previee
2923
3698
0.80
518 (-82%)
693 (-81%)
0.79 (-1%)
QwQ-328
13662
11248
0.94
215 (-98%)
1021 (-91%)
0.80 (-15%)
R1-1.58
5730
4262
0,50
696 (-88%)
882 (-79%)
0.80 (+59%)
R1-7B
3881
4001
0.73
678 (-83%)
948 (-76%)
0.81 (+11%)
R1-8B
4232
5755
0.92
725 (-83%)
1148 (-80%)
0.80 (-13%)
Table 2: Local envelope overthinking (Oenv) and global overthinking (Og) scores, along with accuracy for reasoning models under the Base setting and with Thought Terminator. Relative changes from Base to Thought Terminator are shown in parentheses.
Results & Discussion
Figure 6 shows the performance and token spend of five DeepSeek and QwQ reasoning models in the base setting (triangle marker) and with THOUGHTTERMINATON (star marker). Table shows the change in overthinking scores reasoning models exhibit from base setting to TROUGHTTERMINATONN
4/5 models MATH500, 2/3 models on GPQA, and all models Zebra and DUMB500- MATH see significant decrease in overthinking for effectively equivalent (or better) Pass@10 performance under thinking scores drop dramatically and accuracy increases when THOUGHTTERMINATE used. Considering that the token spend budgets are directly defined by LMs, THOUGHT- TERMINATOR R is a simple and effective tool to dramatically improve token efficiency in reasoning models.
BERLGHTTTRMMAATTT
than under standard decoding. Globally, over-
is
5.1 Calibration of TTHUGHHTERMINAATH
To evaluate how well-calibrated THOUGHT is (i.e., whether the token budget selections are optimal) we compare our difficulty prediction-based deadline estimator against set of baselines. In addition to our trained difficulty predictor and zero-shot gpt4o predictor, we use the previously observed optimal token spends from base models (section 2) and fixed deadlines of 500, 1000, and 2000 tokens with DeepSeek-r1-QQen-- 1.. to assess how performant our predicted deadlines are in the THOUGHTTERMINATOR framework.
Figure 7 shows the performance of the model under those deadline prediction strategies.
8

---

Published as a conference paper at COLM 2025
MATH500
GPQA
1.0
1.0
0.8
0.8
Method
0.6
0.6
base
0.4
0.4
base-32k
0.2
0.2
real-min
0.0
0.0
fix-200
10
10a
fix-500
Tokens
Tokens
fix-1000
ZEBRA
DUMB500
fix-2000
pred-diff-gpt4p
pred-diff-ref
pred-diff-trained
0.6
0.6
0.4
0.4
0.2
0.2
Terminator
0.0
0.0
10
10a
10a
Takens
Tokens
Figure 7: Calibration ablation experiment using DeepSeek-R1-1.55 real-min represents using the previously observed minimum successful answer length (or, a fallback maximum for examples that were never solved correctly) as the THOUGHTTERMINANT fix-(200, 500, 1000, 2000] sigmify using the respective number as a fixed token count dead- ref, trained) refer to using question-level difficulty
deadline.
line for all samples. wred-diff-gppte predictions as deadlines, produced from external LMs, a question-level reference difficulty key of token lengths from the other models, or trained RMs.
Our method, pred-di ff-trained (i.e., using the trained difficulty estimator), achieves op- timal Pass@10 over the other methods on MATH500 and DUMB500, and is within 0.02% of optimal Pass@10 on ZebraLogic and GPQA, for significant savings in compute cost. Note how all four datasets exhibit positive correlation between average token spend and Pass@10 which eventually reaches steady maximum. Under our definition, overthinking mitigation can be thought of identifying the lowest token spend that recovers high-spend performance. Figure 7 confirms that THOUGHTTERMINATON achieves this.
5.2 Utility of interrupt messages and constrained decoding in THOUGHTT TERMINATTRMINNOOR
To identify the performance benefit that the ThOUGHT interrupt messages provide, Appendix Table 3 shows the difference in performance of r1-1.5B in an unmodi- fied base condition, as well as under naïve baseline, and THOUGHTTERMIN question-level randomly assigned deadlines and the core trained-predicted deadlines. In this naive baseline the reasoning model immediately interrupted at the deadline, and without warning forced to generate an answer using the same constrained decoding technique.
with
presents roughly equivalent performance to the naive baseline the non-arithmetic GPQA and ZebraLogic datasets in Pass@10, and wins by 6% on MATH500 and 18% on DUMB500-math, This suggests that the intermediate interrupt do play a role in minimizing performance loss of decoding- based overthinking mitigation.
messages produced by THOUGHITERMINT
In order to further isolate the impact of the specific interrupt messages, and the performance gain that the constrained decoding-based based answer forcing provides, we compare r1-1.5B and AwQ-32B for MATH500 and GPQA using the following modified baselines:
Base Unmodified base condition, but with the prompt "Do not overthink."
Base Base condition prompt " "Answer concisely, avoid unnecessary elaboration."
Chunk THOUGHTTERMINATTO condition, without constrained decoding, with interrupts which simply say "You have used tokens." and no reference to time remaining.

---

Published as a conference paper at COLM 2025
Int THOUGHTTERMINATON condition with the interrupt message "I have used tokens, and I have tokens left to answer." "
Int THOUGHTTERMINATOR condition with the interrupt message "xX time elapsed, y% remaining"."
The results of these ablations are shown in appendix Table 4. They show roughly equivalent performance in token spend at Pass@10 for the models regardless of exactly how the inter- Inf 1, and Int 2), while the conditions
rupt message is phrased where forced stopping never ta kes place (Base Base 2, Chunk) are considerably worse both in performance and token spend.
5.3 There is more to token spend than overthinking
Token length a complicated signal, which is as affected by both a model's base reasoning efficiency (i.e., in problem-solving) as its overthinking for that specific question. A weaker model might not be able to leverage the same token saving strategies of a larger model; this is different phenomenon than overthinking.
By evaluating both local and global overthinking scores, we can effectively control for this phenomenon. As local overthinking only compares model's token spend to its confirmed minimuun, isolates "real overthinking" as a cause, while global overthinking would capture both "model weakness" well as real overthinking.
In Table we report both local and global overthinking for each model, and find that the two methods have PCC of 0.97, suggesting that the impact of relative model strength on global overthinking scores is low. This is fortunate, as it means that overthinking can be assessed on new reasoning model using another model's token spend.
Related Work
Mitigating overthinking. To shorten LLM reasoning chains, Deng et al. (2024) and Liu et al. (2024) propose to internalize intermediate steps by iteratively training the models, introducing additional training overhead. Dynasor technique for terminating chains of thought using the LM's confidence using a probe with constrained decoding (Fu et al., 2024). While our termination process can use similar constrained decoding technique, THOUGHTT THOUGHTTERMINATER TERMINATOR is not reliant a white-box probe, and is much simpler to run. Chen et al. (2024) introduce metrics for overthinking and process efficiency, similar to us, but they focus on important heuristics such as "number of repetitions of the correct answer" or "ratio correct to incorrect answer proposals", while our analysis solely quanitifies overthinking based on the observed distribution of reasoning chain lengths.
Benchmarking reasoning models. A number of benchmarks have been proposed to evaluate the reasoning ability of large language models (LLMs), with focus on challenging, multi-step problem-solving 2021; Zhu al., 2023; Lin al., 2024). Several recent works on efficiency benchmarking of LMs have been proposed, including Mercury, an efficiency evaluation for code synthesis tasks (Du et al., 2024). GSM8k-Zero is an another dataset to evaluate efficiency of reasoning, which contains easy questions from GSM8K (Chiang & Lee, 2024).
solvs-. (Cobbe et al., 2021; Srivastava et al., 2022; Hendrycks et al.,
7 Conclusions
In this work we analyzed the problem of overthinking in reasoning models through an observational lens. Motivated by our observational measures of overthinking, we demon- strated a clear sample-wise relationship between token spend and question-level difficulty. We introduced the DUMB500 dataset to allow us evaluate the robustness of any over- thinking mitigation to simple questions and proposed THOUGHTTERMIIS ATOR, simple inference-tin time technique to ensuring efficient token spend, calibrated by the aforementioned lifficulty-optmma spend relationship.
10

---

Published as a conference paper at COLM 2025
References
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like llms. ArXiv, abs/2412.21187, 2024, URL httttt:://ppli www.steeeechottteeeeee www.eechcoa.rrooooorroeeeeeeeeee orrp/CrrpsSD:225133665
Cheng-Han Chiang and Hung-yi Lee. Over-reasoning and redundant calculation of large language models. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), Pp. 161-169, St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL https: htaps::/aalaateee lontoooloog org/2024. eacl- 15/.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bing-Li Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dong-L. Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Jiong Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, M. Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shao-Kang Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng. Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wen-Xia Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyu Jin, Xi-Cheng Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yi Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yu-Jing Zou, Yujia He, Yunfan Xiong, Yu-Wei Luo, Yu mei You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yao Li, Yi Zheng, Yuchen Zhu, Yunxiang Ma, Ying Tang, Yukun Zha, Yuting Yan, Zehui Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhen guo Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zi-An Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. ArXiv, URL httpt:tt: //api. semanticscholar wwaechhlaarrrg////e wrg. Cong//orppsII::::: 275789995.
2025.
Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arviv preprint ar XwXi::24055114833 2024.
Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. Mercury: A code efficiency benchmark for code large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinay Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, An- thony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru,
11

---

Published as a conference paper at COLM 2025
Bap tiste Roziere, Bethany Biron, Binh Tang, Bobbie Chemn, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Cantón Ferrer, Cyrus Nikolaidis, Damien Al- lonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriele Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guanglong Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Ju-Qing Jia, Kalyan Vasuden Alwala, K. Upasani, Kate Plawiak, Keqian Li, Ken-591 neth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuen ley Chiu, Kunal Bhalla, Lauren Ventala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenk- ins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri S. Chatterji, Olivier Duchenne, Onur cCelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasié, Peter Weng, Pra- jjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Ro main Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Chandra Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whit ney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yiqian Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zhengxu Yan, Zhengxing Chen, Zoe Papakipos, Aa- ditya K. Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adi Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Ben Leonhardi, Po-Yao (Bernie) Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsian Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Shang-Wen Li, Danny Wyatt, David Adkins, David Xu, Davide lestuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzm'an, 'an, Frank]. J. J.Knaytt, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang, Guna Lakshminarayanan Hamid Shojanazeri, Han Zou, Hannah Wang, Han Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold- man, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, (ean-Baptise Gaya, Jeff Marcus, Jeff Tang, Jennifer
12

---

Published as a conference paper at COLM 2025
Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kaixing(Kai) Wu, U KamHou, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, A Lavender, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Raste- gari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollár, Polina Zvyagina, Prashant Ratanchandani, Pritish Yu- vraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Kam, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govin- daprasad, Sumit Gupta, Sung- BBe Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xia Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama herd of models. ArXiv, abs/2407 21882 2024. URL https://apt
cemanticscholarr org/ wtrg/CorpsSI:::22 2711514444
Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. Efficiently serving m1m reasoning programs with certaindex. arXiv preprint ar arXiv:2412.20993, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed KI-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai of system card. Xiv preprint ar arXiv:2412.16620, 2024. 2024.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. ArXiv, abs/2305.19118, 2023. URL https://api. semanticscholar. Bww.laa..rr///orrrrre Orgg/CorpsID::: (CCorpusID:2589775540 258967540.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. ArXiv, abs/2305.20050, 2023. URL https::/aap http:/ppp...mmmniccccooooaaaaa.. www.://rrr.rr/oooouutoo..o :
Bill Yuchen Lin, Ronan Le Bras, Peter Clark, and Yejin Choi. Zebralogic: Benchmarking the logical reasoning ability of language models, 2024. URL https:/ httpss:///gggggacce....m co/ apaces//llenai// /Zoico.
13

---

Published as a conference paper at COLM 2025
Tengxiao Liu, Qipeng Guo, Xiangkun Hu, Cheng Jiayang, Yue Zhang, Xipeng Qiu, and Zheng Zhang. Can language models learn to skip steps? arXiv preprint ar arXi:22411.01855, 2024.
Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, L. Sifre, Morgane Rivière, Mihir Kale, J Christopher Love, Pouya Dehghani Tafti, L'eonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Am's elie H'eliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette- Choo, Clé ment Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian
Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Kather ine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel I. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vladimir Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeffrey Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology. ArXiv, abs/ 2403.08295 2024. URL https: / /api. www.stiisccoooiieeeoe org. org/Corppstin 268379206.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei Li, Hanna Hajishirzi, Luke S. Zettlemoyer, Percy Liang, Emmanuel J. Candes, and Tatsunori Hashimoto. sl: Sim- ple test-time scaling. ArXiv, Abs/2500.115 2025. URL https: //api. semanti cemanticscholar. org/CorpusID: 276879693
Liangming Pan, Michael Stephen Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics, 12:484-506, 2024. URL https://app httpss//pp..smaantissssssssees semanticscholar .rg/ CorpusID:26963651..
Qwen. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm. github. io/blog/qwq /wgb3
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google- proof q&a benchmark. Arxiv, abs 2bb1/22112222. 2023. URL https: / /api. semanticscholar. org/CorpusID: /Bg/CoppsID:226255000 265295009.
Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of- of-to-of-thought helps mainly on math and symbolic reasoning. ArXiv, abs/2409.12183, 2024. URL https: //api. semanti cecelaarrrrg//ee www. orgg/CorpusID:227778333
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazary, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. lyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmuller, Andrew M. Dai, Andrew La, Andrew Kyle Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla
14

---

Published as a conference paper at COLM 2025
Karakacs, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ek- mekci, Bill Yuchen Lin, Blake Stephen Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, C' C'esar Ferri Ram Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison- Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Daniel H Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Mosegu'i Gonz' Danielle R. Perszyk, Danny Heman- dez, Dangi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan J. Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Mart iaaz'innzzPlu Francesca Happ'e, François Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch- L'opez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schutze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, John Kernion, Jacob Hilton, Jachoon Lee, Jaime Fernández Fisac, James B. James Koppel, James Zheng, James Zou, Jan Koco'n n, Jana Thompson, Janelle Wing- Jared Kaplan, Jarema Radom, Jascha Narain Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Oluwadara Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Jane W Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jorg Frohberg, Jos Rozen, José Hernández-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochande Louis- Philippe Morency, Luca Moschella, Luca Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Col'on, Luke Metz, Lutfi Kerem cSenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ram firez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Tvanitskiy, Michael Starritt, Michael Strube, Michal Swkedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Monica Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, T. MakundVarma, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, P'uul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, P. Milkowski, Piyush S. Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel Rahel Habacker, Ramon Risco, Raphael Milliere, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepi-
15

---

Published as a conference paper at COLM 2025
deh Sadeghi, Shadi S. Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Bradley Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Dem- berg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yu Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ArXiv, 0bs/2206.04615, 2022. URL https::t///// /appi. semanticscholar org. wttg/CorpuSD..... :eeuuII:::6288811 263625818.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, ttbs/2201.1190) 2022. BtLhstps::/
semanticscholas org/CorpusID:
246411621.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. 2024. URL https: ptttps://apll. www.attiicccolar.rrgg/ooo//
27 271601023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. Xiv preprint ar Xxx::2241..5511,, 2024.
Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei. Towards thinking-optimal scaling of test-time compute for Ilm reasoning. ArXio, abs/2500.180m 2025. URL https://api. semanticscholar org
Zishun Yu, Tengyu Xu, Di Jin, Karthik Abinav Sankararaman, Yun He, Wenxuan Zhou, Zhouhao Zeng, Eryk Helenowski, Chen Zhu, Si-Yuan Wang, Hao Ma, and Han Fang. Think smarter not harder: Adaptive reasoning with inference aware optimization. ArXiv, abs/2501.17974, 2025.
URL https://///pp http://:///pppssssniieiisseeeeee semanticscholar wrwp/oorpssI.:275594444
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. arXio preprint ar 1iX112339..1716, 2023.
16

---

Published as a conference paper at COLM 2025
A
Appendix
A.1
Additional DUMB500 dataset details
The dataset is categorized into four subsets, each containing multiple fine-grained categories:
Mathematics (Math)
Arithmetic: Addition, Subtraction, Multiplication, Division
Comparison: Greater/Less than relationships
Fractions & Percentages: Simple fraction and percentage comparisons
Exponents Roots: Squaring and square roots
Unit Conversion: Basic metric conversions
Patterns & Sequences: Identifying missing numbers in sequences
Geometry: Recognizing shapes, angles, and basic geometric properties
Logical Reasoning: Basic problem-solving using logic
Conversational Interaction (Chats)
Self-reflective: Questions involving introspection and emotional states
Acknowledgment: Checking system responsiveness (e.g., "Can you see this?")
Greetings & Casual Chat: Common greetings and informal small talk
Commonsense Reasoning: Fundamental knowledge about the physical world (e.g., "Is water wet?")
Object Interaction: Simple cause-effect relationships (e.g, "If drop my phone, will it fall?")
General Knowledge: Basic factual questions (e.g. "What is the capital of China?")
Programming & Computing (Code)
Basic Output: Printing text and numbers
Variables & Data Types: Assigning and manipulating variables (numbers, strings)
Mathematical Operations: Performing basic calculations in code
User Input Handling: Handling user input in simple programs
Conditional Statements: Basic if-else logic and checking conditions
Loops & Iteration: Simple loops for repeated tasks
Data Structures: Lists, dictionaries, sets, tuples
Randomization: Generating random numbers and selections
Task Execution (Tasks)
Communication & Writing: Emails, Messages, Creative Writing, Social Media, Daily-life tasks
Language Text Processing: Paraphrasing, Translation, Sentence Transformations,
Grammar Checking
Analogy & Concept Matching: Identifying similar concepts and words
A.2 DUMB500 Evaluation Rubrics
Each section contains the requirements that are checked by the LM judge to score TASK and CHAT answers in DUMB500. The score for given answer is the rate of "yes".
17

---

Published as a conference paper at COLM 2025
A.2.1 General Requirements
Information must be correct and complete: "Does the response include all essential information requested?"
Accuracy:
Avoid unnecessary elaboration: "Does the response avoid unnecessary explanations and get straight to the point?"
Conciseness:
A.2.2 Task Rubrics
Emails
Formality Appropriateness: Level of formality must match context: "Is the level of formality appropriate for the context?"
Example Question-Specific For "Write a short email to Alice confirming meeting 3pm":
- "Is the email addressed Alice?"
"Does the email mention meeling at 3PM?"
Messages
Tone Appropriateness: Must suit messaging context: "Is the tone suitable for the messaging context?"
Format: Must be formatted as a text message: "Is the response formatted as text message?"
Paraphrasing
Must match requested style/tone: "Does the paraphrase match the requested style/tone?"
Style Appropriateness:
Example Question-Specific For "Make formal invitation casual":
"Does the message instruct to RSVP by Thursday?"
"Is the email addressed to colleagues?"
Translation
Accuracy:
Must provide correct translation: "Is the translation correct?"
Example Question-Specific: For "Translate to French":
"Does the sentence closely resemble: J'aime lire des livres pendant mon temps libre?"
Words
Relevance: Words must fit request context: "Are the provided words relevant to the request?"
Contextual Appropriateness: Words must suit intended use: "Are the words appro- priate for the context?"
Creative-Writing
Contextual Appropriateness: Must match specific context: "Does the response match the specific context of the creative writing lask?"
Length Requirements: Must follow specified length: "Does the response follow the length requirement there's one?"
18

---

Published as a conference paper at COLM 2025
Social-Media
Platform Appropriateness: Must match platform conventions: "Does the content match the conventions of the specified platform?"
Example Question-Specific For "LinkedIn new job post":
-"Does the post mention the job title and company?"
Work
Formality Appropriateness: Must match workplace context: "Is the response contains correct format as required?"
Example Question-Speciffc: For "Slack message to manager":
-"Does the message respectfully address the manager?"
"Does the message omit names?"
A.2.3 Chat Rubrics
Self-reflective
Friendliness: Must show politeness: "Does the response show friendliness and polite- ness?"
Acknowledgment
Conciseness:
Avoid overthinking simple queries: "Does the response avoid overthink- ing the intent behind simple queries?"
Greetings
Contextual Appropriateness: Must sound natural: "Does the greeting sound natural and human-like?"
Daily-Chats
Contextual Appropriateness: Must suit casual conversation: "Is the response appro- priate for casual conversation??
Commonsense
Avoid overthinking obvious answers: "Does the response avoid over- thinking obvious answers?"
Conciseness:
Knowledge
Share knowledge without excessive detail: "Is the knowledge shared without excessive detail?"
Conciseness:
A.2.4 LM judge meta-ecaluation
Following reviewer concern that GPT-4o-mini may not sufficiently performant for use as judge this task, we rerun the evaluation for all CHAT and TASK questions for all Deepseek-rl and QwQ-32B variants. We find an inter-judge agreement of 0,9912 for Deepseek-r1 outputs and 0.9115 for QwQ-32B outputs.
19

---

Published as a conference paper at COLM 2025
A.3 Additional THOUGHTTERMINATOR details
A.3.1 Difficulty estimator details
To produce our questionwise difficulty estimator we train a 1aama-3-88-Instruee model with 1,465 examples across diverse task types to predict the 10-point difficulty level given question. Each training sample consists of question and discrete difficulty label derived from the minimum correct generation length (discretized into bins). The trained predictor achieves mean absolute error (MAE) of 1.57 a held-out test set, indicating that predicted difficulty levels are, on average, within 1-2 bins of the reference.
A.3.2 THOUGHT component prompts
Scheduling prompt:
Please generate answer to the following question in (deadline) tokens: {prompt). Messages of remaining time will be given as messages enclosed in <System></System, tags. Please provide you answer **Answer: **Final Answer: when complete.
Interrupt prompt:
I have used {elapsed) tokens, and I have {remaining) tokens left to answer. To continue:
Terminator prompt:
I'm out of time, I need to provide my final answer now, considering what have computed far. **Final Answer:**
A.4 Supplemental Results
Setting
Acc.
Pass@5
Pass@10 10
Tokens
MATH500
Base
0.47
0.78
0.81
3015
Naïve
0.52
0.78
0.82
1938
THOUGHTTERMINATOR
0.48
0.81
0.87
1590
Zebra-logic
Base
0.03
0.095
0.135
3861
Naïve
0.22
0.575
0.755
1254
THOUGHTTERMINATOR
0.19
0.585
0.75
1368
GPQA
Base
0.15
0.4096
0.5783
2815
Naïve
0.20
0.5783
0.7470
922
THOUGHITERMINATOR
0.21
0.5542
0.7470
1279
DUMB500
Base
0.58
0.9646
0.9735
3570
Naïve
0.37
0.7385
0.8154
377
THOUGHTTERMINATOR
0.67
0.9610
0.9610
447
Table 3: Comparison of performance and token spend of R1-1 .B under the Base Setting, with Naïve, and with THOUGHTTERMINATTON
20

---

Published as a conference paper at COLM 2025
MATH500
GPQA
Setting
Pass@10
Tokens
Pass@10
Tokens
Deepseek- r1-1.58 5B
Base
0.83
2176
0.57
3038
Base 2
0.82
3309
0.58
2919
Chunk
0.79
2287
0.16
3637
Int 1
0,87
1589
0.75
1279
Int 2
0.81
1586
0.76
1264
TERMINATOR
0.87
1590
0.75
1279
QwQ-322 32B
Base Base 2 Chunk
0.92
3262
0.64
4599
0.94
3364
0.61
4559
0.67
2779
0.34
4672
TERMINATOR 0.80
767
0.76
1498
Table 4: Comparison the four interrupt message and prompt ablations and THOUGHT- TERMINATOR of R1-1 and QwQ-32B. The exact formatting of the interrupt message (Int 1, Int THOUGHTT'RMINAN messages, and applying constrained decoding, does.
has limited impact on performance, but including the
Model
Head only
Tail only
Head & Tail
Tokens
Non-reasoning language models
Mwen2-7B-Instruct
0.77
0.73
0.76
923
2-
0.53
0.53
0.53
955
Llama-3 3.---33.-----ttteeee
0.54
0.54
0.55
2069
1-8B-Instruct
0.48
0.41
0,49
9402
gemma-2-2b-it
0.90
0.90
0.90
73
gemma- gemma-2-9b-it 2-9b-it
0.93
0.93
0.93
64
gemma-2-22 2-27b-it
0.76
0.76
0.76
96
0.61
0.60
0.61
314
Reasoning language models
MM0-3BB-Previee
0.72
0.66
0.71
1774
QwQ-32B
0.70
0.49
0.67
6712
-eepSsek-R---isstll--wwwerrr
0.59
0.58
0.58
3570
Beepseek-Ri-Distill- -1wen11-1--00e------------ 0.68
0.66
0.67
2042
DeepSee--R--istill--
0.80
0.80
0.80
2053
Table 5: Accuracy and token usage across different models under different input truncation settings.
21

---

Published as a conference paper at COLM 2025
Pearson Carralation Heatmap (2:85S55NATH5022)
Ovenr 320
3:2
0:42
3.99
0.29
0411
9
9.19
Oevo 525
0:5
1.00
322
0:47
335
022
3.43
020
0.0
0.00
0.79
0.6
0.79
meto Aarvatia 3.2 1-10 Irsst
0.40
9:27
022
392
032
292
042
0443
044
9:39
029
9.29
029
934
D004
mete Reridliaee
3.22
012
5.9
3722
048
0.39
0.44
0.7
MAA
0:24
10.00
281
187
110
161
010
1347
0.11
0.:01
0.11
p.m
0.13
pooplegermmn moe Hilb
4:45
6466
144
0.73
372
D44
LO3
010
GAT
038
0.5
0.2
002
0.11
despoesive Owe Stak Deven 1- 1.50
8:53
0.66
3:35
0.59
344
045
3.5
841
1:3
675
8.74
622
0.33
8.24
6.00
3.24
0:57
344
6:18
3.5
8:58
825
875
1/D
3.87
04
deepsawe and Sealo Cre 20
8.72
0.76
3.24
0.56
3.59
643
874
0.79
1:00
0m0
0.07
Deopp
suarage
6.75
3.51
0.02
322
6.10
044
8:2
0.02
3.12
0.01
22 Yea 1o0 20 Coceooae dee Saazee
Figure Pearson correlation of accuracies across different models on the MATH500 dataset
Poa Correlation Heatmap ((saviineen/pgg))
9:1
1.00
0:25
311
DES
303
632
0:05
B27
0.19
935
011
0.55
mede Aaraatiae 5.2 froi
3:55
5:08
1:2
6:11
6:31
4/21
A14
00.2
4040
0.01
DATI
0.5
00
poople permin Jublit
0:19
41124
3.84
LLD
4.49
Ra
MAC
0:11
0.49
pooplegeemm 36b
0.4
8:35
0.00
6.01
0.16
353
624
100
640
005
8:24
0.22
0.44
041
6.70
despsenie a DD Seck IC: 1.59
0:3
6:32
6:25
325
6334
3.34
8:33
1:33
0:5
8.73
0.42
0.57
dieeps wok an Cwwpeeart 43 Dest lamee-
A20
1:4
035
4:37
6:01
325
6/16
342
648
000
8:1
1,0
3.75
675
4.44
despaent an Seado Cred 1-0 Dowen- 78
1:53
0133
6:01
0.06
3.2
0:4
0.44
8:3
0.75
1.93
0:71
0.75
evarage
0.20
383
0.2
382
042
3.70
8:12
0.74
3.72
0.70
Figure 9: Pearson correlation of accuracies across different models on the GPQA dataset
22

---

Published as a conference paper at COLM 2025
Pearson Correaation Heattrap bleeaaiZeeeaaakkeerch
OONOOOOO Previcn
5.14
1:0
3:24
0/00
339
841
932
0441
0:12
0.99
0.1
0.11
mede nallie hos
0:14
1:1
0:15
112
DvD
0:17
13
0.14
5.13
313
proplegeemma 2-0b
3:23
655
352
110
4442
0.56
0.51
0.99
poople 24lb-
055
2:03
0.5
313
659
353
643
100
646
0.3
8:4
0.20
4.5
0.66
6.1
354
051
despuest Desssedt Dvcce L.2
0.20
3.24
0.3
330
640
0.24
8.28
0.00
0.80
dioposel dl www.epeeet au oleeiii cibmo
MANU
6:44
4.44
0:44
4.44
Bad
4.44
0.4
614
LAM
0.44
0.94
dispacet Deoug
313
0.72
172
0.66
0.54
1.00
0.00
0.03
svarage
0.64
3.44
3.7
800
5.19
0.00
Figure 10: Pearson correlation of accuracies across different models on the Zebra dataset
23


Reviews:
[{"review_id": "jd1h22e61T", "reviewer": "Reviewer_HZWD", "summary": "This paper proposed DUMB500, a dataset of extremely easy tasks to evaluate the calibration of reasoning models, and found that these models often overgenerate tokens, especially on easy problems. It also proposed ThoughtTerminator, a training-free decoding method that improves token efficiency by better calibrating reasoning depth to problem difficulty.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Good motivation to alleviate overthinking in reasoning model without retraining\n2. The solution is very intuitive and straightforward.", "reasons_to_reject": "1. The difficulty of a question is based on the (in)accuracy of model-generated answer, but the question difficulty is not the only factor that affects the answer accuracy, it can also relate to the question/answer memorization. \n2. The task and chat evaluation in DUMB500 is evaluated by gpt-4o-mini, which is not a widely used model for LLM-as-a-judge. Would be nice to present the agreement between gpt-4o-mini and more widely used judge LLM such as GPT-4o to validate the evaluation effectiveness.\n3. In figure 6, the performance of QwQ-32B + Terminator is significantly worse than other methods. Would be nice to have an analysis for that.\n4. Would be nice to present more details on the prediction-based deadline estimator, including zero-shot gpt-4o-mini, and trained instruct-llama-8b model. Would be interesting to see how many data points can make the prediction lead to good performance.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zPXYFWYfxS", "reviewer": "Reviewer_1isg", "summary": "First, thanks to the authors for their work. This paper attempts to first quantity, then mitigate the overthinking problem of current LLMs. The overthinking problem is defined as generating unnecessary tokens during inference which don’t improve the accuracy of a question. The authors take the following actions to tackle this problem:\n- They introduce a new way to estimate the difficulty of a task, by sampling n answers for a given question and computing the rate of inaccurate answers.\n- They quantify overthinking by using the difficulty of a question and the observed token spend per question.\n--Global overthinking score for a given model - the difference between the mean spend tokens for a question by the given model and the absolute minimum observed spend across different models for the same question.\n--Local envelope overthinking score for a given model - the mean difference between the maximum observed spend for a question and the minimum observed spend for the same question by the same model.\n-Introduced a new dataset called DUMB500, consisting of simple questions in four domains: mathematics, conversational interaction, programming & computing, task execution, aimed to measure the overthinking of the models on dumb/easy questions.\n-Introduced the THOUGHTTERMINATOR, a simple text-augmentation method aimed to reduce the general overthinking of models.\n--Doesn’t require re-training of the model.\n--Consists of three stages:\n---Scheduling - estimates the number of tokens to solve a given question.\n---Running - sends/inserts interrupt messages to notify the model of how many tokens have been remaining.\n---Terminating - if a final answer hasn’t been produced when all the tokens have been spent, sends a termination message asking the model to produce the final answer.\n-The results of THOUGHTTERMINATOR are presented on 4 datasets (including DUMB500), using 5 different models, showing improvements.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The novelty of estimating the difficulty of a task.\n- The novelty of quantifying a model's overthinking.\n- The introduction of the DUMB500 dataset.\n- The introduction of THOUGHTTERMINATOR, and the showed improvements of using it.", "reasons_to_reject": "- More ablations for the THOUGHTTERMINATOR are needed. The only provided (and as far as the public is aware, tried) prompt is the one presented in the paper. Since we have seen in the past evidence that the models can’t reliably count letters, count words, compute length of words, etc, maybe the format that is chosen is not best suited for the models. We can’t assume that the model well understands what “x elapsed tokens, y remaining” means. For example, a different ways of how to phare urgency to a model is:\n--“x% time elapsed, y% remaining” - maybe the models have seen more training data in this format\n- From the results, figure 6 looks like when using THOUGHTTERMINATOR the overthinking is reduced, but in some cases the performance is hurt. For example, in MATH500, the QwQ-32B (red dot), has pass@10 of ~0.9 without THOUGHTTERMINATOR, and with pass@10 of ~0.55. This suggests to me that a more thorough analysis of the failure cases when using THOUGHTTERMINATOR needs to be done. I would have appreciated a more systematic and thorough analysis of the failures, to make sure that the trade-offs of using THOUGHTTERMINATOR are well understood and outlined.", "questions_to_authors": "- As a small suggestion, can you improve the readability of section 2.1, where the overthinking measures are introduced? It took me a few reads back and forth between the formulas and text to understand the metrics.\n- In figure 7, for the ZEBRA dataset, the base seems to use the most tokens but produce the worst (by a huge margin) result. This seems like an outlier, and it  wasn't addressed in the text. Can you please address it?\n- Will the DUMB500 dataset be available publicly?", "extra_scores": {}}, {"review_id": "P60jcfRqcA", "reviewer": "Reviewer_aNwS", "summary": "The paper focuses on the issue of overthinking of reasoning models. This work presents a new approximate measures of problem-level difficulty and demonstrates the relationship of difficulty and token spends. The finding reveals the calibration issue of reasoning models and its “overthinking” over easy problems. DUMB500 is a dataset introduced. Finally a decoding trick called ThoughTerminator was introduced.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overthinking and the calibration of reasoning models is a timely research problem and this paper is well-fitted in this gap. \n\nThe paper proposes ToughtTerminator, a prompting + tool use approach to address the overthinking issue. It’s a simple enough and effective baseline and I believe it will be widely adopted. \n\nThe introduction of DUMB500 could be used to evaluate this issue. \n\nThere are many interesting findings for a range of frontier reasoning models. These findings about token spend, accuracy and optimal usage could benefit and inspire the community.", "reasons_to_reject": "There are some baselines and settings not explored in this paper. For instance, adding a prompt like “think accordingly / think as you need / do not overthink” or adjust the sampling parameters. I think it would be essential to deploy some easy fix before claiming it’s a problem. \n\nThere are some key assumptions and I am not sure if they are rigorous. For example, the difficulty of a task is the accuracy of tested models. The overthinking score is the mean diff between the length of each reasoning chain and the global observed min spend for each question. These metrics depend on the choice of models. If there is one specific model never thinks, the score will drastically change. The definition of “overthinking score” should be more intrinsic rather than relative evaluation.", "questions_to_authors": "Can you explain the “A_hat ~ M !=a”? I don’t get the a_hat here. \n\nI get a little confused by the percentage in Table 2. 0.58 -> 0.50, should it be 8%, if it’s 1.00 scaled.", "extra_scores": {}}, {"review_id": "uFJrwFcjxL", "reviewer": "Reviewer_U1jN", "summary": "This work has three main contributions:\n+ It proposes measuring problem-level difficulty by the model’s output token length. Based on this, the paper introduces a method to quantify the severity of a model’s overthinking, using both the model output token length and the problem difficulty.\n+ It introduces DUMB500, a dataset of extremely easy problems. A model that overthinks is expected to answer problems in this dataset without generating too many output tokens.\n+ It proposes a decoding method, THOUGHTTERMINATOR, to improve token efficiency.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The research problem studied in this work is very important. Recently, it has been increasingly recognized that overthinking in strong reasoning LLMs is a serious issue.\n+ The DUMB500 dataset is interesting and seems to be a potentially useful testbed for future work on LLM overthinking.\n+ The proposed decoding method, THOUGHTTERMINATOR, appears to be effective. It also does not require any training, which could make it easily pluggable into different models.", "reasons_to_reject": "+ I am not fully convinced that simply measuring token length is a reliable way to quantify the severity of overthinking. Token length itself seems to be a complicated signal, coupling both the model’s reasoning capability (i.e., problem-solving ability) and overthinking. For example, a weaker model might be unable to come up with a smart solution for a difficult math problem and thus has to brute-force a solution, resulting in a much longer token output compared to a stronger model that leverages a clever trick to simplify computations. In this case, it would not make sense to attribute the longer token output to overthinking, as it simply reflects the weaker model’s limited problem-solving capability.\n+ The paper does not compare THOUGHTTERMINATOR with a simpler baseline:\n   - For instance, instead of estimating the required number of tokens externally, we could instruct the reasoning LLM itself (via system prompts) to try its best to save tokens.\n   - Additionally, when generating every CHUNK of tokens, we could append a reminder to the context such as \"You have consumed {CHUNK × TIMES} tokens\" to make the model aware of its token usage.\n   - In this way, we leverage the reasoning LLM’s own capability to estimate the token length needed, which might be more effective than relying on an external model (e.g., a LLaMA-based predictor or GPT-4o), which may not closely match the internal behavior of the target model.\n+ What are the main challenges in collecting DUMB5000? My assumption is that one could simply compile older LLM benchmarks that have become easy for today’s models. Please correct me if this assumption is incorrect.\n+ I feel that the writing could be improved:\n   - Overall, the paper’s structure could be more coherent. The three main contributions currently feel somewhat disconnected—would it be possible to connect them into a more unified narrative?\n   - Additionally, some figures (which I will discuss further in the Questions section) seem unfinished. For example, some legends are unclear or hard to interpret.", "questions_to_authors": "+ Since different models may use different tokenizers, how do you ensure fair comparisons in Table 1? Also, in Formula 3, token lengths from different models may not be directly comparable or simply averaged. Although this might be a minor issue, clarification would be helpful.\n+ In Figure 4, why do all the models perform poorly on Code, Task, and Chat? As you claimed, DUMB500 should consist of very easy problems. I had previously assumed that DUMB500 is designed so that models can easily solve the problems and the main challenge (for a model that tends to be overthinking) is minimizing token usage. However, the observed poor performance suggests that my assumption might be incorrect. Could you clarify this?\n+ What do all the baselines shown in the legend of Figure 7 represent? I could not find their definitions in the main text. Specifically, what do `real-min` and `pred-diff-ref` mean?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on over-thinking, the problem of reasoning models generating many more tokens than necessary.  The paper contributes both a manually created dataset of easy questions for measuring overthinking, as well as an inference-time technique for reducing overthinking.

Strengths:
- The problem studied is timely and impactful
- The manually created dataset is distinct from existing resources and can be useful for others working on over-thinking
- The interrupt based solution is effective - often not reducing accuracy and at the same time reducing the number of tokens by about 2x over multiple baseline models

Weaknesses:
- The original manuscript was missing some natural baselines and comparisons.  However during the author response period, the authors provided the baselines requested by the reviewers, and the results are all still in favor of the proposed technique.
Decision: Accept