Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?
Authors: Chenrui Fan
Ming Li
Lichao Sun
Tianyi Zhou
Venue: COLM 2025 (2025)
Keywords: LLM
Reasoning Model
Overthinking
Abstain
TL;DR: Reasoning models can’t think critically when premise is missing.

Abstract: We find that the response length of reasoning LLMs, whether trained by reinforcement learning or supervised learning, drastically increases for ill-posed questions with missing premises (MiP), ending up with redundant and ineffective thinking. 
    Such failures are against the ``test-time scaling law'' but have been widely observed on multiple datasets we curated with MiP, indicating the harm of cheap overthinking and a lack of critical thinking. 
    Surprisingly, LLMs not specifically trained for reasoning exhibit much better critical thinking ability, producing much shorter responses that quickly identify ill-posed queries and ask for the MiP. This implies a critical flaw of the current training recipe for reasoning LLMs, which does not encourage efficient thinking adequately, leading to the abuse of thinking patterns. 
    To further investigate the reasons behind such failures, we conduct fine-grained analyses of the reasoning length, overthinking patterns, and location of critical thinking on different types of LLMs. 
    Moreover, our extended ablation study reveals that the overthinking is contagious through the distillation of reasoning models' responses.
    These results improve the understanding of overthinking and shed novel insights into mitigating the problem. 
    Our code and data can be found in: https://github.com/tianyi-lab/MiP-Overthinking.

Published as a conference paper at COLM 2025
Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?
Chenrui Fan"*, Ming Li" Lichao Sun?, Tianyi Zhou University of Maryland; Lehigh University
{cfan24, minglii, tianyi (emyy.)eumd..e @emdd.duu edu
Abstract
We find that the response length reasoning LLMs, whether trained by reinforcement learning or supervised learning, drastically increases for ill-posed questions with missing premises (MiP), ending up with redundant and ineffective thinking. This newly introduced scenario exacerbates the general overthinking issue to a large extent, which we name as the MiP- Overthinking. Such failures are against the "test-time scaling law" but have been widely observed on multiple datasets we curated with MiP, indicating the harm of cheap overthinking and a lack of critical thinking. Surprisingly, LLMs not specifically trained for reasoning exhibit much better perfor- mance on the MiP scenario, producing much shorter responses that quickly identify ill- ill-posed queries. This implies critical flaw of the current training recipe for reasoning LLMs, which does not encourage efficient thinking adequately, leading to the abuse of thinking patterns. To further investigate the reasons behind such failures, we conduct fine-grained analyses of the reasoning length, overthinking patterns, and location of critical thinking on different types of LLMs. Moreover, our extended ablation study reveals that the overthinking is contagious through the distillation of reasoning models' responses. These results improve the understanding of overthinking and shed novel insights into mitigating the problem. Our code and data can be found in: https:// //gi thub. com/ com/tianyi
1 Introduction
Reasoning abilities in large language models (LLMs) have become a comerstone of advanced AI applications (Huang & Chang, 2023; Li et al., 2025b; Ahn et al., 2024; Wang et al., 2025), powering breakthroughs in mathematical reasoning (Xiong et al., 2025; Xia et al., 2025), code generation (Liu et al., 2024), and commonsense question answering (Wang Zhao, model/dataset 2023). These gains often stem from the scaling law of model/ dataset sizes (Kaplan et al., 2020) in both pre-training (Shao al., 2024) and post-training, which unlock emergent capabilities such as step-by-step reasoning and reflection skills witnessed on OpenAl's GPT-ol (OpenAI, 2024b) and the open-source DeepSeek-RI By leveraging supervised fine- tuning (SFT) on expert responses (Li et al., 2025b; Ye et al., 2025; Muennighoff et al., 2025) and/or or reinforcement learning (RI.) (DeepSeek-AT
al., 2025), these models are tailored to produce detailed multi-step reasoning paths, whose length increase usually associated with improved performance on complex tasks such as math reasoning and programming.
Despite the fascinating reasoning capabilities exhibited on recent models, there is growing concern about the efficiency and quality of the long reasoning process (Sui et al., 2025). Chen et al. (2025b) first raises the "overthinking" problem in reasoning LLMs, which is reflected by the excessively long reasoning paths generated for extremely simple queries. For example, even for questions like "What is the answer of 2 plus 3?", existing reasoning models might generate hundreds of response tokens.
In particular, the ill- ill-posed queries are unsolvable due to the lack of a necessary premise or condition. We call the reasoning failure for the ill-posed queries Overthinking under
*Equal Contribution.

---

Published as a conference paper at COLM 2025
Non-Reasoning Model VS Reasoning Model
Normal VS Mip Question
What is the value of a?
James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a woek?
Please provide the context you Abstain
Non- Red-onnggnnng Nodel
are working with.
Reasoning Model
<Thinking> He runs 540m week. Overthink
What is the value
682 Takens
James decides sprints 3 times week. How many total
Ho.... night wont... Altarratieely... Perhapa... Alsernatively... But, Roybe... since... theck egein... alternatvely.... Might... Tricky...
but,
He-runs- wwwe--0e----eeeee-eeeeeeeee-e-ee meters does he week?
Lat's see... Woit It... What it. Alternatively... Assuning 6 sprintt is A, then... It it is then... It it is C, then Noit... Check once egain... But... Perhops.... Altematively.... The user night... win Woit... bouble check... Read carefully... It seens... Alterotiively... Maybe the user sant... Hovever... Han Tricky... The common definstion.... Check runs neters week.
probl once agatn...
Reasoning Model
shysice.... altarnatvelly... In woth... sefence... In chretstry... stology.... Honever.... Moybe..: that's case... Revisit the question ogain... common asy... must give on onawer.... thes...
Reasoning Nodel
O
8030
11742 Tokens 95s
value is 2. MiP-Overthink
Oue--vverhinnk
Figure 1: Illustration of MiP-Overthinking. When queried by questions with missing premises, the response length of reasoning models increases excessively, and they cannot abstain from answering with MiP identified. The left shows a query with an undefined variable, while the right compares a well-defined GSM8K question with its MiP variant. Reasoning models' responses to Mil' questions are much longer than those for well-defined questions and those generated by non-reasoning models. The left corner of each response report the response length and thinking time by DeepSeek-RI.
Missing Premise (Mii-Overthinkinng) For example, the simplest Mip question is What is the value of a? shown on the left part of Figure 1. Without providing any other information regarding a, it is evidently unsolvable. However, DeepSeek-RY generates thousands of tokens and spends several minutes thinking about this question before outputting the final meaningless answer 1, In this paper, we find that trivial type of ill-posed queries will significantly exacerbate the overthinking of reasoning models, resulting in excessively redundant and meaningless thinking. In contrast, humans and even non-reasoning models are often immune to such scenarios and quickly end up by questioning the validity of the given query, indicating the critical thinking capability. This exposes a risk of the abuse of thinking patterns and a lack of critical thinking on the models trained for deep thinking. Ideally, a model with critical thinking skills is expected to identify the missing premise and quickly respond by either requesting clarification or gracefully indicating that it cannot proceed (Cole et al., 2023; Amayuelas et al., 2024).
MiP-Overthinking differs from the widely discussed overthinking issue (Cuadron et al., 2025), in which the query is usually well-defined, but a model applies much more reasoning than necessary for little benefit. MiP-Overthinking, by contrast, happens when the question itself is ill-posed and lacks sufficient information to be solved. For example, the right of Figure 1 presents a well-defined question from GSM8K and a MiP variant, where the latter triggers drastic increase of the generated tokens on recent reasoning models compared with the general overthinking. Overthinking can be presented by the length difference between models addressing the same well-defined questions, while MiP-Overthinking can be presented by the additional tokens generated due to MiP It further reveals the lack of critical thinking that questions the validity of ill-posed questions and quickly identifies MiP, thus abstaining from answering the questions. Moreover, we observe that reasoning models' ineffective and redundant thinking often cannot stop even after successful notice of MiP, violating the expectation of test-time scaling law. Hence, Mii--veethinking indicates potential drawbacks of current training recipes of reasoning models.
To systematically investigate this issue, we construct a suite of MiP questions designed to trigger the overthinking failures in a controlled way. These include synthetic questions generated by Rule-based Formula (queries where a formula reference is empty or nonsen- sical) and careful modifications of established datasets across diverse levels of difficulties, including SVAMP, GSMSK, and MATH500. On the modified datasets of MiP questions,
'The screenshot on how DeepSeek-R1 responds to this question is shown in the Appendix D.
2

---

Published as a conference paper at COLM 2025
empirically evaluate wide range of state-of-the-aa LLMs, from reasoning models to non-reasoning models and from open-sourced models to proprietary models, to ensure the generalizability of our findings. Our analysis is mainly based on three evaluation metrics, the length of generated responses, the accuracy on well-defined questions, and the abstain rate ill-posed questions with MiP.
Main Contributions: We present the first in-depth study of Overthinking under Missing Premise (MiP--Oeethinkkinn)) which reveals a critical shortcoming in existing reasoning models: Although they appear to follow coherent reasoning patterns, they lack genuine crit- ical thinking capabilities. To systematically analyze this issue, we curate four MiP datasets covering various difficulty levels and three ill-posed question generation strategies, i.e., Rule-Based Generation, Body-Question Swapping, and Essential- Dssentia-Premiis Removal. We then evalu- ate wide range of large language models including reasoning- based and non-reasoning ones. Our empirical results illuminate the differences how models handle well-defined vs. MiP questions, ultimately offering insights into the limitations of existing reasoning models.
Our key findings:
Missing premise in questions induces reasoning models to generate significantly longer (2x to 4x more tokens) responses than general overthinking on well-defined questions. The increased tokens fail to help identify MiP in the ill-posed questions, surprisingly contradicting the widely-discussed test-time scaling law.
2. In contrast, given MiP' questions, non-reasoning models generate consistently shorter responses and quickly identify MiP, demonstrating greater robustness to the absence of critical information.
3. Reasoning models respond differently to well-defined vs. Mil questions: they mostly follow stable chain-of-thoughts for the former, but often trapped in a self-doubt loop, repeatedly revisiting the question, and guessing the user intentions under MiP, resulting in an explosion of tokens.
4. Reasoning models often can notice the existence of MiP or identify at an early stage, but they hesitate to commit to this judgment and keep outputting ineffective thinking.
2 Missing Premise Definition and Construction
2.1 Definition of Missing Premise
Prior to introducing the construction our dataset and analyzing the behavior of reasoning models on problems with missing premises, we formally define the Missing Premise (MiP) problem to establish a rigorous foundation for our subsequent analysis.
Definition 1 be a set of premises. Define the function mapping premises and a question to the set of logically valid answers
(Missing Premise Problem). Let Q be question, and let P (P,,.., Pn)
F(P,Q) = {A P+ A, is an answer resolving 2}
(1)
where denotes logical entailment. Consider a proper subset p' = -P) {P.) for some P; E
P.
The tuple (P', 2) forms a missing premise problem if and only iff
[F(, =1 and |F((', 2) #
(2)
This indicates that the removed premise P; is essential for uniquely determining the logically valid answer to the question Q.
According to Definition 1, an ideal reasoning system should efficiently identify the absence of critical premise and terminate its inference process upon recognizing that the available information is insufficient to derive a unique solution to the given problem. However, our empirical analysis in Section 3.2 demonstrates that state-of-the-art reasoning models
3

---

Published as a conference paper at COLM 2025
Dataset
Example
Diff
Count
Pair
Method
MiP-
Rule-Based
What the value of In(a b)?
50
X
Formula
Generation
Paco had 26 salty cookies and 17 sweet cook- ies. He ate 14 sweet cookies and 9 salty cook- ies.
Body- Question
MiP-
300
X
SVAMP
ti How many pencils does she have?
Swapping
James decides to run 3 sprints 3 times week. He-runs 60 meters
Essential-
MiP-
meters does he eaches--ach--piiiiie How many total
582
Premise Removal
GSM8K
week?
There are 360 people in my school. 15 take calculus, physics, and chemistry, and 15 don't take any of them. 180 take calculus. Twice as many students take chemistry as take physics. 75 take both calculus and chemistry, and 75 take both physics and chemistry. Only 30ke----ttttttt physies and-ealu How many students take physics?
Essential-
MiP-MATH
*** 58
Premise
Removal
Table 1: Statistics and examples of our curated MiP datasets. For GSM8K and MATH, premise is removed from the original questions (crossed out) create MiP questions. Diff represents the (estimated) difficulty for models to identify MiP. Count denotes the number of questions in the subset. Pair indicates whether each MiP question is associated with well-defined question. Method indicates the method used to generate the MiP question.
consistently fail to exhibit this capability. Instead, these models engage in extensive, redun- dant reasoning chains that consume significant computational resources without ultimately identifying the missing premise.
2.2 Overview of Data Construction
To systematically investigate this MiP-Overthinking issue, we construct a suite of MiP questions controllable manner. Our MiP questions are sourced from math datasets across different difficulties. In addition, we also construct synthetic dataset consisting of formulas with unassigned variables. Our ill-posed question generation employs three distinct methods covering three difficulty levels and three strategies to create Mil questions: Rule-Based Generation, Body-Quee roay-Questiion Swapping, and Essential-Premise Removal.
Then, we further construct 4 MiP datasets utilizing the above methods: MiP-Formula, MiP-SVAMP, MiP-GSM8K, and MiP-MATH. For comprehensive implementation details and additional methodological considerations, refer readers to Appendix B.
Overthinking under Missing Premise
3.1 Evaluation Metrics
To systematically evaluate model responses under MiP, we conduct experiments with diverse set of reasoning and non-reasoning models. For each model, we analyze calculate the following metrics for the responses across different datasets: Response Length, Abstain Rate for MiP Question, and Accuracy for Well-defined Question. For datasets without reference answers (MiP-Formula and MiP-SVAMP), we only calculate the abstain rate for the questions. Response evaluation performed using GPT-40 as an automated evaluator. Detailed experimental procedures and evaluation protocols are provided in Appendix A.
3.2 Main Results
Figure compares average response length, accuracy well-defined questions, and the abstain rate on MiP questions across a range of state-of-the-art LLMs, revealing several significant patterns in model behavior.
4

---

Published as a conference paper at COLM 2025
Reasoning Models
Now- Reaseeiassggggg Mlddel
Average Response Length (tokens)
9.75
10.3%
2008
10.774
8.0%
100
Queation Rospasac Lengh
123
Well-Defined Quatiion Resporee Lenpth
Open Source Medels
Rote Mil Quaation (%6
Acouacy for Well Defined Question
aB Im
QwQ-328
$1.1-328
DeepSedt-R
DeepSeel-R R--Ditillll ai-mini
o1-mini
GPT-ol
Qwen-32B-IT
GPT-4
Gemini-1.5
B-1T Gemma-2- 9hl-3-medium
Qwen:12B
Figure 2: Response lengths, accuracy on well-defined questions, and abstain rate of
reasoning/ aeasoning/non-reasoninng non-reasoning models on MiP questions from our MiP-GSM8K dataset. (1) Existing reasoning models generate significantly longer responses for MiP questions than well-defined questions, while non-reasoning models generate responses of similar lengths for both types of questions, indicating MiP-Overthinking for reasoning models. (2) For both questions, reasoning models generate longer responses than non-reasoning models, indicating General Overthinking.
(3) Although the longer responses by reasoning models slightly improve the accuracy for well-defined questions, it does not enhance the abstain rate for MiP questions, indicating contradiction on the test-time scaling law.
Firstly, existing reasoning models (left side of the figure) display an explosive increase in response length when facing the Mil' questions, often producing 2 - 4x more tokens than general overthinking on well-defined questions. For example, QwQ-32B (Team, 2025) and DeepSeek-R1 (DeepSeek-At et al., 2025) exhibit a substantial increase from already long reasoning paths on well-defined questions (approximately 0,000 tokens for simple GSM8K questions) to highly lengthy outputs (more than 3, 000 tokens) under missing premise conditions. On the contrary, no similar issues exist for non-reasoning models (right side of the figure), which generate similar token counts for both types of well-defined and MiP questions. This phenomenon directly illustrates the MiP-Overthinking phenomenon as introduced in the paper.
Secondly, comparing the token lengths on well-defined questions between the reasoning and non-reasoning models, reasoning models tend to produce longer responses, even for simple questions, than non-reasoning models, underscoring the inefficient and verbose responses of existing reasoning models. For example, for the non-reasoning models, only takes approximately 200 tokens for them to generate the responses for well-defined questions, while taking 1, 1,000 tokens for DeepSeek-R1 and 1, 1,0 tokens for QWQ-32B to answer the exactly same questions. However, the explosive increase in extra tokens does not lead to corresponding large accuracy im improvements, shown in the green line, highlighting the issue of the General Overthinking.
Finally, the abstain rates (red line) on Mil questions reveal that although some reasoning models (e.g., GITT-ol) have promising capabilities in abstaining from the MiP questions, most of the other reasoning models are not able to abstain from the given MiP questions correctly despite the dramatically long reasoning paths. This phenomenon indicates that although most existing reasoning models have thinking and reasoning capabilities to some extent, they lack the critical thinking capabilities to "reject" ill-posed questions. By contrast, non-reasoning models, though they are not explicitly trained for reasoning, tend to strike a better balance, generating shorter answers that are more likely acknowledge MiP when the question is ill-posed. This phenomenon reveals a surprising contradiction on test-time scaling law.
5

---

Published as a conference paper at COLM 2025
Model
Type
MiP-Formula Length.
Abstain!
Length!
SWPMP
Abstain?
Type
Length)
MiP-GSMIK
Abstain"
Length.
MIP-MATH
Abstainf
Non non-Reasoning Models
Owen2.5-32B-Inster
MiP
288
14.0
128
98.3
MiP Well-defined
246
219
44,0
0.5
1114
525
15.4
1.9
GPT-4o
MIP
338
70.0
122
96.3
Well-defined
MiP
202
212
46.9
0.5
487
472
15.4
1.9
Gemíni 1.5
MiP
453
20.0
99.0
Well-defined
M:P
808
55
568
502
5.8
0.0
Gemma-2-27B-IT
MiP
204
85.2
92.0
Well-defined
Phi-3-medinm-126k
MiP
1465
48.0
125
98.7
Well-defined
216
1549
3.8
Qwen3-328 (Non-Reason)
MiP
496
90.0
184
Well-defiiee
MiP
279
237
32.6
1.0
1287
1571
17.3
0.0
Reasoning Models
GPT-o1
MiP
1123
78.0
581
99.0
Well-defined
MiP
348
838
53.7
0.3
4189
2502
30,8
0.0
GPT-oTimini
MiP
958
66.0
639
96.7
Well-defiieed
MiP
762
449
40.0
122
2193
1913
25.0
0.0
GPT-o3mini
MiP
Well-defined
MIP
1516
2377
3772
11.5
Distill Owen2.5-32B
MiP
22911
Well-defined
MiP
2302
24.6
9876
DeepSeek
MiP
4737
1996
84.3
Well-deffned
MiP
3246
1226
16.8
3200
7268
3.8
1.9
S1.1-32B
MiP
5284
18.0
3358
57.0
Well
MiP
1896
1600
0.2
9322
5037
15.4
0.0
QwQ:32B
MiP
2933
8487
Well defined
MiP
4780
1896
0.2
5037
0.0
Owen3-52B (Reason)
MiP
5293
34.0
1872
58.0
Well
MiP
3149
1723
22.4
0.0
9468
5555
5.7
0.0
Table 2:
Comparing response length and abstain rate across different MiP' datasets. Shorter lengths and higher abstain rates are preferred. For each column, the top-3 preferred values are colored in green, otherwise red. MiP--verthinking reflected by longer response with low abstain rate, commonly observed on most existing reasoning models across all datasets, indicating critical drawback of existing reasoning models.
1,0
Well Defined
Moreover, Table further presents the com- parisons length and abstain rate other MiP datasets we curated. The preferred re- sults are colored green (shorter responses and higher abstain rate for MiP questions), and the worse results are colored red, from which we can easily discover that reason- ing models are prone to generate long re- sponses while having low abstain rates across all datasets, indicating the consis- tent MiP Overthinking issue of existing rea- soning models. In addition, by compar- ing the behaviors of models different datasets, we can observe that for the rela- tively harder dataset (MiP-MATH), all mod- els generate relatively longer responses and obtain lower abstain rates, indicating that harder MiP questions require reasoning ca- pabilities.
70-
Step Indee
60
20
40
Step Indee:
Step Index
Figure 3: The step-level similarity heatmaps for s1.1 responses towards well-defined (left) and MiP (right) questions in MiP-GSM8K dataset. To avoid differences in matrix size, we only consider responses with more than 50 steps and visualize the average simialrity matrix across first 50 steps. The heatmap for MiP questions has a higher averaged similarity and lower standard variance, also shown in the heatmap, which indicates the considerable redundancy in its content when responding to MiP questions.
3.3 Thinking Patterns through Tokens
To gain deeper insight into the MiP- Overthinking issue, we compare the reasoning-related token distribution (Lin et al., 2023; Li et al., 2025c) on the MiP-GSM8K dataset. As shown in Table 3, we break down the average usages of several token patterns related to the thinking process, as well as the number of steps for each model to solve the given questions. Specifically, values of alternatively, wait, check, and but can be directly
6

---

Published as a conference paper at COLM 2025
Alternatively
Wait
Check Cnt.
But
Hypothesis Cnt.
Step
Models
Type
Cnt.
Cnt.
Cnt.
Cnt.
Non Reasoning Models
Qwen2.5-32B
MiP Well-defined
0.0
0.0
0.0
0.0
0.0
0.0
0.3
0.0
0,0
5.6
4.3
MiP
0.0
4.7
GPTT-o
Well-defined
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.2
0.0
0.0
6.2
Gemini 1.5
Well-defined
MiP
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.1
0.0
0.0
0.0
3.8
1.6
-2.2
Gemma-2-27B
Well-defined
MiP
0.0
0.0
0.0
0.0
0.11
0.0
0.1
0.0
0.0
0.0
5.7
5.2
-0.5
Reasoning Models
MiP
11.5
19.7
1.0
40.1
38.4
54.9
DS-Distill Qwen
Well-defined
0.1
11.4
0.4
19.3
0.2
0.8
0.8
39.3
0.4
38.0
12.7
42.2
MiP
16,9
14.4
3.8
49.4
44.7
54.2
DeepSeek R1
Well-defined
15.2
3.5
10.9
2.5
1.3
42.1
4.3
40.4
21.2
33.0
S1.1
Well-defined
MiP
42.0
4.0
38.0
21.9
6.0
15.9
3.0
5.5
2.5
13.1
87.2
74.1
84.8
7.8
79.9
29.0
50.9
QwQ
Well-defined
MiP
47.0
6.7
40.3
19.4
6.4
13.0
3.4
1.6
11.9
54.2
94.1
12.4
81.7
39.2
97.9
58.7
Table 3: Comparisons of reasoning-related token counts on MiP-GSMBK dataset. Hypothesis category includes several key words, including perhaps, maybe, and might. Step represents the step counts, spited by \n where negative values colored in green and positive in red. A denotes the difference between MiP and well-defined questions. When facing MiP questions, reasoning models encounter explosive growths on reasoning-related tokens and steps, indicating a severe abuse of thinking patterns, while non-reasoning models use fewer steps for MiP questions than well-defined ones.
counted from the model responses, including the thinking paths of reasoning models. Hy- pothesis category includes several key words, including perhaps, maybe, and might. Step represents the step counts, spited by \n)
Reasoning models exhibit much higher occurrence of tokens such as alternatively, wait, and check, compared with non-reasoning models, whose frequencies remain close to zero, indicating their advanced thinking capabilities. However, when moving from well-defined to MiP questions, reasoning models encounter explosive growths reasoning- tokens, indicating a large redundancy in thinking patterns. Moreover, when comparing the changes of steps, reasoning models exhibit large increase in step count for MiP questions, while non-reasoning models typically show fewer steps, suggesting they quickly conclude the question is unanswerable. With this gap, together with the consistently better abstain rates of the non-reasoning models, we conclude that the lengthy reasoning steps are mostly redundant and indicate self-doubt thinking patterns for reasoning models.
3.4 Step-level Similarities
To further assess how redundant the generated content becomes under MiP conditions, we examine the step-level similarity within the model's 's responses on our MiP-GSM8K dataset. Specifically, we divide each response into discrete steps, split by (n',,,, nI, and compute pairwise cosine similarity scores with embeddings generated by "all-MiniLLM-L6v22" (Reimers & Gurevych, 2019). The visualization is shown in Figure 3, where each value in the heatmap metrix represents the averaged cosine similarities between the corresponding step index. The average similarity score for well-defined question is 0.45 and 0.50 for MiP response. The variance 7.9e-3 and 8.2e-4 respectively.
As shown in the figure, responses to MiP questions have greater overall similarity across steps and lower standard variance, indicating the considerable redundancy in the content. This means, in many instances, the model revisits similar partial reasoning repeats previous sentences with only minor changes, showing a potential self-trapping issue. Together, these patterns confirm that MiP questions induce high degree of repetitive
7

---

Published as a conference paper at COLM 2025
MiP-Formula
MiP-SWAMP
MiP-GSMBK
MiT-MATH
Model
Type
Length)
Abstainf
Lengthi
Abstaint
Type
Length.
Abstain
Length
Abstaint
Non-Reasoning Models
Qwen3-1.77 (Non-Reason)
MiP
437
64.0
265
77.0
Well-defined
MIP
264
331
22.2
1236
952
11.5
0.0
Qwen3-SB (Non- Reason)
Well-defined
MiP
1538
17.3
Qwen3-32B (Non-Reason)
MIP
496
90.0
184
100.0
Well-defined
MiP
237
279
32.6
1.0
1571
1287
17.3
0.0
Reasoning Models
Qwen3-177 (Reason)
3072
MiP Well-defined
4000
10.2
9272
5.9
Owen3-SB (Reason)
MiP
5483
58.0
2636
49.4
Well-delined
MiP
2620
3851
31.3
2.0
5895
9933
4.0
1.9
Qwen3-32B (Reason)
MiP
5293
34.0
1872
38.0
Well-defined
MIP
3149
1723
22.4
S685
9468
5.7
0.0
Table 4: Comparing the effects of model sizes and reasoning capabilities within the Qwen3 family. MiP-Overthinking is widely observed among models of different sizes consistently, and there is no consistent pattern between size and severity. This phenomenon indicates that Sint-vverhinking can not be mitigated by simply scaling up the model size.
MiP-Formula
MiP-SVAMP (Mix)
MiP-GSMBK
Metrics
Precision
Length
Precision (5)
Length (words)
Recall (%)
Precision (%)
Time (seconds)
Length (words)
(%)
(%)
(seconds)
(words)
(%)
(seconds)
100
100
5.21
20.3
98
100
17.66
34.6
94
96
19.80
42,5
Table 5: Human evaluation on mixed datasets that contain MiP and solvable questions. Humans reliably detect MiP questions while spending only small amount of time per question. This observation highlights the significant gap between the current models and human critical-thnnking behavior when facing ill-posed tasks.
content in reasoning models. Rather than terminating early to conclude for insufficient premise, the models fill their reasoning paths with repetitive re-checks and reiterations, significantly inflating token usage without improving real abstain rates.
3.5 Effects of Model Sizes and Reasoning Capabilities
To investigate how the model sizes and reasoning capabilities affect the MiP-Overthinking issue within the same model family, we conduct controlled experiments on Qwen3 family models as shown in Table 4. The Reasoning and Non-Reasoning represent whether to utilize the reasoning mode for the Qwen3 models. As shown in the table, the overall obersva- tions are consistent with the results in Figure 2, indicating the generalization Moreover, by comparing this overthinking phenomenon across different model sizes within the same model family, we observe that the MiP-Overthinking issue occurs consistently and there is no consistent patterns that are strongly correlated with the model size. This observation especially highlights the significance of the MiP-Overthinking issue, which can not be mitigated by simply scaling up the model size.
this issue,
4 Further Discussion
4.1 How Humans React to MiP Questions?
To evaluate how humans respond to MiP questions, we conducted a small-scale user study with three graduate-level participants. The participants were presented with mixed versions of the MiP-Formula, MiP-SVAMP, and MiP-GSM8K splits, each consisting of 50 MiP problems and 50 ordinary solvable questions2. For every question they were told to decide whether each question is solvable or not and the time spent and the question length in words are recorded for further analysis.
Thee setting is slightly different from LLM's -ieetherthnn setting, which provides only the non-solvable questions. In the setting, even if the participants are not told that some questions might be unsolvable, they can figure this out and quickly annotate all the questions non-solvable accordingly. Thus, we utilize the mixed version to avoid this issue.
8

---

Published as a conference paper at COLM 2025
Mil-Forrulaa DS-Queen
MiP-CSMR
Model
DeepSeek-RT
Qw(
ST.1
DeepSeck-RI
18- Qwen
QwQ
ST.1
In- In-Process Suspicion Rate
100%
100%
100%
100%
95.5%
83.3%
99.6%
100%
In-Process First Suspicion Index
1.32
1.36
1.42
1.16
2.01
3,90
1.77
1.6
Table 6: The in in-process insufficiency suspicion information across different reasoning models on MiP-Formula and MiP-GSMR datasets. The in-process insufficiency suspicion is defined as when the reasoning model suspects the given question unsolvable during its thinking process. In-Process Suspicion Rate represents how many percent of the samples trigger the in-process suspicion. First Suspicion Index is the averaged step index where the model first suspects the question's 's validity.. Most reasoning models do notice the existence of MiP at the very early steps, but they still suffer from low abstain rate and cannot confidently stop the thinking.
As shown in Table 5, humans achieve near-perfect MiP identification: recall of > 94% and precision of 96% across all three datasets, while requiring only a few seconds for short algebra problems (MiP-Formula) and under twenty seconds for longer word problems (MiP-SVAMP /GMM8)) This performance is substantially better than the abstain rates of state-of-the-ane reasoning LLMs reported in Table 2, highlighting pronounced gap between current models and human critical-thinking behavior when facing ill-posed tasks. This observation further highlights the significant gap between the current models and human critical-thinking behavior when facing ill-posed tasks.
4.2 Do Models Know Premises are Missing?
To investigate whether reasoning models recognize the potential unsolvability of ques- tions during their reasoning process, we conducted a detailed analysis of their reasoning chains. We segmented each reasoning chain into discrete steps using \n' as delimiters and performed step-wise verification to detect whether models express doubt on the ques- tion solvability. We introduce two key metrics for this analysis: In-Process Suspicion Rate, which measures the percentage of responses where the model expresses doubt about solvability during reasoning, and First Suspicion Index, which captures the average step number at which the model first suspects the missing premise. To ensure robust evaluation, we employed GPT-4o to assess each step three times, using majority voting for our final step-level result. The quantitative results of this analysis presented in Table 6.
As we can see from the table, most of the existing reasoning models have suspected that the given question might be unsolvable at the very early stage of their reasoning process, demonstrating the ability of reasoning models to recognize the potential MiP. However, these reasoning models lack critical thinking capabilities: they are prone to keep digging the given unsolvable question by re-visiting the question and related definitions again and again and again, rather than question the solvability of the given question. Thus, as visualized in Figure 4, despite existing reasoning models suspecting the solvability of most of the given MiP questions, they only abstain a very small proportion of them.
Based on the above observations, we conclude that reasoning models actually have the capabilities find out that the given MiP question is not solvable, but they do not "dare" to abstain it. Thus, our MiP-Overthinking issue indicates the lack of critical thinking abilities of reasoning models.
4.3 What Caused MiP-Overthinking?
Figure 2 demonstrates that MiP-Overthinking manifests across both RL-based and SFT-based reasoning models. We hypothesize this phenomenon primarily originates from inadequate length constraints during the rule-based reinforcement learning phase of RL-based models, subsequently propagating to SFT-based models through distillation.
Current RL-based reasoning models predominantly employ rule-based training focused on format and accuracy rewards (Shao et al., 2024; Sui et al., 2025), with some incorporating step or length rewards to promote thorough reasoning (Face, 2025). This approach can lead to reward hacking, where models explore excessive reasoning patterns to achieve correct answers (Aggarwal & Welleck, 2025; Shen et al., 2025; Luo et al., 2025).

---

Published as a conference paper at COLM 2025
In- Pr-Prccess
Final Decision
Suspected
Non-abstain
Unsuspected
Abstain
DeepSeek-RA
DS-Distill-Qwen
QwQ
S1.1
Figure 4: The transition flow between in-process suspicion of MiP and the final successful abstention on different reasoning models. For each Sankey diagram, the left bars represent whether the model suspects the given question is unsolvable during its thinking process, i.e, Suspected or Unsuspected; the right bars represent the final abstention, categorized into Abstain (preferred) or Non-abstain.. Most existing reasoning models have suspected that the given question might be unsolvable, but only for a very small portion, the models insist their suspicion.
Average Resporse Length (tokens)
To demonstrate the transmissibility of this be- havior through distillation (Xu et al., 2024), we finetune Qwen-225-7B-Instruuc using scall-scale
95
50 MiP responses generated by DeepSeek-R! R1
on the MiP-Formula dataset. As shown in Fig- ure 5, the fine-tuned model exhibits clear MiP-
X00
Naandl Leagis
Abasia De MP (4)
Overthinking characteristics when evaluated on GSM8K: significantly increased response lengths for both MiP and well-defined questions, emer- gence of a length disparity between MiP and
Naanal
5.2%
Qwen2.5-7B-Luttree
MIP Sarr--@en2.--775 Instruct
Figure 5: Comparison of response length, abstain rate of MiP, and accuracy of well- defined questions before and after tun- ing on 50 responses from DeepSeek-R1 on the MiP-Formula dataset. The re- sults demonstrate a rapid onset of MiP- Overthinking behavior after exposure to a small number of MiP examples during fine-tuning.
well-defined responses absent in the original model, and decreased abstain rates.
Conclusion
We introduce the Overthinking under Miss- ing Premise (MiP-Overthinking) issue, which is a widespread but still under-explored phe- nomenon for current reasoning models, In this phenomenon, when faced with ill-defined un- solvable questions with missing premises, ex- isting models generate dramatically long re- sponses while having very low abstain rates. With systematic investigation of this phe- nomenon, our findings show that while these models sometimes suspect the given MiP question is not solvable the early state of the thinking process, they typically fail to act those suspicions and instead generating repetitive and redundant thinking traces with the final answer that does not address the missing premises, indicating a lack of critical thinking capability. This behavior highlights a pressing gap: current training recipes for reasoning models, which emphasize thorough chains of thought, do not sufficiently reward critical thinking or early exit from unsolvable tasks.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and etc. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https://arxiv. httss///arxi..rr/aab/242000 org/abs//204. 14219.
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning, 2025. URL https://arxiv.. wrr.ii.orr//aee arrg/bbs/2503 04697.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. In Neele Falk, Sara
10

---

Published as a conference paper at COLM 2025
Papi, and Mike Zhang (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 225-237, St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL https: //aclanthology. org org/2024 eacl-srw.
Alfonso Amayuelas, Kyle Wong, Liangming Pan, Wenhu Chen, and William Wang. Knowl- edge of knowledge: Exploring known-unknowns uncertainty with large language models, 2024. URL https:///axiiii wttp://////bb.....// http:://araa...rrgaab//2230 13712.
Maciej Besta, Julia Barth, Eric Schreiber, Ales Kubicek, Afonso Catarino, Robert Gersten- berger, Piotr Nyczyk, Patrick Iff, Yueling Li, Sam Houliston, Tomasz Sternal, Marcin Copik, Grzegorz Kwasniewski, Jürgen Müller, Lukasz Flis, Hannes Eberhard, Hubert Niewiadomski, and Torsten Hoefler. Reasoning language models: A blueprint, 2025. URL https://arxiv.. orggabs/25011 11223.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL ht /aaxxiv http//aww...g/.aa//////////roltt wrgg/bs////e 2407. 21787.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models, 2025a. URL https://arxxv.orrg Abs/2503.09557
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like 1lms, 2025b. URL http::///aaivooor/as///2/2/2
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ar preprint 1rXiv:2110.14168, 2021.
Jeremy R. Cole, Michael J. Q. Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. Selectively answering ambiguous questions, 2023. URL 14613.
Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, Nicholas Thumiger, Aditya Desai, Ion Stoica, Ana Klimovic, Graham Neubig, and Joseph E. Gonzalez. The danger of overthinking: Examining the reasoning-activn dilemma in agentic tasks, 2025. URL https://arxiv
http://www..rg////////////////// 28220.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards, 2025. URL https://arxivv www.//r////////// 01456.
DeepSeek-AT, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, and etc. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL http://raxiiv.orr/////////// 12948.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL www.///uggingaac///ppe....o http://///ttioooooooo/gggngnnann
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D. Goodman. Stream of search (sos): Learning to search in language, 2024.
URL https://arxiv.org/bbbs//// 03683.
11

---

Published as a conference paper at COLM 2025
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. Advancing language model reasoning through reinforcement learning and inference scaling, 2025. URL hitps:://arae http///wxivioo.//.....// org/ abs//501. 11651.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey, 2023. URL https:://axxie http:/aawwiii.grbb/ra/aaaaaaaa/e orgg/bb////12.. 10403.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, January 2025. ISSN org/10 11153/3//1.115/373111.... 1145/3703155
1558-2868.
doi: 10.1145/3703155. URL http: www.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: //axxiv. wrgg/br//////11 083611.
Abhinav Kumar, Jaechul Roh, Ali Naseh, Marzena Karpinska, Mohit lyyer, Amir Houmansadr, and Eugene Bagdasarian. Overthink: Slowdown attacks on reasoning Iims, 2025. URL https://arxiv. http://axxxv..rgabbb////aa/a org/ 02542..
Kuang-Huch Lee, Ian Fischer, Yueh-Hua Wu, Dave Marwood, Shumeet Baluja, Dale Schuurmans, and Xinyun Chen. Evolving deeper Ilm thinking, 2025. URL https: //arxiv.org//as//551. 099811.
Noam Levi. A simple model of inference scaling laws, 2024. URL http:::/://rrx 2410. 16377.
Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. Selective reflection-tuning: Student-selected data recycling for LLM instruction-tuningg In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 16189-16211, Bangkok, Thailand and virtual meeting, August 2024a. Association for Computational Linguistics. URL https://aclantholl// findings- 958.
org/2024.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7595-7628, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. URL https:/:::/ http:/aaclaaa /acaathoooatooyyyo.o org/2024 naacl naacl-long. .411.
Ming Li, Yanhong Li, Ziyue Li, and Tianyi Zhou. How instruction and reasoning data shape post-training: Data quality through the lens of layer-wise gradients. arXiv preprint arXio:2504.10766, 2025a.
Ming Li, Yanhong Li, and Tianyi Zhou. What happened in LLMs layers when trained for fast vs. slow thinking: A gradient perspective. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 32017- 32154, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8-89176-251-0. URL htttps://aclantholllo www.llooy.orgg////0 wrg.2025....2...... acl-long. 1545//
Ming Li, Zhengyuan Yang, Xiyao Wang, Dianqi Li, Kevin Lin, Tianyi Zhou, and Lijuan Wang. What makes reasoning models different? follow the reasoning leader for efficient decoding. Xiv preprint XxX1::5506.06998 2025c.
12

---

Published as a conference paper at COLM 2025
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khy- athi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning. arXiv preprint arXio:2312.01552, 2023.
Changshu Liu, Shizhuo Dylan Zhang, Ali Reza Ibrahimzada, and Reyhaneh Jabbarvand. Codemind: A framework to challenge large language models for code reasoning, 2024.
URL https://arxxi.orr//bb//2/22. wtwp:://///.......///// 09664.
Yue Liu, Jiaying Wu, Yufei He, Hongcheng Gao, Hongyu Chen, Baolong Bi, Jiaheng Zhang, Zhiqi Huang, and Bryan Hooi. Efficient inference for large reasoning models: A survey, 2025. URL https: / http//aaww..oorabbe/// www.oos////////oo/ 2503 23077.
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: Length-harmonizing fine-tuning for ol-like reasoning pruning, 2025. URL
12570.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL httpss//aaxxi weriiiiooor//aeeeeee aa//a/bs///111 19393.
OpenAL. Learning to reason with llms, 2024a. seainng-to-ress-wwittttt
URL https::/oopeaai.com//iniee
OpenAI. OpenAl ol System Card, December 2024b.
URL https://ccn..peeai.ccoo
01 -ar-eeee-ard--2221 pdf.
OpenAl. OpenAlo1-mimnng System Card, September 2024c. URL https:://opeaaa.ccm/iiiii openal-ol-minni-advaciin costee-oostteeeeeee oraaaaccinnnntttttfeeiiieeeeeeen
OpenAL. OpenAl 03-mini System Card, January 2025. URL https://cdn.openall
o3 33-mini-syyse sesiii-yssemmmae card-febl0... -errr-sssyttm-a-d--eeleeee pdd..
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Floren- cia Leoni Aleman, and etc. Gpt-4 technical report, 2024. URL httpss org/ /
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/ m1/2021.naaclmmiin 168. URL https: h org/ 2021.
maacl-maing 168.
Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, Peng Li, Wei Wei, Jing Shao, Chaochao Lu, Yue Zhang, Xian-Sheng Hua, Bowen Zhou, and Yu Cheng. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond, 2025. URL https: //arxiv.org/abs/25555 21614.
Nils Reimers and Iryna Gurevych. Sentence-Bie BERT: Sentence embeddings using Siamese BERT-networks In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982- 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi:
URL http::ps////caaannooooooooo/////
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning open language models, 2024. URL https: / /awxxv..ivvv/// org/ / 2402. 03300.
Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, and Shiguo Lian. Dast: Difficult--daptiie wow-niiiiiig for large reasoning models, 2025. URL https://arxii: //axxvv...iiv. wwwrxxiv.oor//ee abwpg///b////.././//// 04472.
13

---

Published as a conference paper at COLM 2025
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https:// // arxiv. orgg aaaaab///2//8 03314.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models, 2025. URL https:////aviioooo abs/2503. 16419.
Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, and etc. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024a. URL https: http://aaxvvvrrrr/e wrgp/br////////////// 05530.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, and etc. Gemma Improving open language models at practical size, 2024b. URL httpss://arxe http://www..g/////448ee.../ 00118.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. //qwenlm. gi giu.....tt io/blog/ httpbbllgg/wn2..../
URL https:
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm. gi github..o.
Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and
Hao Fei. Multimodal chain-of-thought reasoning: A comprehensive survey, 2025. URL https://arxiv. httgg/bb////0//// 12685.
Yuqing Wang and Yun Zhao. Gemini in reasoning: Unveiling commonsense multimodal large language models, 2023. URL http:p/://aarii..oggabb///////// 17661.
Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. Evaluating mathematical reasoning beyond accuracy, 2025. URL https://arrxi..org/abbsst
05692..
Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang, Self- rewarding correction for mathematical reasoning, 2025. URL https://arxiv org/abs. 2502. 19613.
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models, 2024. URL https://arxinn wttp////www...////////..... 13116.
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models, 2025. URL httts::/arxxv..rrggget 118171
11817.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025. URL http://aaaavv..rrr/a/////2.2...
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models, 2023. URL https: //arxiv. aaaxxiv.org/abss/322.. 13439.
14

---

Published as a conference paper at COLM 2025
Table of Contents for Appendix
A Detailed Experimental Setup
16
A.1 Models
16
A.2 Evaluation Metrics
16
A.3 Generation Setting
16
B Data Construction Details
17
MiP on broader domain
18
D The Example of MiP-Overthinking on R1
19
E Related Work
20
E.1 Reasoning Large Language Model
20
E.2 Test-time Scaling
20
E.3 Models' Behavior Study in Ambiguous Condition
20
15

---

Published as a conference paper at COLM 2025
A
Detailed Experimental Setup
A.1
Models
We leverage a series of non-reasoning and reasoning model for our study, from both open- source and proprietary source with different training recipes. The non-reasoning models we use include 2ween.5532B-Instrucc Team (2024), Gemma-2-27B-it Team et al. (2024b), Phi-3-medium-128k Abdin et al. (2024) ,GPT-40 OpenAl et al. (2024) and Geminil.5 Team et al. (2024a). The reasoning models we use are QwQ-32B Team (2025), DeepSeek-RI- Distill-Qwen-32B DeepSeek-AI et al. (2025), S1.1 Muennighoff et al. (2025), DeepSeek- R1 DeepSeek-Al Al et al. (2025), GPT-ol OpenAl (2024b), GPT-olmini OpenAl (2024c) and GPT-o3mini OpenAl (2025).
A.2 Evaluation Metrics
In Section 3.2, measure response length by considering both reasoning and answer components. For open-source models, we employ model-specific tokenizers to calculate token counts, while for proprietary models, we obtain generation lengths via their APIs. To determine abstain rates, we parse responses by paragraphs (delimited by nnn)n') and analyze the final two paragraphs the model's conclusion. These conclusions, along with reference answers when available, are evaluated by GPT-4o to assess whether the model provides definitive answer or abstains. For data sets with reference answers (GSM8K and MATH), GPT-4o also evaluates the correctness of the response.
A.3
Generation Setting
For all open-source models, we employ greedy decoding and utilize the default chat tem- plate specific each model. We deliberately omit system prompts prior posing questions to maintain consistency across evaluations. For proprietary models, we adhere to their default parameter configurations as provided by their respective APIs. In the case of GPT- olmini and GPT-o3mini, we configure the reasoning_effort parameter to the medium setting by default.
16

---

Published as a conference paper at COLM 2025
B Data Construction Details
To systematically investigate this MiP-Overthinking issue, we construct a suite of MiP questions controllable manner. Our MiP questions are sourced from math datasets across different qualities, including SVAMP, GSM8K, and MATH 500. In addition, we also construct a synthetic dataset, rule-based Formula, for evaluation.
MiP-Formula We construct dataset of 50 synthetic unsolvable formulas in rule-based manner. The formulas are generated recursively through combination of variables and operators, with maximum recursion depth of three. The variable set comprises numerical values, Latin letters, and Greek symbols. The operator set includes arithmetic operators ('+', ""),, set operators """,,"), mathematical functions ('sin', 'sqrt',, and construct operators ('E, 'V). To ensure the formulas are fundamentally unsolvable, we enforce the inclusion of at least one unassigned variable in each formula, excluding commonly recognized math- ematical or physical constants such as 'e', 'I' and 'g' While these formulas may appear complex a glance, their unsolvability should be immediately apparent due to the presence of undefined variables.
MiP-SVAMP We utilize SVAMP (Patel et al., 2021), a benchmark dataset comprising 1,000 mathematical word problems, where each instance consists of problem body and an associated question. The Mil' questions can be generated by randomly permuting the problem bodies and associated questions. To maintain dataset integrity, we manually select 300 permuted questions after thorough human evaluation to eliminate any inadvertently solvable questions that may exist. The resulting problems contain clear logical inconsistencies between their body and question components, making their unsolvability readily apparent without additional context.
elementary-school-E
MiP-GSMsK We further utilize GSM8K (Cobbe et al., 2021), grade school mathematics dataset that presents more complex challenges compared to SVAMP. The questions in GSM8K typically contain multiple numerical conditions and require certain reasoning capabilities to arrive at solutions. The MiP question can be constructed by randomly removing a necessary premise from the original solvable question. We first identify the questions containing two or three numerical conditions and then randomly eliminate one numerical condition per question. Subsequently, a thorough human verification is conducted to filter out those questions that are still solvable in some way and finally obtain 582 MiP questions. Compared with previous MiP questions, questions from this source require the basic logical analysis of models to identify that the question is unsolvable.
MiP-MATH For the MATH dataset (Hendrycks et al., 2021), which comprises challeng- ing competition-level mathematical questions, it is hard to build a rule-based filtering mechanism before human evaluation. Thus, we directly read through all the questions in MATH500 and manually select 58 questions that are feasible for constructing the MiP questions and remove one necessary premise from the question. Due to the sophisticated nature of this data source, identifying the insufficiency of these instances requires substantial mathematical reasoning capabilíties, testing models' ability to recognize unsolvability in complex mathematical contexts.
17

---

Published as a conference paper at COLM 2025
C MiP on broader domain
To evaluate the impact of the Mil' phenomenon in broader domains, we constructed new MiP dataset sourced from different fields in the MMLU dataset, consisting of both commonsense and domain-specific questions, including clinical knowledge, chemistry, and physics. For each sample, we manually removed premise that contributes the answer and made sure the question is unsolvable.
Non-Reasoning Models
Reasoning Models
Qven33-30 (Nom Reason)
Qwren2 Dwen225-22--h 5228-nnst
Gomma 2ommma-2-22B-17
Phi-3-med-122 1288
DS Distil Qwen2.
Qwen3-32B (Reason)
GPT-40
Gemind 1.5
GPE-o
GPT-olmini
GPFe33mini
DerpSack R1
$1.1-228
QuQQ32B
Metric
Type
Length|
Well-defined
MiP
487
382
407
334
288
367
282
294
401
399
764
453
1538
883
1413
793
1977
644
2707
4148
4795
3470
6480
2809
6385
3620
3034
5090
Abstain
Well-defined
0.0
0.0
0.0
4.0
0.0
0.0
0.0
4.1
0.0
0.0
0.0
0.0
Table 7:
We evaluate the models with our constructed MiP-MMLU dataset. The substantial gap in average response length between reasoning and non-reasoning models is consistent with our findings reported in the main paper. This consistency verifies the generalizability of our findings about MiP-Overthinking
As we can see from Table 7, the substantial gaps of response length between the MiP and well-defined questions still exist on the MiP-MMLU dataset, which further verifies our findings.
18

---

Published as a conference paper at COLM 2025
D The Example of MiP-Overthinkngg on R1
Figure 6 shows the example of MiP-Overthinking on DeepSeek-RI. When being asked What the value of a?, R1 spends a dramatically large amount of tokens on this MiP question. This response is generated by the initial Deepseek R1 (2025/01/20)
Figure 6: The example of MiP-Overthinking on DeepSeek-RI. When being asked What is the value of n?, R1 spends dramatically large amount of tokens on this MiP question.
19

---

Published as a conference paper at COLM 2025
E Related Work
E.1 Reasoning Large Language Model
Recent advances in LLMs have sparked significant research interest in enhancing their reasoning capabilities (Ahn et al., 2024; Besta et al., 2025; Chen et al., 2025a). Research has focused on improving these capabilities through various post-training approaches. Several studies have employed reinforcement learning techniques to guide models toward more effective reasoning strategies (Shao et al., 2024; Xiong et al., 2025; Cui et al., 2025). Additionally, researchers have demonstrated that instruction tuning on carefully curated, high-quality datasets can significantly enhance performance (Li et al., 2024b;a; Ye et al., 2025; Muennighoff et al., 2025; Li al., 2025a).
While Reasoning Models have demonstrated impressive performance on various bench- marks, recent studies have begun critically examine the quality and efficiency of their reasoning processes. Xia et al. (2025) conducted comprehensive analysis of RLMs' rea- soning quality, revealing significant redundancy in their solution approaches. Further investigations (Chen et al., 2025b; Cuadron et al., 2025; Qu et al., 2025; Liu et al., 2025; Li et al., 2025c) identified a concerning "overthinking" phenomenon, where reasoning model generate unnecessarily verbose solutions even for simple problems. Building on these obser- vations, Kumar et al. (2025) demonstrated the potential security implications of this behavior by developing slowdown attack that exploits overthinking through input perturbation.
E.2 Test-time Scaling
In contrast to earlier research on training-time scaling laws (Kaplan et al., 2020), recent literature has increasingly focused on test-time performance scaling strategies, which aim to enhance model performance by optimizing inference-time time token generation (Snell et al., 2024; OpenAI, 2024a). These approaches can categorized into several primary methodolo- gies: parallel sampling techniques (Brown et al., 2024; Levi, 2024), which generate multiple candidate responses and select the optimal output; sequential refinement approaches (Snell et al., 2024; Lee et al., 2025), which enable iterative improvement of previous outputs; and tree-based methods (Gandhi et al., 2024; Hou et al., 2025), which combine elements of both parallel and sequential approaches. While the prevailing consensus suggests that increased token generation during inference enhances reasoning capabilities, our investigation re- veals a concerning counterpoint: under certain conditions, extended responses can lead to computational inefficiency and, paradoxically, degraded performance outcomes.
E.3 Models' Behavior Study in Ambiguous Condition
LLMs are prone to hallucination (Huang et al., 2025; Xu et al., 2025), generating non-existent conditions that compromise trustworthinesss. An essential aspect of reliability is the ability to abstain under uncertainty. Prior work (Cole et al., 2023; Amayuelas et al., 2024; Zhou et al., 2023) has proposed benchmarks assessing LLMs' recognition of knowledge limits when facing ambiguous or challenging queries. Different from theirs, our study explores reasoning models under MiP condition. Surprisingly, we find these specialized models exhibit prolonged reasoning and inferior performance.
20


Reviews:
[{"review_id": "zM6sntwXgs", "reviewer": "Reviewer_teYv", "summary": "This paper investigates the phenomenon of overthinking in reasoning-focused large language models (LLMs) when confronted with ill-posed questions containing missing premises (MiP). The authors systematically analyze this issue by constructing datasets with missing premises (MiP) across varying difficulty levels (e.g., MiP-Formula, MiP-SVAMP, MiP-GSM8K, and MiP-MATH). They evaluate both reasoning and non-reasoning models (open-source and proprietary) using metrics such as response length, accuracy, and abstention rates. The results show that reasoning models often fail to identify MiP and continue generating redundant thinking patterns, while non-reasoning models are more likely to quickly identify the missing premises and abstain from answering.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper identifies a previously under-explored issue in reasoning models, namely MiP-Overthinking, which provides new insights into the limitations of current reasoning LLMs.\n2. The construction of MiP datasets across difficulty levels (from synthetic formulas to competition-level MATH questions) provides a systematic framework for evaluating model robustness.\n3. The authors curate diverse datasets (synthetic and real-world) and evaluate a wide range of models (e.g., GPT-o1, DeepSeek-R1, Qwen, Gemini), ensuring generalizability. The metrics (response length, abstention rate, accuracy) are well-chosen and effectively highlight the inefficiency of reasoning models.", "reasons_to_reject": "1. Experiments focus exclusively on mathematical reasoning. While math is a structured domain, extending evaluations to commonsense QA or code generation would strengthen claims about the universality of MiP-overthinking.\n2. Including human performance on MiP tasks (e.g., response length, abstention rates) would contextualize model failures and successes.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ukBxTBTwMp", "reviewer": "Reviewer_hrUF", "summary": "This paper analyzes the issue of overthinking in reasoning models, a topic that has drawn significant attention within the research community and remains a pressing challenge. The authors focus on the specific phenomenon of redundancy in \"overthinking on the MiP problem\" and conduct an extensive investigation. First, they provide a clear definition of the MiP problem and establish evaluation benchmarks and metrics for it. Second, they assess the performance of existing reasoning and non-reasoning models on the MiP problem and analyze patterns in the model outputs. Finally, the authors briefly discuss potential factors contributing to overthinking based on the literature. While the analysis may lack depth, the overall contribution of this work is meaningful, particularly in defining the MiP problem and constructing its evaluation benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A valuable problem with a clear and precise definition: This paper investigates the widely discussed issue of overthinking in reasoning models and provides a clear definition of the MiP problem.\n2. The authors constructed a diverse MiP dataset and designed reliable and robust metrics (Abstain Rate, step-level similarity, In-Process Suspicion Rate, First Suspicion Index) to evaluate and analyze model outputs. Specifically, they examined four distinct overthinking phenomena: excessive token counts (3.2), high frequency of reasoning-related special tokens (3.3), repeated reasoning steps (3.4), and suspicion without abstention (4.1).\n3. The experimental results provide statistically significant support for several inferred patterns regarding redundancy phenomena.", "reasons_to_reject": "1. The analysis in this paper is limited to pattern analysis of model outputs, lacking an in-depth discussion of the factors contributing to overthinking.\n\n   1. The paper discusses four specific redundancy phenomena (Reasons To Accept point 2), covering multiple perspectives but remaining confined to statistical patterns in model outputs.\n   2. In Section 4.2, the authors introduce the viewpoints of \"inadequate length constraints\" and \"format and accuracy reward hacking,\" supported by related literature. However, the experimental results provide little direct evidence for these claims (despite Figure 2 being mentioned on line 253). \n   3. The arguments presented in Section 4.2 fail to directly and concretely explain the observed redundancy phenomena. The logical connection between the hypothesized causes and the observed phenomena is missing.\n   \n   These three points collectively result in an analysis that is descriptive rather than deep.\n\n2.The experimental design has some flaws.\n1. The SFT experiment in Section 4.2 appears to lack purpose. Since DS Distill Qwen2.5-32B is distilled from Deepseek-R1, the evaluations in Section 3 already demonstrate the transmissibility of this behavior through distillation. Fine-tuning SFT Qwen2.5-7B on redundant MiP responses (only 50 samples!) is merely a miniature replication of DS Distill Qwen2.5-32B. Moreover, the ability to replicate behaviors in smaller models through distillation is well-known in the community, making this result entirely predictable and the motivation questionable.\n2. In Section 3.1, the authors classify models as \"reasoning models\" and \"non-reasoning models\" and compare five or seven examples from each category. This design is overly simplistic and lacks rigor. Within this design, reasoning models are limited to only a few examples, such as QwQ, S1.1, the Deepseek series, and the OpenAI series. How can the authors ensure that the observations generalize to other reasoning models and that the significant differences in metrics are not due to architecture, parameter size, or implementation details, but rather the intrinsic differences between reasoning and non-reasoning models? \n\n      In fact, as shown in Figure 2, the proposed metrics exhibit significant differences across different series. For instance, GPT-o1 has a higher abstain rate than all non-reasoning models, suggesting that more tokens can be an acceptable trade-off for improved performance, which does not support the authors' claim (lines 174-177). This indicates that implementation differences may have a greater impact than whether a model is classified as reasoning or non-reasoning. A broader investigation is necessary.\n\n      The authors could consider controlling for irrelevant variables such as parameter size and architecture by comparing non-reasoning models and reasoning models based on the same base model. Additionally, they should conduct broader and more rigorous comparisons across different series and parameter sizes to eliminate these influences and derive robust conclusions. The conclusions should also be softened to ensure they are fully supported by the observed phenomena.", "questions_to_authors": "1. The MiP problem is discussed only in the context of mathematical tasks. Does redundancy also exist in MiP reasoning questions based on knowledge? \n2. The x-axis and y-axis ranges of Figure 3 are labeled as \"0-100,\" which contradicts the caption stating \"across first 50 steps\" (line 219). Please revise to ensure consistency and provide accurate results.", "extra_scores": {}}, {"review_id": "bCoqa70MVk", "reviewer": "Reviewer_4FcV", "summary": "This paper identifies a problem: when the questions are ill-posed with missing premises (MiP), the LMs think redundantly. This contradicts the widely-discussed test-time scaling law. The paper finds that non-reasoning models generate shorter responses and more quickly identifies these MiP. This paper further analyzes the behavior patterns of the reasoning models, and find that they are trapped in self-doubt loops, repeatedly revisiting the questions, and guessing the user intentions. A very interesting finding is that the reasoning models notice the existence of MiP at an early stage, but hesitate to commit this judgment and keeps this overthinking.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper identifies a very interesting problem. Considering the reasoning models are actually more likely than the non-reasoning models to get trapped with MiP.\n- This paper identifies the thinking patterns and documents them.\n- This problem with MiP is tested on multiple datasets (Formula, SVAMP, GSM8k, MATH), and multiple models.", "reasons_to_reject": "- While the current paper is already pretty good, I think one way to continue pushing forward the research field is to hypothesize/propose some methods to improve the reasoning models, defending them against these MiP, while keeping their original high performances. \n- I like the discussion into the root causes of MiP-overthinking (section 4.2). They fine-tuned a non-reasoning model with a reasoning model on merely 50 MiP-Formula responses and observed the response lengths significantly increased. This is good but the discussions can go deeper: what aspects of the reasoning model’s MiP outputs lead to this contagious behavior? While the page number limits the space, I recommend the authors to add more discussions about this, e.g., in the Appendix, if possible.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NQBigXtrrV", "reviewer": "Reviewer_gLKw", "summary": "Signficance:\nThe paper contributes a new dataset and analysis on the overthinking problem. \nAlthough the overthinking problem has been noted before, this paper provides a new dataset that is valuable for empirical research in fixing the problem.\n\nClarity:\nThe paper was easy to read with clear examples of the overthinking problem.\n\nOriginality:\nThe overthinking issue has been raised previously.\nBut, IMO, originality of the idea is not important. The execution is more important.\nThe creation of a dataset to measure the overthinking problem is a notable contribution. This dataset helps the community measure the propensity of the overthinking problem, as new, improved, reasoning models are created.\n\nQuality:\nThe dataset introduces ~ 1000 questions to measure the overthinking problem.\nThe problems come from 4 different sources e.g. MATH 500, and a constructed synthetic dataset.\n\nThese sources have different levels of difficulty:\nIn the easy synthetic problems (MiP-Formula) \"What is the value of ln(a +b)\", it seems obvious that the model is overthinking and should request for more information.\nIn harder problems, e.g. MiP-MATH, it is more unclear how much the model should try to solve the problem, before giving up. Some may think that the model thinking for a long time, trying its best to solve the problem is an intended behavior, is working as designed. \nNevertheless, at least in the easier problems, I think the intended behavior should clearly be to not overthink.\n\n\nOverall, I think the paper is a good contribution to the community. I still have some areas where the paper can be improved.\nI note that I am not an expert in problem of overthinking in reasoning models, so I have indicated my confidence to be lower.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a useful dataset that is relevant to user experience and efficiency of reasoning models. Whenever new reasoning models are released, it would be good to run experiments on this dataset, so that we can gauge how large the problem is.\n-The experiments provide clear insights into the overthinking phenomenon. They test the latest reasoning models, showing that it exists in all of them.  There are further interesting experiments where they show that models may know that some problems are unsolvable, but still choose to continue working on them.", "reasons_to_reject": "- The number of samples in some categories is relatively small, which might limit the statistical power of the findings. For example, on the easiest dataset, the authors only provide 50 examples. I think it would be very helpful for the community to have more examples (e.g. 600), so that whenever models improve on the easy category, the error bars are not so large due to the small sample size. I highlight the easy category because it seems to be the most important category to not overthink.\n- One thing the authors can elaborate on and experiment with is the trade-off between identifying unsolvable cases and not giving up too quickly. It is not always clear to models (and humans) whether a problem is unsolvable. So, maybe you want to err on the side of caution and not give up too quickly?", "questions_to_authors": "The authors say that the reasoning models \"lack genuine critical thinking capabilities\", and well as in title. While this is a catchy headline, I would warn against this claim.\n\nHow do you envision the trade-off between identifying unsolvable cases and not giving up too quickly?\nSuppose you sat for an exam where you are not timed, and one question seems unsolvable to you, but you are not sure. You can give up early within 1 minute, or try to solve it for 10 minutes. Either way, you are not penalized for how much time you spend on the problem. \nIf you were a smart person with critical thinking, what is the rational choice to take?\nI think my choice would be to continue trying to solve it, since you are not penalized for the time you spend, and you are sitll unsure about whether the problem can be solved. (Suppose you don't get tired from spending your brainpower). \nThis analogy applied to the overthinking problem: In the reasoning training setup, the models do not have a large token length penalty. So, there is no incentive to give up early. The authors also mention that sometimes the models realise they are stuck in a loop, but continue to go on. This seems like the rational choice is to continue thinking if length penalities are low. So it does not seem to be that this reflects a lack of critical thinking -- the model is simply playing the game optimally according to the rules of the game.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces "MiP-Overthinking," a novel and significant failure mode in which reasoning LLMs enter inefficient, lengthy thought processes when faced with ill-posed questions containing missing premises. The authors make a valuable contribution by defining the problem, creating several new datasets to measure it, and demonstrating through extensive experiments that specialized reasoning models are particularly susceptible. The authors have sufficiently addressed the concerns raised in initial reviews regarding the limited scope (math-only), the lack of a human baseline, and the need for more rigorous experimental controls. Overall, the paper has been significantly improved through the review process and now stands as a solid contribution to the field.
Decision: Accept