Title: Can Reasoning Language Models Think More Creatively? A Study of Reasoning Ability and Overconfidence
Primary area: generative models
Keywords: LLM, Reasoning model, mathematics, hallucination, overconfidence

Abstract: Recent advances in Large Language Models (LLMs) have been largely attributed to improvements in reasoning abilities. Reasoning models trained with Supervised Fine-Tuning (SFT) and Reinforcemenet Learning (RL), such as Policy Optimization (PO), demonstrate significantly superior performance compared to base models.  However, recent studies raise questions about the ability of these reasoning models to achieve creative thinking beyond that of the base model. In this study, we compare the creative problem-solving abilities in mathematics of two types of models: reasoning models and math models that have been further trained on simple mathematical corpora. Our comparison spans two representative open-source LLM families, DeepSeek and Qwen. The results indicate that reasoning models are less effective in generating creative solutions. We attribute the reasoning models’ limited ability to generate creative responses to Overconfidence (OC)—the tendency of models to exhibit excessive confidence in their own outputs. For example, within the DeepSeek family, reasoning models exhibit 15% higher OC compared to the math model, and within the Qwen family, the gap rises to 80%. Notwithstanding their heightened OC, they fail to generate creative responses as intended. We hypothesize that the high OC may stem from overly aggressive probability adjustments for certain tokens during SFT and PO. To examine this hypothesis, we introduce the notion of a High Entropy Segment (HES), defined as a region in which entropy varies sharply. Within these segments, reasoning models tend to exhibit greater heterogeneity compared to other models. Lastly, we measure the proportion of time steps where the model does not generate the most probable token, and observe that reasoning models show a substantially lower rate than math models. This is largely because their distributions contain a substantially greater share of tokens whose probabilities exceed 80% at each step. Our findings will be of great help in understanding and improving reasoning models.

TL;DR: base language models show more creative thinking than reasoning language models. This is due to overconfidence caused by excessive tuning during SFT and PO.