NeurIPS 2025

CLAWS:Creativity detection for LLM-generated
solutions using Attention Window of Sections
Keuntae Kim1∗
Eunhye Jeong2∗
Sehyeon Lee3
Seohee Yoon1
Yong Suk Choi1†
1Department of Computer Science
2Department of Artificial Intelligence
3Department of Future Mobility
Hanyang University, Seoul, Korea
{ktkpv94, jeh0826, leesehyeon, seohee09}@hanyang.ac.kr
Abstract
Recent advances in enhancing the reasoning ability of Large Language Models
(LLMs) have been remarkably successful. LLMs trained with Reinforcement
Learning (RL) for reasoning demonstrate strong performance in challenging tasks
such as mathematics and coding, even with relatively small model sizes. How-
ever, despite these impressive improvements in task accuracy, the assessment of
creativity in LLM generations has been largely overlooked in reasoning tasks, in
contrast to writing tasks. The lack of research on creativity assessment in rea-
soning primarily stems from two challenges: (1) the difficulty of defining the
range of creativity, and (2) the necessity of human evaluation in the assessment
process. To address these challenges, we propose CLAWS, a novel method that
defines and classifies mathematical solutions into Typical, Creative, and Halluci-
nated categories without human evaluation, by leveraging attention weights across
prompt sections and output. CLAWS outperforms five existing white-box detec-
tion methods—Perplexity, Logit Entropy, Window Entropy, Hidden Score, and
Attention Score—on five 7–8B math RL models (DeepSeek, Qwen, Mathstral,
OpenMath2, and Oreal). We validate CLAWS on 4,545 math problems collected
from 181 math contests (A(J)HSME, AMC, AIME). Our code is available at
https://github.com/kkt94/CLAWS.
1
Introduction
In recent years, Large Language Models (LLMs) have achieved remarkable success across a wide
range of tasks. Among these, the most notable progress has been made in reasoning ability, particularly
in mathematical problem solving. Solving math problems requires cognitive proces...