CLAWS:Creativity detection for LLM-generated solutions using Attention Window of Sections
Authors: Keuntae Kim
Eunhye Jeong
Sehyeon Lee
Seohee Yoon
Yong Suk Choi
Venue: NeurIPS 2025 poster ()
Keywords: llm
reasoning
math
creativity
hallucination
Primary area: deep_learning
TL;DR: LLMs have advanced in reasoning tasks, but creativity assessment has been overlooked. CLAWS uses attention-based features to classify solutions without human evaluation.

Abstract: Recent advances in enhancing the reasoning ability of Large Language Models (LLMs) have been remarkably successful. LLMs trained with Reinforcement Learning (RL) for reasoning demonstrate strong performance in challenging tasks such as mathematics and coding, even with relatively small model sizes. However, despite these impressive improvements in task accuracy, the assessment of creativity in LLM generations has been largely overlooked in reasoning tasks, in contrast to writing tasks. The lack of research on creativity assessment in reasoning primarily stems from two challenges: (1) the difficulty of defining the range of creativity, and (2) the necessity of human evaluation in the assessment process. To address these challenges, we propose CLAWS, a novel method that defines and classifies mathematical solutions into Typical, Creative, and Hallucinated categories without human evaluation, by leveraging attention weights across prompt sections and output. CLAWS outperforms five existing white-box detection methods—Perplexity, Logit Entropy, Window Entropy, Hidden Score, and Attention Score—on five 7–8B math RL models (DeepSeek, Qwen, Mathstral, OpenMath2, and Oreal). We validate CLAWS on 4,545 math problems collected from 181 math contests (A(J)HSME, AMC, AIME). Our code is available at https://github.com/kkt94/CLAWS.

CLAWS: detection for LLM-generated solutions using Attention Window of Sections
Keuntae Kim
Eunhye Jeong"
Sehyeon Lee
Seohee Yoon'
Yong Suk Choil!
Department of Computer Science
Department of Artificial Intelligence
Department of Future Mobility
Hanyang University, Seoul, Korea
(ktkpv94, jeh0826, leesehyeon, Shansangg ac.kr
Abstract
Recent advances in enhancing the reasoning ability of Large Language Models (LLMs) have been remarkably successful. LLMs trained with Reinforcement Learning (RL) for reasoning demonstrate strong performance challenging tasks such mathematics and coding, even with relatively small model sizes. How- ever, despite these impressive improvements in task accuracy, the assessment of creativity in LLM generations has been largely overlooked reasoning tasks, in contrast to writing tasks. The lack of research on creativity assessment in rea- soning primarily stems from two challenges:
(1) the difficulty of defining the range of creativity. and (2) the necessity of human evaluation in the assessment process. To address these challenges, we propose CLAWS, novel method that defines and classifies mathematical solutions into Typical, Creative, and Halluci- nated categories without human evaluation, by leveraging attention weights across prompt sections and output. CLAWS outperforms five existing white-box detec- tion methods Perplexitt Logit Entropy, Window Entropy, Hidden Score, and Attention Score-on five 7-8B math RL models (DeepSeek, Qwen, Mathstral, OpenMath2, and Oreal). We validate CLAWS on 4,545 math problems collected from 181 math contests (A())SMEE AMC, AIME). Our code is available at https: http::://githbb wtt.lkt9//////////....
Introduction
In recent years, Large Language Models (LLMs) have achieved remarkable success across wide range of tasks. Among these, the most notable progress has been made in reasoning ability, particularly mathematical problem solving. Solving math problems requires cognitive processes that go beyond simple calculations, making it an ideal benchmark for assessing how closely Al approximates human intelligence. Recently released frontier LLMs [1 2 appear to approach human-level intelligence in terms of accuracy on mathematical reasoning tasks.
However, human intelligence is not defined solely by accuracy; it also encompasses diverse aspects such as creativity. Within LLM research, creativity has primarily been explored in writing tasks, often evaluated through the Torrance Test of Creative Writing (TTCW) from the Torrance Tests of Creative Thinking (TTCT) 6,, These tests provide framework for assessing creativity beyond factual consistency between input and output g or coherence of generated responses [9] In contrast, creativity remains largely overlooked in reasoning tasks.
[ which is adapted
"quual contribution
Correspondence to: Yong Suk Choi <cys@ hays@hanyang.cc.rr
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Step 1. Generation
Step Feature Extraction
Step 4. Evaluation
Prompt: Given
Reference set
following mudl ennticall
Feonures
e Time Falie
acstact Extended Test
problem vo, Please
nevel sohrion
Hallucinated
Creaiie
TTypicll
Step 3, Labeling
Detection Method
L.Ms
Gemini 1,5 Pro
Hallucinated
Creative
Solution
GIFT o4-mini
Typical
Forere
Fosture Tabel
Refereses
Test sct Estanded Test Set
Figure I: Overview of the proposed framework, Step LLM generates a solution. Step 2: Features are extracted during generation. Step 3: LLM Evaluator assigns labels (Hallucinated Creative / Typical). Step 4: Detection methods are evaluated by comparing predictions with the labels.
Evaluating creativity in reasoning is particularly challenging because it requires human expertise to establish what constitutes a "creative" solution. Assessing mathematical creativity, in particular, demands high-level domain knowledge, making large-scale evaluation costly and difficult to standard- ize. To overcome these challenges, recent studies on mathematical problem solving 10 [1] have attempted to define creativity and measure the creative problem-sol abilities of LLMs. Interest- ingly, their results revealed that even LLMs with similar accuracy exhibit substantial differences in creative ability, motivating further research on creativity assessment in reasoning tasks.
If clear criteria for judging creativity were available, would be possible to detect whether an LLM's response is creative or not. Since the emergence of LLMs, numerous studies have focused on hallucination detection [2 [3.. While detecting and mitigating hallucination crucial for ensuring factual accuracy, excessive restriction of model generations may inadvertently suppress creativity and lead to repetitive, typical outputs 14. Thus, identifying creative responses is essential for maximizing the effectiveness and diversity of LLM generations.
Recent advances in prompt engineering have shown that the structure of input prompts significantly affects LLM performance 16). Building on these findings, we hypothesize that creative generation may depend on which sections of the prompt an LLM attends to to-whether relies more on the given instructions on its self- generated reasoning. Accordingly, we propose CLAWWS, which divides the prompt into sections and quantifies, via attention analysis, the degree to which each section influences generation. This enables detection of Hallucinated, Creative, and Typical solutions based on attention differences across prompt sections and output.
In this study. we present an experimental framework that classifies generated mathematical solutions into Hallucinated, Creative, and Typical categories, and propose CLAWS, a novel white-box detection method that leverages attention over distinct prompt sections and output to perform this classifi- cation without requiring human evaluation. CLAWS achieves saree-way&oo classification with high efficiency-4 capability rarely demonstrated by existing hallucination detection methods. Moreover, consistently outperforms baseline methods on hallucination detection tasks.
To validate the superior performance of CLAWS, we utilize five Reasoning Language Models (RLMs) Dess))-DDeeeee-Mathhhh
(wen-Math(18), Mathstral OpenMMath2-Llamal 200] and Oreal 21 each with 7-8B parameters and trained with reinforcement learning to enhance reasoning capabilities. We conduct extensive validation using dataset of 4,545 mathematical problems spanning Algebra, Precalculus, Prealgebra, Number Theory, Geometry, and Counting & Probability.
Our major contributions summarized as follows:
We propose framework for detecting Hallucinated, Creative, and Typical solutions without human evaluation in reasoning tasks using RLMs.
We introduce CLAWS, a novel white-box method for detecting creativity and hallucination in mathematical reasoning.
We present comprehensive evaluation protocol, consisting of five evaluation strategies and four metrics, to assess the features extracted by detection methods.

---

2 Experimental Framework
An overview of the experimental framework is presented in Figure 1 During the generation process, features are extracted from the model's ' internal representations through the Generator. The generated responses then labeled by the LLM Evaluator, which determines whether each solution is Hallucinated, Creative, or Typical. Finally, the reference set is utilized to perform detection using the selected methods, enabling each method to classify the responses based on the extracted features.
2.1 Problem Formulation
We aim to classify a generated solution It = f(X) into one of three categories Hallucinated / Creative / Typical Solution without relying on human evaluation, where X G| P|S| is the input prompt to the generative model f. As illustrated in Figure 2 input prompt X consists of the following four sections:
Guideline G: Describes the criteria for evaluating the difference between two mathematical
solutions, providing the model with the concept and standard for identifying Creative solutions.
Problem P: The math problem that the model is required to solve.
Reference Solutions S: A of to n typical solutions the problem, provided help the model generate creative solution in contrast to these references.
Instruction I: An instruction to create novel solution that is different from reference solutions for a given problem.
In prior study MM prompt in Figure 2 was used to induce the generative model to create novel cre- ative solutions, and the creative problem-solving ability of LLMs was successfully presented. Build- ing on this setup, we take a further step by inves- tigating whether the model's internal state detect Hallucinated Creative / Typical Solution.
Criteria for evaluating the difference between two mathematical solutions include:
i). If the methods used arrive the solutions fundamentally different, such as algebraic manipulation versus goometric reasoning, they be considered distinct;
ii). Even if the final results are the sume, if the intermediate steps proceases involved reaching solutions can be considered different;
solutions vary significantl,
iij. two solutions rely on different assumptiions or conditions, they likely be distinct:
We define generated response R as Creative solution if it differs from the provided reference solutions S' in way that satisfies the criteria spec- ified in the guideline G. The generative model f receives between 1 and n reference solutions and generates a novel solution accordingly. Each R is then evaluated by LLM evaluator E, using the prompt described in Appendix B.1
iv), A solution might generalize to breader class of problems,
while another solution might specific to conditions. such cases, they are considered distinct: v). I'one solution significantly simpler or more complex the other, they can be regarded esscutially different, even they lead to the same result.
Given the following mathematical problem: largest power that divisor 13* 11
And some typicul solutions:
1. First, we use the difference of squares on
13' 111 (12 (11*'?,...
2. Just like in the ahove solution, we use the difference-of-squure factorization, but only ones to get
2.2 Model
13* - 11f (13 11s) + +11").
2.2.1
RLM Generator
Please output novel solution distinct from the given omes for this math problem.
To generate mathematical problem solutions, we
Example of an input prompt X used to elicit creative solutions from LLMs. Prompt consists of four sections: Guideline (G, yellow), Problem (P, green), Reference Solutions (S, blue), and Instruction (7, purple).
Figure 2:
select five RLMs: DeepSeck-Math-77B-- [17).
Owen2.5-Math-77B- In-sst [18. Mathstral-7B [19],
OpenMath2-Llamas 8B (20 and OREAL-7B
21). These models are released after 2024 and have 7-8B parameter scales, which enables white- box approach of long length of problem and reference solutions. addition to the RIM,, we also tested general LLMS such as LLaMa3-8B (22]. Qwen2.5-144 [23], and DeepSeek D22pSeekV2116i 16 (24] but finally did not adopt them due to their lack of ability to creatively solve mathematical problems or model sizes.
2.2.2 LLM Evaluator
To evaluate RLM's Generations, we employ two frontier-level L.MSs - GPT-o4-mini 25 and Gemini-L.5-Pro [26]- as LLM Evaluators. These two models have shown great performance
3

---

Table 1: Number of samples per class (Hallucinated, Creative, Typical) for each dataset and model. Ha denotes Hallucinated solutions, Cr Creative solutions, and Ty Typical solutions.
Model
DeepSeck
Mathstral
OpenMath2
OREAL
Qwen-2.5
Dataset
Ty
Total
Ty
Total
Cr
Ty
Total
Ha
Ty
Total
Ty
Total
REF
868 206
649 1723 1192
175
437 1804 923
103 785 1811 1244
1706
631 324
752 1707
TEST
798 160
456
1414
961
154
337
1452
815
97
551 1463
932
89
369 1390
578
203
579
1360
AMC
1197 530
1373
3100
1679
434
1049
3180
1330
291
1578
3199
1928
237
935
3100
637
629 1784
3050
AIME
772
126
262
1160
917
68
221
1206
644
67
501
1212
911
47
161
1119
529
159
373
1061
AGDHSME
657
424
763
1844
945
354
606
1905
723
248
943
1914
1005
161
656
1822
281
491 1054 1826
mathematical benchmarks and have already been recognized as human-level Evaluators in prior studies. Based their evaluations, assess the solutions according to the following criteria:
Hallucinated Solution: If both evaluator did not judge the generated solution as 'Correctness', classified as Hallucinated Solution.
Among the generated solutions that both evaluators judged to be 'Correctness', if even one evaluator judged them to be 'Creativity", the solution was classified as Creative Solution.
Creative Solution:
This is a criterion for inclusive acceptance of creativity.
Typical Solution: Among the generated solutions that both evaluators judged as 'Correctness', those that neither evaluator judged 'Creativity' were classified as Typical Solution.
2.3 Dataset
To conduct our study, we adopted publicly available math datasets from CreativeMath [1 and HARP CreativeMath is a dataset of 400 math problems with solutions, sampled and cleaned from 50 questions from eight math contests: AMC 8, 10, 12, A(JHSME,, AIME, USAJMO, USAMO, and IMO. HARP contains 5,409 problems from A()HSME, AMC, AIME, and USA(JMO. Since there is an overlap of 282 problems between HARP and CreativeMath, We reconstructed HARP by removing duplicate entries and excluding problems that were either too difficult (e.g.. proof-based) or whose problems or solutions were excessively long, resulting in a final set of 4,545 problems with solutions.
27.
2.3.1 Dataset Construction
We generate solutions using the Generators described in Section 2.2.1 and construct the reference set and test from the CreativeMath, and the extended test from the HARP. The reference set serves as a low-resource for detection, the test set is used for validation on the same dataset, and extended dataset. For all Generators, we limit the
the extended test set is used for validation input token length to 2,048 and the output token length to fewer than 1,024 tokens, based on each Generator's respective tokenizer.
Reference Set Reference set serves as the standard for classifying generated solutions into one of three categories. We select 29 problems from the CreativeMath dataset, considering the distribution of difficulty levels. Following the approach of prior work [12), we generate 20 responses for each input prompt using stochastic decoding. For each problem, n is the number of reference solutions available in the dataset. We attempted to ensure sufficient diversity in the reference set by providing diverse number of reference solutions for each problem. Because each reference solution for problem differs in both length and content, naturally resulted in variation in prompt length. Such diversity contributes to the effectiveness of the reference set as criterion for classification.
include up to reference solutions, where
Test Set Test set consists of the remaining 371 problems with solutions from CreativeMath. For each problem, three responses are generated using stochastic decoding. The number of reference solutions provided in the input prompt is limited to maximum of two.
Extended Test Set Extended test set is used to evaluate the generalization performance of the method. We utilize problems and solutions from four math competitions A(J)HSEE, AMC, and AIME compiled in the HARP, For each problem, one response is generated. The number of reference solutions included in the prompt is limited to maximum of two, consistent with the test set.

---

Held Guideline
Problem P Solations & Instruction /
Resporse R
Detection Perfermance
K
Ferwand
Hallacinated Solution
Craative
Typical Solution
Solation
& Nomn.
Multi-llead Atteation
Sectice wise attentio rato %%)
CLAWS
Evalustion Stritegy
N-1
Transformer Decoder
72
FT-1
Figure 3: Architecture of CLAWS. All tokens segmented into five sections: Guideline. Problem, Solutions, Instruction, and Response. The average attention weight for each section (AVGAu) is computed, normalized to obtain section- wise attention ratios (CLAWS)), and used features for detecting Hallucinated, Creative, or Typical solutions.
3 CLeWstCreativity detection for LLM-generated solutions using Attention Window of Sections
In Figure 3 we input the prompt X G|P into the RLM f and leverage the decoder attention weights during generation classify the class of the generated response / f(X). As described in Section 2.T X consists of four sections - Guideline (G), Problem (P), reference Solutions (S), and Instruction (I). additionally include the generated Response section (R), thereby dividing tokens into five distinct semantic segments. We hypothesize that the class of R is influenced by which sections the model attends to most during generation. Based on this hypothesis, propose CLAWS, a method that leverages the average attention weights for each section windows. Let A denote attention weights decoding time step from head last layer
(L) (L) (L) (L) |ai, 02hh aggh aete, where k len(X)
To construct the complete attention weight matrix A, stack the attention vectors over all time steps = 1 ... T. Since one output token is generated at each time step, the length of the attention target increases over time. To ensure consistent dimensionality, each A! is padded to fixed length of len(X) T.
..
(
E (1mm((())
2
We then compute the average attention weight for each section E G, P,S, R, by summing the attention weights over all tokens belonging that section, and averaging over all heads H, time steps T, and section-specific token positions Tzx:
AVGAU H.T. IT
ΣΣΣ
A 11, for 4= PG,P,,,,,R
Finally, to quantify the proportion of attention allocated to each section, we normalize the average attention weights AVGA into ratios and use them as input features for the evaluation model:
AVGAu
CLAWSH
EvelG.ns.I AVGAu
CLAWS extracts only the attention layer in the single response generation process and just performs sum and average operations, enabling effective detection without additional calls operations. This

---

Arplodd
1.27
Dpec
4.1
A08
5017
254
0383
Figure 4: Visualization of class-wisg wise average scores each method, computed on the reference set generated using between 0.1 and 0.9 for CLAWS. Visualizations for models are presented in Figure 9
To enhance visual clarity, the normalization range is clipped
is an efficient method with similar level of computational overhead to methods that only use existing hidden state or attention layers.
Evaluation
4.1 Baseline
In previous research, both black-box, including SelfCheckGPT [12], and white-box approaches that use uncertainty- methods [28] have been extensively studied to detect hallucinations LLM generations. In addition, there methods to mitigate hallucinated in LLMs through majority voting by generating multiple generations, such as Self Cofsisseent [29] or INSIDE 30 However, these methods have limitations, they depend external models and require multiple responses per input, resulting substantial computational cost. Therefore, these methods are not suitable as baselines for this study. In light of these limitations, employ five white-box hallucination detection methods as baselines in this study, without relying on external models or using majority voting, to distinguish between Hallucinated, Creative, and Typical solutions. The five methods can be broadly categorized into output token uncertainty quantification and eigenvalue analysis of internal LLM representations. These approaches have been widely used in prior research, and more recently, LLM-Check [3] has further consolidated these methods.
Perplexity Perplexity is measure of the confidence in the model's generation of the the response
x = [xn+1 ... Fm which is generated by the model f given an input prompt Xp- It is calculated based the ligelioel of each output token 2;, where m denotes the length of the generated response and n denotes the start index of the response being evaluated and is defined as follows:
1
Perplexity(x) = exp( m
Σ
log (xg
Xp 2 exxi)
Logit Entropy Logit Entropy represents the uncertainty in the generation by measuring the entropy of the probability distribution over the top-k tokens at each token position, based on the logits:
LogitEntropy(x. k) n
M
(
X, ex<r) logps(z! xp xxci).
(2)
Window Logit Entropy Since Logit Entropy calculates over the entire response, there is problem that the entropy value diluted and hallucinations cannot be detected when hallucinations is short. In order to address the problem, Window Logit Entropy is defined as the calculation of the logit entropy over sliding windows of size & and selecting the maximum value among them, as follows:
1
LindowLogg k w)
max
LogitEnt(1;; k))
(3)
SE.I.....m w+1]
Hidden Score Hidden Score is a measure of variance in representation diversity, computed by performing cigen-decoopossiionn the hidden state matrix H ædxm which consists of m

---

tokens represented embedding in dimensional space, and then calculating the mean of the logarithm
of eigenvalues. So, Hidden Score is defined as the mean log-determinant of I² where o; represent the singular values of H:
HiddenScore log det det(E²)
Σ log F where E² = H
(4)
Attention Score Attention Score quantifies self-attention by analyzing the diagonal entries of the self-attention kernel matrix:
AttentionScore
Σ log (Ker.)u)).
(5)
Ker denotes the j-th diagonal entry of the self-attention kernel matrix from i-th attention head.
4.2 Evaluation Strategy
The standard evaluation strategy for the five baselines is determine threshold that yields the optimal detection performance. However, the limitations of threshold-basee approaches have been consistently noted, and several studies have explored how to better leverage internal model features for detection. In this study, we evaluate the performance of CLAWS and the five baselines using five distinct evaluation strategies. Detailed hyperparameters and experimental settings are provided in Appendix B.3]
Threshold A threshold that yields the best performance among the values computed by each detection method is determined. This strategy applies only to the five baselines that use a single sealar value as feature.
Prototypes A prototype-based classification approach employed using the reference set 332. The
Euclidean distance between each class's 's center embedding and given sample is computed to predict the nearest class. This strategy applies only to CLAWS, which outputs multi-dimensional feature vectors.
XGBOOST tree-based classification algorithm based on Gradient Boosting Decision Trees (GBDT) is employed. XGBoost (33] serves as an advanced implementation of this approach.
to classify the features extracted by each method. The classifier implemented as a Multi-Layer Perceptron (MI.)..
MT.P A conventional trainable classifier is employed
TabM An ensemble-based classification method is employed to mitigate the large variance of MLPs. trains multiple MLPs and aggregates their predictions to generate the final prediction.
TabM 34
4.3 Evaluation Metrics
For a fair comparison, we employ four evaluation metrics: weighted Fl score (Flw), macro Fl score (Flm), Area Under the ROC Curve (AUROC), and macro Average Precision (AP,). Given the limited number of samples and the intrinsic difficulty of classifying the Creative class-whic lies between the Typical and Hallucinated classes-it is essential to adopt metrics that capture the model's ability to recognize rare and challenging categories. Furthermore, in cases where not all three classes were predicted, we separately marked instances where a model achieved an artificially high score by predicting only the majority class, indicated by gray highlighting in the tables.
5 Result
In Table [ five baselines were evaluated using thresholds, while CLAWS was assessed with prototype strategy; CLAWS outperformed all models on the test set across all four metrics. The full result table is presented in the Appendix C, using all strategies and metrics. In particular, CLAWS achieved superior performance Flm m and Am. metrics that measure the macro-average by giving equal weight to all classes. In contrast, none of the five basclines performed well on all models.
7

---

Table 2: Results for creativity detection. The evaluation strategies are Threshold (for PPL: Perplexity, WE: Window Entropy, LE: Logit Entropy, HS: Hidden Score, AS: Attention Score) and Prototype (for CLAWS). Bold values indicate the best performance, underlined values denote the second best, and gray-shaded cells correspond to cases where the model detected only two out of the three classes.
Dataset
TEST
AMC
AIME
ACDHSME
Model
Method
Fl,
Fl,
AP,.
AUROC
Fl,
Fla
AP,.
AUROC
Fl,
Fla
AP,
AUROC
F1
Fl,
AP,
AUROC
PPL
48.09
35.27
37.07
56.49
35.63
36.28
55.12
55.93
36.70
36.59
56.63
42.34
36,52
37.49
56.82
WE
18.59
23.20
35,89
53.89
27.52
26.03
34,67
52.26
9.07
16.36
33,67
50.31
28.72
27.70
34.44
51.92
L.B
40.56
33.21
34,00
50,90
35.69
32,96
33,42
50.31
28.99
28
33.27
48
34.89
33.46
33.45
50.18
DeepSeck
HS
29,56
25.18
32,40
45.03
38.44
32.96
33.60
50.22
38.30
65
33.71
50.67
38.61
35.80
34.54
52.40
AS
33.95
24.99
30.98
42,80
33,43
50.19
43.92
33.58
50.97
33.19
49.70
CLAWS
58.66
46,01
41.17
62.09
46.71
40.99
37.16
56,40
56,90
38,12
35.38
$4.47
38.82
37.64
36.25
$4,40
PPL.
42.45
25.94
31.37
43.26
36.50
25.21
31.81
45.89
56,90
29.97
32.76
47.49
34.79
25.58
32.58
48.15
WE
46.19
89
32.58
46.66
30.02
32.73
48.44
$2.20
30.20
32.91
48.33
31.79
33.86
51.05
LE
41.62
28.17
32.11
45.
35,20
29.55
32.05
46.95
44,77
28:56
33:47
50.50
35.46
30.56
32.40
47.77
Mathstral
HS
49.86
26.53
32.49
47.07
37.33
33.33
49.97
65.96
31.13
33.46
50.23
22.65
33.42
50.14
AS
38.41
24.50
31.23
42.22
36.92
27.53
32.02
46.69
57.35
31.95
33.51
49.82
35.26
27.57
32:41
47.60
CLAWWS
46.05
41.75
63.70
51.47
41.45
37.89
57.69
65.25
36.05
34,43
52.73
49.13
42.29
38.20
58.11
PPL
36.47
27.52
32.72
47,30
41.10
31.45
33.12
49.24
40.44
30.49
32.18
47.57
39.22
30.05
33.13
48.56
WE
40.89
32.14
33.84
50.50
43.44
34.48
33.93
51.19
40.55
31.17
33.37
50.00
42.45
34,16
33.99
51.08
LE
47.48
35.96
35.15
53.15
43.17
36.18
34.28
52.62
41.82
33.10
34.32
52.92
42.38
37,55
34.70
53.28
MpenMath2
HS
30,48
23.20
30.77
41.57
33.02
26.78
31.17
44.52
40.45
32.09
32.63
49.34
31.62
26.55
31.37
44.93
AS
33,20
24.48
30.65
42.17
33284
27.77
46.75
40.42
30.96
48.59
32.59
31.03
27.53
32.09
47.04
CLAWS
60.86
44.27
40.77
60.66
54.32
42.12
38.53
58.06
49,35
34.41
35.35
52,00
50.88
41.36
37.73
57.22
46.52
27.81
31.78
45. (1.
41.68
26.38
31,25
44.27
55.96
11
32.64
47.36
36,90
31.03
43.62
WF
49.57
27.39
32,80
48.26
44.87
27.32
32.87
48.82
66.65
32.63
33.48
51.28
36.37
24.79
32.79
48.44
LE
55,39
36.15
34.46
53.11
49,80
35.95
34:43
53.30
63.53
33.86
34,02
53.06
11.06
31.86
33.29
50.47
OREAL
HS
51,95
29.46
32.60
47,90
48.58
28.28
33.20
49.60
68.10
31.63
33.30
49.23
41.65
28.36
33.12
49.62
AS 45.56
28.24
31,83
45.56
47,92
29.11
32,70
48.25
65.19
32.74
33.20
49.56
40.18
26.41
32.74
48.48
CLAWS
54,19
40.18
38.15
59.46
43.83
34.77
35.57
54.78
59.95
32.74
33.81
51.55
35:70
31.93
35.51
54.41
PPL
25.66
23.30
31.76
42.62
26.40
21.39
31.71
43.31
28.29
25.29
32,00
44.52
24.88
20.34
32.09
44.79
WE
30.79
29.40
34.71
52.50
26.04
33.80
51.15
29.08
33.12
49.24
23.61
33.12
49.51
I.F
50.81
45.29
39.50
59.80
45,86
40.18
36.15
55.81
43,20
39.01
36.40
55.83
45,70
38.64
35.23
$4.09
Qwea-2.5
HS
30.67
28.31
36.25
54.53
47.57
31.98
34.81
$2.98
20.37
34.57
52.83
48.52
32.54
34.78
52.77
AS
30.75
26.96
32.05
45.53
38.61
31.55
32,93
48.78
37.32
33.71
33.92
51.42
33,72
28.55
32.86
48.35
CLAWS
50.35
43.37
39.88
59.32
52.77
41.39
37.45
57.59
39.05
32,31
33,08
49.31
47.90
36.04
35.86
54.94
However, CLAWS demonstrated superiority in Flm and APm. which reflect the macro-average, and showed even more pronounced performance in Fl which assigns more weight larger sample classes. This finding suggests that CLAWS not only effectively detects Creative solutions but also maintains robust performance in classifying Typical and Hallucinated solutions.
As shown in Figure 4 and Figure | the reason for CLAWS superior performance is clearly presented. The visualization illustrates the mean per method for each class based on 20 generations for the same prompt in the reference set. In the reference set, which is the basis for detection, the five baselines record averages that do not differ significantly by class. In contrast, CLAWS effectively distinguishes Creative solutions as distributed between Hallucinated and Typical solutions.
In Figure 4 the Hallucinated solutions are relatively attention in the Guideline and Problem sections, while the Typical solutions are attention in the reference Solution, Instruction, and Response sections. Figure 4 andde show that hallucination focus on the Guideline section in all models. This suggests that ballucinations may be caused by over- wer-ffousing on a part of the input prompt.
A similar result is observed in extended test set. CLAWS performed well in most models, except for the OREAL. This is because the OREAL has a much higher rate of generating Hallucinated solutions than others, as shown in Table Therefore, it was challenging to produce a prototype well, which consequently made the classification difficult. As shown in Table 20 however, it achieves significantly higher performance with evaluation strategies as MLP, XGBoost, and TabM.
8

---

Table 3: Results for hallucination detection. Threshold (PPL, WE, LE, HS, AS) and Prototype (CLAWS) are used as evaluation strategies. Bold and underlined values indicate the best and second- best performance. Gray-shaded cells indicate cases where the model predicted only single class.
Dataset
TEST
AMC
AIMR
ACDHMME
Model
Method
Fly
Fl,,
AP,.
AUROC
Fl,
Fl,
AP..
AUROC
Fl,
Fl,
AP,.
AUROC
Fl,
AP,
AUROC
PPL
37.63
40.32
45.13
53.06
52.18
45.38
62.33
51.95
39.75
42.77
36.16
$5.60
55.31
46,20
65.42
52.25
26.44
30.34
43.56
50.00
38.04
61.39
50.00
16.77
25.06
33.45
50.00
50.42
64.37
50.00
LE
27.66
31.46
43,81
50.50
46.82
38.19
61.40
50.03
17.22
25.35
33.36
49.81
50.37
39.12
64.33
49.92
DeepSock
HS
26.41
30:31
43.52
49.92
38.56
50.20
17.64
25.74
33.54
50.19
39.51
64.34
49.93
AS
26.60
30.45
43.43
49.72
46.68
38,02
61.37
49.97
16.77
25.06
33.45
50.00
50,59
39.43
64.38
50.03
CLAWS
67.46
67.24
55.73
67.78
61.77
59.79
66.64
59.84
61.95
58.17
38.45
58.68
64.93
61.67
70.38
61.62
PPL.
17.09
25.27
33.82
50.00
29.62
31.76
46.58
49.90
9.44
19:45
23,98
50.05
33.84
33.58
50.37
49.95
WE
17.09
25.27
33.82
50.00
29.71
31.84
46.62
49.90 99.996
9.61
19:57
24.00
50.11
33.89
33.63
50:42
50.05
LE
17.36
25.47
33.82
50,00
29.65
31.79
46.62
49.97
19.94
24.06
50.27
33,98
50.50
50.21
Mathstral
HS
17.03
25.08
33.48
49.
30.11
32.18
46.36
49.44
9.26
19:35
23.96
50.00
33.96
33,70
50.39
50.00
AS
17.09
25.27
33.82
29.66
46.64
50,00
19.33
50.00
50.39
50.00
CLAWS
72.99
69.59
49.42
69.30
63.97
63.90
$5.69
64.14
65.62
50.26
24.19
$0.59
61.05
61.04
57.10
61.04
PPL
31.78
34.68
44.16
49.74
15.53
10.17
58.19
49.51
34.68
36.15
45.52
47.20
19.49
41.63
61.65
48.76
WE
27.19
30.70
44.29
50.00
43.09
36.88
58.42
50.00
29.91
31,91
46.86
50.00
47.74
38.36
62.23
50.00
LE
28.14
31.55
44.39
50.20
43.18
36.99
58.41
49.97
30.38
32.95
46.89
50.06
48.09
38.81
62.32
50.21
OponMaih22
HS
27.55
31.01
44.27
49.95
43.40
37.27
58.43
50.01
31,30
33.21
46.95
50.17
47.68
38.37
62.12
49.78
AS
27.46
30.94
44.32
50.05
43.30
37.16
58.39
49.99 92
30.09
32.08
46.90
50.08
48.14
90
62.30
50.15
CLAWS
64,91
64.70
54.19
65.05
62,53
61.65
65.19
61.82
58.10
57.50
52.32
58.47
63.88
61.81
68.70
61.96
PPI.
16.96
25.09
32,61
49.23
21.37
27.87
37,65
49.67
6.79
16.08
18.33
49.13
28.09
31,19
44.51
49.32
23.20
29.92
33.39
50.99
22.04
28.48
37.93
50.26
10.95
18.75
18.58
49.96
31.27
44.88
50.09
LE
20.75
27.91
32.84
49.75
23.17
29.34
37,87
50.14
10.03
18.35
18.77
50.60
30.50
33:45
45.22
50.75
OREAL
HS
16.33
24.78
32.95
50.00
20.69
27.37
37.73
49.83
6.01
15.80
18.60
50.05
27.74
30.93
44.81
49.94
AS
16.47
24.78
32.71
49.45
20.99
27.59
37.71
49.79
6.17
15.85
18.55
49.87
28.04
31.11
44.32
48.94
CLAWS
58.13
56.36
38.36
59.87
53.10
53.06
41.17
56.34
64.02
49.22
18.98
51.22
$3.96
54.41
48.36
56.42
PPL
41.30
35.91
56.88
48,
41.30
35.91
56.88
48.7
33,74
33.64
50.05
49.81
76.90
46.45
84,45
49.38
WE
41.98
36.51
57.50
50.00
41.98
36.51
57.50
50.00
33,70
33.61
50.19
50.09
77.69
46.20
84.66
50.18
47.87
43.32
58.86
52.72
47.87
43.32
58.86
52.72
38.32
38.24
51.16
$1.99
84.62
50.05
Qwen-2.5
HS
42,12
3667
57.51
50.02
42.12
3666
57.51
50.02
33.49
33.40
50.14
50.00
77.53
45.82
84.60
49.97
AS
41,92
36.45
57.44
49.87
41.92
36.45
57.44
49.87
33,49
33.40
50.14
50.00
77.44
46.04
84.58
49.89
CLAWS
54.67
$3.30
59.21
53.35
72.13
55.12
80.75
54.82
47.08
47.10
49.11
47.83
74.68
52.33
85.25
52.44
Analysis
For the Hallucination Detection, we used Typical and Creative classes as Non-hallueinated classes in the dataset presented in Table As shown in Table 3 even though the five basclines are designed to detect hallucinations, CLAWS shows overwhelmingly superior performance. CLAWS showed overwhelming performance not only on the test set but also on three extended test sets, and in particular, on AIME, which contains many difficult math problems, it showed clear discrimination.
When comparing the models, Qwen, which exhibited the most balanced distribution between hal- lucinated and hon-hallueinated samples, consistently achieved high detection performance across all methods. In contrast, all five baselines performed poorly on Mathstral and Oreal, both of which produced approximately 1,200 Hallucinated solutions in the reference set, corresponding to a halluci- nation rate of nearly 70%. DeepSeck and OpenMath2 displayed distinct patterns depending on the dataset difficulty: in AMC and A(J)HSME, which are relatively easy datasets, the baselines achieved moderate performance, whereas in AIME, their performance dropped substantially.
To overcome these problems, We construct balanced dataset for creativity detection. For each reference set, a balanced dataset is constructed by including all samples from the minority class (Creative) and randomly sampling an equal number of instances from each of the two majority classes. For example, in the reference set of DeepSeek, there are 206 samples in the Creative class; accordingly, 206 samples are randomly selected from each of the Hallucinated and Typical classes to ensure balance. As shown in Table 16 CLAWS also achieved the best overall performance. In

---

Table 4: Impact of each prompt section. Perfor- mance comparison of the DeepSeek-Math-7B-RR using the Prototype strategy through ablation study, where each section is selectively removed.
Average Runtime per Method
00
Flw
58.59
58.81
50.01
54.68
58.39
0.526
Flm
46.01
46.35
39.45
43.57
45.97
46.01
APm
41.13
41.04
38.46
40.29
40.60
AUROC
62.12
62.21
59.15
61.61
61.58
62.09
Time
Flw
46.66
46.68
46.48
45.52
46.49
46,71
Ply
39.59
39.33
39.59
98.16
40.20
40.99
0.149
AMC
APm
37.15
37.08
36.91
36,70
37.14
37.16
AUROC:
56.11
56.23
56.00
55.80
56.52
56.40
0V1
WB
LE
115
AS
CLAWS
Flw.
55.88
53.33
54.92
51.37
54.01
56.90
Figure 5: Average runtime for computing input features of each method - PPL. WE, LE, HS, AS, and CLAWS. The Response (R) generation time, identical across methods, is excluded.
Fim
37.35
35.52
82
34,80
34.84
38.12
AIME
AP
35,25
35.20
34.98
35.003
35.15
35.38
AUROC
54,08
53.38
53.51
52.47
52.67
54.47
37.85
39.92
40.49
40.10
34.25
38.82
37,46
35.76
37.65
36.02
34.11
37.64
AUJHSME
AF
35.09
36.25
AUROC
55.05
53.88
54.33
54.18
53.08
54.40
particular, it performed best under the MLP strategy and exhibited the least performance degradation compared other methods when trained small amount of data.
Furthermore, we analyzed the effect each prompt section CLAWS. As shown in Table 4 CLAWS achieved the best overall performance across most datasets and metries when all five sections were utilized. Moreover, as illustrated in Figures and 9 the degree to which each section influenced generation varied depending the model. This observation suggests that leveraging all five sections provides the most stable and robust performance regardless of the dataset or model.
Lastly, We compare the runtime efficiency of our proposed method, CLAWS, with five baseline methods. Unlike other approaches, CLAWS does not require any additional mechanisms after the generation phase. Moreover, during generation, it performs only simple operations such taking the mean or sum of attention weights, without relying on complex computations. As shown in Figure 5 CLAWS demonstrates the highest efficiency among all baselines. Specifically, among entropy-based methods (PPL, WE, and LE). WE, which computes entropy separately for each window, shows the poorest efficiency, whereas LE, which calculates simple logit entropy, achieves the best efficiency in this group. Among layer-level methods, HS, which involves computing a transposed matrix and singular values, exhibits the lowest efficiency. In contrast, AS, which utilizes only the diagonal components of the self-attention kernel matrix, shows the best efficiency except for CLAWS. In conclusion, CLAWS not only achieves the highest creativity and hallucination detection performance but also exhibits the greatest computational efficiency, demonstrating superiority in all respects.
7 Conclusion
In this work, we presented a challenging study on creativity detection. To address this problem, we systematically investigated multiple key components, including the definition of creativity, the design of an experimental framework, the proposal of novel detection method, and the analysis of evaluation strategies for extracted features. These efforts provide a solid foundation for future research this emerging area. Moreover, extending hallucination detection toward creativity detection broadens the scope of LLM research beyond reasoning improvement, introducing new perspective on model evaluation and generation analysis. We anticipate that the proposed experimental framework and our method, CLAWS, will serve as a foundation future research on reliable creativity detection and the improvement of LLM generation quality and diversity.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Institute of Information and communications Technology Planning and evaluation (IIT)) grant (No.RS-202) 20..S22225-25422680, No. RS-2020-11201373), and the National Re- search Foundation of Korea (NRF) grant (No. RS-2025-0052066 funded by the Korean Government (MSIT).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Anthropic. claude-3-7-sonnet, 2025.
[3] Google DeepMind. Gemini 2.5, 2025.
[4] Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng
Wu. Art or artifice? large language models and the false promise of creativity, 2024.
[5] Murray Shanahan and Catherine Clarke. Evaluating large language model creativity from literary perspective, 2023.
[6] Paul Torrance. Torrance tests of creative thinking. Educational and psychological measure- ment, 1966.
[7] Roger Beaty, Paul Silvia, Emily C Nusbaum, Emanuel Jauk, and Mathias Benedek. The roles of associative and executive processes in creative cognition. Memory & cognition, 42:1 1866 1197, 2014.
42:11866
[8] Philippe Laban, Wojeiech Kryšciński,, Divyansh Agarwal, Alexander R. Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. Llms as factual reasoners: Insights from existing benchmarks and beyond, 2023.
[9] Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. Human like summarization evaluation with chatgpt, 2023.
[10] Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on Ilm generalisation and diversity. arXiv preprint arXiv:2310.06452, 2023.
[11] Junyi Ye, Jingyi Gu, Xinyun Zhao, Wenpeng Yin, and Guiling Wang. Assessing the creativity of Ilms in proposing novel solutions to mathematical problems. arXiv preprint arXiv:2410.188 1021)00.8336 2024.
[12] Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black- box hallucination detection for generative large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 9004-9017, Singapore, December 2023. Association for Computational Linguistics.
[13] Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8011):625-633) 2024.
[14] John Joon Young Chung, Ece Kamar, and Saleema Amershi. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. arXiv preprint 20w:::360000444000 2023.
[15] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
11

---

[16] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023.
[17] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
[18] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu. Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement, arXiv preprint arXiv:2409, 12/22, 2024.
[19] Mistral AI. Mathstral 7b, 2024.
[20] Shubham Toshniwal, Wei Du, Ivan Moshkov, Bramislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-22 Accelerating ai for math with massive open-source instruction data, 2024.
[21] Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang. Shuaibin Li. Qian Zhao, Haian Huang. et Exploring the limit outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2.502.677 2025.
[22] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 2/ 1783, 2024.
[23] Qwen, :, An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
[24] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint ar arXiv:2405.0m 04434, 2024.
[25] OpenAl. 04-mini, 2025.
[26] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint vrXiv:2403.005540 2024.
[27] Albert S. Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse, and Aaditya K. Singh. Harp: A challenging numan-annotated math reasoning benchmark. 2024.
[28] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1-38, 2023.
[29] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023.
[30] Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. Inside: Lims' internal states retain the power of hallucination detection, 2024.
[31] Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. Lim-check: Investigating detection of hallucinations large language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances Neural Information Processing Systems, volume 37, pages Curran Associates, Inc., 2024.
12

---

[32] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[33] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '16 page 785-794. ACM, August 2016.
[34] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing Tabular Deep Learning With Parameter-Efffcientt Ensembling. In ICLR, 2025.
[35] Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng false promise of creativity. In Proceedings of the 2024 CHI Conference on Human Factors Computing Systems, CHI '24 New York, NY, USA, 2024. Association for Computing Machinery.
Wu, Art or artifice? large language models and the
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The abstract and introduction clearly articulate the scope and core contributions of the paper, accurately reflecting the proposed framework and findings.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are explicitly discussed in the Appendix A along with direc- tions for future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: All theoretical assumptions are clearly stated in Section 3 and their validity is examined in Section | and supported by empirical tables in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Detailed descriptions of the prompts, the mathematical dataset, and the experi- mental procedures provided in Section and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Both the dataset and source code included in Supplementary Material. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ctipp.ccc/uublic//gr guwdes/ddeeeee CooeeSumeeeeeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Training and evaluation details including data preprocessing, hyperparameter configurations, and optimization choices are described in Section 2 and Appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
The statistical significance of the experiment is presented in Section 4.3 where appropriate metrics are reported. Additional statistical analysis and supporting details are provided the Appendix C
Justification;
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Implementation details including computing infrastructure, and memory specifications are reported the Appendix [ Furthermore, the average runtime required to compute features for each of the six methods in summarized in Table 5 offering clear comparison of computational costs across methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:s////aarrrrs..cc/o//olllttd
Answer: [Yes]
Justification: The research adheres the NeurlPS Code of Ethies by employing publicly accessible, non-personal mathematical datasets, excluding human subjects or surveillance data, and ensuring transparency and reproducibility through a structured code release process.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
17

---

The paper discusses potential positive and negative societal impacts in both the section (Introduction) and section 7 (Conclusion).
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not introduce any new language models or datasets that could pose risk of misuse. It uses only existing publicly available reasoning models and math datasets to evaluate creativity detection method.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper uses publicly available datasets (CreativeMath, HARP) and existing open-source reasoning L.Mss.
Guidelines:
The answer NA means that the paper does not existing assets.
18

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessiithheeeocc thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pase re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any new assets such datasets, models. proposes a detection method evaluated using existing public datasets and pretrained models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: The paper does involve crowdsoureing or human subject research, as all evaluations are conducted using LLM-based based assessments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
19

---

Justification:
The paper does not involve any human subjects or participant participant-based studies, therefore IRB approval not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper makes essential and original use LLMs both as solution generators and evaluators, and explicitly describes their usage in Section 2.2.1
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (ttppe///eerrppeccc///offrrrrrrr for what should or should not be described.
20

---

Appendix
Limitations
We have successfully experimented with many RLMs, but have not been able to experiment with general LLMs of similar size because they do not have sufficient creative bilities. Due to the nature of the White-Boo Box approach, using large-sized models (over 20B) with sufficient performance requires lot of resourees. Lastly, defined 'Creativity" based solving mathematical problems, and expanding it to various tasks will be our future work.
capa-
B Implementation Details
B.1 LLM Evaluator Details
We use an LLM-based evaluator to classify each generated response R into three categories Hallucinated Solution, Typical Solution, Creative Solution following the evaluation protocol introduced in [.. As described in Section 2.2.2 and Figure D the evaluation process consists two stages. First, assess whether the generated response R is mathematically correct. For example, As shown in Figure the evaluator is given two reference solutions and asked to determine whether R valid solution to the given problem.
both evaluators agree that correct, response proceeds to the second stage, where is further classified as either a Typical Solution or Creative Solution. As shown in Figure this decision is made by comparing R against the reference solutions S provided in the original input prompt X, and determining whether it satisfies the criteria outlined in the guideline G.
we used two LLM evaluators. In particular, instead of using GPT-4o one of the evaluators, we adopted o4-mini, which demonstrates superior performance in mathematical evaluation. The LLM evaluators used our study are listed below:
Unlike M.
* GPP-o44mini: models/geminge 5-00-o00002
Gemini-L.5-Pro:
B.1.1 Justification LLMM-based Labeling without Human Evaluation
In this study, we adopted LLM-based labeling instead of human evaluation for dataset construction. The justification is as follows.
Practical constraints The dataset used in our study consisted of mathematically challenging problems, where evaluating the creativity of generated solutions would require human experts with substantial mathematical expertise. However, as shown in Table [ conducting human evaluation all 46,528 data instances would have required excessive time and cost, making it practically infeasible.
Support from prior work Recent studies have validated the of LLMSs for creativity assessment 11 355.
Following these works, we adopted their definitions and evaluation criteria to construct our dataset without relying on human evaluation.
Indirect reflection of human creativity As mentioned earlier, creativity evaluation required a reference solution S. Importantly, was always taken from numan--ritten solutions and never generated by an LLM. Thus, although no direct human evaluation was performed, the diversity and creativity embedded human aomannwritten solutions were indirectly reflected in the evaluation.
Ensuring reliability To enhance the reliability of LLM-based evaluation, we designed the LLM Evaluator not only to output labels but also to provide explicit reasoning. Specifically, we appended the instruction "Additionally, explain the reason for your judgment." at the end of the evaluation prompts (see Figure and 7 Figure shows actual evaluation cases, confirming that the judgments of the LLM Evaluator were consistent and reasonable.
21

---

Criteria for evaluating the difference between two mathematical solutions include:
D. If the methods used arrive at the solutions are fundamentally different, such algehraic manipulation vensus geometric reasoning, they can be considered distinct:
Given following mathematical problem: What is the largest power of 2 that is diviser of (1344-1414
ii). Even if the final results are the same, if the intermediate stops or processes involved reaching those solutions vary significantly, the solutions can he considered different;
Reference solutions:
ii). If tw solutions rely on different assumptioos or conditions. they are likely to be distinet:
Solution 1:
The factons of 36 are 1, 2, 3,4, 6,6%, 12, 18, and 36. The multiples of
iv). A solution ght generalize to a broader class of problems, while another solution might he specific certain conditions. such cases, they are cousidered distinct:
up to are 4, 8. 12, 20, 24, 28, and 36. Only 4. 12, and 36 appear on both lists, the answer 3, which option (B)
v). one solution significanlly simpler other, they can be reganded as essentially different, even if l'they lead the same result.
complex than the
Solution 2:
36 42 3", All possible factors 36 will be here, except for ones
divisiibee by 2 and not by 4. (1+1)- (2+1)-6 6. Subtmet factors not
Given the following mathematical problem:
divisible by 4, which are 1,3%, and 3". 6-3-3, which is B)
What largest power that divisor (13/444
Reference solutions:
New solution:
Solution 1:
To solve the problem of finding how many positive factirs of 36 are also multiples of,, we determining the prime factorization of 36 and then identiffing the
The factors of 1, 2,
18, and 36. The multiples of up to 36 are 4, 8, 12, 16, 20, 24, 28, 32, and 36, Only 4, 12, and 36 appear both lists, the answer 3, which option (B) 3].
systematic approach first
factors that multiples of
Factorization
[36 116-2% 22x 3-]
New solution:
Identiyy the General Form of Factors: Any factor 36 can be written the form (2* 33), where
To solve the problem finding many positive factors of 36 are also multiples of we can use more systematic approach by first deternining the prime factorization of
2) and (0 s b 3. Determine the Conditions for Multiples 4: For a factor to be a multiple 4, must contain at least (29) in its prime factorization.
identifying the
factors that are multiples of
Therefore, (a) must be least 2.
I. Prime Factorization of 36: [36 2 x3'
Possible Values for (a) and (b):
Identify the General Form of Factors: Any factor of 36 can be
-Since (a) must be least 2. (a) can be either 2.
written in the form (2 3"), where (0 i s 2) and (0 is b 2). Determine the Conditions for Multipls 4: For factor be
-(b) be
5. List All Possible Factors:
multiple of 4, must contain at least (22) in its prime factorization.
For 2y:
Therefore, (a) be least 2.
-(b-0): 0): (2 x44)
4. Possible Values for (a) and (b):
-(b- x22333311
-Since (a)
least 2, can be either 2.
-(b- 2): (2= (22xxxxx-3))
-(h) can be 0, or 2.
Count the Factors: There factors of that are also multiples
.List Possible Factors:
of 4:: (4, 12,) and (36). Therefore, the number of posirive factors of
-For (a-2):
36 that are also multiples of (B).
-(b 1 0): (2 44)
-(b- 3223
Please output "YES" the new solution the sume results the reference solutions; otherwise, output "NO", YES or NO?
-(b (b--2y (2 32 36)
6. Count the Factons: There are fictors of 36 that are also multiples
of (4. 12.) and (36). Therefore, the number of positive factors of
36 that are also multiples of (8)).
Figure 6: Example prompt used for correctness evaluation
Please output "YES" the new solution novel solutions: otherwise, output "NO". YES or NO?
Figure 7: Example prompt used for novelty (cre- ativity) evaluation
Addressing potential limitations We acknowledge that LL-M-based evaluation may involve in- trinsic biases. However, we mitigated this concern by grounding evaluations on Human-written reference solutions and by explicitly requiring reasoning for each judgment, thereby reducing the risk of arbitrary or biased labeling.
B.1.2 Criteria for Creativity Judgment
To determine creativity, we combined the results of the two LLM evaluators using a criterion: if either evaluator judged a solution as creative, we labeled it as a Creative Solution. Since creativity inherently involves a certain degree of subjectivity, requiring agreement from both evaluators could risk overlooking genuinely creative responses. By applying the union- based criterion, we aimed to mitigate the risk of underestimating creativity.
To verify the reliability of this approach, we measured Cohen's kappa score between the two evaluators. The score was 0.741, which corresponds to substantial agreement, indicating that the two evaluators produced highly consistent judgments.
22

---

Case 1: LLM Evaluator Output Creative
Case 2: LLM Evaluator Output Not Creative
The new solution uses a completely different approach (Lifting The Exponent Lemma) compared to the reference solutions (difference of squares factorization). This satisfies criterion I for novelty. While the final result is the same, the intermediate steps and the mathematical tools used are entirely distinct, satisfying criterion The LTE lemma also relies different underlying principles and conditions than simple algebraic manipulation, satisfying criterion 3. Therefore, the new solution considered novel.
The new solution is not novel. uses the same logical deductions the reference solution. Both solutions arrive at the same answer through the constraints. The core next Bret, and Abby not being between Bret and Carl- is identical. The new solution merely restates these conditions and elaborates on the placement possibilities without introducing any new mathematical techniques, different assumptions, or more generalized approach. Therefore, it does not meet any the criteria for novelty.
Carl not being
Figure 8: Example outputs from the LLM evaluators for determining creativity. The evaluators not only provide a binary judgment (Creative Not Creative) but also justify their decision with textual rationale based on the predefined criteria.
B.2 LLM Generator Details
For all LLM Generators, the maximum input token length was set to 2000, and the maximum output token length was limited 1023. Top-p was fixed 1.0, and Top-k was fixed 50 across all models. Temperature values were adjusted for each model to encourage the generation of Creative Solutions, and the final settings used for dataset construction are follows:
DeepSeek-Math-7B wwtpsseeeaaa///aaapaaaaaaaaaaaa. 0.7
Mathstral-7B htt.r//aaaaa//aaaaaaaaaaaa-0b000 0.25
OpenNNath--LJaMAA..1BBB (wwiaa/iiiiiiaaaaaaaaaaaaaaaaa
1-8B): 1.0
OREAL-7B (internimm/0RBAL-BBB)
Qwen2.5-Math-77 (Qwen/ Maaa:255-MMMt---IIItttu 0.7
All generations were performed parallel on eight NVIDIA RTX A5000 GPUs (24GB VRAM).
B.3 Evaluation Strategy Details
We evaluate methods using variety of Evaluation strategies, including thresholding, distance- based prototype matching, and trainable models such as MLP, TabM, and Decision-tree based XGBOOST. The implementation and hyperparameter settings for each method are summarized below.
Threshold (only for Baselines) We divide the value range of each baseline measure into 200 intervals and evaluate performance at each threshold. The threshold that achieves the best macro-f1 score on the reference is selected for final evaluation.
Prototype (only for CLAWS) We used an Encoder consisting of two Linear Layers for the prototype-based evaluation method. The input dimension is reduced 16 dimensions through the first Linear Layer and reduced to 8 dimensions through the second Layer. After that, it is expanded to 16 dimensions again and reduced to the dimension corresponding to the final number of classes, and used as the output. The output of each data was averaged by class and used as the class center value. Afterwards, the Euclidean distance between each data sample and the class center was calculated to predict the closest class. We generated prototypes of the reference set using 20 different random seeds and presented the one that achieved the best macro-f1 score.
For classification, adopt the XGBoost algorithm work based on ensembles of decision trees. In the hallucination detection setting, the model optimizes the logistic loss function, which corresponds the binary cross-entropy objective. For creativity
XGBOOST
33 gradient boosting frame-
23

---

Table 5: Comparison of feature generation using attention weights from different layers. CLAWS, leverages the last-layer attention weights, and this table shows how performance changes when attention weights are taken from other layers. Results based applying the Prototype strategy to responses generated by DeepSeek- DeepSeek-Math-77B- Math-7B-RL and Qwen-2.55Matth on the test dataset. Metrics weighted F1 wF),), macro FI m)) macro average precision (AP,, m),, and AUROC.
DeepSeek
Qwen
Layer
F1 w
Flm
APm m
AUROC
Layer
Fl w
F1,
AP m
AUROC
5
57.00
45.02
40.15
60.58
5
48.37
42.05
37.12
57.82
58.03
45.33
40.42
61.40
48.
42.48
38.01
58.65
15
56.94
44.76
39.81
60.22
15
50.64
43.31
39.45
59.10
20
57.89
45.61
41.08
61.93
20
49.85
42.71
38.74
58.98
25
57.25
44.98
40.41
61.37
25
50.01
43.25
39.21
59.26
Last(30)
58.66
46.01
41.17
62.09
Last(28)
50.35
43.37
39.88
59.32
detection, minimizes the softmax cross-entropy loss, producing discrete class label corresponding to the highest posterior probability.
MLP We three-layer ferdad neural network. The model is trained for 10 epochs using cross-entropy loss with class weights to account for class imbalance. Optimization is performed using Adam with a learning rate of 0.001. Depending on the number of classes and the input feature dimension, the model contains up to 133 learnable parameters. We experimented with 20 different random seeds and presented the one that gave the best macro-fl score.
TabM We use TabM [34], an ensemble of 32 independently parameterized MLPs. Each MLP has 5 512 parameters. model in the ensemble is trained for 20 epochs using cross-entropy loss. Optimization is performed using Adam W with learning rate of 2e-3 and a weight decay of 3e-4.
C Experimental Results
C.1 Layer-Wise Analysis
We present the layer-wise performance evaluation in Table The results show that there are no significant differences in performance across layers, and this trend is consistent across all models and metrics. These findings indicate that CLAWS does not rely on any specific layer and operates robustly under various configurations.
C.2 Visualizations for Reference Set
Figure presents class-wise average scores across different methods on the reference set for four models. Results for 2.5e--255Math- are shown separately Figure
C.3 Full Table of Creativity Detection
The results of 3-class detection for each model using different evaluation strategies (see Section 422 are presented Tables T0 The results for the Threshold and Prototype-base based methods are shown separately in Table 2
C.4 Full Table of Hallucination Detection
The results of 2-class detection for each model using different evaluation strategies are presented in The results for the Threshold and Prototype-baa methods are shown separately in Table
Tables 15
24

---

Dpeck
2,57
2013
CBC
DeepSock SaNSSScc--Mt------ Math-7B
k0/ boyly Ba)
Viakk mice
w4
Ohalte
0,71
(b) Mathstral-7B
Pertesty (2:1
Vradtre CIECOY
Loga mremo
Ascuee ICE Libelie
Traed
Casdii
124
255
XH
(c) OpenMath2 1aAaMMtth2-LLMMAS LLaMA3.1-8B
Taplonky
Palie
CLAWS
1.65
6233
1204
B-45
428
Ouatae
G
0BX
0,00
(d) OREAL-7B
Figure 9: Visualization of class-wise average scores across all evaluation methods for four models on the reference set. For CLAWS, the scores are normalized and clipped to the range [0.1, 0.9] to enhance visual clarity.
C,5 Full Table of Creativity Detection for Balanced Reference Set
The results of 3-class balanced detection using the Threshold and Prototype-based methods shown Table I6 followed by model-specifie results using other evaluation strategies in Tables 17
25

---

Table 6: Evaluation results for creativity detection using SeepSeek-Math-i Bold values indicate the best performance, underlined values indicate the second-best. Light gray- shaded cells correspond to results where the model performed detection over only two out the three target classes, while dark gray-shaded cells indicate cases where the model predicted only out of the three target classes.
Dataset
TEST
AMC
AIME
ALDHSME
Strategy
Method
Ft,
Fl,,
AP,
AUROC
Fl,
Fla
AP,.
AUROC
Fl,
Fl,,
AP,
AUROC
Fl,
Fl,
AP,
AUROC
PPL
50.52
36,08
37.52
56.82
40.34
32,65
36.65
53.92
55.57
34.55
36.68
55.38
38.30
33.32
37.23
55.15
WE
59.39
12.20
43.01
63.81
41.53
33.75
38.03
57.36
56.51
31.67
35.92
55.01
58.52
33.52
37.16
56.25
LE
45.62
34.29
34.82
52.23
37,96
31.41
34.63
51.27
50.71
33.27
35.18
52.83
34,09
30.
34,80
52.29
XGBOOST
HS
49.14
37.55
38.40
56.61
39.37
32,70
34.52
51.19
47.59
31.38
32.97
49.50
35.47
31.85
34.74
51.74
AS
53.58
38.76
42.19
63.53
38.0
30.36
34.31
51.09
51.56
66
33,97
49.75
33.85
28.
86
33.85
50.40
CLAWS
57.85
45.59
47.98
68.39
44.37
37,67
38.57
56.99
51.12
34.53
38.65
57.31
39.04
34.49
37.23
54,88
PPL
39.38
56.92
35.93
36.36
39.57
57.42
42.32
56.56
31.67
38.62
33.61
55.99
35.47
35.43
30.22
35.41
MI.P
42.00
62.98
35.61
52.59
32.55
32.00
34.58
36.61
50.68
32.78
CLAWS
50.08
43.78
34.84
60.11
42.26
39.68
PPL
51.79
37.78
41.03
60.03
43.76
35,20
39.29
56.83
56.84
35.80
41.01
61.73
40.89
35.30
39.67
57.42
WE
59.39
42.20
42.93
63.27
41.53
33.75
38.00
57.13
56.51
31.67
36.16
51.97
38.62
33.61
37.17
56.13
LE
42.80
31.66
35.39
53.49
40.44
32.32
35.45
$1.89
48.06
30.82
36.39
54.66
37.27
32.04
36.73
53.64
TabM
H8
$1.25
36.33
42.64
63.04
38.75
31.17
34.86
52. 60
48.77
29.56
32.40
48.14
34.58
29.
35.34
$1.85
AS
51.05
37.63
42.83
63.13
36.61
28.87
33.97
50.77
46.90
30.25
33.76
49.82
32,78
27.70
33.68
49.85
CLAWS
60.78
45.03
51.37
72.82
46.46
37.56
41.06
59.26
57.05
37.77
41.44
60.73
40.81
35.
41.25
59.12
Table 7: Evaluation results for creativity detection using Mathstral-27
AIDHSME
Strategy
Method
Fl,
Fl,
AP,
AUROC
Fl,,
Fla
AP,
AUROC
Fl,
Fl,,
AP..
AUROC
Fl.
Fl,
AP
AUROC
PPI
55.02
31.81
36.77
56.43
41.89
29.03
34.58
51.87
64.84
31.95
34,91
54.42
38.87
28.83
35.90
$3.20
WB
58.25
34.05
44.34
67.71
45.13
31.05
41.15
61.71
65.90
29.74
35.19
54.33
41.84
31.38
39.03
58.88
LE
52.67
29.93
33.46
49.70
43.28
30.38
36.59
54.33
63.76
31.62
32.94
49.05
37.13
27.49
34.06
51.18
XGBOOST
HS
59.40
38.04
41.71
62.34
45.00
34.05
37.12
54.68
61.85
30.67
32.82
49.71
38.97
29.97
35.07
$2.29
AS
57.64
34.26
41.54
60.55
41.61
28.15
39.25
57.78
65.34
28.87
33.94
49.11
38.13
28.34
36.23
53.93
CLAWS
62.05
42,09
44.25
64.43
50.92
38.85
40.31
$7.69
69.30
40.42
41.21
59.53
47.04
38.75
40.66
59.06
PPL
53.42
32.71
39.61
59.24
36.93
28.62
36.25
52.83
54.76
33.05
34.34
52.57
34.71
26.01
35.11
51.70
WB
49.45
28.16
37.02
55.53
37.14
23.20
35.41
52.17
65.47
28.70
34.48
$2.90
33.06
22.45
33.79
49.80
LE
49.54
32.89
37.04
56.02
46.20
34.91
36.55
54.41
57.77
31.25
32.55
48.89
38.70
31.36
34.14
51.37
ML.P
HS
60.32
40.84
45.48
68.58
38.35
30.42
39.90
58.59
63.39
32.59
33.02
50.08
38.05
30.74
35.76
$3.95
AS
62.82
42.48
44.00
63.55
19.70
37.03
38.63
58.10
61.26
33.63
33.59
50.32
41.86
33.38
37.48
55.85
CLAWS
64.65
45.34
47.74
70.25
$2.01
41.33
41.89
61.03
69.54
39.70
41.51
61.86
45.81
40.58
42.67
60.58
PPL
$2.72
26.55
38.26
59.43
37.14
23.20
38.09
56.46
65.
28.
54.48
39.68
28.93
32.67
48
WB
26.55
55.54
53.40
35.90
26.74
54.93
47.47
TabM
42.69
45.11
35.81
39.97
50.08
36.14
57.52
33.81
66.28
66.
CLAWS
43.67
37.58
58.42
46.14
26

---

Table 8: Evaluation results for creativity detection using OpenMath2-LLLaMAA.
Dataset
TEST
AMC
AIME
AUJHSME
Strategy
Method
Fl,
Fla.
APo
AUROC
Fly
Fla
AFe
AUROC
Fly
Flix
APs.
AUROC
Fly
Flu
APs.
AUROC
PPL
47.78
33.14
34.54
52.60
42.29
31,00
33.16
49.65
46.38
32.22
32.55
50.81
40.45
31.02
34,02
51,25
WE
58.84
40.14
43.09
63.58
49.28
36.50
40.17
58.33
43.16
28.33
35,66
$2.80
47.00
36.
38.15
57.20
LE
47.12
33:47
34.59
50.79
44.35
33.34
34.88
52.15
44.58
30.92
33,01
48.48
39.81
30.93
33.49
49,91
XGBOOST
H8
54.90
38.46
41.22
60.99
48.40
35.39
37.51
54.81
48.65
34.10
34,91
$1.78
45.77
35.26
37.85
55.67
49.95
37.26
54.48
35.32
44.56
33.59
52.01
CLAWS
63.20
46.23
48.16
67.22
54.56
40.24
42.68
56.87
40.82
42.51
59.67
50,40
38.13
42.63
61.47
50.26
30.08
51.08
WB
38.02
37.93
32.85
35.74
52.96
23.13
33.27
43.96
33.90
34.99
$1.99
33.52
41.39
MLP
HS
57.11
40.73
46.31
66.72
50.02
36,02
40.98
58.74
49.78
35.33
36.12
52.65
48.00
35.
39.40
56.58
40.33
54.72
34.53
46.34
52.32
CLAWS
65.32
47.69
52.80
73.42
58.31
43.82
48.59
67.67
58.11
42.09
44.17
61.44
52.72
42.31
45.52
64.63
PPI.
35.59
26.46
31.59
47.64
40. 66
28.92
32,32
49 99.46
39.10
27.92
29.78
43.42
32,90
22.11
35.43
52.59
WE
58.46
40.05
40.11
60.38
48. 15
35.64
36.65
54.54
45.85
30.55
35.01
51.67
33,20
22.42
34.08
50.31
LE
47.65
33.30
35.52
32.44
45.31
33.11
34.86
52. 12
44.65
31.01
31.60
45.48
33.40
22.78
35.11
52.27
TabM
H8
58.51
41.39
46.16
66.42
52.12
37,80
41.05
$8.90
50.78
35.73
36.14
52.56
41.22
31.66
36.10
54.01
AS
57.09
39.35
42.58
63.30
51.33
37.50
38.17
55.22
47.05
32.05
35.16
51.83
39.04
28.53
37.16
$5.69
CLAWS
66.45
46.83
52.95
73.17
58,70
42.78
47.88
66.93
58.92
40.95
44.10
60.67
$4.20
41.15
46.42
64.95
Table 9: Evaluation results for creativity detection using OREAL-
Dataset
TEST
AMC
AIME
AUDHSME
Strategy
Method
Fls
Fla
APs.
AUROC
Flu
Fl.
APe.
AUROC
Flu
Fly
AP..
AUROC
Fle
Fla
AP..
AUROC
PPL
56.08
31.31
35.98
54.06
51.46
30.83
36.59
56.38
72.38
33.10
33.81
50.45
44.41
29.62
35.92
$3.99
WE
59.33
33.74
36.28
54.38
55.08
33.66
36,80
55.13
73,10
30.26
33.47
51.08
50.95
34.42
37.19
54.80
L.
54.29
28.42
33.05
48.22
50.04
28.78
33.53
50.16
72.80
32.68
33.59
50.18
43.16
27.57
33,80
$1.14
XGBOOST
HS
61.54
38.44
39.26
58.61
57.30
38.33
38.29
56.62
70.42
33.57
33.90
53.02
52.60
36.87
37.58
55.27
$3.83
26.76
36.00
$3.50
47.70
25.56
35.93
$3.68
23.077
29.92
34.63
55.29
39.22
23.70
34.03
50.52
CLAWS
62.41
40.83
42.20
62.33
54.03
37.02
38.32
57.29
70.94
33.49
33.87
49.05
49.02
35.11
36.37
54.38
37.54
36.16
41.35
26.76
33.11
47.
49.
29.92
33.51
23.70
50.36
26.72
47.70
MLP
37.18
48.60
56.63
31.67
33.29
23.70
33.37
50.04
55.57
53.61
CLAWS
60.12
42.39
51.30
37.14
40.19
59.47
66.67
34.20
33.83
50.25
48.76
37.00
59.78
PPL.
53.83
26.76
37.84
57.08
47.
25.56
38.36
58.89
23.07
29.92
35.26
$6.47
39.22
23.70
38.30
$7.69
WE
59.16
34.17
35.25
$2.75
55.08
33.66
36.22
54.09
73.10
30.26
33.35
50.55
50.95
34.42
36.90
54.54
L.P.
53.83
26.76
31.97
46.90
47.
25.56
32.06
49.43
23.07
29.92
32.20
47.67
39.22
23.70
34.54
$2.61
TabM
HS
62.28
38.75
40.26
58.73
$7.60
38.44
39.42
$7.29
71.32
33.38
33.59
51.54
52.55
36.74
38.95
56.34
A8
$3.83
26.76
39.79
$8.17
47.90
25.56
37.33
55.00 62
73.07
29.92
33.94
49.
39.22
23.70
39.28
56.80
CLAWS
62.86
39.09
42.41
63.48
55.53
37.45
38.75
58.23
71.21
33.92
35.08
$1.07
$1.27
36.49
38.07
57.23
27

---

Table 10: Evaluation results for creativity detection using Qwen-2.5 2.5-Math-7B
Dataset
TEST
AMC
AIME
AJJHSMEE
Strategy
Method
Ft,
APa.
AUROC
Fly
Fl,
AP,.
AUROC
Fly
Fl,
AP..
AUROC
Fl,
Fl,
AP,
AUROC
PPL
41.49
37.65
37.82
55.39
34.63
39.15
43.47
63.00
36.57
32.17
55.51
38.22
32.42
33.83
31.06
35.19
33.38
XGBOOST
39.95
41.80
57.98
37.94
41.31
50.87
33.46
37.91
32.76
33.74
29.96
CLAWS
47.72
50.30
38.98
61.11
39.75
47.54
36.02
59.45
PPL
54.27
46.34
47.90
67.59
50.01
38.01
43.58
61.96
47.58
16.13
41.91
62.02
44.92
37.00
38.23
57.72
WE
45.67
39.58
38.31
54.97
43.32
32,95
36.49
53.87
40.26
30.78
34,22
51.26
42.11
33.31
34.35
50.53
LE
29.48
23.10
28.27
41.43
36.53
26.33
33,96
49.47
31.62
25.13
31.19
47.00
32.30
23.38
35.14
50.73
MLP
H8
54.58
42.77
46.98
65.88
50.29
36.64
41.90
60.61
43.11
32.96
36.01
52.38
46.24
36.2
37,80
56.01
A8
43.80
38.05
40.62
59.13
43.04
32.73
35.52
52.88
39.07
30.08
32,98
47.71
39.86
30.64
34.43
51.33
CLAWS
53.78
45.32
50.44
67.98
52.51
39.65
43.06
63.20
42.76
36.59
39.32
56.49
53.75
41.05
42.29
62.61
PPL
54.53
42.74
47.13
65.44
50.83
38.45
43.07
60.61
48.30
37.01
43.37
60.80
45.58
34.28
38.54
$7.10
WE
48.15
37.73
40.35
58.70
46.62
33.89
36.59
55.06
38.66
30.12
34,82
$1.49
45.58
31.17
34.81
52.06
LE
33.24
26.04
32,55
50.24
41.43
28.20
34.61
52.39
31,50
25.36
33.98
50.30
38.81
25.93
35,89
52.41
TabM
H8
51.47
41.39
45.50
63.64
50.08
36.79
41.58
61.00
43.75
34.48
36.73
53.74
46.01
33.00
37.82
$5.41
AS
45.76
35.86
38.65
56.76
43.53
31.74
36,03
54.26
37.84
29.68
33.06
49.16
41.37
28.
35,60
54.28
CLAWS
$1.93
42,92
45.82
63.83
48.69
38.90
39.23
59.61
45.14
38.28
38.15
55.26
47.31
35.97
39.23
59.28
Table 11: Evaluation results for hallucination detection using DeepSeek-Math-7B. Bold and under- lined values indicate the best and second-bes performance, respectively. Light gray gray-shaded cells indicate cases where the model predicted only single class in 2-class detection setting.
Dataset
TEST
AMC
AIME
AUDHSME
Strategy
Method
F1,,
Fl,
AP,.
AUROC
F1,
Fl,
AP
AUROC
F1,
Fla
AP,
AUROC
F1,
Fla
AP,
AUROC
PPL.
56.31
56.08
52.18
59.45
54.76
53.53
66.15
55.92
57.65
53.32
40.16
56.75
58.61
56.18
70.04
58.75
WE
65.53
63.87
60.55
68.13
50.79
52.21
68.78
60.84
59.51
49.52
38.21
55.66
53.27
53.41
70.15
59.51
LE
50.45
50.49
44,87
52.12
52.12
50.32
62.19
51.76
53.25
50.66
36.64
$3.36
54.86
52.15
65.98
$3.49
XGBOOST
HS
57.91
57.84
49.82
58.86
53.23
49.99
61.94
50.51
46.49
44.95
31.93
46.78
56.00
51.74
64.72
50.91
AS
61.26
61.37
57.39
67.27
$3.47
49.26
61,88
50.82
$2.64
48.97
34,84
50.44
55.74
49.52
64.52
49.93
CLAWS
66.15
66.02
66.06
73.21
$8.95
56,00
68.65
59.39
55.34
52.80
43.24
58.22
61.12
56.83
71.01
$9.99
PPL
53.26
54.05
60.78
63,40
58.89
55.25
57.58
60.10
59.93
57.91
60.78
63.60
63.44
58.87
39.02
61.55
WE
65.53
63.87
62,80
65.76
50.79
52.21
56.97
60.84
60.76
54.86
53.73
55.66
$7.90
56.12
55.34
59.09
LE
47.54
48.57
50.43
50.62
55.57
53.13
53.23
53.82
53.84
51.23
52.66
53.65
58.38
54.61
54.77
55.73
MLP
HS
61.89
62.14
63.48
66.54
$3.86
48.59
52.13
$2.39
42,56
42.32
48.47
46.41
55.68
49.34
50.96
51.83
61.26
67.08
50.82
50.77
49.90
CLAWS
70.69
70.51
76.44
78.35
63.48
61.29
62.59
65.67
$7.34
61.15
62.70
65.12
61.64
64.64
58.42
57.20
69.78
60.62
45.75
63.60
59.47
71.05
61.55
60.57
52.21
60.84
38.17
55.64
53.58
70.20
59.56
41.98
50.62
52.92
63.51
53.82
52.33
36.12
53.71
66.79
55.29
TabM
HS
60.90
61.22
54.83
65.20
$3.59
48.45
63.08
52.67
42,46
42.25
33.84
46.98
55.16
48.54
67.30
$3.87
62.43
57.29
67.33
61.96
50.82
56.37
50.39
35.48
50.12
49.92
64.47
49.94
CLAWS
70.80
70.40
68.96
77.03
61.18
58.52
71.23
62.34
61.27
$7.73
46.94
63.37
64.01
59.84
74.78
64.70
28

---

Table 12: Evaluation results for hallucination detection using Mathstral-7B
Dataset
TEST
AMC
AIME
AIJHSME
Strategy
Method
Flu
Fla
AFa.
AUROC
Fly
Flix
APn.
AUROC
FJ,
Fla
AFe.
AUROC
Fl.
Fln
AP,.
AUROC
PPL
59.16
51.63
39.99
57.91
48.13
46.97
48.19
52.02
66.72
53.03
25.93
53:45
47.97
48.07
54.48
53.54
WE
61.35
52.24
52.44
70.96
48.34
46.73
59.67
63.49
66.05
44.75
26.79
54.60
47.97
48.12
59.81
62.04
LE
55.98
48.86
33.54
50.88
49.70
48.54
50.70
$4.97
63.63
47.62
22.38
47.37
45.96
46.08
50.91
51.38
XGBOOST
HS
64.41
58.97
49.50
66.57
$3.31
52.71
52.16
$7.07
65.91
48.86
23.06
48.36
50.24
50.32
53.49
53.79
63.29
66.47
56.53
66.37
46.10
25.84
45.81
45.96
56.92
CLAWS
68.93
64.74
54.22
70.14
$7.86
57.42
57.63
60.81
68.62
55,25
34,92
59.38
56.64
56.66
61.68
61.07
63.82
57.72
54.01
52:42
55.67
56.78
35.98
53.10
52.71
66.08
44.36
52.01
53.63
50.57
51.02
50.27
55.60
58.33
55.27
55.44
43.69
52.10
54.60
MLP
HS
68.29
65.92
69.13
72.56
53.85
59.92
61.18
65.09
51.03
49,89
49.29
50.87
50.79
53.46
54.78
69.28
65.
68.56
58.45
59.61
63.78
52.56
55.17
56.01
57.89
CLAWS
72.60
70.23
71.58
76.46
63.97
63.91
64.43
67.13
69.36
55.98
60.50
65.42
57.48
57.41
61.13
63.78
PPI.
52.72
39.83
41. .66
63.13
37.14
34.80
52.24
58.09
65.66
43.19
28.13
55.72
32,90
83.16
51.
52.35
WE
53.00
40.24
48.29
67.44
38.28
36.03
55.86
62.68
66.08
44.36
26.90
54.81
33.56
33.82
55.86
56.94
LB
53.17
41.09
37.32
57.25
38.42
36.18
51.03
56.34
65:43
43.89
21.57
46.00)
34.25
34.49
52.13
53.35
TabM
HS
70.78
65.92
55.96
69.92
58.19
57.72
54.09
59.79
65:00
44.02
24.20
$1.09
50.56
50.65
54.18
54.76
AS
65.
57.77
54.07
69.09
52.23
30.99
57.18
62.07
66.85
47.91
25.53
51.90
49.70
49.82
56.85
57.84
CLAWS
71.24
66.85
58.34
74.90
59.83
59.33
57.96
62.41
70.49
$4.75
34.28
59.23
58.69
58.70
60.09
61.38
Table 13: Evaluation results for hallucination detection using OpenMathh-LLaMMA3.11
Dataset
TEST
AMC
AIME
AUDHSME
Strategy
Method
Fle
Flw
AP..
AUROC
Fle
Fla
APs.
AUROC
Fl.
Fla
AP,.
AUROC
Flx
Pla
APw.
AUROC
PPL
50.29
50.23
45.17
51.55
49.68
48.63
57.68
48.70
48.47
48.42
46.56
48.88
50,25
48.29
62.56
49.13
WE
63.59
62.14
61.51
68.35
54.17
55.17
68.09
63.29
43.86
42.25
50.17
54.96
54,85
55.10
69.61
60.47
LE
51.35
51.07
46.90
53.15
51.61
50.97
59.25
51.71
47.46
47.15
45.91
47,80
48.39
46,90
60.89
47.56
XGBOOST
HS
60.00
59.81
54.59
64.55
56.74
55.33
64.64
57.95
51.68
51.65
50.50
$2.95
56.39
53.87
68.48
57.27
A8
63.52
63.06
57.72
66.34 34
58.27
$7.02
63.49
$1.37
$1,99
$4.12
56.36
53.87
64.44
$3.77
CLAWS
67.70
67.18
67.37
72.71
62.63
61.48
69.63
64.47
$9.28
58.87
$8.90
62.40
62,85
60.48
73.57
64.44
PPL
44.60
49.84
49.79
42.76
43.00
58.43
53.62
52.21
52.93
52.34
53.08
53.63
50.32
51.29
46.27
MLP
64.21
69.38
71.21
60.17
58.12
61.24
63.68
53.21
53.96
55.40
60.69
$7.26
58.61
60.95
63.22
66.27
56.78
59.16
52.56
53.79
53.65
55.54
CLAWS
71.11
70.76
77.32
78.66
66.07
64.48
66.29
69.34
60.82
61.13
63.52
60.69
64.97
67.89
PPL.
44.17
44.84
44.73
47.61
49.54
47.44
57.60
47.50
44.17
44.14
43.89
43.50
48.89
45.42
60.63
46.06
WE
59.94
59.40
51.84
61.08
54,28
53.64
61.87
55.35
$1.44
51.03
49.40
52.96
54.54
52.81
63.59
53.08
LE
52,78
52.51
48.32
55.00
52,50
$1.56
59.79
51.76
47.48
47.23
43.22
45.68
51.77
50.04
65.28
52.06
TabM
HS
65.03
64.90
63.72
71.21
60.35
58.89
69.00
63.68
$4.66
54.59
50.40
55.40
60.25
57.57
70.61
60.95
AS
62.90
62.26
57.64
66.27
58.20
$7.27
64.66
59.16
50.65
49.97
48.80
$3.79
55.84
$3.76
64.66
$4.94
CLAWS
69.62
69.19
69.18
74,51
65.20
63.98
74.83
69.17
61.15
60.82
61.13
63,52
65.13
62.79
75.60
67.35
29

---

Table 14: Evaluation results for hallucination detection using OREAL-7B
Dataset
TEST
AMC
AIME
AIJHSME
Strategy
Method
Flu
Fla
AFa.
AUROC
Fly
Flix
APn.
AUROC
F1,
Fla
AFe.
AUROC
Fl,
Fln
AF,.
AUROC
PPL
59.22
49.94
36.97
54,44
54.35
47.79
43.30
$6.57
72.00
49.27
18.63
48.58
49.75
47.51
49.10
54.52
WE
60.40
50.83
38.54
57.13
56.09
49.39
43.11
57,40
73.10
45.26
18.48
49.66
52.84
50.64
50.98
58.15
LE
56.21
44.73
33.26
48.61
51.53
44.15
38.09
49.52
72.
49.29
19.12
50.26
44.64
41.85
45.93
50.64
XGBOOST
HS
64.38
58.48
43.79
60.96
61.46
58.64
46.34
$9.01
72.17
53.12
19.66
50.69
57.84
56.87
51.96
58.27
38.5
40.78
53.26
CLAWS
64.31
58.82
44.59
62.34
57.95
55.58
44.36
$7.67
70:94
50.29
48.47
53.84
53.17
53.54
60.87
72.77
50.52
40.14
53.34
55.76
38.35
50.38
73.07
49.57
48.51
35.55
46.93
43.33
48.33
47.30
73.07
50.10
MLP
59.85
56.95
59.13
61.23
57.76
56.68
59.33
60.93
65.99
49.85
51:27
51.93
55.95
55.86
58.06
59.23
60.03
63.33
56.22
66.15
50.58
57.28
58.53
60.12
CLAWS
64.71
61.29
63.95
66.93
56.54
56.08
59.53
61.73
68:43
51.28
51.04
50.94
56.14
56.45
58.49
60.57
PPL.
59.36
49
41.76
60.30
56,70
50.27
50.12
63.76
72.67
49.88
20.65
56.42
51.83
49.60
56.57
64.01
WE
60.40
50.83
36.82
55.76
56.09
49.39
42.67
55.17
73.10
45.26
18.97
50.41
52.84
50.64
50.67
56.85
LB
53.83
40.14
33.18
51.29
47.70
38. 35
35.34
47.56
73.07
44.88
18.84
52.45
39.22
35.58
45.74
50.69
TabM
HS
65.67
59.53
46.96
61.59
62.04
58.94
51.10
60.995
72.72
51.80
20.36
51.17
57.68
56.55
56.12
59.23
AS
53.83
40.14
42.08
61.99
47.70
38.35
42,90
$7.50
73.07
44.88
20.29
52.54
39.22
35.55
49.02
56.70
CLAWS
66.82
61.04
46.92
63.98
60.38
57.71
44.02
58.87
72,66
51.27
19.93
51.23
57.67
56.93
52.52
59.28
Table 15: Evaluation results for hallucination detection using Qwen- 2.5-Math-78
Dataset
TEST
AMC
AIME
ACDHSME
Strategy
Method
Flu
Flg
APr.
AUROC
Fly
Fla
AP,.
AUROC
F1,
Fla
AP
AUROC
Flu
Fln
AP,
AUROC
PPL
61.27
59.63
70:49
65.17
71.32
55:49
84,41
60.70
53.64
53.62
57.96
57.56
75.31
51.90
87.48
56.02
WE
61.82
59.59
21.87
69.07
74.08
56.51
85.10
63.88
49.14
49.09
52.66
53.82
79.36
54.39
87.86
60.02
I.
50.31
47.63
84.19
46.79
64.87
49.50
78.87
50.02
44.29
44.25
48.77
48.05
68.83
47.76
83.42
47.38
XGBOOST
HS
60.24
58.39
71.75
65.48
73.30
57.63
86.31
65.38
$1.06
51.04
52.35
52.96
76.28
$3.50
86.34
56.05
53.53
50.47
65.72
59.94
$1.53
82.11
55.60
44.77
44.71
48.40
76.44
51.33
86.24
52.55
CLAWS
65.68
64.26
76.73
71.49
70.33
55.37
87.57
64.43
54,20
$4.19
56.73
56.92
74.70
53.65
89.57
61.67
66.79
73.18
65.10
58.59
53.17
49.56
52.77
53.42
42.24
41.58
MLP
68.94
72.35
61.08
52.22
58.05
53.04
53.56
CLAWS
73.19
74.80
57.50
58.89
51.41
51.43
$3.89
77.81
56.34
61.42
PPL.
54.59
$1.04
77.93
74.38
21.03
46.78
88.28
70.25
42.73
42.66
64.11
65.78
77.71
46.49
90.16
65.19
WE
44.28
39.16
66.10
63.23
70.53
45.80
82.92
59.29
35.37
35.28
50.26
50.65
77.97
47.21
85.75
54.04
L.
42.61
38.82
56.70
42.63
62.39
45.80
77.26
44.41
38.08
38.04
47.82
40.96
66.08
44.04
81.65
41.85
TabM
HS
63.90
62.04
76.98
72.73
73.50
55.38
87.61
68.01
51.06
51.04
53.91
54.20
77.91
53.08
88.03
$8.05
AS
41.98
36.51
66.28
57.65
69.89
44.17
82.10
55.16
33.49
33.40
48.82
48.11
77.56
45.83
86.
53.51
CLAWS
63.38
61.74
27.83
71.17
69.70
55-45
87.38
63.99
53.38
53,38
56.58
56.1-4
78.89
53.44
89.54
61.99
30

---

Table 16: Evaluation results for creativity detection on a balanced dataset. The evaluation strategies used Threshold and Prototype. Bold values indicate the best performance, underlined values indicate the second-best, and gray-shaded cells correspond to results where the model performed detection over only two out of the three target classes.
Dataset
TEST
AMC
AIME
AUDHSME
Model
Method
Fl,
Fla
AP,
AUROC Fl,
Fla.
AF.
AUROC Fl,
Fla
AF.
AUROC
Fl,
Fl
APa
AUROC
PPL.
35.22
35.22
34,40
52.03
37.57
37.57 35.04 53.16
42,57
42.57
37.65 57.34
36.21
36.21
34.83
$2.30
WE
30.54
30.54
35.34
53.44
28.46
28.46
34.20 51.60
25,80
25.80
33.15
49.40
28.70
28.70
34.35
51,95
L.
31,89
31.89
32,98
48.91
32,78
32.78
33.35
49.95
29.32
29.32
32.65
47.62
33,06
33.05
33.37
49.94
Deepseck
HS
27.36
27.36
32.21
45.16
31,40
31.40
33.25
49.58
32,09
32.09
33.14
49.21
32.88
32.88
33.58
50.53
AS
31.48
31.48
33.81
49.69
33.31
30.56
30.56
33.56
50.40
33.29
33.55
50.47
CLAWS
35.87
55.95
35.97
54.89
31.55
36.14
54.71
35.05
53.11
33.81
50.74
34.19
34.41
52.05
44.85
Mathstral
49.35
33.12
48.96
25.67
AS
24.86
24.86
31.88
44.16
28.19
28.19
32.16
46.26
28.62
28.62
32.91
48.53
29.09
29.09
32.16
46.82
CLAWS
40.40
60.71
38.13
37.08
56.45
31.86
31.86
34.23
51.84
38.04
37.05
56.43
PPL.
29.78
29.78
33.01
49.23
27.45
27.45
32.15
46.56
25.40
25.40
31.73
44.40
23.79
23.79
31.37
43.75
WB
33.85
33.85
34.18
51.55
33.45
33.45
34.29
$1.89
31.01
31.01
32.73
48.51
29.53
29.53
33.14
49.50
LE
36.34
36.34
34.53
52.32
40.00
40.00
36.16
55.15
31.18
31.18
33.42
50.00
38.06
38.06
35.53
53.93
OpenMath2
HS
25.49
25.49
31.53
44.33
28.34
28.34
32.25
46.91
36.30
36.30
34.92
52.61
28.43
28.43
32.30
47.38
AS
23.92
23.92
31.19
43.04
29.84
29.84
32.75
48.20
38.59
38.59
35.52
54.10
32.32
32.32
33:77
50.30
CLAWS
41.90
41.90
38.92
58.51
37.66
37.66
36.93
56.36
24.86
24.86
33.22
49.63
33.47
33.47
35.60
54.23
PPL.
29.02
29.02
32,41
47.47
23,55
23.55
31,56
44.09
31.64
31.64
33.65
50.00
23.87
23.87
31.48
44.25
WE
25.69
25.69
32.14
46.91
27.60
27.60
33.08
49.37
30.21
30.21
33.34
50.00
27.38
27.38
33.00
49.07
LE
33.34
33.34
33,86
50.84
34.64
34.64
34.96
53.16
29.33
29.33
33.37
49.47
35.79
35.79
35.33
53.88
OREAL
HS
30.03
30.03
32.87
48.60
26.77
26.77
31.91
45.89
33.93
33.93
33.76
50.53
27.64
27.64
32.10
46.58
AS
26.10
26.10
33.10
47.75
31,65
31,65
34.15
51.58
25.07
25.07
33.61
50.53
30.21
30.21
33.59
49.22
CLAWS
25.27
25.27
32.99
48.31
34,08
34.08
35.16
53.48
34,85
34,85
34,69
52.66
37.49
37.49
35.52
54.04
PPL.
27.04
27.04
34.14
50.00
27.75
27.75
33.72
49.52
25.76
25.76
33.34
49.53
35.59
31.09
33.92
50.47
WB
34.62
34.62
52.96
32.83
34.20
31.12
28.56
33.01
I.F
45.25
45.25
39.53
59.24
40.54
40.54
36.60
55.60
39.56
39.56
36.05
54.56
41.31
39.10
38.66
54.32
Qwen-2.5
HS
34.67
51.72
66
53.58
31.55
53.11
32.32
32.32
34.20
51.73
CLAW'S
49.63
38.04
45.28
31

---

Table 17: Evaluation results for creativity detection using DeepSeek-Wath-78 on a balanced dataset. Bold values indicate the best performance, underlined values indicate the second-best. Light gray- shaded cells correspond to cases where the model performed detection over only two out of the three target classes, while dark gray-shaded cells indicate cases where the model predicted only one out of the three target classes.
TEST
AMC
AIME
ACDHEME
Strategy
Method
Fly
Flx
AP
AUROC
Fly
Fla
AP
AUROC
Fl,
Fla
AP,
AUROC
Fl,
Fla
AP
AUROC
PPI.
39.12
39.12
37.18
55.02
35.85
35.85
35.79
53.32
36.68
36.68
35.70
52.40
35.90
35,90
35.24
52.73
WE
33.20
33.20
40.18
60.71
30.24
30.24
37.15
56.72
22.32
22.32
34.51
51.56
31.12
31,12
36.72
56.17
I.
36.50
36.50
35.10
52,32
34.29
34.29
34.37
51.49
32.17
32.17
33.29
49,80
33.12
33.12
34.87
52.15
XGBOOST
HS
36.52
36.52
37.35
54.52
35.71
35.71
34.28
51.47
35.93
35.95
37.00
$4.95
35.13
35.13
35.19
51.92
AS
41.05
41.05
41.24
59.10
34.01
34.01
34.79
51.88
31.78
31.78
37.18
53.95
32.39
32.39
33.68
50.52
CLAWS
43.24
43.24
46.66
63.42
37,50
37.50
37,80
55.29
42.11
42.11
41.87
59.88
35.53
35.55
37.34
54.77
PPL
30.04
30.04
38.58
57.19
23.69
23.09
38.57
56.47
30.73
30.73
40.75
60.01
27.11
27.11
38.43
56.75
WE
33.76
33.76
40.38
60.54
32,25
32.25
35.88
54.05
31,12
31,12
36.99
55.19
31.64
31,64
36.92
56.42
LE
35,27
35.27
36.45
54.32
26.92
26.92
34.54
$1.59
29.76
29.76
36.82
$4.31
34,95
34.95
34.99
51,25
MLP
HS
41.43
41.43
44.90
62.33
27.66
27.66
34.81
51.37
28.28
28.28
36.62
53.21
28.84
28.84
33.96
49.65
AS
38.64
38.64
41.96
60.54
34.57
34.57
34,69
51.74
32.82
32.82
35.22
$1.83
33.22
33.22
33.77
50.43
CLAWS
44.98
44.98
49.40
67.35
41.38
41.38
43.04
60.77
41.75
41.75
42.96
62.88
35,00
35.00
42.14
60.25
PPL
57.59
38.62
56.63
59.97
36.75
38.43
56.73
30,20
30.20
40.38
60.66
56.72
12
36,46
36.91
LE
54.04
51.50
39.61
37,83
55.14
35.20
TabM
HS
40.49
44.16
60.99
32.64
35.74
52.89
34.37
52.06
33.73
41.17
42.68
60.83
34.17
34.17
34.72
51.63
37.28
33.70
CLAWS
64.08
38.87
56.16
45.24
61.62
40.27
Table 18: Evaluation results for creativity detection using Mathstral-7B on a balanced dataset
Dataset
TEST
AMC
AIME
AUDHSME
Strategy
Method
Fly
Fl,
AP..
AUROC
FI,
Fl,
AP..
AUROC
FI,
Fl,
AP,
AUROC
F1,
Flg
AP,.
AUROC
PPI.
33.41
33.41
34.60
51.21
35.84
35.84
34.78
51.49
38.15
38.15
39.75
55.00
34,06
34.05
34.72
51.35
WE
34.18
34.18
37,59
56.00
33.95
33,95
36.71
56.02
32.49
32.49
33,80
50.17
29.30
29.30
37.49
56.35
LE
29.74
29.74
32.30
46.74
34.67
34.67
35.20
52.35
30.95
30,95
35.16
52.07
33.39
33.39
33.54
50.56
XGBOOST
HS
37.82
37.82
38.99
56.58
36.03
36.03
35.64
52.11
32.52
32.52
34.83
49.69
33.65
33.65
34.11
50.75
A8
36.54
36.54
37.89
54.51
36.50
36.50
36.47
$3.84
31,42
31.42
31.34
45.44
34.36
34.96
35.45
$2.44
CLAWS
40,20
40.20
45.46
60.50
40.14
40.14
40.23
57.46
42.73
42.73
41.03
54.54
34.60
34.60
38.38
56.42
PPL
29.20
29.20
41.65
59.58
16.67
16.67
35.93
52.16
16.48
16.48
36.69
$3.09
27.26
27.26
35.73
52.53
WE 27.15
27.15
36.88
51.56
16.67
16.67
35.44
52.37
22,70
22.70
31,40
45.42
19.64
19.64
33.94
49.79
LE
37.59
37.59
37,39
54.32
30.70
30.70
35.27
53.14
32.48
32.48
37.23
52.87
16.67
16.67
33.33
50.00
MLP
HS
45.58
45.58
43.92
62.70
29.61
29.61
37.66
55.65
19.92
19.92
36.00
$2.67
29.42
29.42
35.91
$3.20
AS
38.49
38.49
42.73
62.29
36,60
36,60
36.38
$4.85
31.81
31.81
35.14
$2.07
25.42
25.42
35.28
53.28
CLAWS
46.78
46.78
45.64
63.92
44.79
44.79
44,69
62.51
33.72
33.72
43.75
58.92
39,02
39.02
41.95
60.36
PPL.
42.24
42.24 24
38.66 69
55.69
36.79
36.79
35.46
51.75
37.77
37.72
36.86
53.20
34.14
34.14
35.34
52.11
WB
31.48
36.95
$1. $1.88
30.15
30.15
35.69
$2.18
31.55
31.55
32.46
47.29
222.68
22.68
33.75
49.64
LE
28.89
28.89
36.88
53..45
29.67
29.67
35.14
$2.70
26.54
26.54
37.27
$2.38
28.46
28.46
34.87
52.11
TabM
HS
34.64
34.64
39.16
58.56
31.46
31.46
37.67
$5.25
47.77
52.67
AS
38.21
38.21
40.78
58.34
36.79
36.79
37,88
55.80
31.80
31.80
80
34.94
$0.89
35.48
35.48
35.89
$3.82
CLAWS
45.74
45.74
48.76
66.40
41.22
41.22
43.57
61.17
38.58
44.41
60.86
38.99
40.32
57.89
32

---

Table 19: Evaluation results for creativity detection using OpenMathh-LLaMAA..I balanced
dataset
Dataset
TEST
AMC
ACDHSME
Strategy
Method
Fly
Fln
AP
AUROC
Fle
Fl,
AP
AUROC
Fl,
Fl,
AP,
AUROC
Fl,
Fla
AP,
AUROC
PPL
31.64
31.64
34.48
50.89
29.55
29.55
32.14
48.11
35.35
35.35
38.34
53.75
35.58
35.58
37.00
53.21
WE
37.97
37.97
39.76
57.89
34.03
34.03
35,96
54.12
34.70
34,70
35.60
51.49
29.54
29.54
35.44
53.48
LE
31.75
31.75
34.09
49.66
33.41
33.41
34.72
51.37
32.31
32.31
35.01
51.03
33.04
33.04
34.44
50.80
XGBOOST
HS
32.30
32.30
35.72
53.62
57.06
37.06
38.54
55.35
32.31
32.31
35.01
51.03
33.38
33.38
35.13
52.06
AS
36.01
36.01
57.54
54.58
35.51
35.51
34.71
51.48
31.35
31.35
31.95
47.58
33.35
33.35
33.55
51.22
CLAWS
49.96
49.96
48.76
66.23
42.00
42.00
42.66
60.32
36.91
36.91
37.91
54.72
40.50
40.50
42.87
60.46
PPL
29.07
29.07
35.24
50.63
29.70
29.70
33.56
49.09
23.99
23.99
34.08
48.25
31.27
31.77
42.10
59.10
WB
32.54
32.54
34.84
50.68
28.22
28.22
34.47
50.89
16.67
16.67
34.71
51.53
31.38
31.38
36.24
53.50
LE 31.28 31.28
34.74
49.56
25.55
25.55
32.21
47.23
18.22
18.22
30.51
43.57
28.08
28.08
35.97
52.38
MLP
HS
33.89
33.89
43.40
61.26
29.16
29.16
39.11
56.93
34.24
34.24
37.61
51.56
28.82
28.82
36.98
54,43
AS
40.56
40.56
41.38
60.35
27.80
27.80
36.07
52.92
27.55
27.55
31.38
46.84
23.34
23.34
30.92
45.34
CLAWS
49.51
49.51
49.09
68.00
43.34
43.34
46.49
64.34
27,91
43.29
60.89
43.42
43.42
43.28
47.71
47.23
21.31
46.70
23.14
36.25
34.02
31.37
53.01
36.77
21.50
32.75
TahM
42.31
37.02
40:56
41.36
33.06
31.96
47.37
44.73
Table 20: Evaluation results for creativity detection using OREAL-7B balanced dataset
Dataset
TEST
AMC
AIME
AGDHSME
Strategy
Method
Fl,
Fl
AP..
AUROC
Fl.
Fl,
APs.
AUROC
F1,
Fl,
AP,
AUROC
F1,
Fl,
AP,
AUROC
PPL
33.63
33.63
34.65
51. $1.86
38.34
38.34
37.58
54.99
36.88
36.88
34.53
50.26
35.93
35.93
37.37
53.25
WE
24.25
24.25
33,50
49.75
25.34
25.34
36.10
53.91
28.33
28.33
35.06
52.47
25.40
25.40
36.88
54.24
LE
33.58
33.58
34.26
50.43
32,49
32.49
33,80
50.07
36.88
36.88
36.89
49.59
32,76
32.76
34.25
51.18
XGBOOST
HS
32.32
32.32
33.35
48.58
35.14
35.14
36.56
51.71
28.70
28.70
34.12
50.62
35.73
35.73
35.48
52.58
AS
27.82
27.82
33.13
48.70
33:44
33.44
34.73
51.62
36.34
36.34
34.41
50.94
29.16
29.16
34.10
50.26
CLAWS
39.05
39.05
40.73
59.03
34.99
34.99
36.47
53.69
34.14
34.14
36.99
50.62
35.84
35.84
36.48
53.33
PPL
30.55
39.80
56.69
WB
16.71
16.71
32.98
18.17
34.96
26.00
35.41
$2.23
20.66
20.66
50.55
55.19
16.67
27.58
16.67
MI.P
26.31
41.83
35.34
16.67
23.90
46.50
38.59
36.95
36.95
42.66
59.17
38,03
37.60
40.37
54.71
39.07
WE
21.91
21.91
32.71
47.72
22.22
22.22
33.44
50.59
25.16
25.16
33.97
50.85
20.66
20.66
34.26
51.71
32.76
32.76
34.28
50.15
33.43
33.43
34.97
$2.29
34.93
38.28
52.85
36.
37.31
55.85
TabM
HS
27.86
27.86
34.51
50.49
36.48
36.48
39.10
56.42
31.06
31.06
36.33
$2.99
34.38
34.38
38.41
56.11
AS
31.94
31.94
35.29
51.14
34.14
34.14
35.55
52.82
33.61
33.61
34.50
50.43
29.68
29.68
37.19
55.81
CLAWS
44.21
44.21
44.09
62.58
34,23
34.23
37.20
53.25
29.62
29.62
33.35
46.67
35.21
35.21
36.83
53.68
33

---

Table 21: Evaluation results for creativity detection using Qwen-2.5-Math-77 balanced dataset
Dataset
TEST
AMC
AIME
AGDHSME
Strategy
Method
Fl,.
Fl,
AP,
AUROC
F1,
Fl,
AP,
AUROC
F1,
Fl,,
AP,.
AUROC
F1,
Fl,,
AP,
AUROC
PPL
40.16
40.16
43,09
60.01
37.57
37.57
38.43
55.33
37.99
37.99
39.15
56.60
36.22 35.16 36.07
$2.68
41.38
41.38
40.81
58.87
37.89
37.89
38.51
56.66
35.50
35.50
36.17
52.77
37.33
36.20
36.72
54.31
33,93
33.93
33.71
51,48
32.65
32.65
34.00
50.92
29.51
29.51
32.34
48.28
34.55
33.51
34.05
50.89
XGBOOST
39.46
38.29
33.19
53.33
38.94
37.34
36.63 36.63
37.97
54.72
38.21
38.21
35.99
53.15
30.86
30.86
33.03
38.05
35.96
34.76
$2.47
CLAWS
39.02
39.02
39.06
38.35
37.57
50.93
22.47
27.76
32.19
32.19
39.11
39.11 37.77
56.80
26.66
54.02
26.01
26.01
34.46
33.52
33.52
PPL
40.13
40.13
47.62
66.36
39.52
39.52
43.87
62.15
34.74
34.74
41.71
$9.99
35.79
36.55
39.37
$7.42
WF
38.23
38.23
38.37
55.63
35.77
35.77
37.37
54.90
80.03
30.03
35.10
$1.88
32.93
32,90
35.90
$2.83
26.16
26.16
30.58
46.48
29.14
29.14
32.54
50.05
27.55
27.55
31.08
46.65
28.17
27.31
32.96
48.78
HS
40.42
40.42
44.41
61.94
38.08
38.08
41.85
59.30
34.03
34.03
35.72
$2.50
36.22
34.52
96.96
53.35
AS
36.75
36.75
38.01
56.30
36.24
36.24
36.48
53.82
29.81
29.81
33.21
48.37
34.77
33.98
34.55
$1.32
CLAWS
41.14
41.14
44.03
61.98
40.16
40.16
40.10
39.94
36.52
36.52
37,88
$4.64
40.03
37.98
38.50
57.69
34


Reviews:
[{"review_id": "Z8YsXGTyId", "reviewer": "Reviewer_SEE3", "summary": "The paper presents CLAWS, a novel framework that uses human feedback signals to detect creativity in open-ended language generation tasks. It leverages contrastive learning with weak supervision to distinguish creative responses from conventional ones without relying on labeled data. The authors demonstrate CLAWS’ effectiveness across multiple datasets, showing that it can outperform supervised baselines in creativity detection.", "questions": "How does CLAWS handle variations in human feedback quality across different platforms or domains, and could it adaptively calibrate its learning when feedback signals are sparse, noisy, or biased?", "limitations": "YES", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+Integrates implicit human feedback (e.g., upvotes, ratings) to guide the model without needing extensive annotations.\n\n+Employs contrastive learning in a weakly supervised setting, which is both scalable and efficient.\n\n+Demonstrates robust performance across several creativity-related benchmarks.\n\n-The paper does not fully resolve how to consistently define or quantify creativity, which is inherently subjective.\n\n- Performance may degrade if the human feedback signals (like upvotes) are noisy or biased.\n\n-Focuses mainly on text generation; extension to multimodal or non-verbal creative tasks remains unexplored.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable comments and questions. Below is our detailed response to your concerns:\n\n**Definition of Creativity and Evaluation Quality:**\n\nThe issue you raised regarding human feedback quality across different platforms and domains is indeed one of the core challenges we aim to address in this study on evaluating and detecting creativity. Since creativity inherently involves subjective judgment, high-quality and reliable human annotations are essential, which imposes practical limitations on expanding ‘creativity’ research. In particular, the dataset we used in this study consisted of mathematically difficult problems, which presented an essential limitation in that it required a very high level of expertise for humans to directly evaluate the creativity of the generated solutions.\n\nTo overcome these limitations, we adopted the approach proposed in prior work [1], which evaluates the creativity of model-generated solutions using only LLM evaluators, without direct human involvement. Nevertheless, our evaluation implicitly incorporates human judgment by using high-quality, diverse, and creative reference solutions written by humans as the basis for comparison. Specifically, a generated solution was considered “creative” if it satisfied at least one of the following five criteria (see Appendix B.1 for details): \n- the approach is fundamentally different; \n- the intermediate steps are meaningfully different; \n- it is based on distinct assumptions or conditions; \n- the applicable problem scope is substantially different; or \n- the solution is significantly more concise or more complex than the reference solution.\n\nHowever, we acknowledge that relying solely on LLM evaluators may still raise concerns regarding evaluation quality, as you rightly pointed out. To address this issue, we followed a similar setup to prior work [1], using two LLM evaluators instead of just one to assess the creativity of each generated solution. Notably, whereas [1] used GPT-4o as one of the evaluators, we upgraded this to o4-mini in our experiments, which offers stronger performance in mathematical evaluation.\n\nSince creativity assessment inherently involves a degree of subjectivity, requiring both evaluators to agree that a generated solution is creative may risk overlooking genuinely creative outputs. To mitigate this, we adopted a union-based criterion, under which a solution is deemed creative if either of the two evaluators judges it as such. To ensure the reliability of this evaluation process, we calculated Cohen’s kappa score, which was 0.741—indicating substantial agreement between evaluators and confirming the consistency of their judgments.\n\n**Calibration to Address Sparsity, Noise, and Bias:**\n\nAs you rightly pointed out, the evaluation process for generated solutions can be affected by issues such as sparsity, noise, and bias, which require appropriate mitigation. To address these concerns, we applied the following two types of calibration across the five evaluation strategies:\n1. Adaptive calibration during training: In the MLP strategy, we applied class-weighted loss functions to minimize potential bias and noise caused by class imbalance during training. We also adopted TabM [2], a recent ensemble method that combines 32 MLPs, effectively reducing training noise and improving prediction stability. The corresponding results are reported in Table 2 of the main paper and Tables 4–8 of Appendix section.\n2. Calibration via class balancing: We constructed a balanced dataset by ensuring equal sample counts across the three classes during evaluation. This reduced the risk of class dominance and led to more reliable assessments. The full results of experiments on a balanced dataset are presented in Tables 14–19 of Appendix section.\n\nYour insightful comments and thoughtful suggestions have significantly improved the clarity and persuasiveness of our work. We will actively incorporate all your feedback into the final version.\n\nWe sincerely thank you again for your valuable time and feedback, and we hope this response has satisfactorily addressed your concerns.\n\nThe appendix was submitted during the supplementary materials period and is available in the zip file linked next to *“Supplementary Material”* at the top of this page.  Due to changes in the review system, we are unable to submit a new PDF file—thank you for your understanding.\n\n---\n**References**\n\n[1] Ye, Junyi, et al. \"Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems.\" AAAI 2025.\n\n[2] Gorishniy, Yury, et al.  \"Tabm: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling.\" ICLR 2025."}, {"review_id": "Z5dGwY1lGM", "reviewer": "Reviewer_fJn8", "summary": "The paper proposes CLAW, a novel method for classifying LLM generated answers in mathematical reasoning tasks into hallucinated, creative, or typical categories using attention weight analysis. They segment the input tokens into five sections of guideline, problem, solution, instruction, and responses for attention weights. The evaluation with LLMs such as GPT-4mini and Gemini1.5 pro shows that the proposed method achieves strong performances on benchmark datasets CreativeMath and HARLP with 4 math-specialized LLMs including DeepSeek, Qwen2.5, OpenMath2, etc.", "questions": "1. In lines 103–106, you mention that larger, general-purpose LLMs like LLaMa3-8B, Qwen2.5-14B, and DeepSeekV2-16B were excluded due to their lack of creative solutions. Could this be a consequence of how creativity is defined in your setups? \n- Specifically, that fluent, efficient, and correct solutions that resemble reference answers are penalized under your labeling criteria?\n- If so, does this suggest that the proposed method may be less applicable to stronger models that prioritize clarity and correctness over stylistic novelty? \n- More broadly, how do you distinguish between “lack of creativity” and producing the mathematically optimal solution?\n\n2. This proposed work uses GPT4-mini and Gemini1.5 Pro for dual labeling and report the correctness based on their agreement. Yet, you apply a union-based criterion for creativity. Have you measured the inter-model agreement rate (e.g., overlap or Cohen' k)? This would help understand the reliability and consistency of the proposed labeling protocol.\n\n3. Have you conducted any experiments to manipulate/isolate the influence of each prompt section (e.g., guideline, problem, solution, instruction, etc) on CALW attention weights settings as an ablation?", "limitations": "NA - labeling issues may not be a huge impact for the society but natural limitations due to the design choice for the methodology.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## **Strengths**\n1. Novel problem formulation; the work tackles underexplored task to detect creative responses of LLMs. The work is now a three-way classification instead of a hallucinated/non-hallucinated classification.\n2. Interesting approaches with attention weights; they propose the prompt grouping for better usage of attention weights to obtain section-wise attention ratios and use them as features for three-way hallucination detection.\n3. Strong empirical results; the work compares the method with sufficient number of benchmarks (PPL, logit entropy, windown entropy, hidden scores, attention scores, etc) on multiple LLMs (deepseek, Qwen2.5, Mathstral, OpenMath2, OREAL).\n## **Weaknesses**\n1. Labeling/testing relies on specific LLMs: Creativity annotations are derived using GPT-4-mini and Gemini-1.5 Pro, with no human validation or inter-model agreement reported. This raises concerns about labeling consistency and reliability, especially given the subjectivity of creativity. Esp. the authors even mentioned that they did not report results with other LLMs as they seem not sufficient due to their lack of ability to creatively solve math problems. As scales increase, they may be less creative but likely to generate accurate responses. This may raise concerns regarding LLM specific issues or the necessity of creativity/ valid definition of creativity.\n2. The evaluation approaches relies on class prototypes derived from the reference set, which can be fragile under class imbalance as seen in OREAL where hallucinations dominate.\n3. No ablation studies; the method is empirically studied with performance comparison but no component-wise ablations such as prompt sections of guideline, problem, solution, etc. It would have been interesting to see their individual influence or the order of each chunk matters, etc.\n4. Grammar issues/ language unclarity in writing", "was_revised": true, "final_justification": "My initial concerns regarding labeling and LLM heavy reliance still remains. Yet the task definition with mathematical tasks reduces significant issues possible. \n- \"The reason why it is difficult to evaluate creativity in a reasoning task is because human evaluation is\n32 essential to establish criteria for creativity, and a high-level expert is needed to evaluate whether a\n33 solution to a mathematical problem is creative. However, a recent study on mathematics problem\n34 solving [10, 11], overcame these difficulties and measured the creative problem solving ability of\n35 LLMs, and the results revealed that there was a large difference in creative ability even among LLMs\n36 with similar accuracy.\" from their paper.\n\nI liked authors' response regarding rationales for LLM selection; though it still lingers the risk of LLM-sensitive results, it may provide some criteria to choose which LLM would work for this task.\n\nAlso happy about their ablation study.\n\nHence I raise my score --", "rebuttal": "We sincerely thank you for their valuable comments and questions. Below is our detailed response to your concerns:\n\n**A1. Rationale for Excluding a Particular Model:**\n\n|Model|Correctness(Cor) (%)|Creativity(Cre) (%)|Cre/Cor (%)|\n|---|---|---|---|\n|Deepseek-V2-Lite-16B|21.32|11.57|54.16|\n|Qwen-2.5-14B|52.39|39.00|74.44|\n|Llama3-8B|25.12|16.19|64.47|  \n\nThe table above summarizes the correctness ratio (= Typical + Creative Solutions) and creativity ratio (= Creative Solutions) of various LLMs on the CreativeMath Dataset [1]. The exclusion of certain models from our analysis was not due to how creativity is defined, but rather due to limitations of creative problem-solving ability or experimental resources. Specifically, the reasons are as follows:\n- For LLaMa3-8B and DeepSeekV2-16B, the rate of generating correct solutions for high-difficulty problems included in the CreativeMath [1] and HARP [2] datasets was extremely low. As these models failed to generate a sufficient number of valid responses for meaningful analysis, we excluded them from our experiments. Consequently, the absolute number of Creative solutions was too small, resulting in severe class imbalance. This issue was particularly evident in models that had not undergone additional pretraining on math corpora, Supervised Fine-tuning(SFT), or Reinforcement Learning from Human Feedback (RLHF) or Reinforcement Learning with Verifiable Rewards (RLVR) (including policy optimization-based approaches like PPO or GRPO).\n- In contrast, Qwen-2.5-14B did not exhibit severe class imbalance. However, for models larger than 8B, it was infeasible in our experimental resources to collect the full range of computational values (e.g., hidden scores, attention scores) across all layers and heads. Therefore, to ensure a fair comparison with baseline methods, we excluded these models from our analysis as well.\n\nIn conclusion, CLAWS is not less applicable to models that prioritize correctness. It can be applied to any model, provided that resource constraints allow.\n\n**Definition of Creativity:**\n\nRegarding the definition of creativity, we adopted the criteria proposed in prior work [1]. Specifically, a generated response is considered a Creative Solution if it satisfies at least one of the following five criteria when compared to the provided reference solution (please see Appendix Figure 6 for details):\n- the approach is fundamentally different;\n- the intermediate steps are meaningfully different;\n- it is based on distinct assumptions or conditions;\n- the applicable problem scope is substantially different; or\n- the solution is significantly more concise or more complex than the reference solution.\n\nThese criteria are intended not merely to capture superficial diversity in expression, but to define and evaluate creativity in a comprehensive manner that considers the structure of reasoning, generalizability, efficiency, and strategic diversity. As such, mathematically more general or optimal solutions are also considered sufficiently creative. The evaluation framework adopted in this study was specifically designed to ensure that such solutions are not excluded from the scope of creativity. This means that fluent, efficient, and correct solutions—as well as mathematically optimal ones—can be recognized as creative approaches.\n\n**A2. Reliability of LLM Evaluator-Based Labeling:**\n\nWe fully acknowledge and empathize with your concerns regarding the reliability of LLM evaluator-based labeling. As in prior work [1], this study relied solely on LLM evaluators—without human involvement—to assess the creativity of generated responses. In [1], the authors employed three LLM evaluators and adopted a majority voting strategy. In our case, we used only two evaluators. While [1] employed GPT-4o as one of the evaluators, we upgraded this to o4-mini, which offers stronger performance in mathematical evaluation. Since we could not rely on majority voting with only two evaluators, we instead adopted a union-based criterion: a response was labeled as creative if either evaluator judged it to be so. To verify the reliability of this approach, we measured inter-evaluator agreement using Cohen’s kappa score, which yielded a value of 0.741. This indicates substantial agreement between evaluators and supports the consistency of their judgments.\n\nGiven that creativity assessment inherently involves a degree of subjectivity, we believed that requiring both evaluators to agree could risk overlooking genuinely creative responses. The union-based approach thus serves to mitigate such underestimation. We fully acknowledge your concern regarding the reliability of LLM-based evaluation and will include this clarification in the final version of the paper.\n\n**A3. Impact of Each Prompt Section:**\n\n|Dataset|Metric|w/o $G$|w/o $P$|w/o $S$|w/o $I$|w/o $R$|Full|\n|---|---|---|---|---|---|---|---|\n|TEST|F1 $_w$|58.59|58.81|50.01|54.68|58.39|**58.66**|\n||F1 $_m$|46.01|**46.35**|39.45|43.87|45.97|46.01|\n||AP $_m$|41.13|41.04|38.46|40.29|40.60|**41.17**|\n||AUROC|62.12|**62.21**|59.15|61.64|61.58|62.09|\n|AMC|F1 $_w$|46.66|46.68|46.48|45.52|46.49|**46.71**|\n||F1 $_m$|39.59|39.33|39.59|38.16|40.20|**40.99**|\n||AP $_m$|37.15|37.08|36.91|36.70|37.14|**37.16**|\n||AUROC|56.11|56.23|56.00|55.80|56.52|**56.40**|\n|AIME|F1 $_w$|55.88|53.33|54.92|51.37|54.01|**56.90**|\n||F1 $_m$|37.35|35.52|36.82|34.80|34.84|**38.12**|\n||AP $_m$|35.25|35.20|34.98|35.03|35.15|**35.38**|\n||AUROC|54.08|53.38|53.51|52.47|52.67|**54.47**|\n|A(J)HSME|F1 $_w$|37.85|39.92|**40.49**|40.10|34.25|38.82|\n||F1 $_m$|37.46|35.76|37.65|36.02|34.11|**37.64**|\n||AP $_m$|36.16|35.51|35.78|35.69|35.42|**36.25**|\n||AUROC|**55.05**|53.88|54.33|54.18|53.08|54.40|\n\nIn response to the question regarding the impact of each prompt section, the table above presents the performance changes observed when applying the prototype strategy to responses generated by the DeepSeek model, with each section removed one at a time. While the removal of a specific section did affect performance in some cases, it is difficult to conclude that any single section dominates CLAWS's overall performance. In most cases, the Full configuration, which uses all sections together, exhibited the most stable and superior performance.\n\nThese findings suggest that CLAWS does not rely excessively on any particular section and demonstrates relatively robust performance even under minor variations in prompt composition. We will add the results from all models in the table like above to the final version. Additionally, we have already visualized which sections are given more weight for each model in Figure 4 of the main paper and Figure 7 of the Appendix C.1 section. We hope this will also be helpful in resolving your question, so please refer to it.\n\nYour insightful comments and thoughtful suggestions have significantly improved the clarity and persuasiveness of our work. We will actively incorporate all your feedback into the final version.\n\nWe sincerely thank you again for your valuable time and feedback, and we hope this response has satisfactorily addressed your concerns.\n\nThe appendix was submitted during the supplementary materials period and is available in the zip file linked next to *“Supplementary Material”* at the top of this page.  Due to changes in the review system, we are unable to submit a new PDF file—thank you for your understanding.\n\n---\n**References**\n\n[1] Ye, Junyi, et al. \"Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems.\" AAAI 2025.\n\n[2] Yue, Albert S., et al. \"HARP: A challenging human-annotated math reasoning benchmark.\" 2024."}, {"review_id": "ykCrmOKTKs", "reviewer": "Reviewer_3d1n", "summary": "This paper proposes CLAWS, a white-box method for detecting Creative, Typical, and Hallucinated solutions generated by LLMs for mathematical problems. The approach leverages attention weights segmented by prompt sections (Guideline, Problem, Solution, Instruction, Response) to classify generations based on their attention distribution. Using a structured prompt design and evaluation by strong frontier LLMs (Gemini, GPT-4-mini), the authors label over 4.5K math solutions and demonstrate that CLAWS outperforms five white-box baselines (e.g., PPL, Logit Entropy) across multiple LLMs and datasets.", "questions": "- Could you provide data on the agreement rate between your two LLM evaluators (Gemini 1.5 Pro and GPT-04-mini)? More importantly, have you considered validating their outputs against those of human mathematics experts on a subset of the data to benchmark their reliability for this novel task?\n- For the baseline methods, how is the \"Prototype\" evaluation strategy implemented? These methods produce a single scalar feature, and it is unclear how Euclidean distance to a class prototype is meaningfully applied in a 1D space. Please clarify this.\n- How sensitive is CLAWS to the specific phrasing and structure of the prompt sections? Would minor variations in the \"Guideline\" or \"Instruction\" sections, for example, lead to substantially different attention patterns and degrade classification performance?\n- You note that the prototype-based strategy struggled with the OREAL model due to its high rate of hallucinated outputs. Does this point to a broader limitation of the prototype strategy in cases of severe class imbalance? How does CLAWS perform on OREAL for the 3-class detection task when using a trainable classifier (e.g., MLP) instead of the prototype method?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The idea of using attention weights segmented by prompt sections for creativity/hallucination detection is new and elegant.\n- The empirical evaluation is extensive. CLAWS is tested against five relevant baselines across five different LLMs and on a large, challenging dataset of math problems. The use of multiple evaluation strategies (Prototype, MLP, etc.) and metrics strengthens the claim of the method's superiority.\n- The method is computationally efficient, requiring no additional model calls or complex operations beyond accessing and averaging attention weights from a single generation pass.\n- Despite some language issues, the core idea, setup, and contributions are clearly communicated.\n\n\nWeakness\n- While the paper acknowledges the subjectivity of creativity, it largely sidesteps deeper discussion or formal operationalization of what constitutes “creativity.”\n- The validity of the entire experimental setup rests on the quality of the labels produced by Gemini 1.5 Pro and GPT-4o-mini. While these are powerful models, the paper provides no direct evidence (e.g., a comparison with human experts on a sample, or inter-rater reliability between the two LLMs) to validate their performance on this specific, subjective task of creativity assessment. This introduces a potential source of unquantified noise and bias into the \"ground truth\" labels.\n- Although the guidelines for creativity are reasonable, they are somewhat subjective and rely heavily on the labeling from other LLMs. Some human verification could strengthen the validation.\n- The construction of positive/negative pairs may encode spurious cues or biases (e.g., formality, length, genre), which could confound the learning objective. This is not fully analyzed.", "was_revised": true, "final_justification": "I gave Borderline accept (4) initially and the authors partially addressed my concerns, but I don't think it is worth Accept (5) due to the inherent limitations on some aspects such as evaluation.", "rebuttal": "We sincerely thank you for your valuable comments and questions. Below is our detailed response to your concerns:\n\n**A1. Reliability of LLM Evaluator-Based Labeling:**\n\nWe fully acknowledge and empathize with your concerns regarding the reliability of LLM evaluator-based labeling, as well as the necessity of human evaluation. First, the agreement between the two LLM evaluators was measured by Cohen's kappa score, which yielded a value of 0.741. This corresponds to a statistically defined substantial agreement, indicating a high and reliable level of consistency between the two evaluators.\n\nRegarding human evaluation, the dataset used in this study consists of mathematically challenging problems, which makes it difficult for human annotators to directly assess the creativity of the generated solutions. However, since the reference solutions used as the basis for creativity evaluation were all written by humans, and the generated responses were compared against these solutions, we would like to note that—although no direct human evaluation was conducted—the diverse and creative human-written solutions were indirectly reflected in the evaluation process.\n\nFurthermore, a creativity evaluation method that relies solely on LLM evaluators was proposed and academically validated in prior work [1], which served as the primary motivation and background for our study. Building on this successful research, we propose CLAWS, a framework for evaluating creativity without human intervention. This constitutes one of the main contributions of our work, offering new possibilities for automating creativity assessment. \n\n**A2. Clarification on the Applicability of the Prototype Strategy to Baseline Methods:**\n\nThank you for the insightful question. We provide detailed explanations of the five evaluation strategies used in our study in Appendix B.2, with the prototype strategy specifically described in lines 1105–1112. Although we mentioned in the captions of Table 2–3 that the evaluations were conducted using both the threshold and prototype strategies, your question made us aware that the scope of the prototype strategy was not described sufficiently clearly.\n\nTo clarify, the prototype strategy was applied only to CLAWS, whereas the threshold strategy was applied only to the five baseline methods. Since CLAWS uses the values of the five sections as features, it was able to compute the Euclidean distance accordingly.\n\nWe have supplemented the context in both the main text and the table captions to communicate this distinction more clearly. We sincerely appreciate you for pointing out this weakness where our explanation was lacking.\n\n**A3. Impact of Prompt Section Variations on CLAWS Performance:**\n\n|Dataset|Metric|w/o $G$|w/o $P$|w/o $S$|w/o $I$|w/o $R$|Full|\n|---|---|---|---|---|---|---|---|\n|TEST|F1 $_w$|58.59|58.81|50.01|54.68|58.39|**58.66**|\n||F1 $_m$|46.01|**46.35**|39.45|43.87|45.97|46.01|\n||AP $_m$|41.13|41.04|38.46|40.29|40.60|**41.17**|\n||AUROC|62.12|**62.21**|59.15|61.64|61.58|62.09|\n|AMC|F1 $_w$|46.66|46.68|46.48|45.52|46.49|**46.71**|\n||F1 $_m$|39.59|39.33|39.59|38.16|40.20|**40.99**|\n||AP $_m$|37.15|37.08|36.91|36.70|37.14|**37.16**|\n||AUROC|56.11|56.23|56.00|55.80|56.52|**56.40**|\n|AIME|F1 $_w$|55.88|53.33|54.92|51.37|54.01|**56.90**|\n||F1 $_m$|37.35|35.52|36.82|34.80|34.84|**38.12**|\n||AP $_m$|35.25|35.20|34.98|35.03|35.15|**35.38**|\n||AUROC|54.08|53.38|53.51|52.47|52.67|**54.47**|\n|A(J)HSME|F1 $_w$|37.85|39.92|**40.49**|40.10|34.25|38.82|\n||F1 $_m$|37.46|35.76|37.65|36.02|34.11|**37.64**|\n||AP $_m$|36.16|35.51|35.78|35.69|35.42|**36.25**|\n||AUROC|**55.05**|53.88|54.33|54.18|53.08|54.40|\n\nIn response to the question regarding sensitivity to specific phrasing and the structure of the prompt sections, the table above presents the performance changes observed when applying the prototype strategy to responses generated by the DeepSeek model, with each section removed individually. While the removal of a specific section did affect performance in some cases, it is difficult to conclude that any single section dominates CLAWS's overall performance. In most cases, the Full configuration, which uses all sections together, exhibited the most stable and superior performance.\n\nThese findings suggest that CLAWS does not rely excessively on any particular section and demonstrates robust performance even under minor variations in prompt composition. We will add the results from all models in the table like above to the final version. Additionally, we have already visualized which sections are given more weight for each model in Figure 4 of the main paper and Figure 7 of the Appendix C.1 section. We hope this will also be helpful in resolving your question, so please refer to it.\n\n**A4. Effectiveness of CLAWS under Class Imbalance Induced by a High Rate of Hallucinated Outputs:**\n\nAs you pointed out, in cases where the Hallucinated solutions rate is high and class imbalance is severe—such as with the OREAL model—the prototype strategy may not perform optimally. Considering that certain evaluation strategies may have advantages or disadvantages depending on the scenario, we presented five different evaluation strategies. The full results are presented in Tables 4–8 of Appendix section, including the MLP strategy mentioned in the question. As you know, in the case of learning layers like MLP, class imbalance can be adjusted by weighting the loss, and we have already achieved good results by doing so.\n\nIn summary, CLAWS consistently outperformed the baseline methods in most settings and maintained robust performance even under conditions of severe class imbalance or limited data. Additionally, the results of supplementary experiments conducted under class-balanced conditions are presented in Tables 14–19 of Appendix, and we kindly ask you to refer to them as well.\n\nYour insightful comments and thoughtful suggestions have significantly improved the clarity and persuasiveness of our work. We will actively incorporate all your feedback into the final version.\n\nWe sincerely thank you again for your valuable time and feedback, and we hope this response has satisfactorily addressed your concerns.\n\nThe appendix was submitted during the supplementary materials period and is available in the zip file linked next to *“Supplementary Material”* at the top of this page.  Due to changes in the review system, we are unable to submit a new PDF file—thank you for your understanding.\n\n---\n**References**\n\n[1] Ye, Junyi, et al. \"Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems.\" AAAI 2025."}, {"review_id": "skjTjofthh", "reviewer": "Reviewer_yBrH", "summary": "The paper introduces a new white-box method to classify LLM-generated mathematical solutions into the three categories: 'Typical', 'Creative', and 'Hallucinated'. \n\nIt partitions the context into 5 areas:\n* Guideline: Providing the model with the concept and standard for identifying creative \n* Problem P: The math problem for the LLM to solve.\n* Reference Solutions S: A set of 1 to _n_ ‘typical’ solutions to the problem\n* Instruction I: An instruction to create a novel solution\n* Response R: The output from the LLM\n\nThe average amount of model attention distributed across these five different areas is used to create a 5 element feature vector which is compared against reference features for the three categories. A response’s feature vector used to classify the response based feature vectors of known examples of each class. \nThe authors generate a ‘reference set’ dataset using five 7-8B math-specialized LLMs and labeling their outputs using two larger, state-of-the-art LLMs (GPT-o4-mini and Gemini 1.5-Pro) as evaluators, avoiding any human annotation\n\nExperiments show that CLAWS significantly outperforms five standard white-box hallucination detection baselines.", "questions": "The proposed method is as far as I am aware very significant and original. However there are quality issues that could be substantially improved. I think the quality could be improved if the authors can convincingly address the following critical points.\n\n* The most critical issue is that the results depend on quality labels from LLM evaluators; labels which have not been validated. It would be good to see a measure of the raw agreement between GPT-o4-mini and Gemini 1.5-Pro on the 'Correctness' and 'Creativity' dimensions, e.g. Cohen's Kappa. More importantly, it needs to made clear how these LLM-generated labels compare to judgments from qualified human experts on a representative subset of the reference dataset. Without this, the claim of detecting \"creativity\" is unsubstantiated. The fact the Qwen2.5-14B and DeepSeekV2-16B models ‘lack of ability to creatively solve mathematical problems’ rings alarm bells - could ‘creative’ solutions are ones with correct answers but incorrect reasoning. Providing strong evidence here would be the most effective way to raise your score.\n* Layer-wise Contribution to Attention Signal: CLAWS averages attention weights across all layers. Have you investigated the effect of using different layers? It is known that different layers play different roles so an ablation study showing performance using different layers, e.g. using only top, middle, or bottom layers could provide valuable insights and potentially lead to a stronger method.\n* Definition and Scope of \"Creativity\": What is your rationale for the ‘inclusive criterion’ rather than a stricter one where both LLMs need to agree? How does it affect the results?\n* Completeness of the Submission: The current submission has several \"[TODO]\" markers in the checklist and references appendices that are not included. Can you confirm that a final version would include all prompts, hyperparameter details, and completed sections necessary for full scrutiny and reproducibility?", "limitations": "No, the authors have not adequately addressed the limitations of their work. The paper lacks a dedicated \"Limitations\" section and does not discuss potential negative societal impacts. \n\nI strongly recommend the authors add a dedicated Limitations section that discusses:\n* Evaluation process: The most critical limitation is the reliance on unvalidated LLM evaluators for ground truth. The authors should be upfront about this, discuss the potential for circularity, and state what would be required for proper validation (e.g., human expert studies).\n* Applicability: CLAWS is restricted to open-weight models, preventing analysis of the most SOTA proprietary models behind APIs. This should be stated clearly.\n* Generalizability: The method's reliance on a specific prompt structure with ‘typical’ solutions may not generalize to other creative tasks and this should be discussed.\n* Potential Negative Societal Impact: The authors should discuss the danger in using LLM-generated labels in evaluation high-stake scenarios, e.g., assessing student work, where ground truth validity has not established.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "CLAWS provides a potentially simple yet powerful method to classify LLM responses quality. However, the experiment results rely on an unvalidated assumption about the reliability of LLM-based evaluators, which significantly undermines their credibility.\n\n# Strengths\n* Significance & Originality: Evaluating hallucinations and creative responses is crucial to the progress of LLMs and this paper attempts to tackle this with a relatively simple method to classify responses not based on the attention used to generate them. The method of using attention distribution over prompt sections to infer the nature of the generation (Typical, Creative, or Hallucinated) appears original and intuitive.\n* Simplicity: The CLAWS method is simple and computationally efficient as it does not require multiple LLM calls or external model calls for evaluation.\n* Apparent Rigor of Experiments: The use of 5 open-weight maths reasoning models and multiple evaluation metrics demonstrates a good approach to testing the method with the labels in the reference set (deficiencies in the reference set not withstanding (see below))\n* Figure 4 clearly demonstrates how CLAWS distributes the three categories in feature space.\n\n# Weaknesses\n* Flawed in Evaluation (Quality): The paper's most significant weakness, calling into question its empirical contribution, is its use of a reference dataset’s ‘ground truth’ labels are generated by other LLMs (GPT-o4-mini, Gemini 1.5-Pro). \n  * Unsupported Ground Truth: Using an LLM as a quality judge (e.g. creativity) without any validation is scientifically unsound. No evidence is provided of the LLM evaluators' accuracy, their agreement with each other (e.g., inter-annotator reliability), or, most importantly, their correlation with human expert judgment. \n  * Risk of Circularity: Without grounded labels there is a danger the authors are measuring how well aligned attention distribution in these small models is to GPT-4 and Gemini labels, rather than an objective notion of creativity. \n  * Lack of Transparency: Crucial details for this labeling process, such as the exact evaluator prompts (mentioned as Appendix A.1), are not provided, making it impossible to scrutinize or reproduce this critical step.\n* Clarity & Completeness: The paper is generally well-written, but key details are missing. None of the referenced appendices (detailing hyperparameter settings and the LLM evaluator prompts) are included - making a full assessment of the work's reproducibility and methodological details difficult. Furthermore, The checklist at the end of the paper is filled with \"[TODO]\", indicating an incomplete submission.\n* Limited Scope and Generalization: The method only works with open-weight models and relies on very specific prompt structure. Furthermore, the paper does not discuss how this framework might generalize to other creative tasks ‘typical; examples might not exist.", "was_revised": true, "final_justification": "Thank you for the rebuttal responses which have clarified your submission.\n\nI am happy to up my rating in antipation of the authors updatig their paper to include the information they have provided in the rebuttal.", "rebuttal": "First and foremost, we would like to sincerely thank you for carefully and thoughtfully reading our paper and for providing such an insightful and constructive review. We deeply empathize with the major concerns you have raised, many of which are already addressed in the appendix section. Please note that the appendix, which was submitted during the supplementary materials period, can be found in the zip file linked next to the *“Supplementary Material\"* at the top of this page. We kindly ask for your understanding that, due to the updated NeurIPS review system, it is not possible to submit an additional PDF file.\n\n**Clarifying the Dataset and Expanding Limitation Discussion:**\n\nBefore addressing the individual questions, we would like to clarify one important point. In the first weakness point, it was stated that we used solutions generated by other LLMs as ground truth. However, this is not true. The CreativeMath [1] and HARP [2] datasets contain diverse reference solutions manually written by humans. We evaluated the creativity of model-generated responses based on these human-written solutions. We apologize for any confusion caused by our explanation, and we will revise the dataset description to clarify this point in the final version.\n\nAdditionally, we have separately described the limitations of the paper in Appendix A, and we kindly ask you to review them as well. Furthermore, based on your comments, we will address several limitations that are not currently stated—such as the potential circularity of the LLM-based evaluation, the limitation that CLAWS is restricted to white-box settings, and the risks associated with its application in high-stakes scenarios. Lastly, we would like to note that the limitations concerning generalization to broader generation tasks have already been acknowledged as future work in the limitations section.\n\nWe now provide detailed responses to each of the questions:\n\n**A1. Reliability of LLM Evaluator-Based Labeling:**\n\nAs the you mentioned, we deeply empathize with the concerns regarding the reliability of LLM-based evaluation and the necessity of human evaluation. We also gave careful consideration during the early stages of this study to the extent to which the LLM evaluator could understand the proposed criteria and make consistent judgments.\n\nFirst, regarding human evaluation, the dataset used in this study consists of mathematically challenging problems, posing practical limitations in that evaluating the creativity of generated solutions requires significant expertise and time resources. However, since the reference solutions used as the basis for creativity evaluation were all written by humans, and the generated responses were compared against these solutions, we would like to note that—although no direct human evaluation was conducted—the diversity and creativity embedded in human-written solutions were indirectly reflected in the evaluation process.\n\nFurthermore, a creativity evaluation method that relies solely on LLM evaluators was proposed and academically validated in prior work [1], which served as the primary motivation and background for our study. Building on this successful research, we propose CLAWS, a framework for evaluating creativity without human intervention. This constitutes one of the main contributions of our work, offering new possibilities for automating creativity assessment.\n\nHowever, as you expressed concern, ensuring the reliability of LLM-based evaluation is essential. To this end, we implemented a process to verify the validity of the evaluation by requiring that the LLM evaluator not only output a label but also generate a supporting sentence for each decision. Below is an example of a supporting sentence used in the actual evaluation:\n\n- ```The new solution uses a completely different approach (Lifting The Exponent Lemma) compared to the reference solutions (difference of squares factorization). This satisfies criterion 1 for novelty.  While the final result is the same, the intermediate steps and the mathematical tools used are entirely distinct, satisfying criterion 2.  The LTE lemma also relies on different underlying principles and conditions than simple algebraic manipulation, satisfying criterion 3.  Therefore, the new solution is considered novel.```\n- ```The new solution is not novel. It uses the same logical deductions as the reference solution.  Both solutions arrive at the same answer through the same constraints. The core logic—Carl not being next to Bret, and Abby not being between Bret and Carl—is identical. The new solution merely restates these conditions and elaborates on the placement possibilities without introducing any new mathematical techniques, different assumptions, or a more generalized approach. Therefore, it does not meet any of the criteria for novelty.```\n\nWe will include these supporting sentence examples in the appendix to allow readers to directly assess the legitimacy of the evaluation labels and the consistency of the LLM evaluator’s reasoning.\n\nThe inter-evaluator agreement based on Cohen’s kappa score will be addressed in our response to Comment 3.\n\n**A2. Clarification on Layer Usage and Layer-Wise Analysis:**\n\nWe would first like to clarify that CLAWS uses only the attention weights from the last layer, not from all layers. We appreciate your comment, which made us realize that this point was not clearly stated in the main paper. Regarding the question about layer-wise analysis, as you mentioned, it is generally known that each layer plays a different role. The baseline methods [2] we compared against also analyzed such inter-layer differences, so we considered this point from the beginning.\n\nAccordingly, we saved the attention values for all layers and heads as ```.pkl``` files and used them to compare performance across layers. The resulting performance differences were mostly marginal, so we opted to use the last layer. Due to space constraints, we could not include all experimental results, but the table below presents the prototype strategy results on the TEST set generated by the DeepSeek model as a representative case.\n\n|Layer|F1 $_w$|F1 $_m$|AP $_m$|AUROC|\n|---|---|---|---|---|\n|5|57.00|45.02|40.15|60.58|\n|10|58.03|45.33|40.42|61.40|\n|15|56.94|44.76|39.81|60.22|\n|20|57.89|45.61|41.08|61.93|\n|25|57.25|44.98|40.41|61.37|\n|Last(30)|58.66|46.01|41.17|62.09|\n\nThese results indicate that CLAWS does not rely on any particular layer and operates robustly across various configurations. While we adopted the commonly used last layer, we agree with you that many researchers may be interested in layer-wise performance differences. Thank you for the insightful suggestion. We will include the layer-wise performance table and further clarification in the final version.\n\n**A3. Rationale for Using an Inclusive Criterion in Creativity Evaluation:**\n\nWe would like to explain our rationale for adopting an inclusive criterion—rather than a strict one requiring agreement from both LLM evaluators—when determining creativity.\n\nAs in prior work [1], this study relied solely on LLM evaluators—without human involvement—to assess the creativity of generated responses. In [1], the authors employed three LLM evaluators and adopted a majority voting strategy. In our case, we used only two evaluators. While [1] employed GPT-4o as one of the evaluators, we upgraded this to o4-mini, which offers stronger performance in mathematical evaluation. \n\nSince we could not rely on majority voting with only two evaluators, we instead adopted an inclusive criterion. To verify the reliability of this approach, we measured inter-evaluator agreement using Cohen’s kappa score, which yielded a value of 0.741. This indicates substantial agreement between evaluators and supports the consistency of their judgments. Given that creativity assessment inherently involves a degree of subjectivity, we believed that requiring both evaluators to agree might risk overlooking genuinely creative responses. The inclusive criterion thus serves to mitigate such underestimation.\n\n**A4. Completeness of the Submission:**\n\nWe submitted the full appendix—including all experimental results and implementation details—during the Supplemental Material Submission period. The appendix section includes all necessary information for reproducibility, such as example prompts used in the LLM evaluator’s assessments (Figures 5–6) and hyperparameter configurations (Section B.2). Based on the submitted appendix, the \"Todo\" section  has also been completed. We would like to note that the final version will include a fully consolidated appendix that incorporates your comments.\n\nYour insightful comments greatly helped us enhance the clarity and persuasiveness of our work. We will actively incorporate all your feedback into the final version.\n\nWe sincerely thank you once again for taking the time to provide such thoughtful feedback. We truly hope that our responses have addressed your concerns, and we kindly ask you to reconsider the contributions and completeness of our work in light of the clarifications provided.\n\n---\n**References**\n\n[1] Ye, Junyi, et al. \"Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems.\" AAAI 2025.\n\n[2] Sriramanan, Gaurang, et al. \"LLM-Check: Investigating Detection of Hallucinations in Large Language Models.\" NeurIPS 2024."}]

Author rebuttal:

Meta-review: This paper proposes a new mechanism for evaluating creativity, hallucinations in open-weights models. There were two broad themes of consensus among reviewers (after engaging with the rebuttals). (1) A serious concern is the precise definition of creativity and measurement via LLMs. Authors felt key details were missing regarding evaluation, and also there was no discussion on human inspection for quality or even some light exploratory analysis on the quality of labels. For a paper who's abstract talks about challenges in evaluating creativity, the proposed evaluation should have been carefully assessed. The author responses addresses some of this, points to principles used and prior work etc. (2) All the reviewers also found the method proposed interesting and potentially broadly useful beyond the specific task of creativity (like assessing hallucinations). This could spur more research into related methods and would add value to the Neurips conference. Balancing (1) and (2), I currently lean acceptance but if there's limited space, it would be okay to reject the paper while encouraging the authors to heavily reframe their paper around hallucinations and reduce the emphasis on creativity / more carefully justify the creativity evaluation.
Decision: Accept (poster)