Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation
Authors: Yiming Wang
Pei Zhang
Baosong Yang
Derek F. Wong
Rui Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Label-free Self-Evaluation
AI Reliability
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose the Chain-of-Embedding method for LLM self-evaluation, which enables output-free response correctness estimation during inference time.

Abstract: LLM self-evaluation relies on the LLM's own ability to estimate response correctness, which can greatly improve its deployment reliability. 
In this research track, we propose the Chain-of-Embedding (CoE) in the latent space to enable LLMs to perform output-free self-evaluation. CoE consists of all progressive hidden states produced during the inference time, which can be treated as the latent thinking path of LLMs. We find that when LLMs respond correctly and incorrectly, their CoE features differ, these discrepancies assist us in estimating LLM response correctness. Experiments in four diverse domains and seven LLMs fully demonstrate the effectiveness of our method. Meanwhile, its label-free design intent without any training and millisecond-level computational cost ensure real-time feedback in large-scale scenarios.
More importantly, we provide interesting insights into LLM response correctness from the perspective of hidden state changes inside LLMs.

Published as a conference paper at ICLR 2025
LATENT SPACE CHAIN-OF-EMBEDDING ENABLES OUTPUT-FREE LLM SELF-EVALUATION
Yiming Wang". a.? Pei Zhang 3.7 Baosong Yang". S* Derek F. Wong" Rui Wang" 3**
"Department of Computer Science and Engineering, Shanghai Jiao Tong University [yiming. wang, wangruill 12 @ssjtu. edu. on
Tongyi Lab, Alibaba Group Inc. "NLP'CT Lab, University of Macau yangpaosong. aataaaallaaainnnn ciceaaaiin.ccc.o. com
ABSTRACT
LLM self-evaluation relies the LLM's own ability to estimate response correct- ness, which can greatly improve its deployment reliability. this research track, we propose the Boein-of-Embeddeee
(CoE) in the latent space to enable LLMs to perform output-free self-evaluation. CoE consists of all progressive hidden states produced during the inference time, which can be treated the latent thinking path of L.Ss. We find that when LLMS respond correctly and incorrectly, their CoE fea- tures differ, these discrepancies assist in estimating LLM response correctness. Experiments in four diverse domains and seven LLMs fully demonstrate the effec- tiveness of our method. Meanwhile, its label-free design intent without any training and millisecond- computational cost ensures real-time feedback in large-scale scenarios. More importantly, we provide interesting insights into LLM response correctness from the perspective of hidden state changes inside LLMs. The code is public at: https: thub. com/ A1 sa Baace08/Chie n- o Embeddi ng
INTRODUCTION
teboudding
Layen
Language Model
lead
Large Language Models (LLMs) have ex- cellent abilities generalize across diverse scenarios (Achiam et al. 2023 Guo et al. 2025). However, their outputs can some- times be unstable, leading to incorrect re- sponses that may threaten social safety. Therefore, label-free LLM self-evaluation has emerged as a crucial research area, which estimates the correctness of LLM responses fully through LLMs' own capabilities. It can provide real-time response monitoring and feedback in large-scale employments, enhancing the reliability of LLMs (Sun al. 2024
Inport Text
Figure 1: Chain-of-EEmeeddiie we propose in this paper mirrors the latent thinking path of LLMs, which may reflect LLM response correctness during inference time.
Popular self-evaluation research in the era of LLMs focuses more on output-based forms Zhang 2023 Two typical paradigms that do not assess the internal states of LLMs involve directly
al.
asking LLMs to express confidence in their responses through well-designed prompts (Lin ct al.
2022a Tian al. 2023), and generating multiple responses by perturbing prompts (Gao al. 2024]
or decoding sampling (Wang et al. (023)) to calculating the response consistency (Xiong et al. 2024
Besides the two types, other methods basically draw on uncertainty estimation concepts from the era of deep neural networks, leveraging output logits or probability distributions to gauge the confidence
of model responses (Malinin Gales. 2020 Si et al. 2022] Huang et al. 2023 Kuhn al., 2023
Recently, some research has revealed that the latent space of L.LMs contains substantial amount of untapped hidden state information, they can largely reflect response correctness (Azaria & Mitchell
2023 Liu et al. 2023 Duan et al. 2024), and are usually more interpretable than LLM output (Li et al. 2024a). However, these output-fres free research often require correctness labels 0/1 for training
probing classifiers extract features from hidden states (Burns et al.) [022 (Sky et al.) 2024 Su et al.,
2024). This contradicts our goal of being "label-free" and limits the generalization capabilities on unseen data. To further expand this research line, consider challenging but valuable question:
How to solely utilize hidden states to estimate the LLM response correctness without any label?
"Corresponding Authors. "Wor done during Yiming's 's internship Tongyi Lab, Alibaba Group Inc.

---

Published as a conference paper at ICLR 2025
To answer this question, we start from the perspective of human thinking: In the cognitive theory, human thinking is accomplished collaboratively by intuitive thinking (system 1) and deliberative thinking (system 2) (Evans] 2003): correct thinking tends to activate system 2 to produce more deliberative thinking paths, while incorrect thinking tends to be affected by system I to make more rapid and direct thinking paths (Kahneman 201T). This cognitive phenomenon means that human thinking paths may differ when responding correctly and incorrectly.
Next, we draw an analogy about the latent thinking path from humans LLMs: Some research (Peters et al. 2018 [Tnneyy] 2019) Jawahar et al. 2019] Chen et al. 2020) has demonstrated that large Transformer based (Vaswani et al.) 2017) language models generate text representations by first encoding morphological and syntactic information in the lower layers, then progressing to more complex semantic information in the higher layers. This means that the hidden state changes can mirror the interpretable progressive thinking of LLMs in the latent space. Moreover, some LLM mechanistic studies also utilize multiple hidden states to explore the thinking states at different stages of LLMs (Ye et al. 2024). Therefore, we can treat the progressive hidden states the latent thinking path of LLMs, which we term Chain-of-Embeddeeee Emeeddingg (CoE), as shown in Figure] [
Based on these analyses, we boldly migrate human thinking patterns to LLMs and make the following assumption: CoE discrepancies may happen when LLMs generate correct and incorrect responses. Starting from this assumption, our paper structure and contributions are as follows: In Section [ we first explore the CoF discrepancy when L. respond correctly and incorrectly by quantifying its features demonstrate this assumption. In Section R
we propose comprehensive CoB-based metric for label-free LLM self-evaluation. In Section 4 we verify the performance of our CoE method from four diverse domains that are popular in the LLM ability test: Matherattics, Reasoning, Knowledge, and Understanding, to demonstrate the effectiveness of CoE for self-evaluation. In Section 5 we conduct theoretical analyses further to present more insights about the effectiveness of our method.
Problem Statement. Label-free LLM self- aims to estimate whether the LLM response to given input question is correct or not fully through LLMs' own capabilities without relying
any true label, external tool, and supervised trainer (Chen & Mueler 2023 [ et al. 2024c). We denote the language model as f. For each sample (æ., y) + (question, true answer), the output of the language model is denoted as ÿ = fx=). Then this sample is associated with a decision score = S(x, ú. f). where S(.) is decision function derived solely from the question, LLM output, and language model information - without reference to the true label y. The domain of this function encompasses the entire sample and model space. An ideal decision function aims to achieve the following goal: higher score indicates greater likelihood of a correct answer. Therefore, the self-evaluation task can be formulated as a binary classification problem. threshold, for a question, whether the LLM response is correct can be discriminated as the instruction function x ()) Correct s > 7 else Incorrect. The core goal is to find an optimal threshold to improve the classification accuracy.
Let y be the decision
2
CHAIN-OF-EMBEDIIN REFLECTS RESPONSE CORRECTNESS
2.1 DEFINITION: LATENT SPACE CHALN-OF-EMBEDDIAN (CoE)
Formalization. We start by formalizing the CoE under the language model f. Assume the model has L hidden layers, we can decompose into the following ordered sub-modules:
f f= fhead o fr. o ... ft o o fi o fenh-
(1)
In Eq freat R. - IR MM denoted as fo. is the embedding mapping layer (0-th layer), and each f( s / s L) Rd R is the intermediate hidden layer. Here d is the embedding dimensions and V is the model vocabulary.
the final classification layer, fenb : E
E. which can also be
Given a question æ as input to f. the output 9 consists of T tokens &1 0r. For the t-th token, we denote its hidden state at layer 1(0 s / s L.e.e. the t-th output embedding of function o.... fi o fo- as zf. Following the definitions in Ren et al. 2022) p: Wang et al. (2024a),
define the average embedding at layer as h, + Σi= 2/, which represents the I-th sentence hidden state. Then, the CoE is expressed as progressive chain H of all sentence hidden states formalized follows:
H
ho
h
h,
h
h
(Chain-of-Embeddiing
Input State
Intermediate Hidden States
Output State
2

---

Published as a conference paper at ICLR 2025
Feature Definition. After formalizing the CoE, need to quantify its features so that we can utilize them to validate the assumption proposed in Section I
"CoE discrepancies may happen when LLMs generate correct and incorrect responses". We create continuous CoE trajectory in the latent space by performing segmented linear interpolation (simply connecting adjacent states) on the CoE. To measure the trajectory feature, its geometric information is the most fundamental dimension Hielland-Hanseer & Hampson [2009] Rintoul & Wilson 2015), which usually includes magnitude and angle that can reflect the distance and direction changes produced during the trajectory wandering.
We briefly examine the practical significance of these two features in measuring the LLMs' thinking path. The magnitude feature is undoubtedly the direct feature of the thinking path curvature. In contrast, while the angle feature does not explicitly represent the thinking path curvature, the cosine value between two embeddings indicates semantic similarity (Rahutomo et al. 2012). This suggests that the angle feature can indirectly reflect the thinking path feature at the semantic modeling level.
Now, we first define changes in magnitude and angle between each adjacent state pair (hy, 1 I s L 1). The magnitude change M (h,, hx+1) is quantified using the L2-norm, while the angle change A(h; h1+1) is derived by indirectly calculating the cosine value between the two vectors, and then transformed using the are cosine function. The two measures are formalized as follows:
h hy
M M(,, h(+ 1) || hell2, A(h 1(+1) arccos
(2)
|| 112 1|
Subsequently, the magnitude and angle features of the whole CoE trajectory, denoted as Mag H) and Ang (H). can be defined as the average changes in magnitude and angle between each adjacent state pair. The two features are formalized as follows:
1 L-1 Mag( H) = - Σ
M(h h111)
1 L-1
A(hy, h1t1)
Ang H) (H) =
Σ 1-0
(3)
Zidag
Z
In Eq 3 to reduce potential sample biases, we set the range scaling factors ZMag = M h2) and Z Ang A(ho, h2) for the following reason: The positions in the input space and output space of different samples may vary. if the input and output of one sample are far apart in the latent space, its trajectory naturally has a longer wandering distance. By setting scaling factors, we convert the absolute magnitude and angle changes of each adjacent state pair into relative changes, specifically, the changes of (h, h+++)) is relative to the changes of the onput-output states (ho, ht)), thereby avoiding measurement noise caused by inherent differences between samples.
2.2 How CoE REFLECTS RESPONSE CORRECTNESS
Setup. After defining CoE and its features, we aim to explore the generalized impact of CoE on the LLM response correctness. Therefore, we focus four popular domains: Mathematics, Reasoning, Knowledge, Understanding, with each domain set MATH, TheoremQA, MMLU, and Belebele as domain datasets separately. Dataset details and citations can be found in Section 4. In each domain dataset, we divide the correct and incorrect samples into two sets, with each corresponding to feature set, denoted as V_ = {(Mag .Ang," )>> and V- = {MMgg Ang, } Additionally, we use the Dween--BBInstruuct (Yang et al. 2024) model the backbone. These settings apply to all experimental analyses in this section (Figure and B and will not be repeated hereafter.
CoE Feature Distribution Discrepancy. Now, we quantify the CoE feature discrepancies between correct and incorrect sample sets for each domain. Due to each sample having two CoE features, we employ 2D kernel density estimation to calculate the probability density function (PDF) fv for each feature set V {(Mag, Ang, > which represent the CoE feature distribution of V. We use Gaussian kernel for PDF estimation, expressed as:
1 nhh
1 2x 1=1
exp {-
1 [(Mag Mag,)² (Ang (Ang-Ang,))) Amg.)"
frr (Mag Ang) =
(4)
26
where is the bandwidth, which we set to to maintain consistency with the default value in the Python sklearn library (Pedregosa et al. 201T). We implement Eq using the sklearn library. In each domain, we independently derive the PDFs fv, and fv. for V. and V__ using Eq as illustrated in Figure 2 We find that in all domains, the distributions of correct and incorrect samples do not overlap and show significant discrepancies, with conclusions described below:
3

---

Published as a conference paper at ICLR 2025
0:21
6.00
1.00
Youttuee Aeglle B.10)
Featu s.agg
Raaure 2.10
Feature i.Mag ude
Festure Magnitude
Purtre Sagntade
Feature Magiitude
(a) Mathematics
(b) Reasoning
(c) Knowledge
(d) Understanding
Figure 2: CoE feature distribution correct and incorrect sample sets in four diverse domains. Blue and Red distributions represent the correct samples and incorrect samples, respectively. Datasets and the model used this figure are shown in Section 2.2
"Setup"..
Avee
hrmn Jome
(a) Mathematics
(b) Reasoning
(c) Knowledge
(d) Understanding
Figure 3: CoE trajectory visualization of correct and incorrect sample sets in four diverse domains. Blue and Red distributions represent the correct samples and incorrect samples, respectively. Each trajectory represents the average trajectories of all correct incorrect samples in corresponding datasets, and the shades represent the trajectory standard variances of all samples. We project the CoE into (wo-dimensional space using PCA for dimensionality reduction, which maintains the relative positioning of the data as constant as possible by rotating the coordinate system (Dunteman 1989).
Datasets and the model used in this figure are shown in Section 2.2 "Setup".
CoE magnitude feature Mag of correct samples are more significant than that of incorrect samples, indicating that L.MMs' latent thinking paths are more convoluted when providing correct answers.
CoE angle feature Ang of correct samples are less significant than that of incorrect samples, indicating that LLMs' latent thinking paths at the semantic modeling level are more unstable when providing incorrect answers.
CoE Visualization. These two conclusions directly reveal the essential discrepancies of CoE features. To demonstrate these more intuitively, we visualize CoE as illustrated in Figure Overall, the CoE trajectory is not inclined to choose the shortest path from the input space to the output space; instead, it passes through a semantic space that is quite distant from both the input and output. In detail, the detour phenomenon of the CoE trajectory for correct samples is more pronounced than that for incorrect samples, which verifies its more significant CoE magnitude feature. Furthermore, the intermediate states of the incorrect samples closer to the origin, resulting in larger angle formed before and after the state transition, which verifies its more significant CoE angle feature.
CoE SCORE FOR OUTPUT-FREE LLM CELF-EVALUATION
In Section we have quantified CoE features and highlighted discrepancies between correct and incorrect samples. Next, we wish to create a self-evaluation metric combining the two features, so as to detect the LLM response correctness using comprehensive CoE feature. However, the feature combination is not simple because of their inconsistent magnitudes. We propose two ways as follows:
CoE-R: Real-Space Combination. A straightforward method is calculate numerical summa- tion of the two features. Though adding the two real numbers may seem unmeaningful due to their differing magnitudes, our focus is on the metric relative trends rather than exact numbers, so this way can preserve the metric usability without considering the feature relevance.
we have found that A Ang( (H) is inversely proportional to the response correctness, so for each adjacent state pair, we use 1- A-,(h;, htt1)//(A A(h.. h2) as the direction change measure, then add it with M(h, Mt+1)/Ad M/ (hn, ht).. By removing extrancous constants, we derive CoE-R(H) score by averaging all adjacent state pair changes with the real-space combination way:
In Section 2.2
L-1
M(h, hur) A(hihh44)
Σ
CoE-R(H)) =
(5)
M(hohhi
A(ho, hz)
1=0
4

---

Published as a conference paper at ICLR 2025
CoE-C: Complex-Space Combination. However, while the exact values of the two features may not be critical, they can significantly interfere with each other. particularly one is abnormally large. This interference can weaken the overall impact of another feature, resulting in the instability of the CoF-R metric. Therefore, we aim to combine the two features more seamlessly.
The magnitude and angle features enable clear association of complex numbers in the complex plane, with each point uniquely represented by its complex magnitude and complex argument. Therefore, for each adjacent state pair, We combine M (h,, h(+1) ) and A(h;, h1+1) into a new feature point C( C(hy; h4+1) ) on the complex plane, where M (hy; h+1 represents the complex magnitude and A(hy, h4r1) represents the complex argument. This feature point C(hi, h141) can be expressed as:
C(b,, lu-1)) M(,, l ((12))
MM,,, 1 cos( (A(h;, h)s1)) + + -Mi,,, (M(h,,hi1)) sin (ann(A(hi, hy+1));
(6)
Real Pur: Re(C(h;,Aa-1))
Imaginary Part: Lan(C(tu,
where i is the imaginary unit. Each adjacent state pair corresponds to one feature point, we then average these L feature points by separately averaging their real and imaginary parts. The magnitude of this averaged point yields the final CoE-C(H) score with the complex- space combination way:
COE-C(H)
((CC(() +
(n(C(h))nin)))
I
1=0
1=0
(7)
1 L
Σ M(,, (u-1) cos(((t,, (e-1))
Σ
M h(+1) Sin(A((t (()1))
Min
This combination metric still tends to capture relative trends, while fundamentally aiming to establish more robust mutual constraints between the two features, in contrast to CoF-R. First, the monotonicity of CoE-C and CoF-R remains consistent and does not contradict the conclusions of Section 2.2 Second, CoE-C exhibits less variation than CoE-R when confronted with outliers, allowing for fewer deviations from the feature distribution of the corresponding class. See Section for concrete proofs.
Figure shows the Computation Sketch of the two CoE scores, and the complete Algorithmic Process of the two CoE scores is shown in Ap-
CoE Trajectory
Ru Miti Bea
pendix B. Higher scores indicate a greater likelihood of obtaining the correct response. In
Nizs)
terms of Computational Complexity, after the base LLM inference and the extraction of all
(Origtn)
CoE Score Computation
hidden states, the computations for both CoE-R and CoE-C involve only ((L)) operations for scalar addition and multiplication, along with O(L) operations for square roots and trigono- metric computations. Since these computations can be executed in parallel by the CPU, the over- all computational burden is negligible. Section 4.3 will provide a detailed efficiency analysis of the execution time in practice.
M(Ar Arsa)
(Im)
Point (x.)z)
CoE-R
CoE-C
Aih, Rugs)
Average Point x'
(Re)
Figure 4: Sketch of CoE Score computation.
EXPERIMENTAL VERIFICATION
4.1 SETUP
Dataset. We select six datasets across four domains for our self-evaluation experiments. These domains reflect the four critical dimensions of LLM capabilities (Zheng et al. 2024 Huang et al.)
2024 (1) GSM8K (Cobbe et al. (021) and MATH (Hendrycks et al. 2021) for the Mathematics
domain; (2) CommonsenseeAA (Tlmorr et al., 2019 and TheoremQA (Chen al.
2023) for the
Reasoning domain: (3) MMLU (Hendrycks et al. 2020) for the Knowiedge domain; (4) Belebele (Bandarkar et al. 2023) for the Understanding domain. Dataset details are shown Appendix C.1
Language Model. We use instruction-basse models due to their ability to follow instructions, effectively addressing diverse user needs. We mainly adopt 7B+ parameter models with the Zero-Shod
CoT generation paradigm (Wei al. 2022 Kojima al. 2022h. including Llama2-7B (Touvron
5

---

Published as a conference paper at ICLR 2025
Table I: AUROC, FPR95, and AUPR results of all methods in four diverse domains with different LLMs. Italics means that this method does not assess internal states, "f" means that this method requires multiple stochastic inferences for sampling multiple outputs, and "*" means that this method is output- free and only utilizes hidden states. Additionally, underline represents the SOTA performance among all baselines, bold represents the SOTA performance among all methods.
Demsans Maliecatiess CAUROC FPR8
LAAUPR 13
Nobbll Goaf
CD 35
255 alrdeer
22.34 20.042
24
00:221 20.33
00.33 (2.32
75.64133 79
07:22 (8 23.40
70,22 0.
65,33
Perplsky
58 18:81-90 18-1 63
11 30/55
(3.09 1.25
66 37
401 00
06 96 FAA
Temp.
14.36/13
19.43
03.17
56.77
TEUB 92.0 753.34
08.3 (T000) 1312
66-33
83.46
37.00
06.33 (57.21
90.10
71400
(66.99
06.34 (2200/ 3111
Barry
46
50.32
54.127
F85 70:06
195 63-66-550
50:75 21
L. Eierep
61 (56.3
67.007 0000777333 34 /0
(1.09
07.71
60 16/ 35
05.
65. 35 23./g//
Cat-t (dlan)
64.25 94.47 33.54
12.54
19636
38.507 /33.00
25.15
(66000001110 5167
T1777 63.60/4222
(67.1 $4.10
14.31 (5620 02.37
CaE-C (Dams
63 00.05/35
00100122000 20.57
93.6:
79
66.99
11,22/
58.13
7658
66.48 84.57
72:24
17.11 /43.5
20.35
70653 $4.50
06.34
(51.55
77.22/ T3.64 63.56
Thosain El Beaweing
(AUROC
1/19866
Wetkef Glag
24-22
12 202
44.33
23. 50
51.25
47:08
48.3 00:71
3. Man Podss
55
55
54.33 (1177) 5421
20.82
5 Eaares
0/ 73 23.05
35.31 (2200
54.33
Temps Sculing
15,
8. Joc Bropous
20.40/33
53/25200 22.34
51.21
Pigrafaee
43 mm
16.
60:31
11 ae oo 09
0131
06.8
12,62/
59.30
CaB- (Dan))
59,00/
96.69 34.36
67.07/ 56.44
15.571
06.33 88833 68.31
(0099 56.35
63.54/ BRU12/ $7.51
Dematn BI: Keowledge (AUROC 1:19995
L/AUTR
Nachal Cuas
47.00
99 23143
45.77/90
18.77199351/
42.001
4652
51.23
52.22
30,00 (99 90.07
45.20
(960035444
47.22 107 22
Abedher
$2.39
25007
4 Depplaat
46.74
66-11 34.10
5350/19200
(66-00 1236
55.41
26-55 (10.00
66.93 (332000 $9.31
56.70
6. Teep. Sodling
48 97
00. 20:28
61.77
47 36 (91 35 5.000
(WOT
Droppuu
30.50
Evaregs
43
28.
60.22
63.76
85
45 19
$438 02.05 19.35
63.54 07.20 7481
64303 1135
15.9
01.53
Dansin IV: Enddentarding
(AUBOC 1 FF9805 ./ AUPUP 13
PSA pasdbrr
13.21192 50.22 54.54
52.07 00.64 00.37
57.17 ML///MM// 65.64
62,57 (1 20
54,09
90.12
25.07
31 55:40
3 Peylecay
30
(600)
5 Baing
41
46399
62.53
Kosling
5. Eaagy
-2.45
55.25
Foaveys
Baszkoes
54/0
10 /20 80
35.32/211311
13
48:71/ 60 771.5
03.59 00020/022221
$2.11
61.32/
01.17 /10.3/
71.04 (0000 49
10.31 /90.0//10
55 09.00/00 92 20/33.333 15
58.47/99.11 42 990.0
55.117911 31/6441
70.7
08.3050.0--00 94.30
66.70 67.60
61.45
73.32 08.58 48.45
24.56 (76119/977 5055
66.90 63,00/ 92.57
ct al.
)023). Llama3-8B (MetaAl 2024). Qwen1.5-7B (Qwen- Team 2024). Qwen2-7B (Yang
al. 2024], Mistral-7B (Tiang et al.
2023ap. Additionally, to demonstrate the robustness of model
parameter scaling, we also test two
larger models with 70B+ parameters: LJama3-70B [
2024 and Qwen2-72B (Yang et al.
2024). Implementation details are shown in Appendix C.2
Baseline. We select ten label-free self-evaluation baselines for fair comparisons: The first two
represent typical paradigms that do not assess internal states: (1) Verbal Confidence, we select (2)
(PSA) Pipeline (Xiong et 2024). we refer to Gao et al. (2024) to perturb input prompt with special tokens and aggregate sampling outputs based on lexical similarity (Lin et al. 2022c Kuhn et al., 2023) with Rouge-L (Lin 2004); The remaining eight are centered on the idea of uncertainty estimation, which assesses the internal state: (3) Maximum Softmax Probability; (4) Perplexity (Si al., 2022); (5)
P(true) (Kadavath et al. 2022) for its versatility.
Entropy (Huang et al.] 2023); ; (6) Temperature Scaling (Shih et al. (023); (7) Energy [Liu et al.)
Entropy MMalinin & Gales 2020 (10) Eigenscore (Chen et al. 20044. Baseline details are shown in Appendix
2020); (8) Monte-Carle Dropout (Gal & Ghahramani 2016): (9)
C.3
Among these, (7) require only single output distribution, (2) and (8)-(10) require multiple
stochastic inferences for sampling, and (10) only utilize the hidden states.
Evaluation. We select AUROC (Boyd et al. 2013), FPR95, and AUPR (Manning 1999) metrics to evaluate performances. AUROC focuses on the trade-off off between TPR and FPR under different thresholds; FPR95 focuses on the rate of misclassifying samples when TPR reaches 95%; AUPR focuses on the trade-off between Precision and Recall, placing greater importance on the correct pre- diction of positive cases. These three metrics complement each other and can reflect the classification performances from different perspectives (Cen et al. 2021 Hendrycks et al. 2022). Additionally, we use the exact match to obtain the correctness labels 0/1 of LLM responses for evaluation.
4.2 MAIN RESULTS (TABLE
Method Comparisons. First, our methods achieve SOTA performances across almost all sce- narios. In four domains, our CoE achieves average improvement of 8.30% 5.55% %, and
6

---

Published as a conference paper at ICLR 2025
5.52% across three metries compared to the optimal baseline. Notably, an AUROC value below 60 denotes deviation from ideal performance (Xiong et al. [0244. Our CoE exceeds this threshold in most scenarios, underscoring its practical value. In contrast, most basclines fall short of this standard, particularly in the last three domains.
In contrast to the consistently optimal performance of our CoE method across nearly all scenarios, we observe that the performances of other baselines lack stability, with its effectiveness varying significantly across different scenarios. Below, we provide an analysis of other methods:
First, we examine two typical paradigms (1-2) that do not assess internal states. Verbal confidence often exhibits poor performances, mainly due to confirmed overconfidence issue (Zhang et al.) 2024 Xiong et al. 2024). PSA pipeline lacks stability, it can either perform optimally or under- perform. This may stem from unresolved issues related to effective consistency metrics (Manakul
et al. 2023) and inadequate prompt robustness. See Appendix [..122 for more discussions.
Secondly, we examine the uncertainty estimation methods (3-10) that assess internal states.
Single-output-basee -aased methods (3-7) possess the highest stability. While they do not always achieve best performances, they seldom rank the worst in any scenario. However, on the other hand, their performance limitations align with the assertion by Liu et al. (2024) that "the traditional uncertainty estimation can be extremely challenging when applied to LLMs due to the output diversity". Our chosen tasks include challenging datasets like MATH, where LLMs can generate solutions with thousands of tokens. This complexity greatly limits these methods.
In contrast, multiple-output output-based methods (8-10) exhibit greater instability. For example, the LN-entropy method does not consistently surpass its base entropy version and can sometimes perform the worst, highlighting the inherent uncertainties associated with sampling.
We are especially interested in the performances of Eigenscore (10), as it also utilizes hidden state information. However, it falls short of ideal performance, particularly in the Mathematical domain. The motivation behind Eigenscore is the rich semantic information inherent in the embedding space can be utilized (Chen et al. 2024). However, Wang et al. (2024a) has found that the embedding modeling is often inaccurate for mathematical tasks, which may result in underperformance. In contrast, we emphasize the dynamic changes of hidden states, focusing on their behaviors within the latent space rather than on the specific state representations.
Metric Consistency. Among the three metrics, AUROC emphasizes the trade-off between TPR and FPR. FPR95 emphasizes the FPR, and AUPR emphasizes the prediction of positive cases. They fully simulate the various demands in real-lifé scenarios. In our experiments, we find that in most cases, our CoE method maintains the best performance across all three metrics simultaneously, except in a few cases where the trends of AUROC and FPR95 are inconsistent. This suggests that our method may focus more on positive cases, i.e., correctly responded samples. Overall, our CoE method can adequately adapt to various discrimination needs in real-world scenarios.
Domain Robustness. In vertical comparisons between four domains, the average AUROC perfor- mance improvements compared the strongest baseline are 9,83% 8.36 %, 7.78% %, 7.23% from domains I to IV. Notably, no domain shows a significantly lower improvement, highlighting our domain robustness. More interestingly, the improvement in the Mathematics domain is noticeably greater, and it is exactly the most objective among the four domains, suggesting that our method is likely more effective in objective scenarios. This phenomenon aligns with human intuition: compared to solving subjective problems, humans' thinking path when solving objective problems tends to be less influenced by subjective feelings and biases. This reduces the subjective noise and enhances path systematic (Paul & Elder 2019). thereby increasing the quantification precision of path features.
Model-scaling Robustness. In horizontal comparisons between 70B+ and 7B+ parameter models, we are surprised to find that even with a tenfold increase in parameters, our method still maintains its leading performance and even surpasses the 7B+ model performances in most cases. As the demand for large-scale LLMS surges in the industry. the enhanced model scaling robustness allows our method for widespread deployment real-world scenarios, ensuring its broad generalizability.
Note that the ratio of positive to negative examples varies across different scenarios, as it depends on the model's accuracy for each specific task. Therefore, when performing vertical comparison between various domains of the same model or method, AUROC is the most appropriate metrie for it is insensitive to the ratio of positive negative examples compared the other two metrics.
7

---

Published as a conference paper at ICLR 2025
4.3 EXTENDED ANALYSIS
First, we we further analyze the effectiveness of our CoE methods.
Component Ablation. CoF scores consist of two components: magnitude and angle. To assess their impact on the combination metric, we conduct ablation studies. Table [ the AUROC results for four 7B+ parameter models. We observe that in 14 out of 16 settings, the combination outperforme metric outperforms the individual components, indicating a positive influence from both components. Furthermore, when anomalies arise- such as in Mathematics and Knowledge domains with Qwenl.55 7B CoE-R is more affected by these anomalies, whereas CoE-C demonstrates greater robustness. As result, CoE-C offers more stable performance for real-world applications.
Table 2: Component ablation study of our CoE metric, we report AUROC results in four 7B+ models.
Components
Combination
LJama3-88- In-ntruc
Qwen2-7B-Instrout
Magnitude
Angle CoE-R CoE-C Mathematics
Reasoning
Knowledge
Understanding
Mathematics
Ressoning
Kaowlolge
Undestandlnng
72.54
63.12
64.20
64.81
75.75
66.68
63.14
71,92
73.08
55.85
62.45
58.47
76.68
62.70
61.85
70.87
74.50
55.57
61.79
56.88
75.69
61,95
61.05
70,98
63.91
Components
Combination
Llama2-70. In-strctt
Owenl -n-nstrutt
Magnitude Angle CoE-R CoE-C Muthematics
Reasoning
Knowledge
Understanding
Mathematicss
Reasoning
Knowledge Understanding
64.23
55.51
62.76
60.74
38.44
58.19
49.48
54,69
v
63.63
59,00
59.07
55.19
77.32
67.67
62.11
55.11
57.86
53.97
65.94
$3.97
51.15
59.09
28.90
54.83
44.00
54.71
Task Difficulty Exploration. Tasks within the same domain can vary in difficulty, likely affecting metric performances. In our setup, we select two datasets of different difficulty levels for Math- ematies and Reasoning domains, respectively. In particular, GSM8K and CommonsenseQA are low-difficllyy datasets; MATH and TheoremQA are high-difficulty datasets for they require least college knowledge. Details are shown in Appendix C. Figure Sirressssoww the AUROC results with the Qwen2-7B-Instrucc model, where CoF. has slight edge in low-difficulty tasks, but demonstrates a significant advantage in high-difficulty tasks, and outperforms other baselines by large points. This indicates that CoE is more discriminative on more difficult tasks, which may be because the thinking paths are more complex difficult tasks, increasing the potential informational features for CoE.
Hbx Prch
Pertecty
Comport
Entrapy
flonnesom
CSS-C
Reasoning
80
AUROCVO Volue
AUROC Volue
70
60
60
50
50
40
40
MGSM (Low Difeuuy))
MATH (1ighh Difkcity)
CommannerroAA (low Difficaty) ThearemGA (high Difficulty
Figure 5: AUROC results (w/ Qwen--WB-Insaeeeee model) of all methods for varying difficulty tasks
within the Mathematics and Reasoning domains. See Appendix D for other models.
Secondly, we analyze the reliability and stability of applying CoE methods in real-world scenarios.
Data Ratio Robustness. Self-evaluation differs from other classification tasks because the ratio of positive to negative samples in each scenario not balanced, entirely depends on the accuracy of L.M responses. If we denote the number of positive samples in one dataset 8, and negative samples as s_.. then the ratio can be expressed as 5+ s_ a. To assess the performance robustness of CoE under different data ratios, we match all the CoE results in Table I with the response accuracy of the corresponding models on the corresponding datasets, then observe the AUROC results under different data ratios. Figure O shows the results, there is no significant performance drop in any particular area, especially in areas where a < 0.2 and a > 0.8, indicating that data imbalance does not adversely affect performances. This suggests that CoE is robust against varying data ratios.
High Deployment Efficiency. Existing methods have significant efficiency bottlenecks. Excluding the base LLM inference: For sampling-bast based methods, LLMs must perform at least one additional inference, so the inference time is the lower bound of its execution cost. For sampling-fre free output- based methods, they almost all require the output probability distribution, the Max computation
8

---

Published as a conference paper at ICLR 2025
AUROC Value
70
50
CoE-R
CoE-C
30
10 20 30
40
50
60
70
90
100
(Negative Positive)
LLM Response Accuracy Positive: Negative) (Positive Negative)
Figure 6: AUROC results under different LLM response accuracies (i.e., data ratios).
is unavoidable, which involves large-scale exponential computations as the vocabulary size is huge (iikolov 2013). In practice, find that when the task is difficult and the LLM response is long, this computation time will even be comparable to the inference time.
Table compares the execution time (excluding the first base inference, which is mandatory) of our CoE method with the above two types of methods. Our method requires only simple addition, multiplication, and triangulation oper- ations, with execution costs at the millisecond level, and possesses prominent efficiency and stability ad advantages.
Table 3: Execution Time(s) (w/ Llama3- Instrutttct model) on GSM8K dataset.
Base LLM Inference
12.59 23755
Max Computation
10.32 4551
CoL'' Computation (ours) 1.12c-03 +5.44
Multilingual Scalability. We also test the scalability of CoE in multilingual scenarios, which ensures that CoE effective not only in English environments, thereby en- hancing its universality. We select MGSM (Shi et al.] 2022) dataset subset of the GSM8K with 11 language versions. Figure 7 presents the results in the Llama3- IB-Instruct model, we find that in almost all language scenarios, CoE demonstrates performance advan- tage compared to baselines. Notably, CoE shows a good performance in some low-resoure, languages (e.g.. bn), flecting its adaptability to various language environments.
Perpicsity
de 26
Figure 7: AUROC results (w/ Llama3- 8B-Instruct model) on MGSM dataset with II language versions. Appendix [ shows the results of other models.
5 THEORETICALLY REVISIT COE-C AND COE-R
Monotonicity Analysis. In Section 3 we pointed out that CoE-R lacks physical significance as metric. However, its linearity effectively captures the monotonicity of both magnitude and angle features. In contrast, the monotonicity of these two features of CoE-R R is not as apparent. We assume feature points, each represented
pair of magnitude and angle (L, a,) for 1 s i s n. with po and a = [a,]i-1- The final CoF. feature is denoted as F(L, o.). We consider the
L = [T increments ÅL. applied I, and Aa applied 04, with the CoE feature increments being
AF(L,) = F(L, + AL, L. a.) - F(L,, L.; a);
(8)
AF(a.) F(a; Ao, 04 L) F(a,, + +A0 0: L)
For CoE-R, its final feature Fn= Eブ= L + Σf= ensures F
AL > 0 and AFR(a;)
Aa In contrast, The situation on CoE-C is relatively complex, we formalize its final feature Fo and computer its feature increments (F,(L,) and AFc(a;) by following Eq as below:
F) Σne L,80
Ej= Ly sina;
Σ 13+ [n
2L, Li coo(as a.). Σ
(9)
kit.ha EEgaay L, sin sin a, A sin a²Fela; + Aa, L)+ Fo
AL (2(2 + AL I, s0 2h, costa, o0)) AFo(L:) n²Folli+ + AL, L.,, a)) ao( AFo(a))
See Appendix B.2.1 for the derivation of Eq 10 In practical inference, we statistically find that more than 98% of the cases fall within the range of a; between 0 and x/2. this can also be intuitive in Figure 3 Consequently, is nearly always true that (L. > O. Additionally, the angle difference between correct and incorrect trajectories is often greater than the angle difference within a single trajectory, namely |a; - a.y. When Aa causes a; to deviate from the current class feature, it tends
to be sizable. As result, sin (a: a, + 2 Aa will be greater than 0, leading to (Fc(a;) < 0.
Therefore, in practical scenarios, the CoE-C monotonicity of both magnitude and angle features is consistent with CoE-R and satisfies conclusions drawn in Section 2,2

---

Published as a conference paper at ICLR 2025
Why CoE-C is More Robust Than CoE-R? In Section pointed out that CoE-C may be more sensitive to outliers. This claim has been verified in the ablation study presented in Section 4.3 Here, we delve into the fundamental reasons behind the metric robustness from a theoretical perspective.
We already know that the magnitude changes of the CoE trajectory of a correct sample more significant, which means that for an incorrect sample, if one L, feature point appears abnormally large, it will be easily misclassified as a correct sample. Therefore, if one CoE feature can better control the increment when facing this situation, it will reduce the risk of misclassification Formally, compare AF L;) and AFo L;), the smaller one CoE metric possesses stronger robustness. We first deflate the lower bound of (F,(L,) by fixing the principal element L;:
Fe(L, a)) = #1 [ 13+
Σ 2L cos(as, - a.) 2 v Rit,Eet
L +
Σ J.Jae
Lg Lycos(a - -a))
(1I)
Then, we use this deflation bound to further deflate the (L,) of Eq
8
AL(2L, + +LL+ Egs.. ..: 22, cos(as; - az))
AL
AFo(I.) nn. ( AL + Eg,se L, cos(a; - az)) + n². (L, Iygu by (olla,as)
(12)
We find that the right side of Eq 12 that CoE-C is more robust than CoE-R. The complete derivation can be found in Appendix B.2.2
Eq is exactly Fn(L.), which implies Fc( s Fn(,,), proving
RELATED WORK AND DISCUSSION
Our research focuses on label-free self-evaluation, where the uncertainty estimation in deep neural networks (Gal & Ghahramani] 2016 Guo et al. 2017) and their variants in the era of LLMs (Huang al. 2023 Kuhn et al. 2023) are closely related to us, we categorize them as white-box methods. Additionally, the two typical paradigms that do not access internal states are classified as black-bor -bx methods (Manaul et al., 2023 Li et al. 2024c Among these, white-box and black-box research tracks are usually orthogonal to each other (Li et al. 2024c). They all emphasize the "label-free" condition, and we present a detailed discussion about these related works in Appendix [
From the perspective of research ideas, our research involves the usage of hidden state information. Many existing studies typically utilize hidden states to train correctnessllaell-assee probing classifiers. They can learn useful hidden state features of specific datasets error types, but their generalization ability on outtof-distribution (OOD) data is unpredictable. Despite the differing research intents, considering the overlap of research ideas, also conduct simple comparisons with them.
We select two recent works: (1) ITT (L et al. 2024a). which is trained on the TruthfulQA dataset (Lin et al. 2022b;; and (2) MIND (Su al. 2024), which is trained with hallucinated data sourced from Wikipedia. Both methods focus on detecting factual errors, making mathematics and reasoning tasks likely OOD for them. We select four datasets: (1) TruthfulQA (Lin et al.) (022b) as it is in-distribution (ID) for both ITI and MIND and even fitted by ITT; and (2-4) GSM8K, MATH, and TheoremQA (used in our main experiments), they are OOD for both ITI and MIND.
Table presents the AUROC results with the Llama3- 8B-Instruct model. On the TruthfulQA dataset, ITI outperforms CoE, which is expected given that it is well-suited to its correctness features. However, in the other three datasets, ITI and MIND exhibit significant declines with more than 20 points of MIND and 30 points of ITI due to being OOD relative to their train- ing data. In contrast, our CoE is not tailored to specific datasct features and maintains consistent applicability across diverse datasets, making it robust for complex real-world scenarios. We discuss more related work in this research area in Appendix A.2
Table 4: AUROC Results of our label-free CoE and label-based ITI and MIND (w/ Llama3-8B-Instruel model) on four datasets.
TrudkfulAA
GSMSK
MATH
Thcorem1QA
ITI
83.48
47.49
46.02
48.35
MIND CaE-R (ours)
74.52
50.67
43.96
72.21
69.84
75.23
67.94
CaE-C (ours))
74.74
71.20
74.95
60.47
CONCLUSION
In summary, we propose a lightweight self-evaluation method for LLMs. It does not access the output text or probability distribution but utilizes the progressive chain of all hidden states in the latent space instead, which we term Chain-of-Emmeeddin robustness across various models, domains, task difficulties, and languages. Its low computational cost also ensures real-time deployments for large-scale feedback needs in practical scenarios.
(CoE). Our method exhibits strong performance and
10

---

Published as a conference paper at ICLR 2025
LIMITATION
Our method cannot be extended to a black-box version because the need to access the internal hidden states, this means that our method cannot be applied to closed-source models such GPT-4 (Achiam et al. (023) for the time being. However, from the perspective of research senses, white-box approaches more helpful interpreting the response mechanisms within LLMs, while black-box approaches are more intuitive and better suited to cope with closed-soures protocols. Therefore, we advocate recognizing the parallel contributions of the two research tracks and flexibly choosing between the two types of methods for use in real-world scenarios.
ETHNICS STATEMENT
The data and models used in this work are sourced from the official version of the original paper. and we strictly adhere the provided usage protocol. Regarding the data, no modifications have been made to the original dataset, so they would not involve any sensitive content. From the perspective of research intent, our research aims to detect the incorrect responses generated by LLM in real-world deployment, which has promising implications for social safety.
REPRODUCIBILITY
All models employed in the experiments are official checkpoints, and all implementation details including model sources, hyperparameters, hardware requirements, and prompt instructions are presented in Appendix C.2 to ensure the reproducibility of our method.
ACKNOWLEDGEMENT
This work was supported by the Alibaba Research Intern Program and Alibaba Innovative Research Program. This work was supported by the General Program of National Natural Science Foundation of China (62176153), This work was also supported in part by the Science and Technology Development Fund of Macau SAR (Grant Nos. 0007/2024/AKP, FDCT/0070/2022//MA FDCTT/60/2202//FF)) and the UM and UMDF (Grant Nos. MYRG--GG222--000--SS--DDDD 00165-FST-UMDE, EF2024-00185-FF EF2023-00151-FST, EF2023-00090-FSTT)
MYRG-GRG2024-
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Amos Azaria and Tom Mitchell. The internal state an Ilm knows when it's lying. Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 967-976, 2023.
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. arXiv preprint arXiv:2308. 16884, 2023.
Kendrick Boyd, Kevin H Eng, and C David Page. Area under the precision-recall curve: point estimates and confidence intervals. Machine Learning and Knowledge Discovery Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13, pp. 451-466. Springer, 2013.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.0388 2022.
Jun Cen, Peng Yun. Junhao Cai, Michael Yu Wang, and Ming Liu. Deep metric learning for open world semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 15333-15342, 2021.
11

---

Published as a conference paper at ICLR 2025
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. Inside: Lims' internal states retain the power of ballucination detection. arXiv preprint arXiv:2402.03744, 2024.
Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: Linguistically-inforred interpolation hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2147-2157, 2020.
Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model via intrinsic and extrinsic confidence assessment. arXiv preprint arXiv:2308. 16175, 2023.
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset. In Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7889-7901, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv: 14168,, 2021.
Jacob Deasy, Nikola Simidjievski, and Pietro Liò. Constraining variational inference with geometric jensen-shannon divergence. Advances in Neural Information Processing Systems, 33: 13647-106658,, 2020.
Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 295-302, 2020.
Hanyu Duan, Yi Yang, and Kar Yan Tam. Do llms know about hallucination? an empirical investiga- tion of lim's hidden states. arXiv preprint arXhv:2402.09733, 2024.
Jinhao Duan, Hao Cheng. Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the uncertainty estimation of large language models. arXiv preprint arXiv:230700137) 2023.
George H Dunteman. Principal components analysis, volume 69. Sage, 1989.
Jonathan St BT Evans. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences, 7(10):454-45, 2003.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017)x625-666))) 2024.
Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don't 'tn't hallucinate, abstain: Identifying Ilm knowledge gaps via multi-llm collaboration. arXiv preprint arXiv:2402.00336 2024.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kamalika Das. Spuq: Perturbation- uncertainty quantification for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2336-2346. 2024.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.12948 2025.
W and GJ Hampson. Trajectory analysis: concepts and applications. Basin Research, 21 21(5):554-433 2009.
12

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 00kY::0009003300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mosta- jabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. In International Conference on Machine Learning, pp. 8759-8773, PMLR, 2022.
Yuheng Huang, Jiayang Song, Zhijie Wang. Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv preprint urXiv:2307. 10236, 2023.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neurul Information Processing Systems, 36, 2024.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Compulational Linguistics, 2019.
Haozhe Ji, Pei Ke, Zhipeng Hu, Rongsheng Zhang, and Minlie Huang. Tailoring language generation models under total variation distance. In The Eleventh International Conference on Learning Representations, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a.
Mingjian Jiang, Yangjun Ruan, Sicong Huang, Saifei Liao, Silviu Pitis, Roger Baker Grosse, and Jimmy Ba. Calibrating language models via augmented prompt ensembles. ICML Workshop: Challenges in Deployable Generative AI, 2023b.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfiel-Doodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint ar XrXiv:2207.052 .002210 2022.
Daniel Kahneman. Thinking fast and slow. Farrar, Strauss and Giroux, 2011.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 2022.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-ting time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024a.
Loka Li, Guangyi Chen, Yusheng Su, Zhenhao Chen, Yixuan Zhang, Eric Xing, and Kun Zhang. Confidence matters: Revisiting intrinsic self-correction capabilities of large language models. arXiv preprint arXiv:2402. /2563, 2024b.
13

---

Published as a conference paper at ICLR 2025
Moxin Li, Wenjie Wang. Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. Think twice before assure: Confidence estimation for large language models through reflection on multiple answers. arXiv preprint arXiv:2403.099: 2024c.
Xun Liang, Shichao Song, Zifan Zheng, Hanyu Wang, Qingchen Yu, Xunkai Li, Rong-Hua Li, Feiyu Xiong, and Zhiyu Li. Internal consistency and self-feedback in large language models: A survey, arXiv preprint 1rXx:22407.14507 2024.
Yhin Lin. Rouge: A package for automatic evaluation of summaries. Text summarization branches out, pp. 74-81, 2004.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022a.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, 2022b.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifica- tion for black-box large language models. arXiv preprint arXiv:2305. /9/87, 2023.
Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang. Towards collaborative neural-symbolic graph semantic parsing via uncertainty. In Findings of the Association for Computational Linguisties: ACL 2022, pp. 4160-4173, 2022c.
Kevin Liu, Stephen Casper, Dylan Hadfield-Menee Menell,,,, and Jacob Andreas. Cognitive dissonance: Why do language model outputs disagree with internal representations of truthfulness? Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 4791-4797, 2023.
Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. Uncertainty estimation and quantification for Ilms: A simple supervised approach. arXiv preprint arXiv:2404.15993, 2024.
Litian Liu, Reza Pourreza, Sunny Panchal, Apratim Bhattacharyya, Yao Qin, and Roland Memisevic. Enhancing hallucination detection through noise injection. arXiv preprint arXiv:2502.03799 2025.
Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based out-of-distribution detection.
Advances in neural information processing systems, 33:21 1611444221 21475, 2020.
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. Advances neural information processing systems, 32, 2019.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations, 2020.
Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucina- tion detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004-9017, 2023.
Christopher Manning. Foundations of statistical natural language processing. The MIT Press, 1999.
MetaAl. Introducing meta llama 3: The most capable openly available Ilm to date.
ad.mera.combllommeeetlanm. 2024.
Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-872, 2022.
Tomas Mikolov. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.378. 2013.
Tom Minka et al. Divergence measures and message passing. Technical report, Technical report, Microsoft Research, 2005.
14

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Richard Paul and Linda Elder. The miniature guide to critical thinking concepts and tools. Rowman & Littlefield, 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1499-1509, 2018.
Benjamin Plaut, Khanh Nguyen, and Tu Trinh. Softmax probabilities (mostly) predict large language model correctness on multiple-choice q&a. arXiv preprint arXiv:2402. 13213, 2024.
Qwen-Team. Introducing qwenl.5. 5. gwenlm..ithub.iiog httghhhbbbobboogggeee..... 2024.
Faisal Rahutomo, Teruaki Kitasuka, Masayoshi Aritsugi, et al. Semantic cosine similarity. In The 7th international student conference on advanced science and technology ICAST, volume 4, pp. 1.
University of Scoul South Korea, 2012.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter Liu. Out-of-distributioo detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations, 2022.
Mark D Rintoul and Andrew T Wilson. Trajectory analysis via geometric feature space approach. Statistical Analysis and Data Mining: The ASA Data Science Journal, 865-6))277-31, 2015.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are mul- tilingual thain-of-thought reasoners. arXiv preprint wxXiv22210003057. 2022.
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Long horizon temperature scaling. In International Conference Machine Leurning, pp. 31422-31434. PMLR, 2023.
Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z Ren, and Anirudha Majumdar. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. arXiv preprint arXiv:2412.05563. 2024.
Chenglei Si, Zhe Gan, Zhengyuan Yang. Shuobang Wang, Jianfeng Wang, Jordan Boyd-Grabe and Lijuan Wang. Prompting gpt-3 be reliable. arXiv preprint arXiv:2210.09 150, 2022.
CH-Wang Sky, Benjamin Van Durme, Jason Eisner, and Chris Kedzie. Do androids know they's 're only dreaming of electric sheep? In Findings of the Association for Computational Linguistics ACL 2024, pp. 4401-4420 2024.
Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. Unsupervised real-time hallucination detection based on the internal states of large language models. arXiv preprint arXhv:2403.06445 2024.
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.0556/ 2024.
Saeid Asgari Taghanaki and Joao Monteiro. Explain-query-test:: Self-evaluating lims via explanation and comprehension discrepancy. arXiv preprint arXiv:2501.1172) 2025.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqat A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language 2019.
Technologies, Volume I (Long and Short Papers), pp.
15

---

Published as a conference paper at ICLR 2025
I Tenney. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv: 2019.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5433-5442, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
Yiming Wang, Pei Zhang, Baosong Yang. Derek Wong, Zhuosheng Zhang, and Rui Wang. Embedding trajectory for out-of-distribution detection in mathematical reasoning, In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024a.
Yiming Wang. Zhuosheng Zhang, Pei Zhang, Baosong Yang, and Rui Wang. Meta-reasoning: Semantics-symbol deconstruction for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 622-643, Bangkok, Thailand, August 2024b. Association for Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI. Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? empirical evaluation of confidence elicitation in llms. In The Twelfth International Conference on Learning Representations, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, arXiv:2407.1066 10671, 2024.
Qwen2 technical report. arXiv preprint
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint arXiv:2407.20311, 2024.
Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng. Jun Wang, Yueting Zhuang, and Weiming Lu. Self-contrast: Better reflection through inconsistent solving perspectives. arXiv preprint arXiv:2401.02009 2024.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, al. Siren's song in the ai ocean: a survey on hallucination large language models. arXiv preprint arXiv:2309.012209 2023.
Lianmin Zheng, Wii-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-veriffcation. arXiv preprint 2rXiv:2388.09921 2023.
16

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RELATED WORK AND DISCUSSION
Our research topic is about LLM self-evaluation, where the "self" hides two key constraints: (1) this evaluation is label-free suppose there is an urgent need for evaluation, only a large number of crawled questions, and there is no time and manpower to label high-quality answers. online feedback generated when the model answers these questions to determine the correctness of its answers; (2) this evaluation is not allowed to have any external tools involved, in other words, external scorers or trainers. This is necessary guarantee for the industry to save the cost of model deployment as well as ensure its real-time and scalability when facing large-scale evaluation needs.
A.1 LABEL-FREE LLM SELF-EVALUATION
A.1.1 WHITE-BOX METHODS
The precursor to LLM white-box evaluation can be traced back to Uncertainty Estimation (UE) [Laksmiinrrayanal et al., 2017) during the era of deep neural networks. At that time, the concept
of uncertainty was commonly understood as output probabilities (Gal & Ghahramani 2016 Guo
et al.] [0177] Desai & Durrett 2020), which could be used to measure the model's 's confidence in generating that output. These methods all encounter a key issue that their model output may be overconfident, this is because the Kl.-divergence optimization adopted by model training (including language models) forces models to assign non-zero (often quite high) probabilities to all training samples (Minka et al. 2005 Malinin & Gales 2019). To cover the low-probe wwwprobability regions in the data distribution, LLMs adopt the zero-avoiding solution (Deasy et al., (020) and systematically overestimate the probabilities of almost all text sequences even if they are ill-tormed (Ji et al. 2023). Therefore, confidence calibration is always a popular research topic, aimed at adjusting the output logits or probabilities to align with the true confidence levels.
The emergence of LLMs has introduced new challenges of uncertainty estimation in natural language generation tasks (Shorinwa al. 2024) due to their diverse outputs (Liu al. 2024). Meanwhile, rich evaluation requirements transform this estimation from a continuous probabilisti target (confidence) into a discrete binary target (correctness), emphasizing the importance of accurately predicting the correctness of LLMs' output (Manakul et al. 2023 Li et al. (222c)). Research in this area remains sparse, primarily adhering to conventional ideas uncertainty estimation that leverage the output logits or probability distributions produced by LLMs. Notably, (Si et al. (2022) pioneered the use of perplexity metrics with GPT-3: (Huuang et al. (2023) systematically assessed how entropy can evaluate self-evaluation in language models; (Plaut et al. 2024) used softmax probabilities to predict the correctness of multiple-choice questions: Hendryeks et al. (2022) explored temperature scaling for output probabilities. Apart from the challenges posed by output diversity, more crucial distinction between language models and traditional deep networks is the rich semantic information inherent in LLM outputs (Liu et al. 2024).. To address this. Kuhn et al. (2023) and Farquhar al. (2024) investigated the concept of semantic entropy, and Duan et al. (2023) also introduced a shift towards utilizing semantic information in entropy calculations. These works primarily focus on a specific type of factual error, namely hallucinations.
Beyond traditional uncertainty estimation ideas, LLMs contain rich hidden state information that can enhance self-evaluation. However, existing research typically involves training external classifiers (Discussed in Section A.2 and there has been limited exploration of label-free self-evaluation
methods utilizing hidden states. One such study by (Chen et al. 2024) measures the covariance of hidden states at a certain layer across multiple samples based on the idea of internal consistency (Liang et al.) 2024). Our research fills a significant gap in this research area by going beyond sampling methods and comprehensively examining how the trajectory of changes in model hidden states can inform LLM self-evaluation.
A.1.2 BLACK-BOX METHODS
LLM boack-box self-evaluation methods primarily fall into two paradigms: Verbal Confidence (VC) and Prommp--Samlling-Aggreeg
(PSA).
VC is the more straightforward approach, leveraging the instruction followiin capabilities of LLMs, enhanced by RLHF (Ouyang al. 2022). to generate confidence scores through well-crafted prompts.
17

---

Published as a conference paper at ICLR 2025
This paradigm includes various general methodologies that directly ask LLMs for their confidence
using multi-stage stage pipelines (Lin et al.] 2022a| Manakul et al. 2023 Tian et al. 2023 Li et al. 2024b]
Wang al. [024b; Laghanaki & Monteiro (025) with some additional technical tools like reflection
mechanisms (Feng et al. 2024). Besides this, there are domain-specifie strategies tailored for areas such as code generation (Zhou et al.
2023) and fact-checking (Lin al. 2023 Another notable
indirect verbal method is the P(True) (Kadavath et al. 2022). which assesses the likelihood that the
next token output is True. PSA pipeline (Xiong et al. 2024) estimates confidence by perturbing
prompts (Jiang et al.
2023b Gao et al.) [024 Liu et al. 2025 or by stochastic decoding (Si et al.
2022] Wang et al. 2023) to generate multiple outputs. It then assesses the consistency of these outputs.
The advantages of these methods are their intuitive principles and the lack of constraints from closed- source licenses. However, they also face some unresolved issues. First, it has been confirmed that VC suffers from overconfidence - ..LMs tend to assign high scores to their own outputs (Zhang ct al. 2024 Xiong et al. 2024), which implicitly decreases the credibility of this approach. On the other hand, the PSA pipe ine is considered incapable of finding an effective consistency measurement
(Manakul et al. 2023 Zhang al. 2023), which will increase the instability of this method when
deployed in diverse scenarios.
In our view, the issue of PSA reflects the essential flaw of poor robustness in the selection of sub- modules: The key components of PSA lie in prompt sampling and multiple-answer aggregation strategies. Prompt sampling is mainly achieved through rephrasing. but different research has given different rephrase prompt designs, and some tasks only focus on specific types of tasks, such as multiple-choice questions (Jiang et al. 2023b). This makes it uncertain how to design the most suitable rephrasing prompt when facing a new task; Multiple-answet aggregation techniques mainly assess the consistency between multiple answers, and the simplest way is to match precise answers and calculate frequencies directly. However, this only applies to deterministic answers, and for some generation tasks or answers involving the problem-solving process, the semantic similarity between answers is also crucial. This makes the best aggregation techniques uncertain.
To test the strategy combination robustness, we select two prompt perturbation strategies (rephrased by two people) and two aggregation strategies (exact match and semantic similarity) to test the consistency of eelf-evaluttion results in four (2*2) scenarios. We use the Ilama3- 8B-Instruct and Qwen2-7B-Insste
Jamma-6B-instruce
Owen2-7B-Insteee
Domain Und. Kno.
Stratogs
Strolage
olopyy
Sydway
model for testing in four domains and used AUROC as the metric. Figure g shows the ranking results, which reflect the AUROC ranking for the four strategies under the same domain. We find that no single strategy remains ahead in all settings, which means that applying PSA in diverse real- world scenarios unpredictable.
50
60
70 40
50
60
70
AUROC Value
AUROC Value
Figure 8: AUROC ranking of different prompt sampling and consistency aggregation strategies in four domains with two models. It is clear that no one combined strategy outperforms others in any scenario.
A.2 HIDDEN STATES FOR LABEL-BASED BELF-EVALUATION
Setting aside the intended limitations of being "label-free", the rich information of hidden states is often utilized in supervised correctness estimation studies, which typically focus on a particularly important type of error - factual errors, commonly referred to as "ballucinations". These studies usually use certain hidden states as input and label correctness to train an external classifier. Notably,
Mielke et al. (2022): (Sky et al. (2024) trained a classifier to predict hallucinations; [i et al. (202aa) conducted probing experiments on hallucination issues using the specific TruthfulQA dataset (Lin et al. 2022b and extracted potential detection information from attention modules for training: Su
et al. 2024) collected hallucination corpora from Wikipedia for annotation and training. Some probe analyses also explained the differences in the hidden states exhibited by language models when
producing correct and incorrect answers (Azania & Mitchell 2023: Liu et al. 2023 Duan et al. 2024), but they did not offer extended insights into the correctness estimation.
18

---

Published as a conference paper at ICLR 2025
These methods leverage hidden states to provide valuable correctness estimation insights. However, a common limitation is that they all involve training on supervised corpora. This goes against the intention of being "label-free". More importantly. classifier training domain-restrieted: It is often task-specific (trained on particular datasets) or type-specific (trained on factual errors), which results in unstable performance when faced with Out-of-Distribution (OOD) data, hindering their scalability. In Section 6
we have demonstrated this argument.
B MORE ANALYSIS ABOUT COE SCORES
B. ALGORITIMIIC PROCESS OF COE SCORES
Algorithm 1 CoE-R Computation
Input: L: The number of hidden layers
d: Embedding dimension
ho, h1, ... 1 E R²: All hidden states
1: ZMag -1 hol
2: Zang arcoos (Aaolla-1leell
3: CoE-R - 0
4: for to do
5: Mag - h1111 hell2
6: Ang arccos
Pe by | 12. 1ul12
7: CoE-R CoE-R
Mag
Ang
ZMass
Znng
8: end for
9: CoE-R CoE-R
Output: CoE-R
Algorithm 2 CoE-C Computation
Input: L: The number of hidden layers
d: Embedding dimension
ha h1, ...,,... ht. R²: All hidden states
1: ZMag - hı hell2
2: CoE-C - 0
3: AvgRe, Avglm - 0,0
4: for to do
5: Mag - | hill2
6: Ang + arccos
h A
Aa Aylla
7: Re Mag cos (Ang)
Zsans
8: Im - Mag
sin (Ang)
Zmase
9: AvgRe AvgRe + Re
10: AvgIm - AvgIm + Im
end for
12: CoE-C Output: CoE-C
AvgRe 2
)
Avgim
1.
19

---

Published as a conference paper at ICLR 2025
B.2 THEORETICAL ANALYSIS OF COE-C AND COE-R
B.2.1 DERIVATION OF COE FEATURES AND INCREMENTS
We assume n feature points, each represented as pair of magnitude and angle (L, a;) for & i n, with L [L,]i=] and a [[a,]==. The final CoE feature denoted as F(L, a). We consider the increments AL applied to L, and Aa applied to as, with the CoE feature increments being
AF(L,) F(4 AL, L L-s or) F(L,, L- O)) AF(a;) = F(a; + Aa, a- L) F(a;, + + Ot- L).
(13)
For CoE-R, its final feature FR(L, a) and feature increments A AFR(44),FRR(1))) are:
L
FR(L, Σ
(14)
n2
7
5-1
j-1
AL
AFR(Li)
(15)
Aa
AFR(a,) 7
(16)
For CoE-C, its final feature FR( L, a) and feature increments
(F, are:
2i=1 L a
j= L sin 2
+
Σ 1 Σ
2L cos(ae-ai)
k,t,k4
(17)
AFc(L.)
Σ 13 (L. AL)+
Σ
coston - ax) + Σ 2(L, AL)L, cos(a: aaz)
[ L3
Σ k.t.
2L, Lu coslay - ac)
(1e A1)+ 1, + Ei.i. 2ALL, cos(a, 0,)
(E +1) (L. AL)² (Eu.,emcn 2 cootas - a.)) + E,,,. 2(1, + AL)L, comlas - a,) + VE, L3 Ln 24 Lu castasy - aq)
(2L, + A Er 2L, coolos a,))
A²F(L, AL) F(L,)
(18)
AFcla.)
E6+
Σ coslac az) Σ 2L,4, coolas + Da Da-a)) a,) (kitikacca
Σ 13+ [13
Σ
2 cos(ar oz)
n
3:34:
k.t.het
En 2,, coslos + Da a,) Eyisoi 2h,4, coola. a,)
VE, L3 (En LL cos(as ax) Exm 2n coola, Sa - a,)) +VE, 43 Exn. 2L costas ar) Eigaiey 20,L,(coo(a; Da -aan))ccaaa(a a3))
nFFlas Aa) ((((a)
-Enaany L,,, sin (o: - ag de sin (Aa) n² Da) [(a,)
(19)
B.2.2 ROBUSTNESS ANALYSIS
We already know that the magnitude changes of the CoE trajectory of correct sample is more significant, which means that for an incorrect sample, if one L; of a feature point appears abnormally large, it will be easily misclassified control the increment when facing this situation, will reduce the risk of misclassification Formally, compare AFR(L.) and AF (L.), the smaller CoE metric possesses stronger robustness.
correct sample. Therefore, if one CoE feature better
We find that (L,) as defined in Eq 18 includes Fe a) [7. Therefore, before deflating AFo (T.)) we first deflate Fo (L, a and obtain the lower bound follows:
20

---

Fublished as a conference paper at ICLR 2025
21 cos( - as) ax) cos(a; a.))
Σ 2L [co - aa) - cos(a, - oa)) cos(a, - ac)
E k
Σ L2 cos" - -a))+ +
24 sin(a; - ax) -a) sin(a; - a.)
Σ Ly cos(a, - aa,)
j=l,iti
V.j+i
cos on - aa))
Σ 24 cos - aa)) -
+ Σ + + Σ 21 cos - -aa- -
k.
Σ 21 Los(
+ Σ [111-000(0 - -,,]]+
Σ
+ Σ L sin - -a,)+ +
Σ L, sin(o - ag)
kiikit
Σ 13 + L Σ 21, cos(a; - -a,)+
Σ 13 +
:=,
j-1gMi
j=lj+i
+
L Σ Lg cos(a; - a,))
L Σ L₁ cos((a - aa))
I, Σ Lg cos(a; - - as)
L + Σ Lj cas(a; - ag))
+ Σ Lg cos((s aa - (n)
E L, cos(a, - og))
J.j+i
3,j
3::41
J:f
J:j4i
n
35+1
1 +
Fo(L,a)
L
1
n
11
21

---

Published as a conference paper at ICLR 2025
When a; aj for all 1 = s n. # jnnnj lower bound to deflate the (F( as follows:
Fc a) achieves its lower bound. We can then use this
AFo(L.) = AL (2L, AL+ _ Jure 2L, cos(a,-a,))
FLL AL) + F(L,)
21, + AL + Ey,es 21, cos(a; - aa)) AL (LL,
n2. L, AL + Σ jisss L, cos(aa a,)) + n (L, Eysor Lu cus(a, -a,) aa))
(20)
AL (2L: + AL+ L, 2L, cos(a, a,))
(2L AL + Ex,ees 2L, cos(a: a4))
We can derive that AFo(L,) & AL, Luckily, AL is just the (F)(L,)) as defined in Eq 15 Thus (F)(L.) (L) proved, i.e, it is proved that CoE-C is more robust than CoE-R.
22

---

Published as a conference paper at ICLR 2025
C EXPERIMENTAL SETUP DETAILS
C.I DATASET
We select six datasets across four domains in main experiments. Table D present dataset samples and corresponding LLM output (Produced by Llama3-8B-Insten model) of each dataset. We describe each dataset's information follows:
Mathematics Domain
GSM8K Cobbe al. 2021 consists of linguistically diverse math word problems specifically designed for grade school students and created by human writers. It contains 1318 test problems, each requiring 2-8 steps solve. The challenges primarily involve sequence of basic arithmetic operations (addition, subtraction, multiplication, and division) to derive the final answer.
MATH (Hendrycks al. 2021) consists of challenging competition mathematics problems. contains 5000 test problems from diverse domains including algebra, geometry, counting and probability, number theory, and precalculus.
Reasoning Domain
CommonsenseQAA (Tammor et al. (019)) consists of diverse commonsense question answering questions. It contains 1221 test problems, each with 5 choices. This dataset generated by Amazon Mechanical Turk workers.
questions driven by STEM theorems. contains 800 test problems and covers 350+ theorems spanning across Math, EE&CS, Physics and Finance. The dataset is collected by human experts with very high quality.
TheoremQA (Chen al. 2023) consists of various
Knowledge Domain
MMLU (Hendrycks al. 2020 consists of diverse questions that measure knowledge acquired during pretraining. This dataset covers 57 subjects across STEM, the humanities, the social sciences, and more. It ranges in difficulty from elementary level advanced professional level, can test both world knowledge and groblem-solving ability.
Understanding Domain
Belebele (Bandarkar et al. 2023) consists of multilingual, multitask, and multidisciplinary reading comprehension questions, with each language containing 900 article-question pairs. We use the English version of this dataset.
Table 5: Dataset case and LLM output from GSM8K dataset (Mathematics Domain)
Question
Janet's ducks lay eggs per day. She cats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
True Answer
18
LLM Response (After following the instruction in Section C.2.3
I Let's break down the problem step by step:
Janet's ducks lay 16 eggs per day.
She eats eggs for breakfast, she has 16 3 eggs left.
She bakes muffins with eggs, leaving her with 13 eggs.
She sells the remaining eggs at the farmers' market for 52 per egg.
5. To find out how much she makes, multiply the number of eggs sold (9) by the price per egg (S2): 9 $2
$18.
Answer: 18
23

---

Published as a conference paper at ICLR 2025
Table 6: Dataset and LLM output from MATH dataset (Mathematics Domain)
Question
The sets A = (2 2 1) and B = {w 48 1 are both sets of complex rools of unity. The set C {sw 2 2 E A and w B) is also a set of complex roots of unity. How many distinct clements are in C??
True Answer
144
LLM Response (After following the instruction in Section C.2.3
Let's break this down step by step!
1. The set A {a means that A contains 18th roots unity. which the complex numbers that satisfy z 1.
18 = I} contains all complex roots of unity that have power of 18 equal to I. This
2. The set {w w² 1) contains all complex roots of unity that have a power of 48 equal to 1. This means that B contains 48th roots of unity, which are the complex numbers that satisfy w 48 = 1.
3. The set C' {aw : = E A and w E B} contains all complex numbers that are the product of complex number from A and a complex number from B. Since A and are sets of complex roots of unity, the product of any two complex numbers from A and B will also be complex root of unity.
4. Now, let's think about possible products complex numhers from A and B. Since A contains 18th roots of unity and contains 48th roots of unity, the product of any two complex numbers from A and B will be 144th root unity. This is because the least common multiple of and 48 is 144.
5. Therefore, the set C contains all 1444th roots of unity, which are the complex numbers that satisfy 194 Since there are 144th roots of unity, the set contains 144 distinct elements.
So, the final answer
144
Table 7: Dataset case and LLM output from CommonsenseeA dataset (Reasoning Domain)
Question
A revolving door is convenient for two direction travel, but also serves as security measure at what?
Choices:
(A) bank
(B) library
(C) department store
(D) mall
(E) new york
True Answer
A
L.M Response (After following the instruction in Section C.2.3
Let's break down the question step by step:
1. The question states that revolving door is convenient for two-direction travel, which means it allows people to enter and exit the huilding casily.
2. The question also mentions that the revolving door serves as security measure, which implics that it provides some kind of barrier or obstacle to prevent unauthorized access.
3. Considering the options, a bank is a likely candidate for a security measure, Banks often have high-security
measures in place to protect their customers' money and assets.
4. A library might also have security measures in place, but it's not as likely as a bank. Libraries typically have
more relaxed security measures, as they are public spaces.
A department store or mall might have some security measures, but they are not likely to have revolving door security measure. These types of establishments often have more open layouts and may not require much security.
6. New York is type of building or establishment, so it's viable option.
Based the reasoning process, the most likely answer is:
Answer: A
24

---

Published as a conference paper at ICLR 2025
Table 8: Dataset and L.M output from TheoremQA dataset (Reasoning Domain)
Question
Let undirected graph G with edges = (<1,,,,224225,,,,,<c5..)) which <A,B> represent Node A is connected to Node B. What is the shortest path from node I node 6? Represent the path list.
True Answer
[1, 2, 5,
LLM Response (After following the instruction Section C.2.3
To find the shortest path from node to node 6, will use mreadth-first search (BFS) algorithm. Here's the step-by-step process:
1. Create a queue and enqueue node
Create a set to keep track of visited nodes and add node to it.
While the queue is empty, dequeue node and explore its neighbors.
For each neighbor, check it's the target node (node 6). If it is, return the path from the starting node to the target node.
5. If the neighbor is not the target node, add it to the queue and mark it as visited.
Here's the BFS traversal:
Queue: [1]
Visited: [1]
Dequeue node 1, explore its neighbors:
Neighbor: 2
Queue: [2]
Visited: [1. 2]
Dequeue node 2, explore its neighbors:
Neighbor: 4
Queue: [4]
Visited: [1. 2, 4]
Dequeue node explore its neighbors:
Neighhor: 5
Queue: [5]
Visited: [1, 2, 4,
Dequeue node 5, explore its neighbors:
Neighbor: 6
Queue: []
Visited: [1. 2, 4. 5, 6]
The target node (node 6) is found. The path from node I to node is: 21, 4, 6].
Therefore, the answer is [1, 4, 6).
25

---

Published as a conference paper at ICLR 2025
Table 9: Dataset case and LLM output from MMLU dataset (Knowledge Domain)
Question
Paper will burn at approximately what temperature in Fahrenheit?
Choices:
(A) 986 degrees
(B) 2125 degrees
(C) 3985 degrees
(D) 451 degrees
True Answer
D
LLM Response (After following the instruction Section C.2.3
Let's think step step!
Paper is common combustible material, and it's known burn easily. When think about the temperatures which materials burn, often think of the flash point, which is the lowest temperature which substance ignite and burn.
Paper relatively low-temperature material, and it's commonly reported ignite and burn around 451"F (233'C), This often referred the "ignition temperature" paper.
So, considering the options, correct answer
Answer:
Table 10: Dataset case and LLM output from Belebele dataset (Understanding Domain)
Question
"The Irish government is stressing the urgency of parliamentary legislation to rectify the situation. is now important from both a public health and criminal justice perspective that the legislation enacted as snon as possible"". said a government spokesperson. The Health Minister expressed concern both for the welfare of individuals taking advantage of the temporary legality of the substances involved, and for drug-related convictions handed down since the nww.unconstitutionll changes came into effect."
Question:
What was temporarily legal in Ireland?
Choices:
(A) Burglary
(B) Fraud
(C) Assault
(D) Drugs
True Answer
LLM Response (After following the instruction in Section C.2.3
Let's break down the passage step hy step:
The pussage mentions that the Irish government stressing the urgency of parliamentary legislation to rectify a situation.
A government spokesperson is quoted as saying that the legislation needs to be enacted as soon as possible from both a public health and criminal justice perspective.
The Health Minister is also quoted as expressing concern for the welfare of individuals taking advantage of the temporary legality of the substances involved, and for drugerelated convictions handed down since the now.unconstiutionall changes came into effect.
From these statements, can infer that the substances involved are drugs. The passage does not mention any other illegal activities such as hurglary, fraud, or assault.
Therefore, the correct answer is:
Answer:
26

---

Published as a conference paper at ICLR 2025
C.2 IMPLEMENTATION
C.2.1 MODEL SOURCE
repponsstorrrrroororoooooooo All model checkpoints were downloaded from the official HuggingFace repasitory, with download and license links provided Table [T] We specifically utilize instruction- models, they exhibit superior instruction-followiin ability and excel in zero-shot generation, making them more aligned with user needs in real-world scenarios.
Table 11: Download and license links of all model artifacts we used in this paper.
Madi Nome
Lawk
Alooea Eeao en
5.20 buteet
Geund-m-- kutra
Mixia batret
C.2.2 INFERENCE
Considering the inconsistent difficulty of different tasks, especially since some mathematical tasks may produce longer outputs, we set the maximum output length to 2048 tokens and used the <eos_token> for truncation. The inference process employs greedy decoding without random sampling. We record the answer completion rate of all models across all datasets (i.e., completing responses within the 2048 output token limit), as shown in Table 12 It can be observed that the models are generally able to complete all responses within this specified length.
Additionally, for a 7B+ model, we deploy it using two 32G V100 GPUs, while for 70B+ model, we deploy it using four 80G A100 GPUs.
Table 12: Answer completion rate before 2048 limited output token length.
Dataset
MGSM
MATH CommonsenscQQ
TheoremQA
MMLU
Belebele
Model
100.00%
97.98%
100.00%
99.25
100.00%
100.00%
Llama3-8B Mamma--B--Instruce
100.00%
93.48%
100.00%
97.75%
99.82%
100.00%
Qwenl.5- -5-7B-Instrue
100.00%
99.70%
100.00%
99.88%
100.00%
100.00%
Qwen2-7B Instruct
-went--B-Instrucc
100.00%
96.70%
100.00%
99.50%
100.00%
100.00%
Mistral-7B-Instteee
100.00%
94.86%
100.00%
98.75%
99.65%
100.00%
Llama3-70BB 700ta-3--B--Inttucc
100.00%
97.96%
100.00%
99.88%
99.82%
100.00%
Qwen2-72B- -2nsstrettt
100.00%
99.26%
99.84%
98.88%
99.82%
100.00%
C.2.3 INSTRUCTION
We select instructions followed by LLMs from two open-source projects: OPENCOMPAS and SIMPLE-EVALE They can ensure the professionalism of instructions. Specifically, all instructions used for each dataset are follows:
MGSM
Solve this math problem. Give the reasoning steps before giving the final answer on the last line by itself in the format of "Answer:". Do not add anything other than the integer answer after " "Answer:". Question:
(input_data) data)
MATH
Question: finput__ finput_datal data)
Please reason step by step, and put your final answer within \boxed()
httpp::/hugginngfcee..o/
httpss//giithubb.cmm//p//n 1-wemmppellleeeeeemmmmmmmmmm
27

---

Published as a conference paper at ICLR 2025
CommonsenseQA
Answer the following multiple choice common-sense reasoning question. The last line your response should be of the following format: "Answer: SLETTER" (without quotes) where LETTER is one of ABCDE. Think step by step and output the reasoning process before answering. Linpu__dtta
MMLU
Answer the following multiple choice question. The last line of your response should be of the following format: "Answer: $LETTER" (without quotes) where LETTER is one of ABCD. Think step by step before answering.
Question:
(input_datal
Belebele
Answer the following multiple choice reading- coniinnenomornnniionnn question. The last line of your response should be of the following format: "Answer: SLETTER" (without quotes) where LETTER is one of ABCD. Please fully understand the passage and give explanations step by step before answering.
(input_data)
TheoremQA
Below is an instruction that describes task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
Please read a math problem, and then think step by step to derive the answer. The answer is decided
by Answer Type.
If the Answer type in [bool], the answer needs to be True or False.
Else if the Answer type in [integer, float], The answer needs to in numerical form.
Else if the Answer type in [list of integer, list of float]. the answer needs to be list of number like
[2, 3. 4].
Else if the Answer type in [option], the answer needs to be option like (a), (b), (c), (d). You need to output the answer in your final sentence like 'Therefore, the answer ....
### Question:
(input_datal
### Answer_type:
(answer_type)
### Response:
C.3 BASELINE
denote Y [21 V2. ... yr as the output probability distribution including all tokens, and the output logits corresponding to each token 21, #2, " Zy. The model vocabulary is V, each y; and z; are h dimeennnll vectors, and we have:
y:
(21)
| d=1
IV] Σ
d=1 a
d=1
The baselines in Section
are formalized or described as follows:
1. Verbal Confidence p(True) (Kadavath al.
2022
28

---

Published as a conference paper at ICLR 2025
p(True) estimates the probability that model's generation is correct by "asking" the model if its answer correct. It constructs new natural language question and takes the likelihood of the next token being True the uncertainty measure. We follow prompt templates in Kadavath al. (2022):
Question: [...]
Proposed Answer: [...]
Is the proposed answer:
(A) True
(B) False
The proposed answer is:
2. PromptSaamliig-Aggrgaaioo (PSA) Pipeline (Xiong et al. 2024
The PSA pipeline involves two progressive steps:
Prompt-Sampling: This step requires generating multiple prompt or question formulations (without changing the original question's semantics) for output sampling. To maximize the guarantee of not using external tools (such as external LLMs rephrasing, manual labeling, etc.), we refer to Gao al. (2024) to use token-level perturbations by introducing random perturbation characters (like spaces, tabs, etc.) at random positions in the question. This generates & different question inputs for LLMs, resulting in & output texts: text1, textx-
Aggregation: After obtaining multiple outputs, it is necessary to measure their consistency. A commonly used measurement method lexical similarity (Lin et al. 2022c Kuhn et al.) 2023
Chen al. 2024), which involves using ROUGE-L (Lin 2004 to assess the similarity of these 1: outputs in pairs, and then calculating the average as follows:
2
Rouge L(text,, text;)
(22)
k(k
1= j=++1
We set h = 5.
Maximum Softmax Probability (Si et al. 2022)
Maximum Softmax Probability reflects the maximum probability of the output token probability distribution:
EISET [max [n]
(23)
Perplexity (Si ct al. 2022)
Perplexity reflects the weighted average branching factor of a language:
Eistsr [-log low.maxxxxxmmmoooooo max y:)]
(24)
Entropy Huang et al. 2023
Entropy reflects the distribution uncertainty:
E EISET [ log y. v.]
(25)
Temperature Scaling Shih al. 2023
In the softmax operation before obtaining each probability distribution ys. The logit z, on the exponent is divided by a temperature parameter T to calibrate the final probability. We set T 0.7, the subsequent calculations are consistent with Maximum Softmax Probability:
FulT
FalT
Bists
max
[V] Ed-1 eaaa''
Σd efaalt'
IV] E
E efaaT
(26)
Energy (Liu
2020
Energy maps logits to an energy equation as a substitute for softmax:
V
/T
E EisteT
-T'
log
(27)
d=1
29

---

Published as a conference paper at ICLR 2025
We set T = 0.7.
Monte-Carlo Dropout (Gal & Ghahramani 2016
Monte-Carlo Dropout estimates uncertainty by enabling dropout with different randomness multiple times during the inference phase to obtain multiple output distributions. The variance of these output distributions is used to assess uncertainty. Assuming i random samples yield & outputs y = [Y,, Y2, Y,,",, 1, we estimate the uncertainty as follows:
Eyey [BISISTY [max y.] Eyey [BisisTy [max y.]))
(28)
We set
Length- normal-orra Entropy (Malinin & Gales
2020
Length-normalized entropy utilizes top-k sampling to generate & outputs y [y11 U2. "", yk
then computes the average entropy of these outputs follows:
Eyey [isisry
[E[[ log 9.]1].
(29)
We set
Eigenscore (Chen et al. 2024)
Eigenscore first performs decoding sampling, obtaining k embeddings at the L/2 layer, along with the covariance matrix Σ of these embeddings. The eigenscore measures uncertainty by calculating the determinant of the covariance Σ, perturbed by a small addition:
K log det(E aT).
(30)
We follow (Chen et al. 2024 to set a = 0.001 and k 5.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 TASK DIFFICULTY EXPLORATION
Table 9 to 14 presents AUROC results all seven language models under varying difficulty tasks within the Mathematics and Reasoning domains.
Perpbeiity
Enimrey
Scviing
Eneray
Drspout
Figeneme
CaE
Domainx Mathematics
Domain: Reasoning
70
AUROC Value
70-
AUROC
60
MGSM
Figure 9: AUROC results (w/ Llama2-78 --- Instruct model) of all methods for varying difficulty tasks
within Mathematics and Reasoning domains.
30

---

Published as a conference paper at ICLR 2025
Sciling
Cet Cofc
Reasoning
AUROC Vallue
AUROC
60
80
40
40
MISM JLow Dimicutty
MATH Oligh Difficatt)
Commonseree@ (Low Difcuity)
Theoremihh drigh Diffinuty)
Figure 10: AUROC results (w/ Llama3-88 -B-strrruut model) of all methods for varying difficulty tasks within the Mathematics and Reasoning domains.
Reasoning
80
70
70
AUROC Value
80
AUROC
60
50
30
40
MISH JLOw Dimicuityl
MATH OHigh Difficatty
CommonserseeA (Low Difccuty)
Theeremok High Dificulty
Figure 11: AUROC results (w/ Owen1.5-77 I-ntulnstt model) all methods for varying difficulty tasks within the Mathematics and Reasoning domains.
Cotic
Reasoning
70
AUROC Valuee
AUROC
60
60
50
50
40
40
MCSM
Difficulty!
MATH OHigh Diflicadly
Commonneeneeej (Low Defcuity)
Theoremich (righ Diffiully
Figure 12: AUROC results (w/ Coataal-B-Inntruul model) of all methods for varying difficulty tasks within the Mathematics and Reasoning domains.
Scaling
Dropout
CaE
Cocc
Reasoning
90
90
AUROC value
70
70-
AUROC
40
40
MCCM Now Diticilyy
MATH Origh Defficat))
Corponneeee@Aj flow Difrnity)
Theorenih 01igh Ditinally
Figure 13: AUROC results (w/ Mlama3-70B-Insttee model) all methods for varying difficulty tasks within the Mathematics and Reasoning domains.
31

---

Published as a conference paper at ICLR 2025
Nox
Persboilyy
Erawey
Tema: Scarin
Ereng
MC Dispout
Exgessore
0occ
Mathematics
Domain: Reasoning
90
90
AUROC VUlocvalue Value
HUROC
30
40
(High Defficatt)
CommonaanseAA
Offcuity
Dificully
Figure 14: AUROC results (w/ Qwen2-72B Crres-ttB-Instrruct model) of all methods for varying difficulty tasks
within the Mathematics and Reasoning domains.
D.2 MULTILINGUAL SCALABILITY
Table 15 present the AUROC results in four 7B+ language models the MGSM dataset, which a mathematical task comprising language versions. The language abbreviations are as follows: Bengali (bn). Chinese (zh), English (en), French (frr), German (de), Japanese (ja), Russian (ru). Spanish (cs), Swahili (sw), Telugu (te), and Thai (th).
80
Perplexity
CoE-R
Entropy
CoE-C
75
AUROC Value
70
60
55
50
en
de
fr
ja
ru
SW
te
zh
Language
Figure 15: AUROC results (w/ L model) on MGSM dataset with 11 languages.
Perplexity Entropy
CoE-R CoE-C
75
AUROC Value
65
en
de
fr
zh
Language
Figure 16: AUROC results (w/ Biaamaa--B-Instaree model) on MGSM dataset with 11 languages.
32

---

Published as a conference paper at ICLR 2025
80
Perplexity
CoE-R CoE-C
Entropy
75
AUROC Value
70
65
60
de
te
th
zh
Language
Figure 17: AUROC results (w/ Qwen S-B--Instruct model) on MGSM dataset with 11 languages.
80
Perplexity
CoE-R CoE-C
Entropy
75
AUROC Value
70
65
60
de
fr
te
th
zh
Language
Figure 18: AUROC results (w/ Qwen2-7B-Insten model) on MGSM dataset with 11 languages.
33


Reviews:
[{"review_id": "ju3L8gPjL1", "reviewer": "Reviewer_2nBf", "summary": "The authors propose the Chain-of-Embedding (CoE) in the latent space to enable LLMs to perform output-free self-evaluation, and find that when LLMs respond correctly and incorrectly, their CoE features differ, these discrepancies assist us in estimating LLM response correctness.", "questions": "My only question is,  there are a a lot of interesting findings in this paper, such as\n1. Hidden states to estimate the LLM response correctness without any label\n2. CoE discrepancies may happen when LLMs generate correct and incorrect responses,\nhow can these findings be used to improve the reasoning accuracy of LLMs during pre-training or post-training?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is quite interesting to discover that the hidden states of LLMs can be utilized to estimate the LLM response correctness without any label, which inspires me a lot.\n\nMoreover, it is also reasonable to treat the progressive hidden states as the latent thinking path of LLMs, leading to the following\nassumption: CoE discrepancies may happen when LLMs generate correct and incorrect responses.\n\nOverall, it is quite inspireable paper and give the readers a lot of insights on the thinking system of LLMs", "weaknesses": "I donot think this paper have any significant weaknesses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3EiBqucI3j", "reviewer": "Reviewer_GkSP", "summary": "The authors propose a metric for label-free LLM self-evaluation, that utilizes the observed discrepancies in progressive hidden states when LLMs generate correct and incorrect responses. Intuited by the cognitive phenomenon in human thinking, the authors measure the CoE discrepancy in the two sets, responding correctly and incorrectly. Based on the obvious discrepancies in Magnitude and Angle, the authors give the two metrics, CoE-R and CoE-C. Finally, the authors achieved SOTA in various datasets and backbone LLMs. Furthermore, there are other analyses in many aspects, e.g., efficiency and multilingual scalability.", "questions": "a. How does this metric work? Is there a threshold to decide the classification using metrics? If so, what is this threshold?\n\nb. In line 104, why does the average embedding at layer l represent the l-th sentence hidden state?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The literature of this paper is comprehensive.\n\n2. The exploration is structured and natural. From intuitive cognitive phenomenon to observation, the authors verify the assumpted discrepancy between correct and incorrect generation. Then, the authors proposed a metric and then demonstrated the effectiveness of the metric. The idea is well-founded.\n\n3. The authors verify their proposed metrics on various backbones and datasets.\n\n4. The authors give a detailed analysis of many aspects.", "weaknesses": "1. Missing critical statistic analysis of discrepancies. The foundation of the proposed metrics is based on the existence of the discrepancy between correct and incorrect generation. However, only visualizing this discrepancy is not enough.\n\n2. Why is the targeted task self-evaluation? self-evaluation is a task that seems appealing but is quite ambiguous and questionable; even if the reviewer reads the reference paper cited by authors for self-evaluation, there is no task definition of self-evaluation.\n\nIn the authors' presentation, based on the experiment setting and references cited by the authors, self-evaluation is highly associated with uncertainty estimation, but the authors don't mention their targeted method for uncertainty estimation; furthermore, the authors present self-evaluation as a potential method enabling evaluating LLM's responses without labels in the introduction section, which is associated with llm-as-a-judge and self-rewarding. There is a big gap between the two tasks. For label-free self-evaluation, I advise authors to compare their metrics with llm-as-a-judge or self-rewarding methods (label-based self-evaluation) in some instruction-following tasks, even not sota.\n\n3. Missing verifying on some key datasets, for example, triviaQA and TruthfulQA, which have been tested in many previous works cited by authors;\n\n4. Missing critical robustness analysis on adversarial samples, especially uncertainty estimation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pLSF4j0eqR", "reviewer": "Reviewer_in4E", "summary": "The Chain-of-Embedding (CoE) is a novel approach that allows LLMs to self-evaluate correctness by analyzing their hidden states during inference, forming a latent \"thinking path.\" This label-free method reveals distinct patterns for correct and incorrect responses, enabling real-time, output-free accuracy estimation across various domains. CoE’s minimal computational cost and interpretability make it effective for large-scale applications.", "questions": "1. How does the proposed method handle distributional differences across open-world queries?\n\n2. Can the method reliably distinguish trajectories across diverse domains?\n\n3. What is the evidence for aligning human thinking paths with LLM intermediate paths?\n\n4. How does layer count affect the method's performance?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. CoE uses internal hidden states rather than output-based confidence, enhancing adaptability to new tasks without training data.\n\n2. With millisecond-level computation, CoE scales effectively, offering quick feedback for large-scale deployments.\n\n3. CoE provides insights into LLM decision-making by tracing hidden states, offering a transparent, human-like assessment of response correctness.", "weaknesses": "1. My major concern is the scenario and the necessity for the label-free scenario. If we limited the LLM query into only math question, commonQA questions, then the label-free scenario may not be necessary. A few-shot human labeling can achieve satisfied performance. The key reason for the label-free is the open-world scenario, which the query can be arbitrary from various domains.  It is impossible for us to label across queries with different purposes. Considering the open-world assumption, there are two following-up question as follows:  \n1.1. The figure 2 and 3 try to distinguish the discrepancy between the positive and negative samples. When we consider the open-world assumption, there can be more questions with different distributions from a single domain, which may not be able to distinguish. \n1.2. The positive samples from different domains show different trajectories, is it possible for the method to distinguish them given queries from different domains\n\n2. The motivation in the introduction part seems confusing. There is no evidence to indicate the correspondence between the human thinking path and LLM intermediate path. Moreover, there may many knowledge-intensive task may not need the thinking path. \n\n3. The evaluation part (especially how to identify the correctness) is not so clear to me. I would like to kindly ask the authors provide a more detailed explanation on it. \n\n4. The path length is related with the number of layer. I think an additional analysis on how the number of layer influence the performance of the proposed method could make the paper more solid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)