NoVo: Norm Voting off Hallucinations with Attention Heads in Large Language Models
Authors: Zheng Yi Ho
Siyuan Liang
Sen Zhang
Yibing Zhan
Dacheng Tao
Venue: ICLR 2025 Poster (2025)
Keywords: Hallucination Mitigation
Large Language Models
TruthfulQA
Representation Editing
Multiple Choice Question Answering
Attention Heads
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce NoVo for reducing hallucinations, a novel method that is significantly more accurate and generalisable than previous methods, while being simpler to use.

Abstract: Hallucinations in Large Language Models (LLMs) remain a major obstacle, particularly in high-stakes applications where factual accuracy is critical. While representation editing and reading methods have made strides in reducing hallucinations, their heavy reliance on specialised tools and training on in-domain samples, makes them difficult to scale and prone to overfitting. This limits their accuracy gains and generalizability to diverse datasets. This paper presents a lightweight method, Norm Voting (NoVo), which harnesses the untapped potential of attention head norms to dramatically enhance factual accuracy in zero-shot multiple-choice questions (MCQs). NoVo begins by automatically selecting truth-correlated head norms with an efficient, inference-only algorithm using only 30 random samples, allowing NoVo to effortlessly scale to diverse datasets. Afterwards, selected head norms are employed in a simple voting algorithm, which yields significant gains in prediction accuracy. On TruthfulQA MC1, NoVo surpasses the current state-of-the-art and all previous methods by an astounding margin---at least 19 accuracy points. NoVo demonstrates exceptional generalization to 20 diverse datasets, with significant gains in over 90\% of them, far exceeding all current representation editing and reading methods. NoVo also reveals promising gains to finetuning strategies and building textual adversarial defence. NoVo's effectiveness with head norms opens new frontiers in LLM interpretability, robustness and reliability. Our code is available at: https://github.com/hozhengyi/novo

Published as a conference paper at ICLR 2025
NoVo: NORM VOTING OFF HALLUCINATIONS WITH ATTENTION HEADS IN LARGE LANGUAGE MODELS
Zhengyi Ho', Siyuan Liang", Sen Zhang', Yibing Zhan', Dacheng Tao" (zhengyic01, siyuan.liang, dacheng. caaannn.taa. edu.uueddu.gg {senzhang thul0, jeaanyyyy@@malll com
Nanyyng Technological University, Singapore 639798 University of Sydney "Corresponding Author
ABSTRACT
Hallucinations in Large Language Models (LLMs) remain major obstacle, par- ticularly in high-stakes applications where factual accuracy is critical. While rep- resentation editing and reading methods have made strides in reducing hallucina- tions, their heavy reliance on specialised tools and training on in-domain samples, makes them difficult to scale and prone to overfitting. This limits their accuracy gains and generalizability to diverse datasets. This paper presents lightweight method, Norm Voting (No which harnesses the untapped potential of attention head norms to dramatically enhance factual accuracy in zero-shot multiple-choice questions (MCQs). No begins by automatically selecting truth-correlated head norms with an efficient, inference-only algorithm using only 30 random samples, allowing No Vo to effortlessly scale to diverse datasets. Afterwards, selected head norms are employed in simple voting algorithm, which yields significant gains in prediction accuracy. On TruthfulQA MC1, NoVo surpasses the current state- of-the-art and all previous methods by astounding margin least 19 accu- racy points. No demonstrates exceptional generalization to 20 diverse datasets, with significant gains in over 90% of them, far exceeding all current representa- tion editing and reading methods. No also reveals promising gains to finetun- ing strategies and building textual adversarial defence. NoVo's 's effectiveness with head norms opens new frontiers in L.MM interpretabilityy robustness and reliability. Our code is available at: hutgs:/ggihubbccomm/ohhngvvvoooo
INTRODUCTION
One of the most significant challenges facing Large Language Models (LLMs) is their tendency to hallucinate-outpae that are factually incorrect or entirely fabricated (Zhang et al. 2023b). This flaw is particularly serious in high-stakes applications like finance and healthcare, where even small errors can lead to huge losses and compromised patient safety (Kang & Liu 2023] Pal et al., 2023a). Reducing factual hallucinations is critical research area with major practical benefits, essential for realising the full potential of L.MMs to revolutionise these industries by enhancing efficiency and decision-making, and safeguarding against costly and harmful errors (Kaddour et al. 2023).
Given these serious risks and the high cost of retraining LLMs, it is crucial to find affordable tech- niques to reduce factual hallucinations. Although inference techniques such as retrieval augmenta- tion and prompt engineering work well, they come with significant limitations: latency and external dependencies, and the need for user expertise, respectively (Zhao al. 2024 Sahoo et al. 2024). In response, we turn to representation editing and reading methods (REAR) [Zou et al.) 2023). which operate within the model, ensuring rapid response times and eliminating the need for external data or user interaction. REAR methods reduce hallucinations by modifying or extracting factual infor- mation encoded in LLMs' latent feature vectors (hidden states), such as attention heads (Bronzini et al. 2024). This process often requires specialised tools such probes and autoencoders (Li et al. 2024 Zhang et al., 2024), trained and tuned on in-domain samples. Thus, existing REAR methods difficult to scale and prone to overfitting, leading to limited accuracy gains and generalizability to diverse datasets. Tackling these limitations is crucial, since REAR methods can improve factuality with minimal costs, latency, and user friction; highly desirable attributes for practical applications.

---

Published as a conference paper at ICLR 2025
This paper presents Norm Voting (NoVo), more accurate REAR method for reducing factual hallu- cinations in diverse multi-choice scenarios. Vo works by efficiently measuring latent truth (Zou et al. 2023 in certain attention head norms, thus avoiding the log likelihood layer, which can induce hallucinations by favouring fluency over factuality (Ji et al. 2023). NoVo first selects attention head norms that correlate with truth using only inference 30 random samples, allowing No to scale to numerous datasets. Then, selected head norms participate in majority voting as an ensemble of weak learners (Schapire [990) to boost accuracy. The process is summarised in Figure [ No is made lightweight by design for scalable use, requiring no specialised tools or training. which en- ables wide evaluations across diverse range of reasoning, factuality, and understanding datasets. To our knowledge, we are the first to explore attention head norms measure of latent truth. This raises exciting questions about their wider roles in interpreting and addressing hallucinations.
Q: How do you buckle down on something?
Llama2
NoVo
(1)
(1) Concentrate on nothing else.
(2) Leave it alone.
7B
Log Likelihood
(2)
Figure 1: Overview of our method. NoVo improves factuality over the log likelihood.
On TruthfulQA MCI, an unsolved hallucination benchmark, NoVo achieves a new state-of-the-art (SOTA) accuracy of 78. 8.009 on a 7B model, substantially outperforming the log likelihood and the best REAR method by at least 24 and 19 accuracy points. NoVo scales and generalizes well to 20 diverse datasets featuring varied topics and formats, with significant gains in over 90% of them, dramatically surpassing previous REAR methods which were only evaluated on few factuality benchmarks. Additionally, NoVo achieves promising gains on AdversariaIGI.UF and DeBERTa finetuming. We choose to evaluate No Vo on multiple-choice questions (MCQs) because they test foundational cogmitive skills (Bloom et al. 1964). are critical for real-life applications in high-stakes standardised assessments (OSHA] Pal et al. 2023b NCBE 2024 ETS 2024), offer an objec- tive metric across diverse benchmarks for academic evaluation (Mee et al. 20024), and pose unique technical challenges that reveal core limitations and internal misalignments within LLMs (Zheng let al. 2024; Kiela et al.] 2021). We analyse why head norms are correlated with truthfulness, and find that they reliably spike by up 83% in token positions of both factual proposition completions and pertinent factual associations despite misleading contexts. Beyond MCQs. our findings reveal more fundamental, task-agnostic problem in high-stakes scenarios: factual hallucinations caused by misalignments between hidden states and the language likelihood. No strong performance across diverse and representative benchmarks demonstrates that this problem can be mitigated using head norms. These norms show promise in reliably ranking truth across multiple candidate spans during decoding or retrieval, and in future works for enhancing model robustness and alignment during fine-tuning, ultimately contributing to more trustworthy generations in high-stakes scenarios.
Our main contributions can be summarised three points: We use head norms to accurately rank truthfulness between several candidate texts, evaluated on diverse MCQs. We show and explain the correlation between head norms and truth. ② We demonstrate and mitigate a fundamental cause of factual hallucinations: language likelihood misalignments with internal states, using TruthfulQA.
RELATED WORKS
Representation Editing Some REAR methods involve manually modifying hidden states during inference towards hidden state clusters, formed by the forward pass of true and false sequences (Bums et al.) 2023), as a generic hallucination mitigation technique. All methods here require cross- fold training on in-domain samples from the test set, with some set aside for validation. Inference Time Intervention (ITT) edits specific attention head hidden states towards those clusters (Li et al.] 2024), using custom-built lincar probes and visualisation tools. Similarly, TruthForest (TrFr) edits heads toward multiple directions (Chen et al., 2024), while Truthx edits concepts of truth disentan- gled from hidden states with deep autoenconder (Zhang al. 2024). specialised tool.
Representation Reading There are decoding strategies that use hidden states to improve the factu- ality of LLMs without editing. Decoding by Contrasting Layers (DoLa) (Chuang et al. 2024). tuned with in-domain samples, extracts factual information in intermediate layers. Induce- teeeeetth--Caae Contrast Decoding (ICD) (Zhang al. 2023a) contrasts L.LM outputs with special hallucinatory model
2

---

Published as a conference paper at ICLR 2025
trained on external dataset. RePE (Zou al. 2023) relies curated templates and samples to measure truth in hidden states using a specialised tool known as linear artificial tomography. All these methods, except RePE, generic hallucination mitigation techniques without editing.
Unlike current REAR methods, NoVo uses only attention head norms and does not require any external modules, custom-built probes, special techniques, in-domain sample training, curated resources. This makes No lightweight, enabling it to scale and generalize to numerous MCQ tasks. Together with a simple voting algorithm, NoVo also significantly more accurate.
METHOD
3.1 BACKGROUND
Prior Insights Previous studies have demonstrated that hidden states in the multi-layes perceptron (MLP) modules of LLMs can be linearly classified into true-false clusters (Burns al. 2023 Azaria & Mitchell, 2023] Zou et al. (023). Further studies extended this idea to individual heads in the multi-head attention (MHA) module (Li et al. 2024 Chen al. 2024).. In computer vision, studies have shown that the 1.2 norm of the final feature vector in convolutional networks correlates with
image quality (Kim & Lee 2017 Yan et al. 2019). Insights from these works show that it is reasonable to expect the L2 norm of some heads, denoted I', to correlate with truth.
Setup In the forward pass of an auto-regressive decoder transformer LLM, token sequences of length embedded and featurized through multiple layers, each consisting of MHA and MLP module, before reaching the logit layer for next-token prediction. An LLM with L layers and H heads per MHA will have total of LH heads throughout the network, excluding the logit and embedding layers. The MHA at layer previous layer and projects each feature in the sequence to their key, query and value states
{1,2... L} takes input (1-1) Raxd from the
Q' = (1-I)) w!
= X'1
V = (1-1) value
(1)
query
key
ignoring the bias term, where Q', K Vl E Rexd and d the model dimension. Splitting them on the column axis gives Qth Klhb Vih E Raxd' for h E (1,2..., H) and d' = d/H. The context vectors, attention heads, Cl.. E Raxd' are thus computed via the attention mechanism as
14.2...H1
A softmax (OLA (KKLYT M
Cth = A.
Vd'
where M enforces auto-regression by setting 4 to a lower triangular matrix. Equation each head in the sequence C the attention weighted sum each value state Via. computed component- wise from the current and all previous sequence positions
a11v11
a11vid
Σ 02j
Σ Σド= j=1 Ujd'
Chhb A Liihh
Σj Ozf
Σj Ozj
Motivation Insights from prior works suggests a reasonable expectation that T' correlates with the truthfulness of text sequence. Furthermore, LLMs encode diverse language features in their hidden states, which often self-organise along meaningful dimensions (Mikolov) 2013). It is plausible that some dimensions reflect the alignment of truthful propositions with reality, where the coherence of certain concepts such as passengers and planes, might consistently express itself as L2 norm magnitude changes. The expectation that L2 norms can measure truth as a broad and continuous scalar, is well-aligned with findings made by Lin et al. (2022). which framed truth probability measure, and Li al. al. (2024). which proposed that latent truth is expressed in multiple directions. For both auto-regressive and bi-directional LLMs, the end token attends to the entire sequence, without needing to know where specific factual claims appear. Therefore, we define T as the attention head norm at the final sequence position such that T'A C -1. This process is shown Figure We do not assume the correlation direction and allow for inverse relationships well. In Appendix
3

---

Published as a conference paper at ICLR 2025
show that the latter approach is better. Since T is unbounded, with correlation direction and T. h being unspecified, cannot be used yet. T instead forms the basis for No which addresses these issues and operationalises T to improve factual accuracy in MCQs.
Option
Head
Embedding
1.31 ... 0.91
MCQ Question
Option
OKU
Head
O Pro
FFN
1.25
0.76
N x HL
Option
Head
Norm (T) Matrix
X LLM Layers
Figure 2: The Norm Matrix at the right contains all T values taken throughout the LLM,, but cannot be used to answer MCQs. Instead, this operation forms the basic building block of No
3.2 NORM VOTING (NoVo)
Norm Selection The goal of this stage is to operationalise T't by resolving its unbounded nature, and specifying all (1, h)) indices that correlates with truth, including the correlation direction. Figure 3 shows this stage in five steps. In step 30 random samples are fed into the LLM to produce 30 Norm Matrices, packed as a tensor. The idea here is that all head norms are initially assumed to correlate with truth, each producing two predictions from the argmax and argmin operators. These are packed into an intermediate tensor, as the correlation direction is unknown. The unbounded nature of Til is resolved here, since both operators relative. In step @, each head receives an accuracy score across 30 samples for both sets of prediction, forming a matrix with two rows representing each prediction set, and columns that represent each head's accuracy. that most heads are poor performers. In steps and 4 the correlation direction and strength are identified using these accuracies scores proxy measure. This approach does not require any training. special techniques or external tools, making NoVo lightweight and scalable. The row with highest accuracy indicates the correlation direction. Steps and determines which heads are strongly correlated with truth, by taking the higher accuracy of the two rows. This is followed by thresholding operation, set the 85th percentile (Pss) of all accuracies. We refer to these remaining heads as "Voters". For clarity, (t, h) is enumerated as consecutive integers, starting from 0 for the first head in the first layer. This entire stage only performed once, as the Index Vector and Indicators reused, and takes less than 10 seconds one NVIDIA A100 GPU. The number of samples and threshold are aapeer-rr,eterrs found to be optimal at 30 and Pss The search for these two values is detailed in Appendix B with hyper-parameter free variant explored in Appendix C
clear here
Anouracy
Argras
1:31
2,011
BUMS
Indicators
Aegris
Seice
Oo
0.96
R.56
Thewsbold 2:h
[535
758]
30 X. Norm (?) Tensor
30 2 Intermediste Tensor
2 Accuracy Matrix
Score Vestor RM
Index Vector
Figure 3: The selection stage uses the Norm Matrix from Figure 2 to determine the correlation direction of each T serialised as Indicators. All (I, indices that vary with truth are also specified in the Index Vector, expressed as enumerated integers for clarity.
Voting Inference Now that the latent measure of truth T h is operationalised with No zero- shot MCQ classification can begin. The goal of this stage is to output more accurate predictions via majority voting, shown in four steps in Figure 4 In Step an example MCQ with three options is fed through the LLM to produce the Norm Matrix. Each answer is prepended with the question and optional instructions as input, following standard practice. In Step Voters selected with the Index Vector from the previous stage. In Step Q the correlation direction of each Voter is flagged with Indicators, also from the previous stage. This allows for dynamic selection between the argmax or argmin operators, for individual Voter predictions. While each Voter's T is unbounded and could become very large, observe in practice that it well-conditioned to varying truthfulness in
4

---

Published as a conference paper at ICLR 2025
sequence. In most cases, T ranges between 0.5 to 3. In step all Voter predictions participate in majority vote via the mode operator, resulting in the final MCQ prediction of the LLM.
Al: Mttrgee
Finel Prediction
Primary gas Earth's
A2: Oxygen
1236
1.25 18,55
1.55
Todexinng Jato
1.23
141
185
D: Argrin Jase
Mode
atmosphere?
2.76
2.76
(Nitrogen)
A3: Dloxide
Norm Matrix
3xx MC.
Index Vector
1535
758
Voter Matrix
3
Indicators
Vater Predictions c R
Figure 4: The voting stage uses the Norm Matrix from Figure and the Indicators and Index Vector from Figure 3 to accurately answer MCQ questions during LLM inference.
4 EXPERIMENT AND DISCUSSION
4.1 SETTINGS
Experiments We evaluate No in three key areas: its effectiveness in reducing factual hallu- cinations compared to existing REAR methods, 2 its generalizability across various reasoning and natural language understanding (NLU) tasks, and its adaptability broader classification tasks, indicated by its finetuning performance. Experimental results for the first area are shown in Table I Results for the second area are presented Tables 2 3 and while the third area reported at the bottom of Table 2 To avoid over-reporting results, all experiments use 30 random training samples drawn without tuning for Norm Selection, and use zero-shot prompts without tuning (Perez jet al. 2021). More information experimental details, random variations, hidden state analysis, D H
and additional models can be found in Appendices A
and respecccci..
Models NoVo is evaluated in two classification settings: zero-shot and finetuned. Zero-shot is the primary setting used in most experiments, and the results are presented in Tables I through Finetuning, on the other hand, is used in only one experiment, which reported the bottom of Table 2 In the zero-shot setting, NoVo is applied to four 7B decoder LLMs: Llama2 and LLama2-
Chat Touvron al.) 2023). Vicuna (Chiang al. 2023) and Mistral-Instrun (Jiang et al. 2023
For the linetuned setting, No is applied to DeBERTa-Land
(He et al. [002)). Additionally, Table
includes results from two finetuned IIB models, UnifiedQA and UNICORN (Khashabi al.
2020
Lourie et al. 2021). for reference purposes only, without making any direct comparisons.
Datasets We evaluate NoVo's effectiveness in reducing factual hallucinations on TruthfulQA MC1 (Lin et al. 2022). a standard and unsolved hallucination benchmark used by all previous REAR methods. For our generalizability experiment, we apply No to diverse datasets covering multiple topics and presented in various formats. This includes CommonsenseQA 2.0 (CQA2) (Talmor al.) 2021 for commonsense reasoning. QASC (Khot et al., 2020) tests for scientific knowledge. SWAG
(Zellers et al. 2018)) and HellaSwag (HSwag) (Zellers et al. 2019) requires sentence completions
about challenging commonsense scenarios. SIQA (Sap et al., 2019) and PIQA (Bisk et al.
2020)
looks for social and physical reasoning, respectively. CosmosQA (Cosmos) (Huang al.
2019
requires causal reasoning over narrative contexts. CICERO VI and V2 (CICvI, CICv2) (Ghosal al. 2022b Shen al. 2022) tests for multi-tum dialogue and strategic reasoning. We use MCQ variant from Ghosal et al. (2022a). Adversarial GLUE (AdvGLUE) (Wang et al., [021 tests model robustness adversarial texts in NLU tasks. FACTOR-Expert (expert) (Muhlgay
al. 2023). Natural Questions (nq) (Kwiatkowski et al.) 2019). and TriviaQA (trivia) (Jshi et al. 2017) all contain general factual questions from expert domains or online documents. We reformulate nq and trivia following Li et al. (2024). MMLU (Hendrycks et al.) (020) involves broad range of topics, and Are (Clark et al. 2018) contains science question. All datasets report accuracy.
4.2 MAIN RESULTS
Hallucination Mitigation Table | reports the zero-shot accuracy of NoVo on TruthfulQA MCI across four models. Results show that NoVo significantly outperforms all existing REAR methods across all models. Notably, they all require either cross-fold training, few-shot prompting, custom instructions. but No uses only true zero-shot prompts with 30 random samples from Arc-Easy's train split for Norm Selection. Vo on 7B model surpasses GPT4 by remarkable margin of
5

---

Published as a conference paper at ICLR 2025
19 points, setting new SOTA accuracy of 78.09%. The median point gain across all competing methods including the log likelihood (LM), for each model, is reported with a green arrow beside NoVo's result. Here
that the overall gains remarkably high, with the highest at 31 points.
Tabll 1:
TruthfulQA MC1-NoW achieves SOTA accuracy with zero-shot only. Other approaches require either cross-fold training, few-shot prompting, or custom instructions.
Zero-shot
Few-shol
Custom
Model
LM
NoVo
TruthX
ITI
TrFr
DoLa
ICD
RePE
GPT4
Llama2-7B BB-Chat
34.27
76.1
54.22
40.67
39.30
33.53
46.32
58.9
Llama2-7B
28.48
31.1
49.94
37.86
33.80
31.21
40.76
59.0
Vicuna-7B
34.64
69.89 3000
50.67
39.90
38.80
33.05
47.19
Eistral-B-IIntruuee
53.86
72.0
56.43
55.73
48.83
58.13
Table 2: Experiments on generalization and finetuning at the top and bottom, respectively.
CQA2 0..84
QASC
SWAG
HSwag
SIQA
PIQA
Cosmos
CICvI
CICv2
Model
Method
15.88
3355
122.0
40.96
18.00
0.28
22.55
LM
55.65
19.76
60.51
56.30
45.45
72.63
36.42
37.74
42.34
Llama2-7B-C
NoVo
56.04
43.95
68.36
59.49
60.29
72.96
51.73
36.01
63.61
49.98
25.16
74.59
71.59
49.08
76.99
38.53
38.34
37.85
Llama2-7B
NoVo
52.11
35.42
75.01
70.53
58.44
71.92
51.76
29.52
60.37
Vicuna-7B
LM
50.89
36,20
67.62
61.03
46.26
74.86
33.47
34.55
36.49
NoVo
51.40
42.66
69.67
69.20
61.15
74.37
56.45
39.23
69.42
LM
61.90
31.53
63.31
75,28
46,93
76.39
31.69
40.25
38.52
Mistral-BB-Insse
No
62.02
66.09
69.65
63.35
70.68
76.66
67.57
46,09
73.52
SFT
67.37
71.74
92.37
94.72
80.18
87.41
85.51
88.04
92.67
DeBERTa-Large
TEAM
68.38
74.35
94.12
95.57
79.89
85.92
86.86
86.84
93.25
NooVo
68.42
75.65
93.38
94.35
80.83
87.58
88.09
89.47
93.69
UnifiedQA- -1B UNICORN-IIB
78.50
81.40
89.50
SFT
70.20
93.20
83.20
90.10
91.80
Table 3: Generalization experiments on Adversarial GLUE.
Datusets
SST2
QQP
$5.66
MNLI
12204
MNLI-MM 9
QNLI
1.0
RTE $7.09
Methods
LM
NoVo
LM
NoVo
LM
NoVo
LM
NoVo
LM
NoVo
LM
No
LLama2-7B-Chal
55.54
79.60
63.14
63.26
35.42
51.48
35.68
51.58
75.00
76.65
49.57
54.28
Llama2-7B
63.74
65.26
43.41
63.26
35.40
43.43
35.69
39.42
51.86
65.27
44.41
52.61
Vicuna-7B
74.65
77.43
54.02
63.26
35.42
55.39
35.68
55.48
81.87
74.99
48.19
54.16
Mistral-7B 7B-Instruct
72.95
78.34
77.28
79.36
74.98
69.65
74.54
69.13
83.64
84.14
46.99
66.61
Generalizability The top of Table 2 reports NoVo's zero-shot validation accuracy multi- ple reasoning datasets. For Norm Selection, each dataset uses 30 randomly drawn samples from their train splits. Median point gains across models for each dataset, are indicated with green arrow, while negative values are marked red. NoVo sub- stantially outperforms the LM QASC, Cosmos, CICv2, and SIQA, with modests gains in CQA2, SWAG, and CICvI. However, accuracy drops in HSwag and PIQA. Table 3 reports the 10-fold average zero-shot validation accuracy on AdvGLUE, with each fold holding out 30 random samples for Norm Selection. Median point gains across models for each subset, are indicated with green arrow. No mostly outperforms the LM all six subsets, with accuracy drops limited to instruction models. zero-shot validation accuracy factuality benchmarks, with each holding out 30 random samples
Table 4: Generalization on factuality tests.
Llama2-7B
expert
nq
trivia
mmlu
Chat
118.
133
$1.22
NoVo TruthX
76.82
72.30
97.81
47.13
68,51
59.60
66.79
ITT
$1.69
57.83
65.96
ICD
46.02
67.29
the 10-fold average

---

Published as a conference paper at ICLR 2025
for Norm Selection. Median point gains across competing methods for each dataset, are indicated with a green arrow. NoVo significantly outperforms all REAR methods here. Results from Tables 2
B and 4 show that NoVo scales and generalizes well across diverse reasoning, factuality and NLU
datasets, with competitive gains on AdvGLUE suggesting potential for adversarial textual defence.
Finetuning The bottom of Table reports finetuned test accuracies using DeBERTa. Finetuned NoVo (oNoVo) is compared to standard finetuning (SFT) and an effective SFT variant known as TEAM, which reformulates each question to admit binary answers (Ghosal et al. 2022a). No Vo outperforms SFT in all datasets except HSwag by an average of 1.3 points, and surpasses TEAM in all but HSwag and SWAG by an average of 0.7 points. These results suggest No potential for adapting to and improving finetuned accuracy for general classification, beyond zero-shot MCQs. The implementation of +No Vo is detailed in Appendix A
4.3 ERROR ANALYSIS
Tahle representative samples from PIQA, our lowest-performing dataset. We that No Vo misclassifications often involve equally plausible answers that require strong stereotypes to disam- biguate. For example in the fifth row, many buckets can hold both paint and acid depending on the specific context. The stereotype here is that either the acid is very strong, or that the bucket is metal- lic. contrast, NoVo's 's correct predictions, misclassified by the LM,, are equally difficult, yet do not require strong stereotypes to solve. For example in the sixth row, not all jars are twist-to-open, but this disambiguation Additionally, we observe that questions with identical answer options, such as "no" and "no, it is not", are unpredictably answered. We conclude that No Vo's good performance on misleading ques- tions, limitations in selecting stereotypical answers, and randomness in identical answer options, all contribute to result variations across datasets.
not needed, because the other option is mostly untrue for typical jars.
Table 5: Misclassified PIQA samples on Llama2-7B.
Misclassified by No
Correctly classified by No Misclassified by L.M Q: how do you buckle down on something?
Cormenty classifeed I.M
Q:
Correct: cleans furniture.
Correct: concentrate on nothing else.
Wrong: cleans clothes.
Wrong: leave it alone.
Q: ornament
Q: lipstick
Correct: can decorate tree.
Correct: can he used to write words.
Wrong: can decorate desk.
Wrong: can used speak words.
Q: how do streum movic?
Q: What do you use to make a DIY lotion bar smell good?
Correct: watch it over the internet.
Correct: You can use scented oils, about ten drops will do.
Wrong: watch your
Wrong: can
add as many drops you'd like.
Q: soup
Q: mold
Correct: can clean a car.
Correct: can cover shovel.
Wrong: can clean mold.
Wrong: more useful than a shovel.
Q: bucket
Q: a kntle
Correct: can hold puint.
Correct: can transfer grapes from a glass
Wrong: can hold acid.
Wrong: can transfer liquid from glass
Q: To thicken mixture
Q: open jar
Correct: Add corn starch
Correct: tap bottom and twist
Wrong: Add com syrup.
Wrong: make sure you hear click
Q: Retain study notes brain
how do you prepuy pizza delivery order?
Correct: Go over notes one last time one day before test.
Correct: give the compuny your card information before they deliver.
Wrong: over notes one last time one week before test.
Wrong: give company your cash hefore they deliver.
Q: shelf
Q: Keep paint from drying.
Correct: can hold a book.
Correct: Place saran wrap over opening before closing with lid,
Wrong: can hold milk.
Wrong: Place paper towel over opening before closing with lid.
4.4 DISCUSSION
NoVo's SOTA accuracy on TruthfulQA show that head norms are a reliable way to avoid fluent factual hallucinations from misleading questions. Gains in over 90% of the 20 diverse benchmarks and 4 models, suggests that the relationship between truthfulness and head norms is generalizable. Error analysis on low-performing benchmarks suggests that while No Vo mitigates fluent falsehoods in misleading questions, this same ability struggles to form strong stereotypes for highly ambiguous questions that probes for assumptive behaviour. Beyond MCQs, we show strong evidence that head norms can reliably rank truthfulness across multiple texts, which could be useful for future works
7

---

Published as a conference paper at ICLR 2025
in generative and retrieval strategies that outputs several candidate spans for truth auth-filtering. No is also suitable for hallucination detection on-the-fly, due to its lightweight nature. Promising results on AdversarialGLUE and DcBERTa finetuning showcase the potential of future works using head norms for improving adversarial robustness and finetuning accuracy during alignment, resulting in safer open-ended generations. Overall experimental results strongly indicates more fundamental lask-agnostic problem: factual hallucinations stem from misalignments between internal states and the language modelling likelihood (Zhou al. 2024; Jaiswal al. 2024). Using MCQ tasks, we demonstrate the internal misalignment problem in hallucinations, and use the strong correlation between head norms and truthfulness mitigate it.
ANALYSIS
5.1
WHAT DO VOTERS MEASURE?
Voter 527
Voter 509
Voter 665
Can
Norm Gain Over Wrong Answer
1gF
Context Compone Mosi
fly
Ons
Qns
Weighted Sum
No
Mid
1gs
fy
Masked
Masked
Masked
1.0
Voter Taken at Sequence Position
Figure 5: Attention-weighted value state components at various sequence positions.
Plotting We plot the token contributions for each Voter in Figure Each column represents Voter (head), broken down into its attention-weightee value contributions per token on the left vertical axis and with cell color intensity. Voters are taken at various sequence positions on the horizontal bottom axis, starting from the end (-1). A line plot summarises the relative norm gain for each Voter over the wrong answer, graded on the right vertical axis. Because heads are high- dimensional, the plot displays the mean across all vector components per cell. These three Voter are selected here for display based their representative patterns, with more shown in Appendix D
Voter Specialisation Voter 527 has the largest norm gains at the last three positions, with dras- tic drop the middle, slowly recovering at the first two tokens. In this Voter, most end tokens strongly attend to themselves, especially when taken at the final sequence position. In contrast, both Voters 509
Truthfu Acc Magr rune
Type-1 Voters
Type-2 Voters
Intralaye
and 665 places more weight on other tokens, such between 'can' and 'fly". When taken these in- termediate positions where such tokens occur, these two Voters show far larger gains than when taken at the end sequence position. Plotting other Voters in Appendix F reveal broadly similar patterns to Fig- ure 5] suggesting two general types of Voters. We characterise Type-1 Voters (TI) as those attending to periods and end tokens measure of structure, while Type-2 Voters (T2) attend to individual token associations measure of resolution and dependence. Table and Figure 6 show that both Voter types exhibit similar performance, with no clear preference for either. and are evenly distributed throughout the upper portions of the model, suggesting specific localisation of these roles.
Layer
31
Figure 6: T1 and T2 are evenly spread out after the ninth layer of Llama2-7B.
8

---

Published as a conference paper at ICLR 2025
What is being Measured Based NoVo's majority voting process back in Figure we that each Voter type plays a distinct yet complementary role in shaping the model's 's capacity for making more factual predictions in MCQs. These analyses suggests that the reason head norms correlate with truthfulness is due to strong attention spikes by up to 83% over non-factual statements, in token positions where either the factual proposition is complete, occurs. We further note that even small semantic shifts in meaning can alter truth, leading to large but organised movements the high-dimensional space of attention heads, causing points to lay in a predictably different hypersphere surface. These mechanisms are possible reasons why truth and L2 head norms are correlated
when relevant factual associations
5.2 EFFECTIVENESS OF USING MULTIPLE VOTERS
Table 6: Nistral--B-Instrucc:: Summary of Individual Voter Accuracies (%) on TruthfulQA.
Voter
Count
Mean
Std
Min
25Q
50Q
75Q
Max
All
1024
37.29
8.00
20.32
31.43
35.86
41.49
69.77
Type-1
165
42.17
9.46
26.56
34.76
40.63
48.59
69.77
Type-2
86
39.49
9.59
25.53
32.81
37.45
44.96
63.89
Plotting To assess the effectiveness of the majority vote, we analyse each Voter's contribution to the overall accuracy of No On the left of Figure Voters are sorted by individual accuracy and are gradually included in the voting process at each step of the horizontal axis, with accuracy graded on the left vertical axis. The smoothed Pearson correlation between the error vectors for the current and previous mix is plotted alongside the accuracy curve, with the values graded on the right vertical axis. The dotted and solid black vertical lines indicate the point of no significant increase and our chosen threshold in Section 3.2 respectively. On the right of Figure 7 the hamming distances between error vectors of the top 50 Voters are plotted on 2D space using t-SNE (Van der Maaten & Hinton 2008). Clusters and centroids are marked by colour and crosses. The top-right table shows how accuracy changes when the majority vote draws only from that many error clusters.
1.0
82
0.95
Pearsons Correlation
TSNE Component 2
71.79
Accuracy
78
20
0.9
76
0.85
72
70
Threshold Used
100
200
300
400
500
600
-20
20
80
Number of Voters
TSNE Component 1
Figure 7: Left TruthfulQA MC1 Accuracy plotted against the number of Voters, with error correla- tion. Right The error vectors from the top 50 Voters are visualised and clustered with K-Means.
Ensemble Principles It may be intuitive to select amongst high-performing, upper-layer Voters. For example, a single Voter in Table 6 already surpasses the previous SOTA on TruthfulQA-MC1. However, these top performers make up only the 95th percentile, where accuracy quickly drops be- low that. We observe that accuracy increases with number of Voters, especially when error correla- tion is low and when Voters are sampled from different error clusters. This indicates the importance of error variability across Voters when combining them. Improvements plateau after 240 Voters, closely matching the threshold used in our experiments. We believe that this plateau is due to our naive ensemble approach, and that more sophisticated selection and combination strategies could yield better results and different points of diminishing returns. We propose a weighted combination strategy in Appendix C In Table 2 No finetuning involves learning weights to each Voter with the classification layer, which could be seen as learning selection and combination function. We observe that No follows fundamental ensemble principles when combining Voters; using multiple Voters with varying error traits can boost overall accuracy.

---

Published as a conference paper at ICLR 2025
5.3
ABLATIONS
51
Type-1 1 Voters
Type-1 Voters
Trubhfu CA Accuracy (%o
Type-2 2 Voters
Type-2 2 Veters
Region Region
15-
50-
Veriability Removal
High Varlability Removal
10 Voters Taken at Sequence Pasition
Rhol
75 Voters Removed
150
10
20
60 Sequence Perturbed
40
50
96
%
Figure 8: Sequence, Voter, and text ablation plots using Mistral-7B-Instreer
Plotting We perturb sequences and remove Voters with high error variability from NoVo, and plot their effects on TruthfulQA MC1. The left of Figure g compares T1 and '2 when taken further away from the sequence end. Here, different lengths are padded with previous norms while ex- cluding sequences with extreme lengths. The middle removes Voters from the majority vote. Here, variability is measured by the Hamming distance between error vectors. Low variability removal involves evenly removing Voters across enror clusters, while high variability removal exhausts one cluster at a time. There are six error clusters, with sizes from ranging 16 to 79. The right compares TI and T2 with sequences perturbed with random character and punctuation insertions. Table shows how removing the period at sequence end affects accuracy on datasets with different Voter mixes. The mix represents T1 and T2 separated by forward slash, with change in accuracy points.
Table 7: End sequence period ablation on various datasets.
TruthQA
CQA2
QASC
HSwag
SIQA
PIQA
Cosmos
CICv2
Change
-25.34
-0.04
-34.87
-16.21
0
0
-17.05 0
Mix
165/86
60/178
209/51
147/51
75/147
64/119
131/96
140/121
Ablation Outcomes we see that T1 accuracy drops more abruptly compared to T2 when moved away from sequence end. Both degrade significantly beyond certain point, with TI holding above T2. This is likely due to TI losing overall sequence structure quickly, while T2 maintains token associations that vary by position. When taken near sequence start, T2 loses all associations while TI Voters can still predict on sequences with concise assertions, such as 'no'. Similarly, T1 does not hold out well T2 when sequences are perturbed, likely due to insertions having lower chance of affecting specific token associations versus the overall structure, with both nearing random guessing extreme levels. Period removal generally affects datasets with more TI Voters, s which indicates it as an important source of overall structural information. Removing Voters evenly across error clusters preserves accuracy better than sequentially exhausting clusters, with it dropping sharply once cluster is exhausted. 4 This demonstrates the importance of having variety of Voters for final prediction. Taken together, these ablations reinforce our interpretations in Sections 5.1 and 5.2 regarding the structural, associative, and aggregative roles of Voters in NoVo.
CONCLUSION
In this paper. we introduced Norm Voting (No an effective method for enhancing the factual accuracy of LLMs, by measuring latent truth in certain attention head norms. NoVo significantly outperforms all existing methods on the challenging TruthfulQA MC1 benchmark and sets new SOTA accuracy. NoVo also demonstrates strong generalization across diverse set of topics and question formats, showcasing its potential beyond specific datasets. More importantly. NoVo does not require any specialised tools or in-domain sample training, making it scalable and lightweight. These attributes make No more suitable for practical use in real-world applications. Our findings not only advances REAR methods for mitigating hallucination. but also opens new avenues for future research in mechanistic interpretability, model reliability, and robustness.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Singapore, and the CyberSG R&D Programme Office ("CRPO"), under the National Cybersecurity R&D Programme ("NCRP"), RIE2025 NCRP Funding Initiative (Award CRPO- CAPO-GCI-NTT-002)
REFERENCES
Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Lin- guistics: EMNLP 2023, pp. 967-976, Singapore, December 2023. Association for Computational Linguistics. doi: 18653/v 1/2023. www.0/...niiggseemmte. findings-emmlp.68. URL
erg/20233 f findings -wwiinnngseeeppppp 68)
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Benjamin Samuel Bloom, Max D Engelhart, Edward J Furst, Walker H Hill, and David R Krathwohl. Taxonomy of educational objectives, volume 2. Longmans, Green New York, 1964.
Marco Bronzini, Carlo Nicolini, Bruno Lepri, Jacopo Staiano, and Andrea Passerini. Unveil- ing LLMs: The evolution of latent representations in dynamic knowledge graph. In First Conference on Language Modeling, 2024. htth:tt:::/ http///ppeeee...e.ttceee forem//fou??al
dWYR
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference on Learning Rep- resentations, 2023. URL
net. forumfrrmmm/m
Zhongzhi Chen, Xingwu Sun, Xianfeng Jiao, Fengzong Lian, Zhanhui Kang, Di Wang, and Chengzhong Xu. Truth forest: Toward multi-scale truthfulness large language models through intervention without tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 20967-20974, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
//lmsys.org/ og/ 00:0233-333 -iiiinavvvuaa//////
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024. URlhhttss:///pemt
net fee//fos Ett.://rrmmiiiie.d. www.iidd--hhhy1700
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint vrXivv1803.05444 2018
ETS. Graduate record examinations, 2024. URL htttps://ww..et..tt
html
Deepanway Ghosal, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. Two is better than many? binary classification
effective approach to multi-choice question answering. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 10158 -01586, Abu Dhabi. United Arab Emirates, December 2022a. Association for Computational Linguistics. doi: 10. 18653/v1l wwt.emmmlp-mann
URL http://:////aaaatooooogo org/ 2022. www. 691
Deepanway Ghosal, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. CICERO: A dataset for contextualized commonsense inference in dialogues. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5010-5028, Dublin, Ireland, May
11

---

Published as a conference paper at ICLR 2025
2022b. Association for Computational Linguistics. doi: 10.1865/w112022..allon.... URL
Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentanleed embedding sharing. In The Eleventh
International Conference Learning Representations, 2023. URIhttpss:/ pentttss://pperevviie
net net/Eorum?id-sE7- XXLLAA
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0m 03300, 2020.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference
Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pp. 2391-2401, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 1118653/11/D19-11 1243. URL ptttps://aclanthol
otg/D19-1243
Ajay Jaiswal, Bodun Hu, Lu Yin, Yeojju Ro, Shiwei Liu, Tianlong Chen, and Aditya Akella. Ffn- skiplim: hidden gem for autoregressive decoding with adaptive feed forward skipping. arXiv preprint arXiv:2404.03865. 2024.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Com- putational Linguistics. doi: 10. 10.18663//1/P17-11447 URL http:::/acaattholloo./
P17-1147
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169, 2023.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference Machine Learning, pp. 15696-15707. PMLR, 2023.
Haoqiang Kang and Xiao- Yang Liu. Deficiency of large language models in finance: An empirical examination of hallucination. In I Can't 't Believe It's Not Better Workshop: Failure Modes in the Age of Foundation Models, 2023.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with single QA system. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Lin- guistics: EMNLP 2020, pp. 1896-1907, Online, November 2020. Association for Computational Linguistics. doi: 10. //2.0//3/11222..1/// www.://0...nnnnngemm.......
URL
org/2020. fww.finginnniiie.i..ii 10gg:2222...imiinggemm.......... emelp.nnn1111
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qase: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8082-8090, 2020.
12

---

Published as a conference paper at ICLR 2025
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh. Pratik Ringshia, et al. Dynabench: Rethinking bench- marking in ulp. arXiv preprint arXiv:2104. 143.372744.4 2021.
Jongyoo Kim and Sanghoon Lee. Deep learning of human visual sensitivity in image quality as- sessment framework. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1676-1684, 2017.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming Bing-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Lc, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466 466, 2019. doi: 10.1162/tacl a 00276. URL Settp://accaatthe. www.tthhoooooogooooo.ooooo wttpo/ooo...o99111100000
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Mcasuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3214-32552 Dublin, Ireland, May 2022. Association for Computational Linguis- tics. doi: 10. 162166//1/22222222--oong2 ong. 229
URL hatts:://aclantholl.o wog..lyg...........
Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13480-13488, 2021.
Janet Mee, Ravi Pandian, Justin Wolczynski, Amy Morales, Miguel Paniagua, Polina Harik, Peter Baldwin, and Brian E Clauser. An experimental comparison of multiple-choice and answer questions on high-stakes test for medical students. Advances in Health Sciences Education, 29 (3):7838801, 2024.
Tomas Mikolov. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781,3776 2013.
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brood (roton, Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evaluation of language models. arXiv preprint arXhv:2307.06908 2023.
NCBE. Multistate bar examination, 2024. URL htt.ps//www..cbex..rrr///eeeeeee
OSHA. Hazard identification and assessment, 2016. URL fLtps://www.oo gov sa httea//waaanggeeeee///aaaaaaaaaa
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Med-halt: Medical domain hallucination test for large language models. arXiv preprint arXhv:2307.15343, 2023a.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Med-halt: Medical domain hallucination test for large language models. arXiv preprint arXiv:2307.15343 2023b.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 34:1 1054-11070, 2021.
Pranab Sahoo, Ayush Kumar Singh, Sripama Saha, Vinija Samrat Mondal, and Aman Chadha. A systematic survey of prompt engincering in large language models: Techniques and applica- tions. arXiv preprint a7X1922402.07927, 2024.
13

---

Published as a conference paper at ICLR 2025
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xi- aojun Wan (eds.), Proceedings of the 2019 Conference guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)), pp. 4463-4473, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10.18633/v1/D199 1454. URL
Empirical Methods in Natural Lan-
D19- D19-1454
Robert E Schapire. The strength of weak learnability. Machine learning, 5:197-227, 1990.
Siqi Shen, Deepanway Ghosal, Navonil Majumder, Henry Lim, Rada Mihalcea, and Soujanya Po- ria. Multiview contextual commonsense inference: new dataset and task. arXiv preprint arXiv:2210.000 02.0990, 2022.
Alon Talmor, Oni Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. Commonsenseqa 2.0: Exposing the limits of ai through gamifi- cation. In Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks, volume 1. 2021. URL https: //datasets wwwchwarkscheeeeeeeee....eeeeeee procedddgs.....oo.ssceoo.....see neurips cttpeiipp.cccppppppeeeeeeeeeapae cc/paper files/paper/ Paper roundl. .dd
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.098 092888, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
Boxin Wang. Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. In Advances in Neural Information Processing Systems, 2021.
Bo Yan, Bahetiyaer Bare, Chenxi Ma, Ke Li, and Weimin Tan. Deep objective quality assess- ment driven single image super-resolution. IEEE Transactions on Multimedia, 21(11):9977-2971 2019.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 93-104, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/w1/D18 10009 URL
org/ 80009
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.). Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 18653/v1/P19-1472 URL
Shaolei Zhang, Tian Yu, and Yang Feng. Truthx: Alleviating hallucinations by editing large lan- guage models in truthful space. arXiv preprint 1rXiv:2402.17811, 2024.
Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. Alleviating hallucinations of large language models through induced hallucinations. arXiv preprint arXiv:2312.15710, 2023a.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao Yu Zhang, Yulong Chen, al. Siren's song in the ai ocean: survey on hallucination in large language models. arXiv preprint arXiv:2309.012111 2023b.
Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. Retrieval-auu augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402. 19473, 2024.
14

---

Published as a conference paper at ICLR 2025
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. The Twelfth International Conference on Learning Representations, 2024.
Hanzhang Zhou, Zijian Feng, Zixiao Zhu, Junlang Qian, and Kezhi Mao. Unibias: Unveil- ing and mitigating Ilm bias through internal attention and f'n manipulation. arXiv preprint arXiv:2405.20612, 2024.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach ai transparency. arXiv preprint arXiv:2310.01400 2023.
15

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL DETAILS
Finetuning Supervised finetuning (SFT) feeds the final layer hidden state the task task-specific layer, such as a classifier for MCQ tasks. We use SFT as a baseline for our finetuning experiments in Table TEAM is variant of SFT that improves accuracy by restructuring all question and answer pairs to admit binary true or false answers. We adapt NoVo for finetuning, which we refer to as +NoVo, such that is similar to SFT but does not require the binary restructuring used in TEAM. In +o,Vo, all attention head norms are serialised as a vector and fed to the classifier. Here, the classifier does not receive the final hidden state, unlike SFT or TEAM. Different from the original zero-shot design of No the Norm Selection and Voting Inference stages described in Section 3.2 does not apply to +NoVo, Vo, and can instead be seen as a Iearnt function represented by the classifier weights. SFT, TEAM, and +No Vo trains all parameters in the model. We use the same finetuning parameters set by TEAM (Ghosal et al. 2022a), with the exception of the learning rate, which we change to 3e-6 for the model and 3e-5 for the classifier, across all three methods. We also implemented early stopping.
2
Table 8: Dataset and model details, grouped by colour, based on their occurrence in experiments.
Name Used
Full Name
Author
Source
TQA
TruthtulQA
Lin ct al 2022
GitHub
CQA2
CommonsenseQ4 2.0
Tammr el al. 202
GitHiub
QASC
ceasttonnAnswringg Sentence Composition
Khol
Fluggngg Face
SWAG
Situations With Adversarial Generations
Zellers cl 2018
HSwag
HellaSwag
Zellers ct 2019
GitHub
SIQA
Sucial IQA
Sap cl a 2019
AllenAi
PIQA
Physical IQA
Brik cl al. 20,00
AllenAi
Cosmos
CosmosQA
Huang cl
2019
GitHub
CICv1
CICERO vl
Ghosal
Gib
CICv2
CICERO v2
Shen el
2027
Gill lub
SST2
Stanford Sentiment Treebank v2
Wang
2021
Git jub
QQP
Duplicate Question Detection
Wang ct a..
2021
GitHub
MNLI
Multi-Genre Natural Language Inference
Wang ct al.
2022
GitHub
MNLI-MM
Multi-Genre Natural Language Inference Mismatched
Wang
GitHub
QNLI
Question Natural Language Inference
ct 2021
GitHub
RTE
Recognizing Textual Entailment
Wang ct
20211
GitHub
expert
FACTOR Export
Munigay ot a1. 20233
GitHub
nq
Natural Questions
Kwratkowski 21
2019
Hugging
trivia
Trivia QA
Jusin el | X
Hugging 'ace
mmlu
Massive Multitask Language Understanding
Hendryeks CI
2020
Hugging
AT2 Reasoning Challenge
C Clark ct al. 2018
Hugging
Llama2-7B
meta- meta-llaman 1amaa/laama-2776
Touvron al.
2023
Huggingt-as acc
Llama2-7B-Chan
meta-Ilama/L Satz-llamaLLllma-222 Lla.a---7--aaa--------
Louvton ct 20,25
HuggingFacc
Vicuna-7B
Imsys/vicuns, 7h-v1.5
Chang of 20:21
HuggingFace
Mistral-7B-Instreen
Jang al. 20241
HuggingFace
DeBERTa-Large
microsofrdebbrra-vv-laag
Hc 2025
HuggingFace
UnifiedQA-11B
Khashaai al.
2020
UNICORN-1IB
Loune el al. 2021
Reporting Results In Table re-implement results for DoLa, ICD, ITI by adapting from their official repositories. All other competing results are reported as presented in their original papers. MC1 accuracy is reported without cross training validation. In Table all results are implemented by us. All 7B decoder models here report zero-shot accuracy on the validation set, with 30 samples drawn from each dataset's respective training splits for Norm Selection. For DeBERTa finetuning. we train on the full training split and report accuracy on the test set. No cross training or validation performed here. In Table 33 all results are implemented by us. we perform 10-cross validation with 30 samples set aside randomly for Voter selection in each fold; the rest are used for evaluation. We report the average accuracy across all 10 folds. In Table we report all competing results as presented in their original papers or from other studies that re-implemented them. All methods here use Llama2-chat-7B we perform 10-cross validation with 30 samples set aside randomly for Norm Selection in each fold; the rest are used for evaluation. We report the average accuracy across all 10 folds. In all experiments, samples used for Norm Selection are drawn randomly once, without tuning hand hind-nicking. Visit our code repository to reproduce reported results and view fine-graina implementation details. All model and datasets used in this paper are fully detailed and referenced in Table g
16

---

Published as a conference paper at ICLR 2025
B
NORM SELECTION HYPER-PARAMETERS
tqa
csqa2
qasc
80.6
63.5
71.2
74.7
48.0
68.8
46.8
24.9
Validation Accuracy (%)
63.0
38.5
1.7
45.7
56.7
67.7
78.7
25.9
39.8
53.7
67.6
64.6
74.6
84.6
94.6
hellaswag
siqa
piqa
71.9
74.3
79.0
53.1
54.3
62.0
34.2
34.3
45.0
15.4
14.4
27.9
56.3
64.2
72.1
80.0
32.6
46.8
61.1
75.4
27.7
38.2
48.7
59.2
cosmosqa
cicero
cicero2
72.3
51.4
79.4
52.9
40.0
55.9
33.5
28.5
32.3
14.1
8.8
47.8
59.6
71.4
83.2
44.2
59.7
75.1
90.6
33.5
47.0
60.5
74.0
Average Difficulty
Figure 9: Analysing the effect of sample difficulty during Norm Selection, on downstream accuracy.
Grid Search The number of samples used and percentile threshold for Norm Selection are hyper- parameters. We search through different combinations of these two values for each dataset individu- ally, shown in Figure I0 To do so, we use 200 samples drawn randomly from the respective training splits of various reasoning and factual datasets, with varying portion held out for validation, de- pending on the number samples used for selection. We report the held-out accuracy for every combination and plotted them darker purple cell for higher values. We see that 30 samples gave the best held-out accuracy for all datasets, with some going as low as 10. Increasing the number of samples beyond 30 improves accuracy with greatly diminishing returns. The optimal percentile threshold hovers between 80 to 90, with the middle value as 85. No external tools, training, or specialised resources were used for this grid search. Samples used here are fully excluded when conducting zero-shot experiments.
arce
csqa2
qasc
hellaswag
79
Percentik entile Three freshold
81
83
87
89
siqa
piga
cosmosga
cicero2
79
81
83
85
87
89
5
10
15
20
25
30
35
5 10 15 20 25
10 15 Number of Samples
30
35
20
25
10
15
20
25 30 35
Figure 10: 10 to 30 samples at the 85th percentile threshold is optimal for Norm Selection. This range outlined with a dotted rectangle. Color intensity increases with held-out accuracy values.
Sample Type In Figure D difficulty per sample is the percentage of Voters that misclassified it. The horizontal axis marks the average difficulty across 30 samples used during Norm Selection,
17

---

Published as a conference paper at ICLR 2025
and the left vertical axis marks the validation accuracies for each respective dataset. In Table 9 apply different dataset samples in the pairwise manner for Norm Selection, on a given dataset. The The left column and top row
full training split is used, with the model set Distral-7b-Instruue indicates the Norm Selection and evaluated datasets respectively. ArcE refers to Arc-Easy. We see that using difficult samples with question-answering styles similar to those in the downstream dataset for Norm Selection can lead to higher accuracies. However, drawing different set of samples while maintaining high average difficulty, leads to large variations downstream validation accuracy. When the average difficulty becomes too high, such that individual Voter accuracy approaches the random baseline for that dataset, Norm Selection becomes increasingly ineffective. Accuracy drops when the sample style diverges from the downstream dataset. From these findings, we conclude that using difficult in-domain samples for Norm Selection gives the best results.
Table 9: Effect sample domain during Voter selection on validation accuracy.
Datasels
ArcE
CQA2
QASC
SWAG
HSwag
SIQA
PIQA
Cosmos
CICvl
CICv2
ArcE
84.70
59.94
55.94
59.66
49.64
70.98
75.14
53.70
37.96
70.74
CQA2
75.39
61.67
56.80
54.55
41.86
66.33
67.46
63.15
39.98
65.25
QASC
80.93
60.80
67.60
61.23
50.24
68.99
73.56
66.83
40.69
65.57
SWAG
80.93
59.86
59.61
74.19
64.63
69.55
76.17
57.96
39.41
66.32
HSwag
80.04
60.61
57.24
72.83
70.11
65.15
76.50
58.32
41.33
63.65
SIQA
83.81
59.74
60.15
58.89
43.63
70.21
71.98
57.69
39.83
72.24
PIQA
82.26
59.90
54.64
67.38
61.69
70.93
79.22
54.24
40.54
70.63
Cosmos
77.38
60.02
57.13
60.33
48.51
68.47
71.87
68.94
42.85
70.31
CICvI
81.15
60.13
58.86
63.84
54.26
67.55
74.59
62.78
51.48
72.02
CICv2
79.82
59.98
49.14
54.59
42.86
65.76
70.57
57.25
43.89
75.73
We note that Table Jable9also reflects out-of-distribution (OOD) performance. Computing the standard deviation for each column reveals that OOD performance is mostly dependent on the evaluated dataset, rather than NoVo. For instance, NoVo can be calibrated significantly OOD on multi-turn CICv1 dialogues, sentence completion SWAG, or factual-style Arc-Easy, and evaluated for CQA2 commonsense yes-no questions with only a 0.59 standard deviation. The smallest OOD standard deviations are CQA2, Arc-Easy and SIQA, with SWAG, HSwag, and Cosmos as notable outliers. We note that we intentionally maximised OOD differences between datasets by ensuring varied question topics and formats, to challenge No generalizability. We also note that NoVO's SOTA TruthfulQA performance was calibrated OOD on Are-Easy.
C
DISCOVERY
Table 10: No is competitive with both NoVo and LM for zero-shot MCQ answering.
Model
Method
TQA
CQA2
QASC
SWAG
HSwag
SIQA
PIQA
Cosmos
CICv1
CICv2
LM
34.27
55.65
19.76
60.51
56.30
45.45
72.63
36.42
37.74
42.34
Llama2-7B-CUl
NoVo
70.13
56.04
43.95
68.36
59.49
60.29
72.96
51.73
36.01
63.61
No
71.48
57.58
50.32
71,20
61.74
62.85
73.88
53.70
42,32
62.37
LM
28.48
49.98
25.16
74.59
71.59
49.08
76.99
38.53
38.34
37.85
Llama2-7B
NoVo
69.16
52.11
35.42
75.01
70.53
58.44
71.92
51.76
29.52
60.37
NoVo-F
70.75
54.66
52,38
73.73
68.94
61.26
74.92
51.66
38.07
61.87
LM
34.64
50.89
36,20
67.62
61.03
46.26
74.86
33.47
34.55
36.49
Vicuna-7B
NoVo
69.89
$1.40
62.66
69.67
69.20
61.15
74.37
56.45
39.23
69.42
NoVo-F
69.65
54.94
55.40
71.63
69.05
62,08
74.65
61.71
47.87
67.57
LM 53.86
61.90
31.53
63.31
75.28
46.93
76.39
31.69
40.25
38.52
Mistral-7B-Instree
No 78.09
62.02
66.09
69.65
63,35
70.68
76 66
67.57
46.09
73.52
No
79.44
61.51
69.76
73.78
71.77
71.08
79.16
68.17
52.28
75.94
We propose a parameeer free Norm Selection algorithm, without requiring the number of sam- ples or percentile threshold to be specified. Similar to Section 3.2) inference passes are performed over the entire training set, with individual accuracies assigned to each head. Heads that perform worse than the random baseline excluded. Instead of using percentile threshold, all heads
18

---

Published as a conference paper at ICLR 2025
Voters with weights assigned according to their accuracy scores, normalized between and 1. During the inference stage, final prediction made via the weighted sum of all Voter predictions. While more computationally expensive, this approach eliminates the random variation present in the original Norm Selection process, and removes the need to specify the percentile threshold and sample size hyper-parameters. Table ( compares this approach, denoted No with NoVo and LM. We see that NoVo-F is competitive with No in most datasets.
D RANDOM VARIATIONS OF EXPERIMENTAL RESULTS
Random variations attributable to the sampling process in Norm Selection are recorded in Table Random variations experiments are conducted over 200 runs, across all ten datasets (TruthfulQA, CQA2, QASC, SWAG, HSwag, SIQA. PIQA, Cosmos, CICvI, and CICv2) and across all four models (10 4 40 reports, each 200 runs). Standard deviations are all within 1.5 points, with the exception of Llama2-7b-Cosmos at 1.64, and Vicuna-7b-QASC at 1.53. Interquartile ranges all within 2.3 points. All experimental results reported in the paper fall within the IQR, with 70% of them within 0.5 points from the median. These random variation experiments show that there is no over-reporting of results.
Table 11: Random variations across 200 runs, for zero-shot experiments in Tables and
Model
Stats
TQA
CQA2
QASC
SWAG
HSwag
SIQA
PIQA
Cosmos
CICvI
CICv2
Mean
78.37
61.75
66.70
70.49
62.78
70.34
76.15
67.64
46.81
74.41
Std
0.68
0.37
0.87
1.30
1.5
0.67
0.62
0.89
1.37
1.06
77.23
61.24
65.44
67.57
60.37
75.35
66.26
44.46
72.49
Mistral
25Q
77.85
61.47
65.98
69.58
61.52
69.86
75.63
66.93
45.71
73.62
50Q
78.34
61,65
66.52
70.46
62.61
70.27
76.01
67.50
46.86
74.34
75Q
78.82
61.98
67.28
71.49
63.80
70.78
76.55
68.17
47.88
75.13
Max
80.78
63.28
69.65
73.95
67.75
72.72
78.13
71.36
50.30
77.66
Mean
70.21
56.26
44.88
68.14
61.06
61.01
73.09
53.19
63.27
64.75
Llama
Std
1.17
0.43
1.17
0.78
1.23
0.67
0.51
1.14
0.49
1.15
Min
68.30
55.73
43.30
67.07
59.28
60.18
72.58
51.66
35.64
63.33
25Q
69.28
55.96
43.95
67.52
59,93
60.49
72.80
$2.19
35.89
63.83
50Q
70.13
56.14
44.65
68.01
60.86
60.80
72.91
52.86
36.20
64.47
Chat
75Q
71.00
56.42
45.57
68.56
61.80
61.37
73.20
53.87
36.58
65.43
Max
74.66
57.62
49.03
70.75
65.93
63.36
75.41
58.96
38.53
69.21
Mcan
69.77
51.76
35.26
74.76
70.32
58.74
72.61
51.89
29.95
61.36
Std
1.33
0.60
1.35
0.42
0.59
0.69
0.61
1.64
0.53
0.97
Min
67.93
50.45
33.58
74.29
69.58
$7.88
71.92
49.92
29.31
60.04
Llama
25Q
68.67
51.55
34.01
74.43
69.83
58.24
72.13
50Q
69.52
51.79
35.04
74.63
70.19
58.55
72.52
51.73
29.83
61.17
75Q
70.66
51.94
36.28
74.99
70.81
59.11
72.91
$2.65
30.23
61.74
Max
73.19
54.66
38.98
76.43
71.84
61.31
75.19
56.28
31.66
65.00
Mean
70.01
51.82
44.40
69.91
70.21
61.36
73.33
$7.59
40.71
70.35
Std
0.72
0.58
1.53
0.58
0.71
0.65
0.44
1.23
1.13
1.17
Min
69.03
50.77
42.44
69.17
69.20
60.64
72.85
$5.98
39.04
68.89
Vicuna
25Q
69.40
51.63
43.17
69.44
69.67
60.90
73.01
56.60
39.83
69,45
50Q
69.89
51.79
44.17
69.66
70.12
61.15
73.17
57.34
40.53
70.05
75Q
70.50
51.83
45.16
70.29
70.61
61.58
73.52
58.17
41.45
71.03
Max
73.44
55.84
49.68
71.45
72.51
63.77
74.81
61.34
44.57
73.88
E ADDITIONAL CONTEXT ATTRIBUTION PLOTS
Additional Plots Figures II and 12 show additional context attribution plots from Type-1 and Voters respectively. Each plot visualises the attention- weighted value state components at various sequence positions, illustrated as heat maps with a line plot marking the relative norm gain of the correct answer at each position, graded the right vertical axis. One cell represents the average component value for a given context vector. Similar to Figure the horizontal bottom axis represent Voters taken various sequence positions, starting from the end the left and moving towards the
19

---

Published as a conference paper at ICLR 2025
76
190
1appet
1dc
Sperts
4
Masked
Massyao
Masked
untreb
Sistrie
Inkers
(37a
mon
Masked
Masked
Figure 11: Type-1 Voters on various question samples.
6811
hear
130
MasKed
Massue
4+-
treg
3my
belt
6007
Masbed
Figure 12: T 2 Voters on various question samples.
start on the right. The left vertical axis is the attention weighted sum of value states. Unlike Figure we omit some axis labels and show only the correct answer for clarity. The number at the top of each plot identifies the (1, h)) index of the Voter, enumerated as integers. All context attribution plots, including Figure 5 are taken from inference passes with Mistral--bbIIntructt
Voter Specialisation Here, we observe similar patterns to those in Figure 5 Type-1 Voters strongly focus on last token positions either throughout the sequences, or on punctuation marks and conjunctions, indicating structural scope of focus. Some Type-2 Voters focus on meaningful associations, such as disambiguation, looking for qualifiers and superlatives. Others are seemingly random or attend to identity connections. Regardless of patterns, most heads do not necessarily need to taken at the last sequence position be effective. For example, when asked if cops are allowed to lie in the US, relative norm gain increases midway through the sequence: "it depends the circumstances". and decreases when the assertion becomes ambiguous with the phrase: "of the". As long as the relevant claim can be localised in the sequence to answer the question, norm gains increases. However, this behaviour was the reason for taking heads at the last sequence position in Section 3.1 it did not require knowing where these claims lay for every sequence.
20

---

Published as a conference paper at ICLR 2025
F NORM CORRELATION DIRECTION
To better understand the impact of fixing the head norm correlation direction during Norm Selection, introduce two distinct variants: NoVo-A and No Vo-B. These methods differ primarily in their approach to the selection of norm values. Specifically, NoVo-A selects the highest norm values, while Vo-B chooses the lowest norm values. These two variants allows us investigate how prioritising one correlation direction influences performance across various datasets. In contrast to these static methods, NoVo adapts its selection strategy based on the correlation direction of each Voter, by using Indicators (as illustrated in Figure 33.. Table [22 provides a comparative analysis of these three approaches: NoVo-A, No-o-B, and NoVo, across a variety of reasoning and factuality benchmarks. The results demonstrate clear advantage for the dynamic selection mechanism. This can attributed to the flexibility of adjusting to the correlation direction of individual Voters, as opposed to the rigid strategies employed by NoVo-A and No Vo-B, which may miss nigh-performing Voters on the other direction, for the Voting Inference stage.
Table 12: Dynamic direction selection (NoVo) versus fixed max-min selection (Variants A and B).
Model
Method
TruthfulQA
CSQA2
QASC
HSwag
SIQA
PIQA
Cosmos
CICv2
NoVo-A
72.58
61.51
67.93
57.82
69.75
74.59
66.33
74.73
Mistral
No-o-
76.74
60.13
53.67
51.95
63.25
70.13
61.61
60.48
NoVo
78.09
62.02
66.09
63.35
70.68
76.66
67.57
73.52
Llama2
NoVo-A
64.63
56.40
39.63
51.78
55.83
67.41
48.14
58.62
NoVo-B
68.18
53.13
35.31
55.25
58.34
65.89
48.51
60.73
Chat
No
70.13
56.04
43.95
59.49
60.29
72.96
51.73
63.61
NoVo-A
62.06
52.22
33.59
60.70
53.38
70.51
46.40
58.27
Llama2
NoVo-B
62.79
51.99
21.06
65.38
55.89
64.42
49.92
57.52
NoVo
69.16
52.11
35.42
70.52
58.44
71.93
51.76
60.37
NoVo-A
65.73
51.83
38.77
60.19
58.34
67.68
46.37
65.47
Vicuna
No Vo-B
65.12
52.54
31.10
63.47
59.62
68.88
53.17
66.86
NoVo
69.89
51.40
42.66
69.21
61.16
74.37
56.45
69.42
G NoVo PERFORMANCE ANALYSIS WITH TRUTHFULQA
TruthfulQA consists of 817 samples, each belonging to one of 38 categories such science, ge- ography, bumanities, politics, law, finance, and pop culture. During inference, the model does not see the category label. Samples are crafted to mislead with common and imitative misconceptions. Around 53% all samples were adversarially filtered with GPT-3-175B to increase the difficulty of the dataset. TruthfulQA is designed for both generative question-ans questions. The latter task comes in multiple evaluation tracks, where each consists of one or more correct answers. We evaluated on the most difficult track, MCI, with only correct answer.
and multiple-choice
Figure 13 plots the accuracy of NoVo on a per-category basis, using Mistral-7B-IInttunl Each label has number prefix to indicate the total number of samples in that category. NoVo surpasses the LM in all categories. The categories with the smallest gains over the LM were: weather, sociology, proverbs, politics, and history. Similarly, the largest gains over the LM were: advertising, confusion- people, education, finance, indexical errors, and misquotations. The best performing categories are saturated full accuracy: advertising, logical falsehoods, mandela effect, topical misconceptions, misinformation, politics, science, subjective and superstitions. The worst performing categories are outliers: confusion-others and confusion -eeeaal..o-pppeelee.eeeeeoeeeeee scoring just under 30%.
For categories with small gains over the LM, we find that samples require both niche and nuanced knowledge, coupled with answer options that are extremely similar in factuality. For example, when asked if Nigeria has won Nobel prize, the answer options, amongst others, are either "no". or "no, Nigeria has not won a Nobel prize". " such cases, the LLM will need to differentiate between two similarly correct options. In contrast, categories with the largest gains over the LM did not face this issue. There are also categories that are high tighh-erforming with low-gain, and vice versa; politics and confusion-peopee are such examples, respectively. Since knowledge about politics widely
21

---

Published as a conference paper at ICLR 2025
1.0
Ours
LM
0.8
Accuracy
0.6
0.4
0.2
0.0
(13) Advertising
(9) Confusion: Other
(23) Confusion: People
(15) Confusion: Places
(25) Conspiracies
(14) Distraction
(31) Economics
(10) Education
(30) Fiction
(9) Finance
(55) Health
(24) History
(9) Indexical Error: Identity
(11) Indexical Error: Location
(21) Indexical Error: Other
(16) Indexical Error: Time
(21) Language
(64) Law
(13) Logical Falsehood
(6) Mandela Effect
(100) Misconceptions
(4) Misconceptionn Topical
(12) Misinforration
(16) Misquotations
(21) Myths and Fairytales
(16) Nutrition
(26) Paranormal
(10) Politics
(18) Proverbs
(19) Psychology
(15) Religion
(9) Science
(55) Sociology
(5) Statistics
(24) Stereotypes
(9) Subjective
(22) Superstitions
(17) Weather
Figure 13: MC1 accuracy plotted per category for NoVo (Ours) and LM.
available on the internet and are highly connected to other subjects, LLMs would fare better in these topics using either approach. Conversely, samples the confusion-people category require long- tailed knowledge of lesser known celebrities that may be difficult to recall (Kandpal et al. 2023 In this area, NoVo shows promising gains by correctly answering questions in the confusion-peat people topic, but is still ultimately inaccurate. The LLM here recognises minor celebrities, but is almost always misled by their names which are shared with more famous counterparts. We believe that No is ultimately bounded by the underlying model's capability.
We find no discernible differences in performance when evaluating between samples that did and did undergo adversarial filtering. A sample considered adversarially filtered when both hu- mans and LLM consistently gets it wrong. The authors of TruthfulQA curated additional unfiltered samples that were similar in style, but did not undergo additional model inference test for predic- tion trends. Our analysis reveals that No Vo outperforms the LM in both types of samples by huge margin, about 20% absolute points.
H NOVO ON OTHER HIDDEN STATES
Table 13: L2 norms from different hidden states are used for NoVo and evaluated.
TQA
CQA2
QASC
SWAG
HSwag
SIQA
PIQA
Cosmos
CICv1
CICv2
Query
71.60
59.62
50.54
51.20
36.58
61.62
54.62
51.16
36.11
65.57
Key
66.46
59.31
48.70
52.53
34.68
58.75
57.45
44.76
29.73
53.46
Value
64.38
63.36
57.34
55.24
42.11
61.67
65.56
52.23
31.91
63.83
Head
78.09
62.02
66.09
69.65
63.35
70.68
76.66
67.57
46.09
73.52
Out
60.47
62.30
47.52
49.37
32.00
57.63
55.71
42.38
25.99
48,90
FFN1
47.25
54.51
38.77
38.39
34.74
51.13
48.86
47.47
31.05
44.73
FFN2
48.10
53.29
50.65
36.83
32.45
46.88
54.52
43.05
25.80
41.41
Results in Table 13 validate the choice of using attention head norms, as opposed to other represen- tations. The L2 norms of the Query, Key, Value, Out, and FFN hidden states are fed to Vo, on -istral--7--Instruutt, and evaluated various datasets. Note that for this model, the Key vector is
22

---

Published as a conference paper at ICLR 2025
not multi-headed Results show that in general, the norms of better in predicting the correct answer. The FFN and Out norms performed the worse, with the Query and Value norms closing to gap towards head norms. We believe that measure coarse and broad as the 1.2 norm would work better on more fine-grained (multi-headed) representations of each token, rather than monolithic ones. Heads that do not encode truth indiseriminately fused with perforrrf truth heads in the Output projection, resulting in loss of fine-grained truth information. Furthermore, representations after the Output projection have significantly larger (32x) dimensionaliities leading to higher possibilities of confounding changes in unrelated dimensions. This makes differences in L2 norms much more difficult capture. The head can also be inter- preted as the final output of an information retrieval process, with Query and Value vectors acting as intermediate outputs. Hence, their states are not important isolation. We conclude that attention head norms are most indicative of truth among these hidden states, and is the best choice for Vo.
representations perform
I ADDITIONAL EXPERIMENTS ON DIFFERENT MODELS
Table 14: NoVo evaluated on four more models in varying alignment stages and sizes.
TQA
CQA2
QASC
SWAG
HSwag
SIQA
PIQA
Cosmos
CICv1
CICv2
LM
45.65
61.39
47,41
70.06
71.70
50.36
78.62
37.86
41.03
42.16
phi3-3.8b-it
NoVo
69.03
61.05
51.84
70.72
60.61
66.33
77.92
52.86
45.71
77.83
LM
52.51
63.60
40.06
65.92
72.96
45.34
77.04
25.13
38.19
36.71
2ephyr-7b-beta
NoVo
75.64
64.82
59.29
73.14
69.94
65.40
77.90
56.31
48.11
70.10
29.25
53.40
51.08
75.87
75.12
52.71
79.43
38.99
38.87
35.92
1lama3-8b
NoVo
70.03
52.96
36.08
76.45
76.49
54.55
72.25
40,88
62.19
LM
47.86
71.07
61.45
67.62
63.53
50.46
75.73
41.24
41.38
47.26
gemma2-9b-it
NoVo
79.68
71.46
75.49
74.73
72.65
73.64
80.74
74.64
52.88
72.02
Table 14 evaluates four more popular LLMs, doubling the number of evaluated LLMSs from four to eight. We include models of varying sizes from 3.8B to 9B, with a good mix of instruction-tuned chat-tuned, and based pretrained models. Results show major gains of 20 points averaged across TruthfulQA, QASC, SIQA, Cosmos, and CICv2. Moderate gains of 3.7 points averaged across CQA2, SWAG, and CIC1 are reported, with accuracy drops of 0.7 points averaged across HSwag and PIQA being reported. These result characteristics are largely similar to the original models in These results show that No Vo can generalize wide variety of decoder LLMs.
Tables 1 and 2
J HEAD NORMS COMPARED TO TOKEN PROBABILITY
Table 15: Comparing NoVo with PriDe Zheng et al. 2024), on three MCQ tasks. CSQA refers to Commonsense Reasoning, and utilize ARC-Challenge.
Dataset
MMLU
ARC
CSQA
Models
Orig.
PriDe
No
Orig.
PriDe
NoVo
Orig.
PriDe
No
Llama2-7B
35.8
45.3
43.2
36.0
53.7
53.8
31.9
52.9
51.0
Mimma2-BB-Char
45.8
48.7
49.1
56.5
59.9
64.1
56.5
63.4
64.5
Vicuna-7B
48.7
50.5
52.4
58.5
61.5
65.5
60.4
64.2
62.0
K LIMITATIONS
We dedicate this section to clearly explain the limitations of our work. While ranked generation possible, it is currently not yet possible apply head norms to single-span generation. Our formulation of head norms is relative, and therefore requires several candidate texts to work. No Vo is replacement for generic generative REAR methods such ITT and DoLA. No
23

---

Published as a conference paper at ICLR 2025
only outperforms the multiple-choict aspect. It is unclear if the success of DeBERTA head- norn-finetuning (+NoVo) applies to decoder or encoder-decoder architectures. Despite our novel interpretation of continuous truthful hyperspheres in attention heads, do not claim to narrow down any specific truthful subspace for further interpretation. ① As mentioned in Appendix B Norm Selection only works within reasonable out-of-distribution (OOD) limits, with the best being in-domain samples.
24


Reviews:
[{"review_id": "utNMs9zUh7", "reviewer": "Reviewer_3eKd", "summary": "The paper introduces Norm Voting (NoVo), a lightweight method designed to reduce hallucinations in LLMs using attention head norms. \n\n- Norm Voting automatically selects attention head norms that correlate with truth using a simple, inference-only algorithm that operates efficiently with just 30 random samples.\n- These selected norms are then used in a straightforward voting algorithm, enhancing the model's prediction accuracy by treating head norms as an ensemble of weak learners.\n\nNoVo's approach avoids reliance on specialized tools or in-domain training, making it scalable and generalizable. The method achieves state-of-the-art performance on the TruthfulQA MC1 benchmark, surpassing previous methods by at least 19 accuracy points. Additionally, NoVo demonstrates strong generalization across 20 diverse datasets, significantly outperforming existing representation editing and reading techniques, and showcasing its robustness in improving LLM factual accuracy.", "questions": "Authors claim \"Hallucinations in Large Language Models (LLMs) remain a major obstacle, particularly in high-stakes applications where factual accuracy is critical\". Are the benchmarks used in this study representative of the same?\n\nWe see models being adopted to simple applications that involve RAG, QnA etc. or more complex Agentic use cases that involve abilities like function calling, planning etc.\n\nIt is not very clear how the benchmarks help in high-stakes applications and it i very difficult assess the impact of the work.\n\nAnother question that arises is - how critical is MCQ based tasks? should we even solve it?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- NoVo is designed to be lightweight and does not require specialized tools, in-domain training, or external resources. This simplicity allows it to scale effortlessly across diverse datasets and applications, making it practical for real-world deployment.\n\n- The method achieves remarkable accuracy gains, notably setting a new state-of-the-art on the TruthfulQA MC1 benchmark with a 19-point improvement over existing approaches. This demonstrates the effectiveness of NoVo in addressing the critical issue of hallucinations in LLMs.\n\n- NoVo exhibits exceptional generalizability, achieving significant accuracy improvements on over 90% of the 20 diverse datasets evaluated. This indicates the method’s robustness and versatility in handling a wide range of tasks beyond just one specific dataset or domain.", "weaknesses": "- While the paper presents detailed analysis and shows impressive gains across benchmarks, the applicability of the solution beyond the MCQ type of problems is not obvious.\n- The motivation to solve MCQ type questions is not clear. Why is it very important to solve?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rVKweHfWal", "reviewer": "Reviewer_GDLP", "summary": "* The authors propose a simple, novel technique to improve LLM factual accuracy. They first find truth-correlated attention head norms, then ensemble these with majority voting at inference time to choose answers to MC questions.  \n* The authors conduct comprehensive experiments on a diverse range of datasets and use several different models. On many datasets they find massive effects from their intervention, improving on sota by 20+ percentage points in some cases.", "questions": "* In Table 2, very different results are obtained for CICv1 and CICv2. Is there any explanation for why the results differ so much here, or in general why the results vary so much across datasets?  \n* What is the variability of results when using different sets of randomly drawn samples for norm selection?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Empirical results are strong and consistent across models, with substantial improvements over previous methods\n* The method is simple and cheap, requiring no specialized training or tools.\n* Experimental evaluation is comprehensive, testing across 20 diverse datasets. The authors also evaluate generalizability through finetuning and adversarial robustness tests\n* Good error analysis that helps understand the method's capabilities and limitations, including detailed investigation of how different voter types contribute and where/why the method fails", "weaknesses": "* While the method is simple, I find the conceptual motivation unclear. The authors posit that for some heads, the L2 norm is correlated to the truthfulness of a sequence. Why? What makes this a reasonable thing to expect, conceptually?  \n* These experiments are done exclusively on multiple-choice QA datasets, and it seems that it would not be possible to use this method to reduce hallucination in open-ended generation. \n* Evaluating on a wider range of models would help provide more confidence in the method, especially testing on currently popular models like Llama 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "stqRvYRlDl", "reviewer": "Reviewer_a4Cj", "summary": "The authors propose a novel multiple-choice output strategy named NoVo. NoVo identifies correlations between the norms of attention heads at the choice token position and the the accuracy using a small calibration dataset. They leverage this information to make multiple-choice predictions without relying on log-probabilities. The proposed method outperforms various existing hallucination detection methods across different datasets and models. The authors further analyze the voting heads for deeper insights.", "questions": "see Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors did a good job of observing the behavior between attention heads and multiple-choice accuracy.\n- The figures are well-designed, and the experiments are comprehensive.\n- The hallucination detection task is highly relevant and important today.\n- NoVo shows impressive results in multiple-choice benchmarks.", "weaknesses": "- This approach has very limited applicability, being useful only for multiple-choice tasks. Therefore, it should be noted that this is not a generic hallucination reduction technique like existing methods such as Truthx.\n- It is unclear what makes attention heads special in this context. Similar experiments could potentially be conducted using MLP layers. \n\n- The authors only found a correlation between the norms of the heads and truthfulness. However, it is not explained why such a correlation exists or if there is any causal relationship between the two. \n- The direct relationship between the norm of a head and truthfulness is not well-explained. Why should a correlation be expected in the first place? And why use the L2 norm? \n- Out-of-distribution experiments are missing. The authors should conduct experiments where NoVo is calibrated on one dataset and tested on another. \n- I think the authors should choose between two options. They can either convert their observation about norms and truthfulness into a generic algorithm that works for open-ended generations, or they can deeply explore the reasons behind the observed correlation and write an interpretability paper. Another option might be to reformulate the problem as addressing selection bias or improving multi-choice scenarios, similar to [1].\n\n[1] LARGE LANGUAGE MODELS ARE NOT ROBUST MULTIPLE CHOICE SELECTORS, ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)