Hallucination Detox: Sensitive Neuron Dropout (SeND) for Large Language Model Training

Authors: Shahrad Mohammadzadeh, Juan David Guerra, Marco Bonizzato, Reihaneh Rabbany, Golnoosh Farnadi
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLMs, Hallucinations, Dropout, Reliability, Efficiency

Abstract:
As large language models (LLMs) are increasingly deployed across various industries, concerns regarding their reliability, particularly due to hallucinations—outputs that are factually inaccurate or irrelevant to user input—have grown. Our research investigates the relationship between the training process and the emergence of hallucinations  to address a key gap in existing research that focuses primarily on post hoc detection and mitigation strategies. Using models from the Pythia suite (70M–12B parameters) and several hallucination detection metrics, we analyze hallucination trends throughout training and explore LLM internal dynamics. We introduce Sensitivity Dropout SenD, a novel training protocol designed to mitigate hallucinations by reducing variance during training. SenD achieves this by deterministically dropping embedding indices with significant variability, referred to as Sensitive Embedding Indices. In addition, we develop an unsupervised hallucination detection metric, Efficient EigenScore (EES), which approximates the traditional EigenScore in 2x speed. This efficient metric is integrated into our protocol, allowing SenD to be both computationally scalable and effective at reducing hallucinations. Our empirical evaluation demonstrates that our approach improves LLM reliability at test time by up to 40\% compared to normal training while also providing an efficient method to improve factual accuracy when adapting LLMs to Wikipedia, Medical, and LegalBench domains.

Decision: reject
Reviewer scores: 5, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1729581566440

thread_type:
official_review

<thread_object>
creator: reviewer_97yw

<content>
summary:
This paper empirically validates the oscillatory nature of hallucinations during the training process of LLMs, despite being discovered by previous work. Subsequently, this paper introduces Sensitive Neuron Dropout (SeND), a training-time method for hallucination reduction, and  Efficient EigenScore (EES), a more efficient hallucination detection metric.

strengths:
1. Different from existing post-hoc detection and mitigation strategies, this paper focuses on the relationship between the training process and the emergence of hallucinations, trying to provide interpretation from a relatively new perspective.
2.  Before introducing the specific method, this paper conducts motivational experiments, consolidating the rationale of the method.

weaknesses:
1. Despite focusing on hallucination, this paper does not test on any hallucination dataset and metrics directly, but utilizes some indicative alternatives. This reduces the credibility of the results.
2. Despite the claims of reduced hallucination, it's unclear whether this technique would hinder the performance of models.
3. For LLMs, it's rare to train models for several epochs to prevent overfiting and catastrophic forgetting, while it seems that this method can only be used for multi-epoch training settings.
4. Xsum is not suitable to evaluate hallucination of LLM, and Rouge1 score is a bit out-of-date/ineffective to evaluate the performance of LLMs.
5. The writing for "Sec. 1.2 Related Work" is quite strange. Here, the 2nd and 4th paragraphs focus on motivation and implementation details instead of the comparison with peer methods.
6. In Sec. 2.2, I observed that generally, the metrics change positively with the increase of LLM sizes, inconsistent with the observations of authors. Could you please provide further explanation?
7. The number of models and datasets is too small (i.e., only 1) to validate the robustness of the method.
8. There is a lack of baseline and performance comparison with post-hoc solutions.

questions:
1. For line 227, the meaning of H needs to be further explained.
2. For line 229, a citation is needed to support the claim.
3. Based on my understanding, Sensitive Neurons refers to specific indices, rather than neurons. This name could be misleading.
4. For line 284, the details of removing operation are unclear.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730265383215

thread_type:
official_review

<thread_object>
creator: reviewer_82a9

<content>
summary:
This paper proposes a novel training protocol called Sensitive Neuron Dropout (SeND) to address hallucinations in Large Language Models (LLMs). The work presents three main contributions: (1) empirical validation of oscillatory hallucination behavior during training, (2) development of SeND for reducing hallucination variance during training, and (3) introduction of Efficient EigenScore (EES), a computationally efficient approximation of EigenScore for hallucination detection. While the theoretical framework is interesting, the empirical validation relies heavily on proxy metrics and limited evaluation data, making it difficult to assess the real-world impact on hallucination reduction.

strengths:
The paper proposes a novel approach tackling hallucinations during training rather than post-hoc, representing a interesting shift in addressing this critical challenge.
The foundation is solid, with clear mathematical derivations for both SeND and EES.
The development of EES shows practical value by providing a computationally efficient approximation for hallucination detection with demonstrated speedup.

weaknesses:
The empirical evaluation is severely limited with only 100 test datapoints and lacks validation on more than one established hallucination benchmarks, like HaluEval, raising concerns about result reliability.
The work relies heavily on EES as a proxy metric without sufficient evidence that improvements in EES correlate with actual reduction in model hallucinations.

questions:
1. Could the authors provide results on a significantly larger test set beyond 100 datapoints?
2. What is the correlation between EES improvements and actual hallucination reduction as measured by standard benchmarks?
3. How does SeND compare to other hallucination reduction methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686440082

thread_type:
official_review

<thread_object>
creator: reviewer_u3rz

<content>
summary:
In this paper, the authors are attempting to solve the hallucination problem called confabulations where the LLM generates different responses given the same or similar inputs. Specifically, the authors propose two main contributions including the triaining protocal named Sensitive Neuron Dropout (SeND) and the enhanced unsupervised hallucination detection metric namd Efficient EigenScore (EES).
In SeND, a novel training protocol, aimed at alleviating the phenomenon of hallucinations in large language models (LLMs) by reducing the variance during the training process. This method reduces the variance of illusions and enhances the factual certainty of the model by deterministically discarding neurons with significant variability on the dataset (known as sensitive neurons) as a regularization technique. The developed an unsupervised hallucinations detection metric EES that is twice as fast as traditional EigenScore while minimizing its impact on accuracy. This efficient metric is integrated into the SeND protocol, making SeND computationally scalable and effective in reducing illusions. In the experiments, the study demonstrated that its method improved the reliability of LLMs during testing, increasing reliability by up to 40% compared to normal training, and providing an effective approach to improve real-world accuracy when adapted to fields such as Wikipedia and medical datasets.

strengths:
● Innovation: This study proposes a new training protocol, SeND, which may have a significant impact on the reliability and security of LLMs, making it an important research area.
● Practical application: Empirical evaluations on Wikipedia and medical datasets have demonstrated the potential of SeND in improving factual accuracy, which is particularly important for applications in high-risk industries.
● Computational efficiency: The development of EES has significantly improved the computational efficiency of hallucination detection, which is particularly important for large models and datasets.
● Paper writing: This paper has a smooth writing structure and clear logical expression, allowing readers to quickly understand the relationship between this paper and previous related works.

weaknesses:
● Lack of discussion on other training stages. The authors assume that the existing research that focuses primarily on post hoc detection and mitigation strategies. However, the training stages in current works mainly contain three import paradigms including pre-training, continue pretraining and SFT. All of them may produce the hallucination phenomenon, and thus the discussion about other two training stages should be considered.
● To evaluate the OSCILLATORY BEHAVIOUR, the authors use the two tasks including self-consistency and summarization, the other important metrics (e.g., PPL) or tasks (e.g. QA) should be considered.
● Mismach parameters size between SENSITIVE NEURONS discussion and main experiments. In experiments settings, the range of paramerts' size of LLMs is from 70M to 12B. However, the theoretical analysis and experimental results of SENSIIVE NEURON are conducted using the Pythia 1B model in the main body (Sec. 3), and there is concern about the lack of generalization of the SeND to larger scales model.
● The effectiveness of SeND experiment is weak. Firstly, the authors only select two datasets (general domain and medical domain), and the effectiveness of this method needs to be proven on more authoritative hallucination benchmarks. In addition, as shown in Fig. 4, the FT method is not too weak compared to the SeND, and thus more datasets are needed to prove the effectiveness of the method.

questions:
See the Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730734788990

thread_type:
official_review

<thread_object>
creator: reviewer_tcyx

<content>
summary:
The paper introduces Sensitive Neuron Dropout (SeND), a novel training protocol aimed at reducing hallucinations in large language models (LLMs) by minimizing variance during training. Unlike existing post-hoc hallucination mitigation methods, SeND operates during training, specifically targeting neurons—referred to as Sensitive Neurons—that exhibit high variability across training epochs. By selectively dropping these neurons, SeND helps stabilize model outputs, thereby enhancing factual confidence and reducing the likelihood of confabulations, which are hallucinations where models inconsistently produce factually correct and incorrect responses.

strengths:
1. Innovative Approach: Introduces SeND, a new training method, and EES, an efficient hallucination detection metric.
2. Robust Evaluation: Demonstrates SeND’s effectiveness across multiple models and datasets.
3. Computational Efficiency: EES is scalable, supporting application in large LLMs without adding significant computational costs.
4. Clear Methodology: The paper clearly explains the theoretical background and provides step-by-step details for SeND implementation.

weaknesses:
1. While the paper introduces Efficient EigenScore (EES) as an approximation of the EigenScore metric for hallucination detection, it largely focuses on a single metric. Expanding the scope of metrics could provide a more comprehensive understanding of SeND’s performance. For instance, incorporating metrics like Semantic Entropy or FactScore alongside EES would allow a nuanced evaluation of hallucinations across different aspects of factuality and consistency. 
2. The paper’s experimental setup lacks an ablation study on SeND’s dropout parameters, such as the percentage of neurons dropped and the interval for identifying sensitive neurons.
3. Although the paper tests SeND on the Pythia model series, this restricts its applicability to similar architectures. Testing SeND on diverse LLM architectures, such as LLaMA, would better establish its generalizability across model types with varying parameters and configurations.

questions:
1. What guided the specific selection of neuron dropout parameters (e.g., dropout percentage, sensitivity threshold)? Could the authors provide insights into how the dropout parameters for SeND were chosen? Was there an empirical process for selecting these values, and did the team explore different configurations to determine the optimal settings? 
2. What impact do Sensitive Neurons have on downstream tasks, especially when a high percentage is dropped?
3. Can the authors share any qualitative examples of how SeND changes model outputs? Including specific examples of model outputs before and after training with SeND, particularly for hallucination-prone prompts, would help illustrate the model’s qualitative improvements.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732252182251

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We would like to thank all reviewers for their time and for providing constructive feedback. We are happy that the reviewers found our work novel and coherent, and have implemented the suggestions to the best of our abilities. All changes in the newly uploaded paper revision are highlighted in **blue**. Here we summarize the main changes.

## Sensitive Neurons Term

For the suggestion of a more clear naming for what we refer to as Sensitive Neurons, the terminology has changed from **Neuron** to **Embedding index** as it clarifies the entities of our study. Hence, we modified the title of the paper to reflect this change and it is now “Hallucination Detox: Sensitivity Dropout (SenD) for Large Language Model Training”.

## State-of-the-art Methods and Baselines

We highly appreciate the feedback on the need for the comparison of our method with SOTA methods and baselines. We break the related changes down into three areas:

### 1. Models and Datasets Scope

To better evaluate the applicability of SenD to other SOTA LLMs, multiple LLM sizes, and more datasets, we extended our experiments to include Llama 3.2 1B and Llama 3.1 8B models as well as training the models on an extra corpus for reasoning task: LegalBench. These additions are discussed in **Section 4.1** and the results could be found in **Section 4.2, Figure 4, and Appendix D**. In summary, our general observation stays the same: SenD compared to normal training, results in less hallucinations during the training and even makes the end model factually more confident on all datasets which explains the transferability of the method to different domains and model sizes. An example from **Figure 4** for Llama 3.1 8B trained on the new dataset, **LegalBench** can be found in: https://ibb.co/hYWktQy .

### 2. End Model Evaluation with Other SOTA Hallucination Detection Metrics and Tasks

First, we would like to clarify SenD’s purpose and positioning. SenD is designed to reduce hallucination variance ***during the training*** itself, rather than simply reducing the ***final model’s***  hallucinations. By minimizing oscillations in hallucinations throughout training, SenD facilitates a more stable factual convergence, enabling a reliable stopping criterion when the Efficient EigenScore (EES), along with the training loss, converges. Without SenD, the high variance in hallucinations makes it challenging to identify an optimal stopping point, leading to potential over- or under-training when training an LLM from scratch. Additionally, SenD is not intended as a post-hoc method to reduce hallucinations in an already trained model. Since, nevertheless, it is important to verify the reliability of the end model trained by SenD, we employ **FactScore on 100 and 1000 testing data points, and HaluEval Summarization** on Pythia 1B. The results are appended to **Section 4.2** and can be found in **Table 1** and can be seen below:

| Task | Metric | SenD | Normal |
| --- | --- | --- | --- |
| **HaluEval Summarization (LMEval)** | Accuracy | ***0.016*** | 0.014 |
|  | Correctness | 0.027 | 0.027 |
|  | Exact Match | ***0.589*** | 0.496 |
| **FactScore (100 points)** | Score | ***0.07*** | 0.05 |
| **FactScore (1000 points)** | Score | ***0.08*** | 0.06 |

In addition to showing the better performance of the model trained with SenD compared to normal training, the table proves the **accuracy of EES** for evaluating the hallucinations recalling that in most cases, the EES of the end model was less in SenD compared to normal training.

### 3. Comparison of SenD with SOTA Post-hoc Methods

Reemphasizing that SenD is intended as a training-stage approach rather than a post-hoc method for hallucination reduction in a fully trained model, traditional post-hoc methods may not align directly with SenD’s objectives. However, to respond to the reviewers’ comments, we ran additional experiments by treating SenD as a post-hoc method and comparing its performance to a model augmented with RAG on the Pythia 1B model on the same data domain as test set as in **Section 4.2**. Our new results indicate that when tested on 1000 datapoints, the SenD finetuned model achieves FactScore 0.07, while the normally trained base Pythia model scores 0.25 with RAG. This shows RAG’s better performance within in-context settings. However, SenD, not being a mere post-hoc method makes this an **unfair** comparison. To have a fair comparison, **the model augmented with SenD and RAG outperforms the regularly augmented model with RAG by 12%, achieving a FactScore of 0.28 when tested on 1000 datapoints compared to the baseline score of 0.25**. This suggests that SenD, not only reduces during-training variance, but also allows other post-hoc methods such as RAG to more effectively mitigate hallucinations compared to traditionally trained models.

Once again, we appreciate the reviewers time and hope that the new revision reflects the desired changes. We are happy to discuss any further questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732561126116

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Discussion Reminder: Request for feedback and re-evaluation based on recent changes

comment:
As the rebuttal phase concludes, we wanted to gently remind the reviewers of our updated submission. Since the initial reviews, we have conducted extensive experiments that we believe significantly enhance our work and address key points raised in the reviews.
We encourage you to review these updates, as we hope they may influence your evaluations and scores. We appreciate your feedback.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733160500896

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Call for Comments: Request for feedback and re-evaluation as deadline approaches

comment:
As the extended deadline for rebuttals approaches, we deeply appreciate the time and effort reviewers have dedicated to evaluating our submission. We kindly invite all reviewers to revisit our updates, additional experiments (inclusion of Llama 3.2 1B, Llama 3.1 8B), and detailed justifications (difference between SenD and post-hoc method performance as well as a hypothesis surrounding optimal dataset performance in Section 4.2). We have worked diligently to address your valuable feedback, and we have provided robust comparisons to SOTA methods such as RAG (see Section 4.2 Performance of SenD on Pythia and Llama Models) and demonstrating the impact of our work across diverse metrics (HaluEval, FactScore, and EES) and an additional domain (LegalBench - a new domain of different legal reasoning tasks). We hope these updates help clarify any remaining questions and potentially inform your final evaluations. We believe our revised version addresses all the concerns and requests raised by the reviewers and we hope these efforts will increase their evaluation scores. Thank you again for your thoughtful and constructive insights.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a novel training protocol called Sensitive Neuron Dropout (SeND) to mitigate hallucinations in LLMs. SeND works by deterministically dropping the "sensitive" embedding indices - those with high variability across training epochs. The paper also introduces an efficient approximation of the EigenScore metric, called Efficient EigenScore (EES), to enable computationally scalable hallucination detection during training.

Overall, the paper proposes an interesting and novel approach to address the critical issue of hallucinations in large language models. The theoretical underpinnings and the initial empirical validation are promising. However, the limited scope of the evaluation and the lack of more comprehensive comparisons to related work prevent a conclusive assessment of the full impact and applicability of the proposed methods. Therefore, I recommend rejecting it considering the following factors:
- The need for more extensive evaluation on established hallucination benchmarks and a broader set of LLM architectures and datasets to demonstrate the generalizability of the findings.
- The lack of detailed comparisons to other state-of-the-art hallucination mitigation techniques, which makes it difficult to assess the relative merits of the proposed approach.
- The absence of rigorous ablation studies and analyses on the impact of SeND on downstream task performance, which would provide a more holistic understanding of the method's benefits and limitations.

With additional work to address these gaps, the paper could become a strong contribution to the field.