On-Policy Fine-grained Knowledge Feedback for Hallucination Mitigation

Authors: Xueru Wen, Xinyu Lu, Xinyan Guan, Jie Lou, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, Le Sun
Primary area: foundation or frontier models, including LLMs
Keywords: Hallucination;Large Language Model;

Abstract:
Hallucination occurs when large language models (LLMs) exhibit behavior that deviates from the boundaries of their knowledge during the response generation process.
Previous learning-based methods focus on detecting knowledge boundaries and finetuning models with instance-level feedback, but they suffer from inaccurate signals due to off-policy data sampling and coarse-grained feedback.
In this paper, we introduce \textit{\b{R}einforcement \b{L}earning \b{f}or \b{H}allucination} (RLFH), a fine-grained feedback-based online reinforcement learning method for hallucination mitigation.
Unlike previous learning-based methods, RLFH enables LLMs to explore the boundaries of their internal knowledge and provide on-policy, fine-grained feedback on these explorations.
To construct fine-grained feedback for learning reliable generation behavior, RLFH decomposes the outcomes of large models into atomic facts, provides statement-level evaluation signals, and traces back the signals to the tokens of the original responses.
Finally, RLFH adopts the online reinforcement algorithm with these token-level rewards to adjust model behavior for hallucination mitigation.
For effective on-policy optimization, RLFH also introduces an LLM-based fact assessment framework to verify the truthfulness and helpfulness of atomic facts without human intervention.
Experiments on HotpotQA, SQuADv2, and Biography benchmarks demonstrate that RLFH can balance their usage of internal knowledge during the generation process to eliminate the hallucination behavior of LLMs.

Decision: reject
Reviewer scores: 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730774886195

thread_type:
official_review

<thread_object>
creator: reviewer_hgoa

<content>
summary:
This paper presents RLFH, a method that decomposes the output of large language models into atomic facts and validates them with reference documents. Then the sentence-level evaluation feedback is provided. After that, RLFH traces the feedback back to the original response and builds a token-level reward signal. Finally, an online reinforcement algorithm with these signals adjusts the model behavior to alleviate the illusion. The main contribution is to propose an online reinforcement algorithm based on fine-grained feedback signals and a non-manual, LLM-driven reward model.

strengths:
1.The motivation of the paper is reasonable, and the problems to be solved are relatively important.

2.The model adjusts the model itself based on the evaluation feedback of the results, rather than just modifying the results. The method is interesting.

3.The analysis of the experimental results is detailed. Both the overall summary analysis and some detailed research are given.

weaknesses:
1.The paper fails to fully solve the problem it raises. Evasive ignorance, for example, is a type of illusion that can't be detected or changed.

2.The model simply gives feedback on the correctness of each sentence based on the original answer. If the original answer itself has information missing, the model will not be able to correct and give feedback.

3. Please discuss the potential challenges or benefits of extending their method to other LLMs.

questions:
For page 5 line 267: “Table ??” , please specify which table you are referring to.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730932150746

thread_type:
official_review

<thread_object>
creator: reviewer_ryqp

<content>
summary:
The paper introduces Reinforcement Learning for Hallucination (RLFH), an innovative approach aimed at reducing hallucination in large language models (LLMs). It focuses on instance-level feedback and relies on off-policy data sampling, RLFH provides fine-grained, on-policy feedback at the token level. This method decomposes responses into atomic facts and applies statement-level evaluations to assign feedback to specific tokens in the output, guiding the model to reduce hallucinations. To facilitate on-policy optimization, RLFH includes an automated fact assessment framework to evaluate the truthfulness and helpfulness of these facts. Experimental results on HotpotQA, SQuADv2, and Biography datasets show that RLFH effectively mitigates hallucination in LLMs by balancing their use of internal knowledge during response generation.

strengths:
1.RLFH provides a novel approach to feedback by breaking down outputs into atomic facts and assigning token-level rewards, which allows for a more precise correction of hallucination errors.

2.By using on-policy data sampling, RLFH ensures that the feedback provided is more aligned with the model’s real-time behavior, improving the relevance and effectiveness of the adjustments made to the model.

3.the experimental results have shown the effectiveness of the proposed approach.

weaknesses:
1.the proposed method is not a fully reliable on-policy method, as the feedback is provide by the LLM itself, not trustworthy feedbacks. Thus, it is hard to say after multi-turn iteration, if the model will be better, and maybe the wrong knowledge will accumulate.

2.the compared baselines should involve RLHF-based methods, i.e, DPO and RLAIF. Although these methods are not specially designed for on-policy optimization, it is necessary to show how they will perform under this occasion, to verify the motivation in the introduction part.

3.the writing of this paper needs to be polished. There are several typos: e.g., "Turning" -> "Turing"

questions:
Please refer to the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730938354521

thread_type:
official_review

<thread_object>
creator: reviewer_xpnr

<content>
summary:
The paper, proposes a novel approach called RLFH (Reinforcement Learning for Hallucination) to reduce hallucinations in Large Language Models (LLMs) by fine-grained, token-level  reward. RLFH leverages an online reinforcement learning framework and was tested across multiple datasets and showed improved fact-based accuracy compared to existing hallucination mitigation strategies.

strengths:
1. **Fine-Grained Feedback:** RLFH's use of atomic fact decomposition and token-level rewards enables precise correction of hallucinated responses, surpassing the efficacy of previous, more coarse-grained approaches.

2. **Automated Fact Assessment:** The model employs an LLM-driven feedback mechanism to evaluate truthfulness and informativeness, potentially eliminating human intervention and thus increasing scalability.

weaknesses:
1. **Unsuitable Baselines:** The paper selects "advanced aligned models" such as Zephyr, Orca, Llama2 Chat, and Vicuna 1.5 as baselines (line 311). However, this comparison is unsuitable, as these models differ not only in alignment methods but also in their underlying architectures and training parameters. For a fair comparison, the authors should have evaluated different alignment and hallucination mitigation methods on a consistent model architecture rather than comparing across varied pre-trained models. 

2. **Unsignificant effect:** Despite the methodological novelty, the empirical results indicate only marginal improvements in factuality compared to some baseline methods. The reported gains, although statistically significant, may not justify the increased computational complexity and reinforcement learning setup of RLFH, especially given the modest FactScore improvements (e.g., +2.0% on some benchmarks). This raises questions about the practical advantages of RLFH over simpler, less resource-intensive approaches.

3. **Dependence on External Knowledge:** The model’s reliance on external datasets and existing knowledge boundaries raises concerns about its adaptability in real-world scenarios where such data may not be readily available or up-to-date.

questions:
Could the authors replace the current baseline models with a consistent model architecture and compare only different alignment and hallucination mitigation methods?
 This would provide a clearer evaluation of RLFH’s effectiveness by removing confounding factors related to model architecture and pre-training differences.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732566766495

thread_type:
official_comment

<thread_object>
creator: area_chair_ivuq

<content>
comment:
Dear Reviewers, 

The rebuttal discussion period is coming to a close and the paper currently has a mix of positive and negative reviewers. The authors have spent a lot of time responding to each concern -- can you take a look at the author responses and let them know any remaining concerns you have?
 
Best,  
AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes Reinforcement Learning for Hallucination (RLFH), a method to reduce hallucination in large language models (LLMs) by using token-level rewards and on-policy optimization. It breaks down responses into atomic facts, assigns token-level feedback, and employs an automated fact assessment framework for real-time learning. Experiments on benchmarks such as HotpotQA, SQuADv2, and Biography show modest but statistically good improvements in factuality. RLFH's token-level feedback provides more precision than existing methods, and its on-policy optimization aligns rewards with model behavior without requiring human intervention. However, reviewers pointed out issues with inconsistent baselines, small factuality improvements that might not be worth the added computational cost, reliance on external knowledge that affects scalability. While the authors provided a thoughtful rebuttal, the reviewers still have concern about the paper, leading to a rejection decision. Reviewers recommend refining the paper to address these issues for future submission.