KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality
Authors: name: Baochang Ren, name: Shuofei Qiao, name: Wenhao Yu, name: Huajun Chen, name: Ningyu Zhang

Large Language Models (LLMs), particularly slow-thinking models, often
exhibit severe hallucination, outputting incorrect content due to an inability
to accurately recognize knowledge boundaries during reasoning. While
Reinforcement Learning (RL) can enhance complex reasoning abilities, its
outcome-oriented reward mechanism often lacks factual supervision over the
thinking process, further exacerbating the hallucination problem. To address
the high hallucination in slow-thinking models, we propose Knowledge-enhanced
RL, KnowRL. KnowRL guides models to perform fact-based slow thinking by
integrating a factuality reward, based on knowledge verification, into the RL
training process, helping them recognize their knowledge boundaries. KnowRL
guides models to perform fact-based slow thinking by integrating a factuality
reward, based on knowledge verification, into the RL training process, helping
them recognize their knowledge boundaries. This targeted factual input during
RL training enables the model to learn and internalize fact-based reasoning
strategies. By directly rewarding adherence to facts within the reasoning
steps, KnowRL fosters a more reliable thinking process. Experimental results on
three hallucination evaluation datasets and two reasoning evaluation datasets
demonstrate that KnowRL effectively mitigates hallucinations in slow-thinking
models while maintaining their original strong reasoning capabilities. Our code
is available at https://github.com/zjunlp/KnowRL.