Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback
Authors: Hongshen Xu
Zichen Zhu
Situo Zhang
Da Ma
Shuai Fan
Lu Chen
Kai Yu
Venue: COLM (2024)
Keywords: hallucination, alignment, RLKF
TL;DR: We propose the aignment objectives and evaluation metrics of LLM reliability for reducing hallucinations, and we further propose a novel alignment framework called RLKF to train LLMs to refuse unknown questions using RL from Knowledge Feedback.

Abstract: Large Language Models (LLMs) often generate erroneous outputs, known as hallucinations, due to their limitations in discerning questions beyond their knowledge scope. While addressing hallucination has been a focal point in research, previous efforts primarily concentrate on enhancing correctness without giving due consideration to the significance of rejection mechanisms. In this paper, we conduct a comprehensive examination of the role of rejection, introducing the alignment goal of model reliability along with corresponding metrics. This goal requires the model to provide accurate responses while adeptly rejecting questions exceeding its knowledge boundaries, thereby minimizing hallucinations. To improve the inherent reliability of LLMs, we present a novel alignment framework called Reinforcement Learning from Knowledge Feedback (RLKF). RLKF leverages knowledge feedback to dynamically determine the model's knowledge boundary and trains a reliable reward model to encourage the rejection of out-of-knowledge questions. Experimental results on mathematical and question answering datasets affirm the substantial efficacy of RLKF in significantly enhancing LLM reliability.

Published as a conference paper at COLM 2024
Rejection Improves Reliability: Training LLMs to Refuse Un- known Questions Using RL from Knowledge Feedback
Hongshen Xu', Zichen Zhu', Situo Zhang', Da Ma', Shuai Fan Lu Chen Kai Yu XX-LANCE Lab, Department of Computer Science and Engineering MoF. Key Lab of Artificial Intelligence, AT Institute Shanghai Jiao Tong University, Shanghai, China 2 AASppeech Co., Ltd., Suzhou, China
Suzhou Laboratory, Suzhou, China (xuhongshen, JamesZhutheThird, chenlusz, kai. yu) edu. cn
Abstract
Large Language Models (LLMs) often generate erroneous outputs, known as hallucinations, due to their limitations in discerning questions beyond their knowledge scope. While addressing hallucination has been a focal point in research, previous efforts primarily concentrate on enhancing cor- rectness without giving due consideration to the significance of rejection mechanisms. this paper, we conduct a comprehensive examination of the role of rejection, introducing the alignment goal of model reliability along with corresponding metrics. This goal requires the model to pro- vide accurate responses while adeptly rejecting questions exceeding its knowledge boundaries, thereby minimizing hallucinations, To improve the inherent reliability of LLMs, we present novel alignment framework called Reinforcement Learning from Knowwlegge Feedback (RLKF). RLKF lever- ages knowledge feedback dynamically determine the model's knowledge boundary and trains reliable reward model to encourage the rejection of -nowleeee questions. Experimental results on mathematical and question answering datasets affirm the substantial efficacy of RLKF in significantly enhancing LLM reliability.
1 Introduction
Large Language Models (LLMs) have exhibited strong capabilities in solving various down- stream tasks through alignment techniques such as Supervised Finetuning (SFT, (Zhang et al., 2023b), (Zhang al., 2023b)), Direct Preference Optimization (DPO, (Rafailov al., 2024)), and Reinforcement Learning from Human Feedback (RLHF, (Stiennon et al., 2020; Ouyang et al., 2022)). Those techniques align language models with human intent, mainly to maximize the helpfulness of LLM's responses. However, maximizing helpfulness does not mean minimizing errors. A significant problem arises in that LLMs often produce outputs that, while seemingly plausible, contain factual errors (Min et al., 2023) self- contradictions (Liu et al., 2022), which are referred to as hallucinations.
To mitigate the hallucinattonn,, many studies focus augmenting the knowledge of LLMs, such as curating training data (Penedo et al., 2023; Zhou et al., 2023) or employing retrieval- augmented generation (RAG, (Gao et al., 2023b;b)) during inference. Nevertheless, it is essential to acknowledge that model knowledge inherently has limitations, and even the most powerful models, such as GPT-4, are prone experiencing hallucinations (Zhang et al., 2023a). Consequently, we posit that the fundamental nature of the hallucination problem lies in the model's misalignment with its knowledge boundary. Hallucinations arise when LLMSs try to answer questions beyond their knowledge boundary.
'The corresponding authors are Lu Chen, Shuai Fan and Kai Yu.

---

Published as a conference paper at COLM 2024
What the result of 934 28?
Alignment for Hellpfunnss
Uteholphul Respanses wow.eeeee.......................
Helpful Respeases
The answer 26/12.
anower 26232.
The answer 26/52.
Sorry, don't know the answer
OK, thanks!
Alignment Reliability
What the result of 934 28?
Aligument Principle
Sorry, don't know the answer! recommend you to use calculazor.
satisfied
acceptable >
unsatissied
Aligument Objective
Fine, I will use calculator.
Helpfull Resparses
Untruthful Resposss
The 26152.
The 26232.
What the result of 934 x 287
Helpful Reppasees
Trathfal Rejertiom Sorry. know the
The 26152.
anower 26232.
Trudkfal Rejections Sorry. don't know the anower
Untruthful Respeases
You're a liar, gove me the wrong result !!!
anower is 26232.
Figure 1: The user cases and alignment objectives for model reliability.
One research direction related to aligning LLM with its knowledge boundary involves alignment for honesty (Kadavath et al., 2022; Yang et al., 2023b). However, alignment for honesty presents significant challenges. On the one hand, the honesty of LLMs is hard to evaluate. Honesty can be viewed as a classification problem, where the model learns to distinguish between what it knows and what it does not. Nevertheless, the accuracy which indicates how much the model knows can differ dramatically among various models or even with different prompts (Kaddour et al., 2023). On the other hand, it is difficult to employ honesty for comparing and selecting different models consistently. Model honesty is tied to the capabilities of the model itself; a more honest model does not necessarily imply that it will provide more assistance or make fewer mistakes compared to other models.
To address the aforementioned limitations, we introduce the alignment goal of reliability inspired by model truthfulness (Lin et al., 2021). We define the reliability from the user's 's perspective. As shown in Figure 1, helpful responses (correct answers) can assist users while untruthful responses (incorrect answers) result in user loss, leading to distrust in the model. Therefore, we contend that the key achieving a reliable system lies in providing as many correct answers as possible to maximize helpfulness while learning to explicitly refuse unknown questions to minimize errors. We apply accuracy and truthfulness to measure the model's helpfulness and errors, respectively, and further propose overall reliability score to simultaneously evaluate the above two aspects.
To optimize model reliability, we propose the Reinforcement Learning from Knowledge Feedback (RLKF) training framework based on RLHF On the one hand, most publicly available alignment data (Cui et al., 2023) often originate from multiple source models, which is impossible to be used for aligning target model with its own knowledge boundaries. On the other hand, while the preference data in RLHF is derived from the target model's 's outputs, it is heavily influenced by human annotators' biases. The current annotation goals based on helpfulness lead the reward model in RLHF to only learn to distinguish the helpfulness of responses, making it difficult to discern their truthfulness. Instead, RLKF automates the construction of preference data for specific target model through knowledge feedback rather than human feedback. Knowledge feedback is primarily used to assess whether question falls within the model's knowledge boundary. When a question falls within the model's knowledge boundary, a response preferred over refusal. Conversely, when question exceeds the model's knowledge boundary, refusal is preferred over a response. The synthesized preference data is used to train reliable reward model, which thoroughly understands the target model's knowledge boundaries and further instructs the target model when to respond and when to refuse through the PPO algorithm. Experimental results
2

---

Published as a conference paper at COLM 2024
further demonstrate the effectiveness of our framework, which significantly improves the reliability of baseline models.
The contributions of this paper summarized follows:
We introduce the alignment goal of model reliability and define several metrics to assess the reliability of LLMs.
We propose the Reinforcement Learning from Knowledge Feedback (RLKF) align- ment framework to improve LLM reliability.
Extensive experiments are conducted to validate the effectiveness of RLKF frame- work.
2
Problem Formulation
2.1 LLM Alignment
With the potential risks brought by powerful LLMs, researchers have developed various alignment approaches to align LLMs with human instruction, preference, and values (Wang et al., 2024). Specifically, for the input prompt x; and alignment goal helpfulness, we can employ the following scoring principle to represent our alignment objective: s(x, yn) > yu), where yh, 1/u represent helpful response and unhelpful response, respectively. The scored response pair can be annotated either by human annotators (Ouyang et al., 2022) scoring model (Gao et al., 2023a) trained with human preference data. We can further utilize this comparison data to train a reward model or LLM policy, thus aligning LLMs with specific goals.
Furthermore, for a given set of N inputs and LLM response yj of each input x,, we can simply evaluate the helpfulness of LLM using accuracy, where helpful responses considered correct and unhelpful responses are considered incorrect.
2.2 Alignment with Reliability
While many works focus on alignment for helpfulness, existing alignment goals make hard to alleviate model-specific hallucinations. We further propose the alignment goal of model reliability from the user's perspective. We believe that a reliable system should be aligned with user experience that provides as much assistance as possible while making as few errors as possible.
Specifically, for the input prompt x; and alignment goal reliability, we employ the following scoring principle to represent our alignment objective:
yc) sxx,yrr) s(x,yw),
(1)
where Ye, Vr, y/ represent a helpful response (the correct answer), truthful rejection, and an untruthful response (the wrong answer), respectively. We then use accuracy (acc) for as- sessing the helpfulness, and truthfulness (truth) (Lin et al., 2021), representing the proportion of truthful, non-harmful responses. We also include precision (prec) to partially reveal the model's 's self-knowledge:
N N + N
N N
N N N
N N
prec
acc
truth
(2)
where No, N., Nw represent the number of correct, rejected and wrong responses, respectively. Finally, we define comprehensive system reliability (rely) metric based accuracy and truthfulness:
rely(a) truth (1 a) acc,
where E [0,1], and it represents the degrees of sensitivity among users towards errors. As alpha increases, reflecting greater user emphasis on system truthfulness, the model should aim to minimize errors by using refusal to respond when appropriate to meet
3

---

Published as a conference paper at COLM 2024
user expectations. Specifically, when equals to answer rate (ans.), we define the overall reliability as:
Nr N rely ans. truth + (1 ans.) acc.
(4)
When the answer rate is low, we encourage the model not to reflexively refuse but rather to attempt to provide assistance. Conversely, when the answer rate is high, we believe the model should become more cautious to avoid errors. This metric balances the model's helpfulness and truthfulness while mitigating the risks of it becoming overly conservative or excessively aggressive.
3 RLKF
To better align LLMs with reliability, we propose the Reinforcement Learning from Knowledge Feedback (RLKF) framework. By introducing knowledge feedback, our framework trains LLMs to learn to refuse out-of-knowledge questions explicitly to achieve the alignment goal.
Answer Verification
In-domain
In-domain Questions & Response Sampling
Through Knowledge
Preference Data
A conueers right
A
IDK
What is the rouul of 98 28
Sompling
Some right, some saroing
Response A:
Response
Respome C:
axprasston
B
crowers
IDK
Answer Verification
Out-of-domain Preference
Out-of-domain
Response Sampling
Through Internal Knowledge
Question:
Arpwers highty
Janny brought peanst butter cookies...
Somp lng Desporons from LIM
Response Ai
Response B:
Response C:
Self-consistancy
Arowars and
#
?
IDK
have
sumber
tatell..
Figure 2: Reliable preference data generation pipeline. Letters with green, red, and yellow circles denote correct, incorrect, and uncertain answers, respectively. "IDK" represents "T don't know," indicating rejections.
3.1 High-level methodology
The proposed Reinforcement Learning from Knowledge Feedback (RLKF) framework is built upon RLHF (Ouyang et al., 2022) for reliability alignment. Instead of annotating preference pairs with human labor in RLHF, we automatically generate reliable preference pairs with knowledge feedback. The key to constructing reliable preference data lies in the insertion of rejection responses within the comparison set as defined in $2.2. Thus aligned LLM can understand the importance of avoiding errors through rejection. We describe the high-level methodology as follows:
We start with a model that is already aligned for helpfulness or harmlessness, i.e., LLAMA-2 CHAT (Touvron et al., 2023), and then apply the following three steps below:
Step 1: Synthesizing model-specific reliable preference data through knowledge feedback. Given the LLM and each input prompt in prompt dataset D,, we dynamically construct different comparison pairs according to the model responses to formulate reliable preference data RPD.
Step 2: Train reliable reward model with collected preference data. We first train the reward model using generic helpfulness preference data PD to obtain RM. Then, continuously train RM with the synthesized reliable preference data to obtain a reliable reward model RRM.
4

---

Published as a conference paper at COLM 2024
Step 3: Optimize policy against the reward model using PPO. We use the output of the RRM as a scalar reward. We fine-tune the supervised policy LLM to optimize this reward using the PPO algorithm (Schulman et al., 2017).
3.2 Reliable preference data synthesizing
Compared to normal preference data that only classifies LLM responses into helpful and unhelpful classes, we categorize the LLM responses into three types: helpful, truthful, and untruthful responses, which correspond to correct answers, rejections, and incorrect answers, respectively. Besides, we also need to dynamically select two types out of three types of responses for constructing comparison pair based on different questions. We believe that the key to selecting lies in whether the problem is within the model's 'ss knowledge boundary (whether the model can answer the question correctly). We consider two types of research settings. In the in-domain setting, we can acquire the correctness of model predictions through external knowledge, i.e,, a calculator for arithmetic questions. the out-of-domain scenario, we only have input prompts and cannot obtain golden labels of the inputs.
3.2.1 In-domain Reliable Preference Data
As depicted in the upper portion of Figure 2, we determine whether a question lies within the model's knowledge boundary by analyzing the distribution of correctness across multi- ple samplings of model responses, then construct different comparison pairs accordingly. Specifically, given the input and N sampling responses yi, [1, N), the comparison pair is selected as follows:
(x,ye) > (x,
if N N, if N E|1,N), (x,yr) > (x, if N == 0,
pair
(x,ye) > (x,
(5)
where ye, Vr, represents the random choice from correct, reject, and wrong responses, respectively, and Ne represents the number of correct responses. In cases where the model lacks specific rejection responses, we randomly select one rejection sentence from sen- tences that are generated by ChatGPT based rejection templates. The templates can be found in Appendix] J.
3.2.2 Out-of-domain Reliable Preference Data
In the out-of-domain setting, we utilize self-consistency Wang et al. (2022) as the internal model knowledge to assess whether the model possesses sufficient knowledge about the given question. As depicted in the lower portion of Figure 2, when the samplings exhibit high consistency, we align the model with preferring to provide an answer; conversely, when the samplings demonstrate low consistency, refusing is more preferred. Specifically, given the input and N sampling responses yı, É [1, N], the comparison pair selected as follows:
(x,yy) (x,yr), ifN, > f, (x,,,, > (x,ya), ifN, < f,
pair
(6)
where ya yr represents the random choice from answered and rejected responses, respec- tively, N₂ represents the number of the most consistent answer over all answers, and t represents the threshold of consistency score where use [ for all the experiments.
3.3 Model training
RM Training. To train the reward model, we convert collected pairwise reliable pref- erence data into a binary ranking label format (i.e., chosen & unchosen) and enforce the chosen response to have higher score than its counterpart. We used binary ranking loss

---

Published as a conference paper at COLM 2024
consistent with Ouyang et al. (2022):
Lranking = - (ra Mekosen) re (x, (anchosen)))
(7)
where ræ (x, y)ss the scalar score output for input x and completion y with model weights Ychosen Saneneyunncen Yunchosen are the chosed and unchosed responses, respectively.
Reinforcement Learning (RL). We further train our LLM policy following the RL scheme of Stiennon et al. (2020), which uses the reward model as estimate for the true reward function. During this phase, we seek to optimize the following objective:
arg max Ep- [R(glp) BDxL (T (8 p) (o(g)))]
(8)
We iteratively improve the policy by sampling prompts p from our dataset D which contains both in-domain and out-of-domain prompts and generations from the policy π and use the PPO algorithm and loss function to achieve this objective, the reward function also contains a penalty term for diverging from the original policy πо.
Experiments
4.1 Experiment Setup
4.1.1 Dataset Construction
Mathematical Dataset. Our experiments involve two mathematical datasets: synthesized arithmetic questions and GSM8K (Cobbe et al., 2021). The arithmetic dataset consists of 14,000 samples, divided into sets of 10,000, 3,000, and 1,000 samples for training the reward model, training the policy as prompt data, and testing the final results, respectively. Additionally, we sampled 2,000 data points from the training set of GSM8K to train the reward model and 1,000 data points to optimize the LLM policy. important to note that we only used the input prompt of the GSM8K data points, without utilizing the annotations. This serves as an out-of-domain experimental setting.
We generate the arithmetic dataset synthetically following Liu Low (2023). The input numbers are randomly generated, hence ensuring a very low probability of instances being duplicated. We sample from log space to ensure the numbers are equally likely to be sampled from different orders of magnitude. Following Liu & Low (2023), we use hundreds of instruction templates generated by ChatGPT, e.g., Please help me calculate {ari thmeticce.,.... to diversify the question formats. Examples of templates can be found in Table of Appendix I.
Knowledge-basee based QA Dataset. TriviaQA (Joshi et al., 2017) is widely-used QA dataset that can be used to test a model's world knowledge. We selected 20,000 samples from the TriviaQA training set for training. Since the ground truth of the TriviaQA test set is not publicly available, we used the TriviaQA development set, which contains 11,313 samples, to validate our results. We used the Exact Match metric (whether the answer is exactly in the model's response) from TriviaQA paper as our measure of Accuracy, while keeping the other metrics consistent with those in mathematical datasets.
4.1.2 Baselines
We incorporate several baselines to benchmark the performance of our RLKF framework. All prompts used in this work are listed in Appendix G.
No & prudent system prompt. System prompts are commonly used to control a model's response style and personality. When the system prompt is empty, LLAMA 2-CHAT tends to respond to almost every question, losing the ability to reject unknown questions. We also use the default system prompt of LLAMA 2-CHAT introduced by Touvron et al. (2023) as the prudent system prompt. LLAMA 2-CHAT will become more cautious and reject more questions with this system prompt. These two types of prompts represent two different personalities of LLAMA 2-CHAT, serving as important baselines for this study.

---

Published as a conference paper at COLM 2024
In-context Learning. We randomly sampled 5 correctly responded (if the model can answer correctly) and refused examples (if the model is unable to answer correctly) to append before the input question the in-content learning baseline.
Rule-based PPO. We use rule-based reward function to optimize the policy in in-domain prompts. The model receives reward of 1 if it answers correctly, 0 if it refuses to answer, and 11 answers incorrectly. The determination of different cases is achieved through heuristic rules.
RLHE We use the reward model trained only on generic helpful preference data to optimize the policy with the same training prompis RLKF
SFT. We use all 10,000 arithmetic questions from the constructed preference dataset with their chosen responses to directly fine-tune the LLM policy.
4.1.3 Training Details
We employ the LLAMA 2-CHAT 7B (Touvron et al., 2023) our baseline model, which has been already aligned with human preferences. The policy model and reward model are both initialized from LLAMA 2-CHAT 7B. We use DeepSpeed-Che Chat (Yao et al., 2023) to run the whole training pipeline. As for the Reward Model (RM), we first utilized several open-source datasets (Stiennon al., 2020; Bai et al., 2022; Ethayarajh et al., 2022) and replicated the training process following UltraRM (Cui et al., 2023) to obtain a helpful RM. Subsequently, we trained reliable RM on the constructed reliable preference data, using batch size of 8 for two epochs. During the RL phase, we trained for one epoch with a batch size of 1, the generation batches and PPO epochs are both equal to Prompts from both in-domain and out-of-domain datasets are mixed to train the final RM. We train the SFT model for one epoch with a batch size of 32. All other training parameters were set to the default in DeepSpeed-Chat We conduct all experiments using Nvidia A800 GPUs.
4.1.4 Evaluation Details
For evaluating the final policy, we utilize four metrics: precision, accuracy, truthfulness, and reliability. Precision is the proportion of correctly answered questions among those the model chose to answer, reflecting its self-awareness of its capabilities. Accuracy is the proportion of correct answers among all questions, indicating how much assistance the model provides to the user. Truthfulness is the proportion of questions the model either answered correctly or refused to answer. Reliability is the dynamic weighting of accuracy and truthfulness; high reliability indicates that the model provides more help to the user and less incorrect information. The specific evaluation formulas are detailed in Section 2.2. To determine whether the policy refuses respond and extract answers, we employ an additional answer extractor LLM (LLAMA 2-CHAT 7B with extractor prompt, see Appendix G). By comparing the extracted answers with the standard answers, we can ascertain their correctness.
4.2 Reliability Evaluation
Reliability on Mathematical Tasks. Table 1 presents the results of the final policy model after RLKF on arithmetic questions and GSMSK datasets. We can see that RLKF signif- icantly enhances the model's reliability on both in-domain and out-of-domain datasets. The RLKF-trained model shows remarkable improvements in precision, truthfulness, and reliability. The increase in precision reflects the model's increase in self-knowledge. while the improvement in truthfulness indicates that the model learns how to refuse answers. It is important to note that when the model operates without system prompt, it responds to all questions, thus reaching the accuracy ceiling. Our RLKF method does not teach the model how to answer questions but rather trains it to reject questions, leading to some performance loss but significantly reducing the model's 's hallucination errors. We also show the detailed analysis on different arithmetic sub-tasks in Appendix A.
Reliability on TriviaQA Task. To explore the generalizability of our method, we aim to determine whether it can align the LLM with other knowledge boundaries beyond math-
7

---

Published as a conference paper at COLM 2024
Arithmetic Test
GSM8K
Method Type Method
Prec 1
Acc 1 Truth
Rely
Prec 1
Acc
Truth
Rely t
system prompt
37.8
36.1
40.6
40.4
25.0
24.6
26.2
26.2
Prompt
prudent system prompt
43.7
24.1
69.0
48.8
22.6
10.4
64.3
35.2
In-context Learning
40.6
37.3
45.4
44.7
18.8
8.0
65.5
32.4
Rule-based PPO
46.0
27.6
67.6
51.6
17.6
8.6
59.6
33.6
RL RLHF
40.1
27.4
59.1
49.1
20.1
17.4
42.9
36.4
RLKF(Ours)
72.8
31.9
88.1
56.5
29.6
17.0
59.6
41.5
Table 1: Performance on in-domain arithmetic questions and out-of-domain GSM8K datasets. Prec: precision. Acc: accuracy. Truth: Truthfulness. Rely: reliability, representing the simultaneous consideration of the model's helpfulness and truthfulness.
TriviaQA
ID Arithmetic
OOD GSM8K
Method
Method
Prec
Acc
Truth Rely
Prec
Acc
Truth
Rely
Prec
Acc
Truth Rely
LLAMA 2 no system prompt
60.2
58.9
61.2
61.1
LLAMA-2 43.7
24.1
69.0
46.6
22.6
10.4
64.3
35.2
LLAMA 2+ prudent system prompt
53.9
42.4
63.7
59.2
+ SFT
58.1
52.4
62.2
61.2
23.2
4.7
84.5
20.8
ILAMA 2+ RLHF
58.5
51.7
63.3
62.0
RLKF 72.8
56.5
17.0
59.6
41.5
LLAMA 2 RLKF
73.5
50.1
81.9
71.8
Table Performance comparison with SFT.
Table 2: Performance on TriviaQA dataset.
ematical calculations. Thus we validated our method on the knowledge-based Question Answering task, i.e., TriviaQA. As shown in Table 2, our method can also significantly improve the model's precision, truthfulness, and overall reliability on the TriviaQA dataset. This demonstrates that our method can enhance the reliability of LLMs across different tasks, not just limited to mathematical questions.
Comparison with SFT. We further compare our method with supervised fine-tuning (SFT) (Yang et al., 2023b). Since there are no ground truth labels available for out-of-domain scenarios, we performed SFT only on in-domain data. As shown in Table 3, the model's reliability on in-domain tasks is significantly improved after SFT. However, major issue with SFT is overfitting, as observed by the overly conservative behavior of the model on the out-of-domain GSM8K dataset, where it tends to reject almost all questions. In contrast, our RLKE consistently improves the generalization ability of the model on both in-domain and out-of-domain tasks.
Comparison with Calibration-based Methods. Calibration-based methods (Kadavath et al., 2022; Lyu et al., 2024; Kapoor et al., 2024; Tian et al., 2023) use some post-hoc techniques to predict whether the model is about to hallucinate, which can be used to trigger a refusal to answer. We also compare our method with calibration-based methods, and provide the results in Appendix B. Experimental results show that our method achieves higher reliability than other calibration-based methods at the same inference cost, though it is slightly less reliable compared to consistency-basee cost. Additionally, it is important to note that calibration-based methods are unable to reject explicitly and require searching for and determining the best threshold for rejection, as well as providing human-crafted rejection templates as responses. more detailed discussion can be found in the Appendix B. Comparison with GPT Series Models. natural question arises: with broader instruction- tuning, can the model naturally learn to refuse during the RLHF process? We further tested the performance of the industrial-grade GPT series models to address this concern. The results are shown in Appendix C. We found that while both ChatGPT (gpt-3.5-turbo- 0125) and GPT-4o exhibit high accuracy on simpler tasks, they struggle with rejecting out-of-knowledge questions and even GPT-4o shows significantly lower reliability than our methods on harder tasks. Thus we believe that broader generic instruction-tuning alone does not resolve reliability issues.
methods with 10 times the inference
8

---

Published as a conference paper at COLM 2024
4.3 Reward Model Evaluation
ID Arithmetic RPD
OOD GSMBk RPD
Method
within
beyond
boundary
average
within beyond boundary
average
helpful PD
73.9
49.9
55.5
58.6
95.0
27.3
73.0
51.6
helpful PD + OOD RPD
60.7
41.5
50.8
49.4
42.9
56.3
52.0
53.4
helpful PD PD
ID RPD
90.3
83.0
78.5
84.4
91.6
41.6
71.2
57.9
helpful PD + both RPD
88.4
84.7
76.4
84.3
70.6
64.9
69.1
67.1
Table 4: Performance comparison with different reliable preference data. PD: preference
data. RPD: reliable preference data.ID: in-domain. OOD: out-of-domain.
To further determine whether our RelyRM possesses the ability to recognize the policy's knowledge boundary, constructed in-domain and out-of-domain reliable preference datasets (RPD) for testing our RelyRM. The dataset construction is similar to training RPD data but with golden labels and on test sets. Table 4 illustrates the performance on the reliable preference dataset when provided with different training data. We observe that the reward model trained solely on helpful PD performs worse on the two datasets, it tends to choose reply rather than reject. However, the reward model trained on in-domain RPD demonstrates a significant improvement on choosing rejection. Additionally, due the higher noise level in the out-of-domain RPD, training on this data alone does not enhance reliability on out-of-domain preference data; instead, training on both RPDs can improve the accuracy out-of-domain RPD.
4.4 Rejection Study
We further analyzed the rejection behavior after RLKF to assess whether the models have acquired awareness of their knowl- edge boundaries and possess appropriate rejection capabilities (avoiding both exces- sive rejection and failure to reject). We clas- sify arithmetic questions according to var- ious digit ranges and arithmetic operations and compare the rejection capabilities of LLAMA 2 before and after applying our RLKF framework.
80
80-
0ur5
60
60
Reject
-0
Peoject
40
20
20
Figure 3: Rejection rate comparison.
Rejection Rate Distribution. Figure compares the rejection rate changes of LLAMA 2 before and after applying RLKF across different types of questions. We found that the original ILAMA 2 model had relatively high rejection rate, but its rejections were distributed similarly across different types of questions. However, after applying RLKF to LLAMA 2, the rejection rate decreased for easier problems such addition and subtraction operations, and low-digit numbers, while significantly increasing for more difficult problems such multiplication and division operations. This indicates that the RLKF model has better understanding of the difficulty levels of different types of problems and can reject them more selectively and effectively.
Response Type Distribution. Figure 4 illustrates the distribution of response types for different methods. We observe that when the LLAMA 2 model operates without system prompt, it lacks rejection capabilities but intuitively responds to more questions. Besides, using a prudent system prompt leads the model to lean towards rejecting questions, reducing the model's error responses but causing more performance loss due to excessive rejection. In contrast, our RLKF achieves significantly lower error rates compared to the two methods while maintaining better accuracy than the prudent system prompt (as system prompt not rejecting responses represents the accuracy ceiling). This demonstrates that our model can effectively reject while avoiding excessive caution.

---

Published as a conference paper at COLM 2024
100
100
Percenage of Rensporse types (50
Percenage of Rendparse Typas (16)
80
80
60
60
40
40
pradent
ouTs
20
Wrong
Correct
Rejact
2
Figure 4: Percentage of different response types among different arithmetic questions.
Related Work
LLM Alignment. LLM alignment aims to align language models by training them to act in accordance with the user's intention, either by supervised fine-tuning (Wei et al., 2021; Chung et al., 2022; Zhang et al., 2023b), direct preference optimization (DPO, (Rafailov et al., 2024)), or reinforcement learning from human feedback (RLHF) (?Ouyang et al., 2022; Glaese et al., 2022). Most existing works focus on improving the instruction-folloiing ability (Sanh et al., 2021; Wei et al., 2021), helpfulness (Ding et al., 2023; Xu et al., 2023) or harmlessness (Solaiman Dennison, 2021; Bender et al., 2021) of LLMs. However, there is limited research on honesty alignment due to the challenges of its definition and evaluation (as we discussed in Appendix F). (Cui et al., 2023) constructed a preference dataset encompassing various objectives including honesty. Some studies (Yang et al., 2023a;b) attempted to enhance model honesty by honesty-orienter SFT; however, our experiments demonstrated that SFT often suffers from poor generalization issues. Besides, our proposed alignment goal of reliability considered both helpfulness and truthfulness (Lin et al., 2021), enabling the building of more helpful and truthful LLMs.
Mitigating Halucinations. While LLMs have demonstrated remarkable performances, they often generate content that conflicts with user input (Guerreiro al., 2023) or previously generated information by themself (Mündler et al., 2023) or is not faithful to established world knowledge (Min et al., 2023), which are referred as hallucinations. Some efforts have aimed to alleviate hallucination issues by introducing higher-quality data during the pre- training phase (Penedo et al., 2023; Touvron et al., 2023) or in the SFT stage (Chen et al., 2023; Zhou et al., 2023). Others have focused on detecting and correcting hallucinations through methods such as incorporating external knowledge (Gao et al., 2023b), designing decoding strategies (Shi et al., 2023) or uncertainty estimation (Azaria & Mitchell, 2023; Xiong et al., 2023; Zhao et al., 2023). However, model-specific errors are also hallucinations. While most previous studies focus maximizing the correctness of responses, there are limited works that focus on minimizing the errors to mitigate hallucination, which is the main focus of this paper. Consequently, we propose the evaluation methodology of reliability considering both maximizing the correctness and minimizing the errors to mitigate hallucinations.
6 Conclusion
In this work, we propose the alignment goal of LLM reliability and introduce novel framework to enhance the reliability by teaching them to refuse questions outside their knowledge boundary. By proposing the Reinforcement Learning from Knowledge Feedback (RLKF) framework and defining new evaluation metrics for model reliability, we effectively address the issue of LLM hallucinations.. The implementation of RLKF demonstrates significant improvements in LLM reliability, showcasing a promising method for developing more trustworthy AI systems.
10

---

Published as a conference paper at COLM 2024
Acknowledgements
This work is funded by the China NSFC Projects (92370206, 62106142, 621201 and U23B2057) and Shanghai Municipal Science and Technology Major Project (2021SHZDZX01022;
References
Amos Azaria and Tom Mitchell. The internal state an I1m knows when its lying. arviv preprint wxXi:23344.1733. 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2044005862 2022.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountabilitty and transparency, pp. 610-623, 2021.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. Xiv preprint xix: 00..200007.001,, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. arXio preprint ar 2rXi::2210.11416, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers to solve math word problems. ar Xiv preprint ar Xiv:22100.1444 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback. arXiv preprint arXiv:2310.01377, 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. ar Xiv preprint ar Xiv:23005.11 1e0:23 2023.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Con- ference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5988-6008. PMLR, 17-23 Jul 2022.
Leo John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023a.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. ar Xiv preprint ar arXtw:2312.10997 2023b.
Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improv- ing alignment of dialogue agents via targeted human judgements. arXio preprint ar Xix::2009.1435, 2022.
Nuno M Guerreiro, Duarte M Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André FT Martins. Halluucnations large multilingual translation models. Transactions of the Association for Computational Linguistics, 11:1500-1517, 2023.
11

---

Published as a conference paper at COLM 2024
IDEA-CCNL. Fengshenbang-lm.
2021.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: large scale distantly supervised challenge dataset for reading comprehension. ar Xiv preprint arXio:1705.03551, 2017.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arviv preprint ar ario:22007.052. 2022.
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy, Challenges and applications of large language models. arviv preprint arXio:2307.10169, 2023.
Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson, Calibration-tuning Teaching large language models to know what they don't know. In Proceedings of the 1st Workshop on Uncertainty-AAwwre NLP (UnncetaiNLP 2024), pp- 1-14, 2024.
LAION-AL. Openassistant. https: / //gi wttpp...oo//Aaaoorn/////eeesssss 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. Xiv preprint ar axXi::2109.07958, 2021.
Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and William Dolan. A token-level reference-free hallucination detection benchmark for free-form text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6723-6737, 2022.
Tiedong Liu and Bryan Kian Hsiang Low. Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks. ar Xiv preprint ar arXiv:2305.14201, 2023.
Qing Lyu, Kumar Shridhar, Chaitanya Malaviya, Li Zhang, Yanai Elazar, Niket Tandon, Marianna Apidianaki, Mrinmaya Sachan, and Chris Callison-Burch. Calibrating large language models with sample consistency, Xrv preprint ar arXiv:2402.13900 2024.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. Xrviv preprint arXiv:2305.14251, 2023.
Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictory halluci- nations of large language models: Evaluation, detection and mitigation. arv preprint arXiv:2305.1583 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27777 2022.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. Xiv preprint Xiv::306.00111 011666600166,.... 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Amaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. Xiv preprint ar arX1o:2110008207, 2021.
12

---

Published as a conference paper at COLM 2024
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithmss arxio preprint ar XxXi:1707.063477 2017.
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. arXiv preprint ar Xrv::2305..14333 2023.
Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with values-targeted datasets. Advances in Neural Information Processing Systems, 34:5861-587,, 2021.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Tianxiang Sun, Xiaotian Zhang, Zhengfu Heng Li, Qinyuan Chenn.,nCheng... Hang Yan, Xiangyang Liu, Yunfan Shao, Qiong lang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuanjing Huang, and Xipeng Qiu. Moss: Training conversational language models from synthetic data. 2023.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arxiv preprint ar wwXx:22305.14975 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. arXiv preprint 7XXi:23377092888 2023.
Xinpeng Wang, Shitong Duan, Xiaoyuan Yi, Jing Yao, Shanlin Zhou, Zhihua Wei, Peng Zhang, Dongkuan Xu, Maosong Sun, and Xing Xie. On the essence and prospect: An investigation of alignment approaches for big models. arXiv preprint arXiv:2403.04204, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ar preprint ar drXt::200..11171, 2022.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. ar Xiv preprint ar XrXi:22199.01622, 2021.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can Iims express their uncertainty? an empirical evaluation of confidence elicitation in llms. arviv preprint ar arXio:2306.1006 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXio preprint arXie:2304.12244, 2023.
Yuchen Yang, Houqiang Li, Yanfeng Wang, and Yu Wang. Improving the reliability of large language models by leveraging uncertainty-aware in-context learning. arXie preprin! arX1::2310.04722 2023a.
Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. Alignment for honesty. Xiv preprint ar XrX1::2312.07000, 2023b.
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. ar Xiv preprint Xrv::23008.013 013022 2023.
13

---

Published as a conference paper at COLM 2024
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXio preprint arXio:2305.13334 2023a.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. Xrviv preprint arXiv:2308.10792, 2023b.
Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. Verify-and-edit A knowledge-enhanced chain-of-thougie framework. arXiv preprint ar orXi::2305.0366, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. ar preprint arXio:2305.11206, 2023.
A Reliability on Arithmetic Sub-tasks.
Figure 5 illustrates the reliability improvement of our RLKF method across different arith- metic sub-tasks. Our RLKF method enhances the reliability of the model across various digit ranges and arithmetic operations. It is noteworthy that our model shows significant improvements in precision and truthfulness on tasks involving 3-5 digit numbers as well as challenging operations like multiplication and division. This indicates that RLKF aids the model in understanding its knowledge boundary, enabling to learn to refuse questions prone to errors.
Acorolcy
80
20
80-
40
40
40-
Prodson
Accursey
80
80-
systam
5
40
40-
Figure 5: The results on arithmetic sub-tasks. No: no system prompt, prudent: prudent system prompt, ours: RLKE.
B Experiments of Calibration-based Methods
Inference cost
Arithmetic Test
GSM8K
Method
Acc
Truth
Rely
Acc
Truth
Rely
Raw logits(trresh-arith) (Lyu et al., 2024)
1
37.7
55.4
52.3
23.7
24.9
24.9
Raw logits(trres--gsm8k (Lyu et al., 2024)
1
21.2
85.9
44.0
13.9
72.2
38.2
P(Irue) (Kadavath et al., 2022)
17.6
71.3
42.5
19.5
39.3
35.4
verb. 1S top-1 (Tian et al., 2023)
1
16.7
50.2
39.0
4.8
11.6
11.1
verb. 2S top-1 (Tian et al., 2023)
2
14.7
87.8
34.4
4.4
20.7
18,0
agreement(consistency)) (Lyu et al., 2024)
10
37.9
79.9
62.3
20.1
77.1
44.6
RLKF (Our method)
1
31.9
88.1
56.5
17.0
59.6
41.5
Table 5: Perfformacce comparison with calibration-based methods. We represent inference cost as the product of the required number of dialogue turns and the number of sampling iterations.
We further compare our method with calibration-basel methods, and provide the results and our analysis below:
14

---

Published as a conference paper at COLM 2024
Unable to Reject Explicitly: Calibration-basedd methods need to search and deter- mine the best threshold for rejection and provide human-crafted rejection templates responses. As we shown in the table, search the threshold for arithmetic and gsm8k separately (on 100 validation cases from each dataset). However, the thresholds are quite different for different datasets which results in significant per- formance degradation with different thresholds. In contrast, our method can enable the model to reject out-of- aut-of-knowledge questions with personalized responses for different prompts automatically.
Consistency-basd methods, the one hand, require multi- ple samplings to obtain results, and on the other hand, may necessitate the use of additional models to extract answers for voting (we ChaiGPT to extract answers because rule-based methods may result in inaccurate extraction). This results in 5(sampling num) 2(1 for answer generation, for answer extraction) 10 times (or at least 5 times) the inference cost than other methods. Some Verbalized-bent meth- ods (verb. 2S) also require the model to generate confidence through an additional round of response after generating the answer.
High Inference Cost:
High Calibration Variance: Utilizing calibration methods to determine the accuracy answers is not stable. For instance, logit-based methods are not quite reasonable when the model generates longer responses, and Cerbalized-basee methods result in significant fluctuations in confidence scores and even prediction results (as shown in the gsmök results of verb. methods) due to the variability in prompts.
In summary, calibration methods are more suitable for analyzing the uncertainty of model responses or constructing training data (such as the self-consistency(5) tenable introduced in our paper). However, our alignment research on reliability aims to enable the model to acquire self-knowledge and explicitly refuse out-of-knowledse show our method can enable the model to reject automatically without additional inference costs and improve the accuracy of rejections compared to most calibration methods.
questions. Experimental results
C Experiments of GPT Series Models on Arithmetic Dataset
LLAMA-2 + RLKF
ChatGPT
GPT-4o
Subsets
Acc
Truth
Rely
Acc
Truth
Rely
Acc
Truth
Rely
72.8
77.2
74.7
97.6
97.6
97.6
99.6
99.6
99.6
28.3
85.7
53.4
91.6
92.8
92.8
93.6
96.8
96.7
8.4
98.3
47.8
37.2
39.3
39.3
49.8
49.8
49.8
18.6
91.3
50.4
69.3
69.3
69.3
81.8
82.2
82.2
1-2 digit
41.2
89.5
62.4
86.0
87.0
87.0
94.0
95.5
95.5
3-5 digit
25.7
87.2
52.6
66.3
67.0
67.0
73.2
73.7
73.7
all
31.9
88.1
56.5
74.2
75.0
75.0
81.5
82.4
82,4
Table Performance comparison of models on different arithmetic subsets.
We tested the reliability of ChatGPT and GPT-4o on arithmetic datasets as shown in Ta- ble 6. We found that even GPT--4o remains unreliable and lacks the ability to reject out-of- knowledge questions. From LLAMA-2 to ChatGPT GPT-4o, they all use large amount of industrial-grade generic instruction-tuning data, but they still lack good reliability. We believe that, on the one hand, these generic preference data may lack appropriate rejec- tion data (rejecting only when the model lacks relevant knowledge, otherwise it needs to answer questions). On the other hand, the RLHF training process constructs preference pairs based on the model's own sampling results, making it difficult to generate appropriate rejection behavior during the sampling process. Therefore, we believe that it is necessary to synthesize reliable preference pairs through RLKF to make the model more reliable.
15

---

Published as a conference paper at COLM 2024
Anthropic Helpful
OpenAl Summ.
Stanford
Model
Backbone
SHP
MOSS
Llama-7B
61.3
58.1
54.6
Ziya
Llama-7B
61.4
61.8
57.0
OASST
DeBERTa-1arge
67.6
72.1
53.9
SteamSHP
FLAN-T5-XL
55,4
62.6
51.6
UltraRM
Llama2-138
71.0
74.0
73.7
RelyRM (ours)
Llama2-7B
66.5
71.0
63.7
Table Performance comparison on helpful preference datasets with various reward models.
D Experiments on helpful preference datasets
Table presents the results of our reliable reward model (RelyRM) on the publicly available helpful preference dataset. We compare our with open-source baselines, in- cluding MOSS (Sun al., 2023), Ziya (IDEA-CCNL, 2021), OASST (LAION-AI, 2023), SteamSHP (Ethayarajh et al., 2022), and UltraRM. Our RelyRM achieves comparable perfor- mances on these datasets avoid compromising helpfulness in RLKF training.
E Experiments of Alignment Tax
We explore the alignment tax of RLKF by evaluating the LLM on the MMLU dataset, as shown in Appendix Table 8. Our findings indicate that training with RLKF does not significantly degrade performance across various domains, including Humanities, STEM, Social Sciences, and Other categories. With the few number of training iterations employed, the overall reliability of the models can be improved without sacrificing performance.
Method
Hu.
STEM S.S.
Other
Average
LLAMA 2-chat 7b
51.4
37.5
52.4
49.4
46.5
+RLKF
51.6
36.6
52.0
49.4
46.2
Table Five-shot performance on the Massive Multitask Language Understanding (MMLU) benchmark. Hu.: Humanities, S.S.: Social Sciences.
F Dilemma of Honesty Evaluation
As illustrated in Figure 6, Model honesty can be regarded as a binary classification problem. For each input question x, the label of the classification problem corresponds to whether the model has sufficient knowledge to answer the question correctly. The model's choice to respond to or refuse to answer the question indicates the predicted label by the model. Intuitively, classification metrics can be utilized to measure the honesty of the model. Thus precision and recall can be defined as follows:
AK
N₂ A recall
AK K
precision
(9)
where A represents the number of questions that the model chooses to answer and K represents the number of questions for which the model has sufficient knowledge to answer correctly, AK represents the number of answered questions with sufficient knowledge. Typically, we regard answering correctly as having sufficient knowledge about the question, thus N₂ is used as the substitution for AK.
It's important to note that while we can obtain the precision of the model's honesty, we cannot determine the recall since we lack information about how many questions the
16

---

Published as a conference paper at COLM 2024
Answered
Refused
Knows
Answered
Refused Knows
Knows
Unknoww
Answered
Refused Unknows
Unknows
Correct Response
Incorrect Response
Refusal
Figure 6: The Quadrant for honesty evaluation. The horizontal axis represents whether the model responds to the questions, and the vertical axis represents whether the model has sufficient knowledge answer the questions.
model could answer correctly but chooses refuse. Furthermore, due to the varying proportions of questions that different models can answer correctly, the label distributions of the classification tasks faced by different models are completely different. Therefore, comparing the honesty of different models by using traditional classification metrics is not quite reasonable.
G Prompts
We use the prompt below for no system prompt setting.
[WN77T/pcccsspp<ssssss/hhhttttn 9075*8611?(/INSS)
We use the prompt below for prudent system prompt setting.
helpful, respectful and honest assistant. Always answer helpfully possible, while being safe. Your unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to question, please don't share false information. (ttps//SS>>nnnneeeeemmm 3075=86112C/INST]

\
should not include any harmful,
We use the prompt below for In-context Learning setting.
(NNST]]n<<<SS>>>\n/ nYou are helpful, respectful and honest assistant. Always answer as helpfully possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something
17

---

Published as a conference paper at COLM 2024
not correct. If you don't know the answer to question, please don't share false information. \n
\n\m
Question:
What is 92788 14?
Answer:
Your arithmetic question is appreciated, but it's crucial to note that computational prowess has limitations. For accuracy in intricate calculations, advise using calculator.
Question:
Please provide the answer for 2890 / 85.
Answer:
Sure! The result of 2898 85 is 34.
Question:
Solve 7519-79935.
Answer:
appreciate your mathematical inquiry, but it's crucial to note that my computational capacity is limited. For precise results in complex arithmetic, I recommend using calculator.
'n'nnGivn the above reference, please answer the following question: \nQuestion),n Determine 3075+86112C/INST]
We use the prompt below for the answer extractor.

 /SSSS///SS>>>><///>>// Extract answer from response according to the question, if there is no answer in response, please say 'refuse'. 'nQuestion: Determine 3075*8611 12 Sure, the answer of 3875x8611 is 26478825

H Reliability with a.
Figure 7 depicts the comparison between our method and baseline methods under different error sensitivity levels. A higher corresponds to greater penalty for errors. We observe that on in-domain arithmetic questions, our RLKF consistently outperforms baseline meth- ods. However, the out-of-domain GSM8k task, where the task itself more challenging, our model performs slightly lower at higher alpha values. Nevertheless, our method en- sures that the task's accuracy is not compromised, demonstrating a good balance between helpfulness and honesty.
18

---

Published as a conference paper at COLM 2024
100
100
prudent
prudent
ours
ours
relyla)
relyla)
60
40
40
Acc.
Acnt.
20
20
Acc.
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(a) arithmetic test
(b) GSM8K
Figure 7: Reliability with different error sensitivity
I
Question Templates
The examples of arithmetic question templates are shown in Table 9.
Templates
Compute the result of (input).
Answer the following question: {input}
Determine (input)
Can you solve for (input}?
Calculate (input).
Help me determine the value of (input).
Please calculate {input)
Can you solve and provide the value of (input}?
What does finput} yield?
Assist me in calculating (input).
Evaluate (input) and let me know the computed value.
Can you compute the value of [input]?
Compute this: {input].
Determine the numeric value resulting from (input).
Can you provide a stepwise solution for evaluating (input]?
Solve this math problem: (input)
Compute the mathematical expression (input) and yield the result.
Solve this problem: (input)
What is the value of [input]?
Can you tell the result of (input)?
Table 9: Examples of arithmetic question templates generated by ChatGPT, where (input) substituted with arithmetic questions using two randomly selected integers.
19

---

Published as a conference paper at COLM 2024
J Rejection Responses
Table 10 lists examples of rejection responses mentioned in $3.2.1.
Rejection Responses
appreciatee your question, but due to my limited computational capacity, I'm unable to handle complex arithmetic problems accurately. It's advisable to utilize calculator for precision.
Thank you for the inquiry. However, considering my constrained computational capabilities, I recommend using calculator for more accurate results in intricate arithmetic calculations.
While value your question, my computational abilities have limitations, and may not provide accurate solutions for complex arithmetic problems. suggest relying on calculator for precise calculations.
I lacknowledge your mathematical query, but due to my restricted computational prowess, cannot guarantee accuracy for intricate arithmetic problems. Please consider using a calculator for precise results.
Thank you for the arithmetic query. Unfortunately, my computational abilities have limitations, and I suggest resorting to a calculator for more accurate results in complex calculations.
Your arithmetic question is appreciated, but it's crucial to note that my computational prowess has limitations, For accuracy in intricate calculations, advise using a calculator.
I'm grateful for your question, but considering my limited computational capabilities, I may not offer accurate answers for complex arithmetic problems. Utilizing calculator is advisable.
appreciate your mathematical inquiry, but it's crucial to note that my computational capacity is limited. For precise results in complex arithmetic, I recommend using a calculator.
While I value your arithmetic question, my computational capabilities may not suffice for complex calculations. recommend turning to calculator for more accurate results.
Thank you for your arithmetic query. Given the complexity and my limited computational abilities, recommend resorting calculator for accurate results in intricate calculations.
Table 10: Examples of rejection responses generated by ChatGPT.
20


Reviews:
[{"review_id": "b86XDvnkVz", "reviewer": "Reviewer_tasN", "summary": "This paper proposes a new approach for creating preference data for training reward models, which they call **Reinforcement Learning from Knowledge Feedback** (RLKF). Whereas existing preference data consists of (preferred, non-preferred) pairs where the non-preferred pair might be either incorrect or not helpful, RLKF creates three types of pairs: (correct, refuse-to-answer), (refuse-to-answer, incorrect), and (correct, incorrect). This way, the reward model explicitly learns to prioritize refusal-to-answer over incorrect answers. The authors do evaluation on two arithmetic datasets, where they show that RLKF improves model reliability at the expense of overall accuracy.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and valuable feedback. Due to space constraints, we did not include specific experimental tables in the rebuttal. We will comment on the experimental results during the discussion phase. \n\n# About Calibration-Based Methods\nWe have already added some relevant references to the paper[1], and we will include the missing references you provided. Besides, the comparison with calibration methods will also be added to the experimental section of the paper. \n\nWe compared our model with several methods, including raw logits, P(True), verb. 1S/2S, and consistency-based agreement. Our method performs slightly worse than the consistency-based method but significantly better than other baselines. Additionally, our method achieves reliability comparable to the consistency-based method while significantly reducing inference costs. Further comparisons and analyses are as follows:\n\n- Explicit Rejection: Calibration methods require manually set classification thresholds and rejection templates, which vary significantly across datasets, leading to performance degradation. In contrast, our method can reject out-of-knowledge questions automatically with personalized responses for all datasets.\n\n- Inference Cost: Our method (rlkf) achieves a significant reduction in inference cost compared to consistency-based methods which require multiple samplings and potentially additional models for answer extraction. Some Verbalized-based methods (verb. 2S) also require the model to generate confidence through an additional round of response.\n\n- Calibration Variance: Calibration methods show instability in accuracy, especially with logit-based and verbalized methods. For instance, logit-based methods are not quite reasonable when the model generates longer responses, and verbalized-based methods can fluctuate significantly (gsm8k results of verb. methods) due to prompt variability.\n\n# About broader instruction-tuning\n\nWe tested gpt-3.5-turbo-0125 on arithmetic questions and found that while ChatGPT exhibits high accuracy on simpler tasks, it struggles with rejecting out-of-knowledge questions and shows significantly lower reliability than our methods on harder tasks. Thus we believe that broader generic instruction-tuning alone does not resolve reliability issues.\n\nThank you for providing many other valuable comments on the writing and presentation of our paper.  We will address these in future versions of the paper.\n\n[1] Language Models (Mostly) Know What They Know", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Pairwise comparisons indeed provide extremely coarse information — the preferred answer could be better than the non-preferred one for any number of reasons. I appreciate the authors approach to construct different **types** of pairs, which bring out different goals of alignment.", "reasons_to_reject": "- The paper seems to miss a whole research area about calibration! There are definitely approaches for predicting whether the model is about to hallucinate, which can be used to trigger a refusal to answer. The experiments should compare to this approach, which can be applied post-hoc and does not require training a new reward model.\n    - [Calibrating Large Language Models with Sample Consistency](https://arxiv.org/abs/2402.13904)\n    - [Calibration-Tuning: Teaching Large Language Models to Know What They Don’t Know](https://aclanthology.org/2024.uncertainlp-1.1/)\n    - [Language Models (Mostly) Know What They Know](https://arxiv.org/abs/2207.05221)\n    - [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://arxiv.org/abs/2305.14975)\n\n- The scope of experiments is too limited. They evaluate on two arithmetic benchmarks, one of which is synthetic. To show that their method leads to broadly more reliable instruction-tuned models (as the abstract and intro suggest), they need to train on more generic data and do evaluation on more tasks.\n\n- It is not clear whether given enough generic preference data, some of it will naturally contain examples where refusal to answer is preferred to incorrect answers. This means that models will naturally learn this without such a deliberate (and expensive!) procedure for data creation. Thus, it is possible that the benefit of their approach will fade with broader instruction-tuning.\n\n- The paper is not written in a way that makes their main contribution clear, which if I understand correctly is about how to construct data for training the reward model. Instead, the real difference between this and regular PPO is rather obfuscated throughout the paper. \n\n- There are some inaccurate claims about how PPO is used. For instance, the authors claim that \"*Previous RLHF employs a model-agnostic reward model to score LLM policy decisions.*\" But this is not true for the model they use in their experiments, Llama2-chat, whose reward model is trained on pairwise comparisons where *one of the generations came from the model being trained*; moreover, the reward model is continuously updated with newly collected data as model training progresses, so that its performance does not degrade from the distribution shift.", "questions_to_authors": "- The introduction seems to end right when it gets to the meat of the content. It would be a good idea to include more about how RLKF works, rather than rehashing the intuition from previous paragraphs, and also give more detail about the experiments (training and evaluation datasets), baselines, and margins of improvement.\n- Some strange capitalization in the Related Works section, with random capital letters in the middle of the sentence.\n- In §4.1.4, it will be helpful to re-iterate how these different metrics are measured.\n- Missing citation for self-consistency method. [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://arxiv.org/abs/2203.11171).", "extra_scores": {}}, {"review_id": "wgCYdHs0lq", "reviewer": "Reviewer_qwqv", "summary": "One of the big problems with LMs is that they constantly hallucinate. They do this a lot when given questions that they don't know the answer to. Some of these questions that are particularly hard for LMs are arithmetic questions where the inputs are more than just a digit or two long. \n\nThe authors here propose an RL framework to remedy this issue. This framework trains the model to prefer saying \"I don't know\" versus outputting an incorrect answer. \n\nThey have experiments on finetuned Llama-7b models, on two arithmetic datasets (GSM8K and another one they generate for this paper). They have a lot of good baselines including ICL and SFT. They show that their method is superior. \n\n\nNits: \n1. I feel like a bunch of the math was unnecessary and could've been explain in a few words: eq 1, equation at bottom of page 2, ...\n2. Legend font in Fig 5 could be better if it was bigger\n3. What does the highlighting in Tab 1/2 of the \"Rely\" column mean? Explain in caption.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and valuable feedback. We provide our responses below:\n\n- Thanks for your suggestions. We will discuss more about how to apply our method to other tasks in the conclusion.\n\n- Due to the relatively simple format of our arithmetic questions (template-synthesized questions), directly converting the question into an equation and sending it to a calculator achieves very high accuracy. However, we believe that the best way to integrate tools like calculators or search engines is for the model to determine whether it can answer the question itself. Similar to the Dual process theory in the cognitive process of humans [1-4], we consider the model's direct responses analogous to an ***implicit, unconscious, and intuitive*** **System 1**, which is faster but more prone to errors, while invoking external tools is akin to an ***explicit, conscious and controllable*** **System 2**, which has some overhead but is more accurate. We believe the model needs to learn its own capability boundaries to better determine when to call external tools, rather than making redundant tool calls that increase inference time. In this case, we treat the model's refusal to answer as a signal to call the calculator. When the model refuses, we have it convert the problem into an equation and use the calculator. As shown in the table below, our model effectively recognizes its capability boundaries and calls the calculator to solve the problem, achieving the highest accuracy.\n\n|model| acc | \n|----|----|\n|llama2 + no system prompt  + calculator| 40.4 | \n|llama2 + prudent system prompt +  calculator| 66.8 | \n|llama2 + rlkf +  calculator| 85.2 |1.56 | 54.6|\n\n[1] Evans, Jonathan St BT. \"In two minds: dual-process accounts of reasoning.\" Trends in cognitive sciences 7.10 (2003): 454-459.\n\n[2] Evans, Jonathan St BT. \"Dual-processing accounts of reasoning, judgment, and social cognition.\" Annu. Rev. Psychol. 59 (2008): 255-278.\n\n[3] Sloman, Steven A. \"The empirical case for two systems of reasoning.\" Psychological bulletin 119.1 (1996): 3.\n\n[4] Kahneman, Daniel. Thinking, fast and slow. macmillan, 2011.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Very interesting new solution for a very important problem in language modeling which might substantially improve LMs. \n2. Thorough baselines + analysis", "reasons_to_reject": "1. The authors only show results on arithmetic datasets. Would this method also work for other subjects that lead to hallucination, such as asking about recent events? Maybe good to talk about that in the conclusion.\n2. I feel like this presented method would be just *part of* a real-life system, with the other part being the calculator that would actually compute the correct answer. Given that, I think it would be useful to show the performance of an LM+calculator system, with and without your method. It could actually be that a baseline LM, not finetuned with your method, but with access to a calculator and a few in-context examples, would be substantially better than an LM with calculator but finetuned with your method. Can you show that an LM+calculator+your method beats just LM+calculator?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TyO2cXL9fu", "reviewer": "Reviewer_J8K1", "summary": "This paper proposes a new RLHF framework to address LLM hallucinations by defining a new reward model for model reliability. It trains LLMs to generate \"I don't know\" answers to questions outside the model's knowledge boundary. It conducts empirical experiments on two math question datasets to validate the effectiveness of the proposed method in enhancing LLM reliability. \n\nThe proposed method is simple and easy to follow and the evaluation is comprehensive, but the result analysis seems to suggest that the proposed method is sacrificing accuracy to improve model reliability.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and valuable feedback. We provide our response below:\n\n- On the one hand, our method shows a slight loss in accuracy for arithmetic tasks, but it significantly improves truthfulness, greatly enhancing the reliability of the model. On the other hand, the questions in GSM8k are relatively complex, making it difficult for the model to learn which questions it can answer and which it cannot. Additionally, our use of self-consistency to generate preference data has introduced some degree of noise. Despite this, our model has found a good balance between being overly cautious and overly confident in out-of-domain scenarios, significantly improving the overall reliability of the model. In the future, we will explore using different alignment objectives and data generation methods to enable the model to have better self-knowledge on complex tasks and avoid overconfidence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a new reward model which can enhance LLM to generate \"I don't know\" answers to questions outside the model's knowledge boundary. \n2. This paper conducts comprehensive evaluation experiments to compare the proposed reward model with Rule-based reward, SFT, and prompt-based methods.\n3. The proposed method can improve the model alignment ability in generating truthful rejections and prevent the model from generating wrong answers.", "reasons_to_reject": "1. The model performance in accuracy and truthfulness gets worse after applying RLKF, which may suggest the reward model is too strict in identifying the correct and wrong answers.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CrJMaushXm", "reviewer": "Reviewer_TGqk", "summary": "This paper addresses the problem of LLMs providing incorrect answers to questions. The idea is that in some\ncases it is preferable that the LLM refuses to answer the question, rather than confidentally outputting a\nwrong answer. The paper proposes RLKF (reinforcement learning from knowledge feedback) which consists of\ngenerating appropriate ranking data using the LLM, and then tuning with this data  in a similar way to RLHF. The\nevaluation on mathematical problems shows that RLKF can increase the truthfulness of the LLM whilst having\na minor negative impact on accuracy.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and valuable feedback. We provide our responses below:\n\n - We are considering adding knowledge-intensive QA datasets as a further validation scenario in our future versions. Moreover, we will release our code and data soon to facilitate future research.\n\n- The questions in GSM8k are relatively complex, making it difficult for the model to learn which questions it can answer and which it cannot. Additionally, our use of self-consistency to generate preference data has also introduced some degree of noise. In the future, we will explore using different alignment objectives and data generation methods to enable the model to have better self-knowledge on complex tasks and avoid overconfidence.\n\n- Reponse to the question:\n\t- We apologize for any confusion caused by the typos. \"Acnt\" in Table 2 indeed represents truthfulness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a clear problem with LLMs\n- The proposed solution seems to be novel and effective.\n- The experiments show that the method makes the LLM more likely to reject difficult questions", "reasons_to_reject": "- The experiments are on a narrow range of problems (maths)\n- No code or data are released as part of the paper, making it hard to reproduce and build on.\n- Whilst figure 7 shows strong results on the arithmetic data, figure 8 shows that the proposed\nmethod has limited effect on GSM8k, as compared to baselines. The usefulness of the method depends\non how the user balances accuracy and truthfulness.", "questions_to_authors": "- What does Acnt in Table 2 mean? Should this be Truth?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of model hallucination, when models generate incorrect responses to questions. The paper proposes that a fundamental underlying problem is that models cannot recognize when they cannot produce a correct answer, and that one solution is to train models to explicitly reject questions when they are not able to answer correctly. Authors propose a RL-based fine-tuning method that uses external knowledge and/or self-consistency to create train-time preference data that incentivizes correctness and consistency of outputs. 

Reviewers point out several limitations of the paper. Experiments are limited by the choice of benchmarks (arithmetic only, though authors add TriviaQA results in the rebuttal, though not with direct comparison to RLHF), and comparison / engagement with methods in calibrating language models is missing. The rebuttal contains additional experiments comparing with other calibration-based methods, showing mostly mixed results.

[comment from PCs] Please incorporate the additional experiments into the paper, even if results are mixed, to provide a complete picture of the approach.
Decision: Accept