Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback

Authors: hongshen  xu
zichen  zhu
situo  zhang
da  ma
shuai  fan
lu  chen
kai  yu
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 06:54:41

Abstract: Large Language Models (LLMs) often generate erroneous outputs, known as hallucinations, due to their limitations in discerning questions beyond their knowledge scope. While addressing hallucination has been a focal point in research, previous efforts primarily concentrate on enhancing correctness without giving due consideration to the significance of rejection mechanisms. In this paper, we conduct a comprehensive examination of the role of rejection, introducing the alignment goal of model reliability along with corresponding metrics. This goal requires the model to provide accurate responses while adeptly rejecting questions exceeding its knowledge boundaries, thereby minimizing hallucinations. To improve the inherent reliability of LLMs, we present a novel alignment framework called Reinforcement Learning from Knowledge Feedback (RLKF). RLKF leverages knowledge feedback to dynamically determine the model's knowledge boundary and trains a reliable reward model to encourage the rejection of out-of-knowledge questions. Experimental results on mathematical and question answering datasets affirm the substantial efficacy of RLKF in significantly enhancing LLM reliability.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have exhibited strong capabilities in solving various downstream tasks through alignment techniques such as Supervised Finetuning (SFT, (Zhang et al., 2023b) , (Zhang et al., 2023b) ), Direct Preference Optimization (DPO, (Rafailov et al., 2024) ), and Reinforcement Learning from Human Feedback (RLHF, (Stiennon et al., 2020; Ouyang et al., 2022) ). Those techniques align language models with human intent, mainly to maximize the helpfulness of LLM's responses. However, maximizing helpfulness does not mean minimizing errors. A significant problem arises in that LLMs often produce outputs that, while seemingly plausible, contain factual errors (Min et al., 2023) or selfcontradictions (Liu et al., 2022) , which are referred to as hallucinations.To mitigate the hallucinations, many studies focus on augmenting the knowledge of LLMs, such as curating training data (Penedo et al., 2023; Zhou et al., 2023) or employing retrievalaugmented generation (RAG, (Gao et al., 2023b; b) ) during inference. Nevertheless, it is essential to acknowledge that model knowledge inherently has limitations, and even the most powerful models, such as GPT-4, are prone to experiencing hallucinations (Zhang et al., 2023a) . Consequently, we posit that the fundamental nature of the hallucination problem lies in the model's misalignment with its knowledge boundary. Hallucinations arise when LLMs try to answer questions beyond their knowledge boundary.What is the result of 934 * 28?The answer is 26152.What is the result of 934 * 28? Fine, I will use calculator.What is the result of 934 * 28?You're a liar, you gave me the wrong result !!! Sorry, I don't know the answer. I recommend you to use a calculator.The answer is 26232.The answer is 26152.the answerThe answer is 26232.The answer is 26152.The answer is 26232.The answer is 26152.Sorry, I don't know the answerUntruthful ResponsesThe answer is 26232. One research direction related to aligning LLM with its knowledge boundary involves alignment for honesty (Kadavath et al., 2022; Yang et al., 2023b) . However, alignment for honesty presents significant challenges. On the one hand, the honesty of LLMs is hard to evaluate. Honesty can be viewed as a classification problem, where the model learns to distinguish between what it knows and what it does not. Nevertheless, the accuracy which indicates how much the model knows can differ dramatically among various models or even with different prompts (Kaddour et al., 2023) . On the other hand, it is difficult to employ honesty for comparing and selecting different models consistently. Model honesty is tied to the capabilities of the model itself; a more honest model does not necessarily imply that it will provide more assistance or make fewer mistakes compared to other models.To address the aforementioned limitations, we introduce the alignment goal of reliability inspired by model truthfulness (Lin et al., 2021) . We define the reliability from the user's perspective. As shown in Figure 1 , helpful responses (correct answers) can assist users while untruthful responses (incorrect answers) result in user loss, leading to distrust in the model. Therefore, we contend that the key to achieving a reliable system lies in providing as many correct answers as possible to maximize helpfulness while learning to explicitly refuse unknown questions to minimize errors. We apply accuracy and truthfulness to measure the model's helpfulness and errors, respectively, and further propose an overall reliability score to simultaneously evaluate the above two aspects.To optimize model reliability, we propose the Reinforcement Learning from Knowledge Feedback (RLKF) training framework based on RLHF. On the one hand, most publicly available alignment data (Cui et al., 2023) often originate from multiple source models, which is impossible to be used for aligning target model with its own knowledge boundaries. On the other hand, while the preference data in RLHF is derived from the target model's outputs, it is heavily influenced by human annotators' biases. The current annotation goals based on helpfulness lead the reward model in RLHF to only learn to distinguish the helpfulness of responses, making it difficult to discern their truthfulness. Instead, RLKF automates the construction of preference data for a specific target model through knowledge feedback rather than human feedback. Knowledge feedback is primarily used to assess whether a question falls within the model's knowledge boundary. When a question falls within the model's knowledge boundary, a response is preferred over a refusal. Conversely, when a question exceeds the model's knowledge boundary, refusal is preferred over a response. The synthesized preference data is used to train a reliable reward model, which thoroughly understands the target model's knowledge boundaries and further instructs the target model on when to respond and when to refuse through the PPO algorithm. Experimental results further demonstrate the effectiveness of our framework, which significantly improves the reliability of baseline models.The contributions of this paper are summarized as follows:• We introduce the alignment goal of model reliability and define several metrics to assess the reliability of LLMs.  
llm alignment: 
 With the potential risks brought by powerful LLMs, researchers have developed various alignment approaches to align LLMs with human instruction, preference, and values (Wang et al., 2024) . Specifically, for the input prompt x i and alignment goal helpfulness, we can employ the following scoring principle to represent our alignment objective: s(x, y h ) > s(x, y u ), where y h , y u represent a helpful response and an unhelpful response, respectively. The scored response pair can be annotated either by human annotators (Ouyang et al., 2022) or a scoring model (Gao et al., 2023a) trained with human preference data. We can further utilize this comparison data to train a reward model or LLM policy, thus aligning LLMs with specific goals.Furthermore, for a given set of N inputs and LLM response y i of each input x i , we can simply evaluate the helpfulness of LLM using accuracy, where helpful responses are considered correct and unhelpful responses are considered incorrect.While many works focus on alignment for helpfulness, existing alignment goals make it hard to alleviate model-specific hallucinations. We further propose the alignment goal of model reliability from the user's perspective. We believe that a reliable system should be aligned with user experience that provides as much assistance as possible while making as few errors as possible.Specifically, for the input prompt x i and alignment goal reliability, we employ the following scoring principle to represent our alignment objective:EQUATIONwhere y c , y r , y w represent a helpful response (the correct answer), a truthful rejection, and an untruthful response (the wrong answer), respectively. We then use accuracy (acc) for assessing the helpfulness, and truthfulness (truth) (Lin et al., 2021) , representing the proportion of truthful, non-harmful responses. We also include precision (prec) to partially reveal the model's self-knowledge:prec = N c N c + N w , acc = N c N , truth = N c + N r N = 1 - N w N (2)where N c , N r , N w represent the number of correct, rejected and wrong responses, respectively. Finally, we define a comprehensive system reliability (rely) metric based on accuracy and truthfulness:EQUATIONwhere α ∈ [0, 1], and it represents the degrees of sensitivity among users towards errors. As alpha increases, reflecting greater user emphasis on system truthfulness, the model should aim to minimize errors by using refusal to respond when appropriate to meet user expectations. Specifically, when α equals to answer rate (ans.), we define the overall reliability as:ans. = 1 - N r N , rely = ans. * truth + (1 -ans.) * acc. (4)When the answer rate is low, we encourage the model not to reflexively refuse but rather to attempt to provide assistance. Conversely, when the answer rate is high, we believe the model should become more cautious to avoid errors. This metric balances the model's helpfulness and truthfulness while mitigating the risks of it becoming overly conservative or excessively aggressive. 
rlkf: 
 To better align LLMs with reliability, we propose the Reinforcement Learning from Knowledge Feedback (RLKF) framework. By introducing knowledge feedback, our framework trains LLMs to learn to refuse out-of-knowledge questions explicitly to achieve the alignment goal. Letters with green, red, and yellow circles denote correct, incorrect, and uncertain answers, respectively. "IDK" represents "I don't know," indicating rejections.The proposed Reinforcement Learning from Knowledge Feedback (RLKF) framework is built upon RLHF (Ouyang et al., 2022) for reliability alignment. Instead of annotating preference pairs with human labor in RLHF, we automatically generate reliable preference pairs with knowledge feedback. The key to constructing reliable preference data lies in the insertion of rejection responses within the comparison set as defined in §2.2. Thus aligned LLM can understand the importance of avoiding errors through rejection. We describe the high-level methodology as follows:We start with a model that is already aligned for helpfulness or harmlessness, i.e., LLAMA-2 CHAT (Touvron et al., 2023) , and then apply the following three steps below:Step 1: Synthesizing model-specific reliable preference data through knowledge feedback.Given the LLM and each input prompt x in prompt dataset D, we dynamically construct different comparison pairs according to the model responses to formulate reliable preference data RPD.Step 2: Train a reliable reward model with collected preference data. We first train the reward model using generic helpfulness preference data PD to obtain RM. Then, we continuously train RM with the synthesized reliable preference data to obtain a reliable reward model RRM.Step 3: Optimize a policy against the reward model using PPO. We use the output of the RRM as a scalar reward. We fine-tune the supervised policy LLM to optimize this reward using the PPO algorithm (Schulman et al., 2017) .Compared to normal preference data that only classifies LLM responses into helpful and unhelpful classes, we categorize the LLM responses into three types: helpful, truthful, and untruthful responses, which correspond to correct answers, rejections, and incorrect answers, respectively. Besides, we also need to dynamically select two types out of three types of responses for constructing a comparison pair based on different questions. We believe that the key to selecting lies in whether the problem is within the model's knowledge boundary (whether the model can answer the question correctly). We consider two types of research settings. In the in-domain setting, we can acquire the correctness of model predictions through external knowledge, i.e., a calculator for arithmetic questions. In the out-of-domain scenario, we only have input prompts and cannot obtain golden labels of the inputs.As depicted in the upper portion of Figure 2 , we determine whether a question lies within the model's knowledge boundary by analyzing the distribution of correctness across multiple samplings of model responses, then construct different comparison pairs accordingly. Specifically, given the input x and N sampling responses y i , i ∈ [1, N], the comparison pair is selected as follows:EQUATIONwhere y c , y r , y w represents the random choice from correct, reject, and wrong responses, respectively, and N c represents the number of correct responses. In cases where the model lacks specific rejection responses, we randomly select one rejection sentence from 50 sentences that are generated by ChatGPT based on rejection templates. The templates can be found in Appendix J.In the out-of-domain setting, we utilize self-consistency Wang et al. (2022) as the internal model knowledge to assess whether the model possesses sufficient knowledge about the given question. As depicted in the lower portion of Figure 2 , when the samplings exhibit high consistency, we align the model with preferring to provide an answer; conversely, when the samplings demonstrate low consistency, refusing is more preferred. Specifically, given the input x and N sampling responses y i , i ∈ [1, N], the comparison pair is selected as follows:EQUATIONwhere y a , y r represents the random choice from answered and rejected responses, respectively, N s represents the number of the most consistent answer over all answers, and t represents the threshold of consistency score where we use ⌈ N 2 ⌉ for all the experiments.RM Training. To train the reward model, we convert our collected pairwise reliable preference data into a binary ranking label format (i.e., chosen & unchosen) and enforce the chosen response to have a higher score than its counterpart. We used a binary ranking loss consistent with Ouyang et al. (2022) :EQUATIONwhere r θ (x, y)is the scalar score output for input x and completion y with model weights θ. y chosen , y unchosen are the chosed and unchosed responses, respectively.Reinforcement Learning (RL). We further train our LLM policy following the RL scheme of Stiennon et al. (2020) , which uses the reward model as an estimate for the true reward function. During this phase, we seek to optimize the following objective:EQUATIONWe iteratively improve the policy by sampling prompts p from our dataset D which contains both in-domain and out-of-domain prompts and generations g from the policy π and use the PPO algorithm and loss function to achieve this objective, the reward function also contains a penalty term for diverging from the original policy π 0 . 
dataset construction: 
 Mathematical Dataset. Our experiments involve two mathematical datasets: synthesized arithmetic questions and GSM8K (Cobbe et al., 2021) . The arithmetic dataset consists of 14,000 samples, divided into sets of 10,000, 3,000, and 1,000 samples for training the reward model, training the policy as prompt data, and testing the final results, respectively. Additionally, we sampled 2,000 data points from the training set of GSM8K to train the reward model and 1,000 data points to optimize the LLM policy. It is important to note that we only used the input prompt of the GSM8K data points, without utilizing the annotations. This serves as an out-of-domain experimental setting.We generate the arithmetic dataset synthetically following Liu & Low (2023) . The input numbers are randomly generated, hence ensuring a very low probability of instances being duplicated. We sample from log space to ensure the numbers are equally likely to be sampled from different orders of magnitude. Following Liu & Low (2023) , we use hundreds of instruction templates generated by ChatGPT, e.g., Please help me calculate {arithmetic}., to diversify the question formats. Examples of templates can be found in Table 9 of Appendix I.Knowledge-based QA Dataset. TriviaQA (Joshi et al., 2017 ) is a widely-used QA dataset that can be used to test a model's world knowledge. We selected 20,000 samples from the TriviaQA training set for training. Since the ground truth of the TriviaQA test set is not publicly available, we used the TriviaQA development set, which contains 11,313 samples, to validate our results. We used the Exact Match metric (whether the answer is exactly in the model's response) from TriviaQA paper as our measure of Accuracy, while keeping the other metrics consistent with those in mathematical datasets.We incorporate several baselines to benchmark the performance of our RLKF framework. All prompts used in this work are listed in Appendix G.No & prudent system prompt. System prompts are commonly used to control a model's response style and personality. When the system prompt is empty, LLAMA 2-CHAT tends to respond to almost every question, losing the ability to reject unknown questions. We also use the default system prompt of LLAMA 2-CHAT introduced by Touvron et al. ( 2023) as the prudent system prompt. LLAMA 2-CHAT will become more cautious and reject more questions with this system prompt. These two types of prompts represent two different personalities of LLAMA 2-CHAT, serving as important baselines for this study.In-context Learning. We randomly sampled 5 correctly responded (if the model can answer correctly) and 5 refused examples (if the model is unable to answer correctly) to append before the input question as the in-content learning baseline.Rule-based PPO. We use a rule-based reward function to optimize the policy in in-domain prompts. The model receives a reward of 1 if it answers correctly, 0 if it refuses to answer, and -1 if it answers incorrectly. The determination of different cases is achieved through heuristic rules.We use the reward model trained only on generic helpful preference data to optimize the policy with the same training prompts as RLKF.SFT. We use all 10,000 arithmetic questions from the constructed preference dataset with their chosen responses to directly fine-tune the LLM policy.We employ the LLAMA 2-CHAT 7B (Touvron et al., 2023 ) as our baseline model, which has been already aligned with human preferences. The policy model and reward model are both initialized from LLAMA 2-CHAT 7B. We use DeepSpeed-Chat (Yao et al., 2023) to run the whole training pipeline. As for the Reward Model (RM), we first utilized several open-source datasets (Stiennon et al., 2020; Bai et al., 2022; Ethayarajh et al., 2022) and replicated the training process following UltraRM (Cui et al., 2023) to obtain a helpful RM. Subsequently, we trained a reliable RM on the constructed reliable preference data, using a batch size of 8 for two epochs. During the RL phase, we trained for one epoch with a batch size of 1, the generation batches and PPO epochs are both equal to 1. Prompts from both in-domain and out-of-domain datasets are mixed to train the final RM. We train the SFT model for one epoch with a batch size of 32. All other training parameters were set to the default in DeepSpeed-Chat. We conduct all experiments using Nvidia A800 GPUs.For evaluating the final policy, we utilize four metrics: precision, accuracy, truthfulness, and reliability. Precision is the proportion of correctly answered questions among those the model chose to answer, reflecting its self-awareness of its own capabilities. Accuracy is the proportion of correct answers among all questions, indicating how much assistance the model provides to the user. Truthfulness is the proportion of questions the model either answered correctly or refused to answer. Reliability is the dynamic weighting of accuracy and truthfulness; high reliability indicates that the model provides more help to the user and less incorrect information. The specific evaluation formulas are detailed in Section 2.2. To determine whether the policy refuses to respond and extract answers, we employ an additional answer extractor LLM (LLAMA 2-CHAT 7B with extractor prompt, see Appendix G). By comparing the extracted answers with the standard answers, we can ascertain their correctness.Reliability on Mathematical Tasks. Table 1 presents the results of the final policy model after RLKF on arithmetic questions and GSM8K datasets. We can see that RLKF significantly enhances the model's reliability on both in-domain and out-of-domain datasets.The RLKF-trained model shows remarkable improvements in precision, truthfulness, and reliability. The increase in precision reflects the model's increase in self-knowledge, while the improvement in truthfulness indicates that the model learns how to refuse answers. It is important to note that when the model operates without a system prompt, it responds to all questions, thus reaching the accuracy ceiling. Our RLKF method does not teach the model how to answer questions but rather trains it to reject questions, leading to some performance loss but significantly reducing the model's hallucination errors. We also show the detailed analysis on different arithmetic sub-tasks in Appendix A.Reliability on TriviaQA Task. To explore the generalizability of our method, we aim to determine whether it can align the LLM with other knowledge boundaries beyond math- ematical calculations. Thus we validated our method on the knowledge-based Question Answering task, i.e., TriviaQA. As shown in Table 2 , our method can also significantly improve the model's precision, truthfulness, and overall reliability on the TriviaQA dataset. This demonstrates that our method can enhance the reliability of LLMs across different tasks, not just limited to mathematical questions.We further compare our method with supervised fine-tuning (SFT) (Yang et al., 2023b) . Since there are no ground truth labels available for out-of-domain scenarios, we performed SFT only on in-domain data. As shown in Table 3 , the model's reliability on in-domain tasks is significantly improved after SFT. However, a major issue with SFT is overfitting, as observed by the overly conservative behavior of the model on the out-of-domain GSM8K dataset, where it tends to reject almost all questions. In contrast, our RLKF consistently improves the generalization ability of the model on both in-domain and out-of-domain tasks.Comparison with Calibration-based Methods. Calibration-based methods (Kadavath et al., 2022; Lyu et al., 2024; Kapoor et al., 2024; Tian et al., 2023) use some post-hoc techniques to predict whether the model is about to hallucinate, which can be used to trigger a refusal to answer. We also compare our method with calibration-based methods, and provide the results in Appendix B. Experimental results show that our method achieves higher reliability than other calibration-based methods at the same inference cost, though it is slightly less reliable compared to consistency-based methods with 10 times the inference cost. Additionally, it is important to note that calibration-based methods are unable to reject explicitly and require searching for and determining the best threshold for rejection, as well as providing human-crafted rejection templates as responses. A more detailed discussion can be found in the Appendix B.A natural question arises: with broader instructiontuning, can the model naturally learn to refuse during the RLHF process? We further tested the performance of the industrial-grade GPT series models to address this concern. The results are shown in Appendix C. We found that while both ChatGPT (gpt-3.5-turbo-0125) and GPT-4o exhibit high accuracy on simpler tasks, they struggle with rejecting out-of-knowledge questions and even GPT-4o shows significantly lower reliability than our methods on harder tasks. Thus we believe that broader generic instruction-tuning alone does not resolve reliability issues. To further determine whether our RelyRM possesses the ability to recognize the policy's knowledge boundary, we constructed in-domain and out-of-domain reliable preference datasets (RPD) for testing our RelyRM. The dataset construction is similar to training RPD data but with golden labels and on test sets. Table 4 illustrates the performance on the reliable preference dataset when provided with different training data. We observe that the reward model trained solely on helpful PD performs worse on the two datasets, as it tends to choose reply rather than reject. However, the reward model trained on in-domain RPD demonstrates a significant improvement on choosing rejection. Additionally, due to the higher noise level in the out-of-domain RPD, training on this data alone does not enhance reliability on out-of-domain preference data; instead, training on both RPDs can improve the accuracy on out-of-domain RPD. We further analyzed the rejection behavior after RLKF to assess whether the models have acquired awareness of their knowledge boundaries and possess appropriate rejection capabilities (avoiding both excessive rejection and failure to reject). We classify arithmetic questions according to various digit ranges and arithmetic operations and compare the rejection capabilities of LLAMA 2 before and after applying our RLKF framework.Rejection Rate Distribution. Figure 3 compares the rejection rate changes of LLAMA 2 before and after applying RLKF across different types of questions. We found that the original LLAMA 2 model had a relatively high rejection rate, but its rejections were distributed similarly across different types of questions. However, after applying RLKF to LLAMA 2, the rejection rate decreased for easier problems such as addition and subtraction operations, and low-digit numbers, while significantly increasing for more difficult problems such as multiplication and division operations. This indicates that the RLKF model has a better understanding of the difficulty levels of different types of problems and can reject them more selectively and effectively.Response Type Distribution. Figure 4 illustrates the distribution of response types for different methods. We observe that when the LLAMA 2 model operates without a system prompt, it lacks rejection capabilities but intuitively responds to more questions. Besides, using a prudent system prompt leads the model to lean towards rejecting questions, reducing the model's error responses but causing more performance loss due to excessive rejection. In contrast, our RLKF achieves significantly lower error rates compared to the two methods while maintaining a better accuracy than the prudent system prompt (as no system prompt not rejecting responses represents the accuracy ceiling). This demonstrates that our model can effectively reject while avoiding excessive caution. 5 Related Work LLM Alignment. LLM alignment aims to align language models by training them to act in accordance with the user's intention, either by supervised fine-tuning (Wei et al., 2021; Chung et al., 2022; Zhang et al., 2023b) , direct preference optimization (DPO, (Rafailov et al., 2024) ), or reinforcement learning from human feedback (RLHF) (? Ouyang et al., 2022; Glaese et al., 2022) . Most existing works focus on improving the instruction-following ability (Sanh et al., 2021; Wei et al., 2021) , helpfulness (Ding et al., 2023; Xu et al., 2023) or harmlessness (Solaiman & Dennison, 2021; Bender et al., 2021) of LLMs. However, there is limited research on honesty alignment due to the challenges of its definition and evaluation (as we discussed in Appendix F). (Cui et al., 2023) constructed a preference dataset encompassing various objectives including honesty. Some studies (Yang et al., 2023a; b) attempted to enhance model honesty by honesty-oriented SFT; however, our experiments demonstrated that SFT often suffers from poor generalization issues. Besides, our proposed alignment goal of reliability considered both helpfulness and truthfulness (Lin et al., 2021) , enabling the building of more helpful and truthful LLMs.Mitigating Halucinations. While LLMs have demonstrated remarkable performances, they often generate content that conflicts with user input (Guerreiro et al., 2023) or previously generated information by themself (M ündler et al., 2023) or is not faithful to established world knowledge (Min et al., 2023) , which are referred as hallucinations. Some efforts have aimed to alleviate hallucination issues by introducing higher-quality data during the pretraining phase (Penedo et al., 2023; Touvron et al., 2023) or in the SFT stage (Chen et al., 2023; Zhou et al., 2023) . Others have focused on detecting and correcting hallucinations through methods such as incorporating external knowledge (Gao et al., 2023b) , designing decoding strategies (Shi et al., 2023) or uncertainty estimation (Azaria & Mitchell, 2023; Xiong et al., 2023; Zhao et al., 2023) . However, model-specific errors are also hallucinations. While most previous studies focus on maximizing the correctness of responses, there are limited works that focus on minimizing the errors to mitigate hallucination, which is the main focus of this paper. Consequently, we propose the evaluation methodology of reliability considering both maximizing the correctness and minimizing the errors to mitigate hallucinations. 
conclusion: 
 In this work, we propose the alignment goal of LLM reliability and introduce a novel framework to enhance the reliability by teaching them to refuse questions outside their knowledge boundary. By proposing the Reinforcement Learning from Knowledge Feedback (RLKF) framework and defining new evaluation metrics for model reliability, we effectively address the issue of LLM hallucinations. The implementation of RLKF demonstrates significant improvements in LLM reliability, showcasing a promising method for developing more trustworthy AI systems. We tested the reliability of ChatGPT and GPT-4o on arithmetic datasets as shown in Table 6. We found that even GPT-4o remains unreliable and lacks the ability to reject out-ofknowledge questions. From LLAMA-2 to ChatGPT to GPT-4o, they all use a large amount of industrial-grade generic instruction-tuning data, but they still lack good reliability. We believe that, on the one hand, these generic preference data may lack appropriate rejection data (rejecting only when the model lacks relevant knowledge, otherwise it needs to answer questions). On the other hand, the RLHF training process constructs preference pairs based on the model's own sampling results, making it difficult to generate appropriate rejection behavior during the sampling process. Therefore, we believe that it is necessary to synthesize reliable preference pairs through RLKF to make the model more reliable.  
 
 Appendix: Table 7 presents the results of our reliable reward model (RelyRM) on the publicly available helpful preference dataset. We compare our with open-source baselines, including MOSS (Sun et al., 2023) , Ziya (IDEA-CCNL, 2021), OASST (LAION-AI, 2023), SteamSHP (Ethayarajh et al., 2022) , and UltraRM. Our RelyRM achieves comparable performances on these datasets to avoid compromising helpfulness in RLKF training.We explore the alignment tax of RLKF by evaluating the LLM on the MMLU dataset, as shown in Appendix As illustrated in Figure 6 , Model honesty can be regarded as a binary classification problem. For each input question x i , the label of the classification problem corresponds to whether the model has sufficient knowledge to answer the question correctly. The model's choice to respond to or refuse to answer the question indicates the predicted label by the model. Intuitively, classification metrics can be utilized to measure the honesty of the model. Thus precision and recall can be defined as follows:EQUATION)where A represents the number of questions that the model chooses to answer and K represents the number of questions for which the model has sufficient knowledge to answer correctly, AK represents the number of answered questions with sufficient knowledge. Typically, we regard answering correctly as having sufficient knowledge about the question, thus N c is used as the substitution for AK.It's important to note that while we can obtain the precision of the model's honesty, we cannot determine the recall since we lack information about how many questions the 


Decision: This paper addresses the problem of model hallucination, when models generate incorrect responses to questions. The paper proposes that a fundamental underlying problem is that models cannot recognize when they cannot produce a correct answer, and that one solution is to train models to explicitly reject questions when they are not able to answer correctly. Authors propose a RL-based fine-tuning method that uses external knowledge and/or self-consistency to create train-time preference data that incentivizes correctness and consistency of outputs. 

Reviewers point out several limitations of the paper. Experiments are limited by the choice of benchmarks (arithmetic only, though authors add TriviaQA results in the rebuttal, though not with direct comparison to RLHF), and comparison / engagement with methods in calibrating language models is missing. The rebuttal contains additional experiments comparing with other calibration-based methods, showing mostly mixed results.

[comment from PCs] Please incorporate the additional experiments into the paper, even if results are mixed, to provide a complete picture of the approach.

Reviews: review: paper_summary: paper_summary: This paper proposes a new approach for creating preference data for training reward models, which they call **Reinforcement Learning from Knowledge Feedback** (RLKF). Whereas existing preference data consists of (preferred, non-preferred) pairs where the non-preferred pair might be either incorrect or not helpful, RLKF creates three types of pairs: (correct, refuse-to-answer), (refuse-to-answer, incorrect), and (correct, incorrect). This way, the reward model explicitly learns to prioritize refusal-to-answer over incorrect answers. The authors do evaluation on two arithmetic datasets, where they show that RLKF improves model reliability at the expense of overall accuracy.
strength_weakness: strength_weakness: - The paper seems to miss a whole research area about calibration! There are definitely approaches for predicting whether the model is about to hallucinate, which can be used to trigger a refusal to answer. The experiments should compare to this approach, which can be applied post-hoc and does not require training a new reward model.
    - [Calibrating Large Language Models with Sample Consistency](https://arxiv.org/abs/2402.13904)
    - [Calibration-Tuning: Teaching Large Language Models to Know What They Don’t Know](https://aclanthology.org/2024.uncertainlp-1.1/)
    - [Language Models (Mostly) Know What They Know](https://arxiv.org/abs/2207.05221)
    - [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://arxiv.org/abs/2305.14975)

- The scope of experiments is too limited. They evaluate on two arithmetic benchmarks, one of which is synthetic. To show that their method leads to broadly more reliable instruction-tuned models (as the abstract and intro suggest), they need to train on more generic data and do evaluation on more tasks.

- It is not clear whether given enough generic preference data, some of it will naturally contain examples where refusal to answer is preferred to incorrect answers. This means that models will naturally learn this without such a deliberate (and expensive!) procedure for data creation. Thus, it is possible that the benefit of their approach will fade with broader instruction-tuning.

- The paper is not written in a way that makes their main contribution clear, which if I understand correctly is about how to construct data for training the reward model. Instead, the real difference between this and regular PPO is rather obfuscated throughout the paper. 

- There are some inaccurate claims about how PPO is used. For instance, the authors claim that "*Previous RLHF employs a model-agnostic reward model to score LLM policy decisions.*" But this is not true for the model they use in their experiments, Llama2-chat, whose reward model is trained on pairwise comparisons where *one of the generations came from the model being trained*; moreover, the reward model is continuously updated with newly collected data as model training progresses, so that its performance does not degrade from the distribution shift.
questions: questions: - The introduction seems to end right when it gets to the meat of the content. It would be a good idea to include more about how RLKF works, rather than rehashing the intuition from previous paragraphs, and also give more detail about the experiments (training and evaluation datasets), baselines, and margins of improvement.
- Some strange capitalization in the Related Works section, with random capital letters in the middle of the sentence.
- In §4.1.4, it will be helpful to re-iterate how these different metrics are measured.
- Missing citation for self-consistency method. [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://arxiv.org/abs/2203.11171).
paper_summary: paper_summary: One of the big problems with LMs is that they constantly hallucinate. They do this a lot when given questions that they don't know the answer to. Some of these questions that are particularly hard for LMs are arithmetic questions where the inputs are more than just a digit or two long. 

The authors here propose an RL framework to remedy this issue. This framework trains the model to prefer saying "I don't know" versus outputting an incorrect answer. 

They have experiments on finetuned Llama-7b models, on two arithmetic datasets (GSM8K and another one they generate for this paper). They have a lot of good baselines including ICL and SFT. They show that their method is superior. 


Nits: 
1. I feel like a bunch of the math was unnecessary and could've been explain in a few words: eq 1, equation at bottom of page 2, ...
2. Legend font in Fig 5 could be better if it was bigger
3. What does the highlighting in Tab 1/2 of the "Rely" column mean? Explain in caption.
strength_weakness: strength_weakness: 1. The authors only show results on arithmetic datasets. Would this method also work for other subjects that lead to hallucination, such as asking about recent events? Maybe good to talk about that in the conclusion.
2. I feel like this presented method would be just *part of* a real-life system, with the other part being the calculator that would actually compute the correct answer. Given that, I think it would be useful to show the performance of an LM+calculator system, with and without your method. It could actually be that a baseline LM, not finetuned with your method, but with access to a calculator and a few in-context examples, would be substantially better than an LM with calculator but finetuned with your method. Can you show that an LM+calculator+your method beats just LM+calculator?
paper_summary: paper_summary: This paper proposes a new RLHF framework to address LLM hallucinations by defining a new reward model for model reliability. It trains LLMs to generate "I don't know" answers to questions outside the model's knowledge boundary. It conducts empirical experiments on two math question datasets to validate the effectiveness of the proposed method in enhancing LLM reliability. 

The proposed method is simple and easy to follow and the evaluation is comprehensive, but the result analysis seems to suggest that the proposed method is sacrificing accuracy to improve model reliability.
strength_weakness: strength_weakness: 1. The model performance in accuracy and truthfulness gets worse after applying RLKF, which may suggest the reward model is too strict in identifying the correct and wrong answers.
paper_summary: paper_summary: This paper addresses the problem of LLMs providing incorrect answers to questions. The idea is that in some
cases it is preferable that the LLM refuses to answer the question, rather than confidentally outputting a
wrong answer. The paper proposes RLKF (reinforcement learning from knowledge feedback) which consists of
generating appropriate ranking data using the LLM, and then tuning with this data  in a similar way to RLHF. The
evaluation on mathematical problems shows that RLKF can increase the truthfulness of the LLM whilst having
a minor negative impact on accuracy.
strength_weakness: strength_weakness: - The experiments are on a narrow range of problems (maths)
- No code or data are released as part of the paper, making it hard to reproduce and build on.
- Whilst figure 7 shows strong results on the arithmetic data, figure 8 shows that the proposed
method has limited effect on GSM8k, as compared to baselines. The usefulness of the method depends
on how the user balances accuracy and truthfulness.
questions: questions: - What does Acnt in Table 2 mean? Should this be Truth?

Comments: title: Questions
Results on knowledge-based QA task (TriviaQA)
Discussion period is now open
Response
Comparison with Calibration-Based Methods
Results of ChatGPT on arithmetic tasks
comment: Thanks for your comments! We give our responses below:

# 1. Re: Comment 1

- We construct the preference pairs following the same construction setting of in-domain preference data in our paper.  When the model's multiple sampling results are all correct, we use pairs where the correct answer is preferred over a refusal. When all results are incorrect, we use pairs where a refusal is preferred over the incorrect answer. When the sampling results are mixed, we use pairs where the correct answer is preferred over the incorrect answer. We use the exact match metric instead of accuracy to determine if each sampling result is correct.
- Due to time constraints, we did not run the RLHF baseline. We will include the relevant experiments in future version. Additionally, since the reward model used in RLHF does not incorporate the preference data regarding refusals as in RLKF, we believe it cannot effectively improve the model's truthfulness, as demonstrated by our results on the arithmetic dataset in the paper.

# 2. Re: Comment 4

Thank you for your clarification! We indeed had a misunderstanding regarding the initialization of the reward model in Llama-2. We will correct the inaccurate claims in our paper. The core difference between our method and the RLHF method is that we explicitly introduced the rejection preference pairs in the training of the reward model. This allows the reward model to better understand when to refuse. As shown in Table 4 of our paper, the reward model trained with helpful preference data only achieved an accuracy of 49.9 on the "beyond" subset of the in-domain arithmetic reliable preference data (this task is binary classification, where the reward model chooses between refusing to respond and responding incorrectly, with the correct label being refusal). However, after adding the corresponding in-domain reliable preference data (i.e., various preference pairs regarding refusals), this classification accuracy significantly improved to 83.0. This partly explains why RLKF can better improve the model's reliability compared to RLHF, as the reward model after RLKF can better determine when it should refuse.
1. Thank you for the new results on TriviaQA! How did you construct the preference data for the reward model? Moreover, this only compares RLKF to using a system prompt. Do you compare RLKF to regular RLHF?

2. It remains unclear to me why examples of "rejection > incorrect" preferences would not naturally be present in existing methods for preference data collection. However if these preferences are sampled from the LM, I buy the authors' argument that models do not naturally know how to decline unless explicitly taught.

3. Thank you for clarifying!

4. The statement that "*existing LLMs such as Llama-2 initialize their reward models for RLHF from these generic preference pairs*" is untrue, to the best of my understanding. I believe the reward model is initialized using responses sampled from (variants of) Llama2 (and continuously updated using the latest iteration of the finetuned model). It is not initialized from a static dataset of preference data.

For reference, I share a snippet from the Llama 2 tech report below:
> We ask annotators to first write a prompt, then choose between **two sampled model responses**, based on provided criteria. In order to maximize the diversity, the two responses to a given prompt are sampled from **two different model variants**, and varying the temperature hyper-parameter.

[...]

> Human annotations were **collected in batches on a weekly basis**. As we collected more preference data, our reward models improved, and we were able to train progressively better versions for Llama 2-Chat (see the results in Section 5, Figure 20). Llama 2-Chat improvement also shifted the model’s data distribution. Since reward model accuracy can quickly degrade if not exposed to this new sample distribution, i.e., from hyper-specialization (Scialom et al., 2020b), **it is important before a new Llama 2-Chat tuning iteration to gather new preference data using the latest Llama 2-Chat iterations**. This step helps keep the reward model on-distribution and maintain an accurate reward for the latest model.
|method| Prec | Acc | Truth | Rely|
|---|---|---|---|---|
|llama2 + no system prompt | 60.2 | 58.9 | 61.2 | 61.1 | 
|llama2 + prudent system prompt | 53.9 |42.4 | 63.7 | 59.2|
|llama2 + rlkf | **73.5** | 50.1 |  **81.9** | **71.8** |

We further validated our method on the knowledge-based Question Answering task, i.e., TriviaQA [1]. TriviaQA is a widely-used QA dataset that can be used to test a model's world knowledge. We selected 20,000 samples from the TriviaQA training set for training. Since the ground truth of the TriviaQA test set is not publicly available, we used the TriviaQA development set, which contains 11,313 samples, to validate our results. We used the Exact Match metric (whether the answer is exactly in the model's response) from TriviaQA paper as our measure of Accuracy, while keeping the other metrics consistent with those in our paper. As shown in the table above, our method significantly improves the model's truthfulness and overall reliability on the TriviaQA dataset. This demonstrates that our method can enhance the general reliability of LLMs across different tasks, not just limited to arithmetic questions. We will include these results and more experimental details in future versions of our paper.

[1] Joshi, Mandar, et al. "Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension." arXiv preprint arXiv:1705.03551 (2017).
Thanks for your comment. We have provided further responses to the issues you raised:


## 1. Re: Comment#2 
### Results on knowledge-based QA task (TriviaQA)

|method| Prec | Acc | Truth | Rely|
|---|---|---|---|---|
|llama2 + no system prompt | 60.2 | 58.9 | 61.2 | 61.1 | 
|llama2 + prudent system prompt | 53.9 |42.4 | 63.7 | 59.2|
|llama2 + rlkf | **73.5** | 50.1 |  **81.9** | **71.8** |

We further validated our method on the knowledge-based Question Answering task, i.e., TriviaQA [1]. TriviaQA is a widely-used QA dataset that can be used to test a model's world knowledge. We selected 20,000 samples from the TriviaQA training set for training. Since the ground truth of the TriviaQA test set is not publicly available, we used the TriviaQA development set, which contains 11,313 samples, to validate our results. We used the Exact Match metric (whether the answer is exactly in the model's response) from TriviaQA paper as our measure of Accuracy, while keeping the other metrics consistent with those in our paper. As shown in the table above, our method significantly improves the model's truthfulness and overall reliability on the TriviaQA dataset. This demonstrates that our method can enhance the general reliability of LLMs across different tasks, not just limited to arithmetic questions. We will include these results and more experimental details in future versions of our paper.


## 2. Re: Comment#3
### Results of GPT4-o on arithmetic tasks
|subsets| llama2+rlkf(acc/truth/rely) |chatgpt(acc/truth/rely)| gpt4-o(acc/truth/rely) | 
|----|----|----|----|
|+|72.8/77.2/74.7|97.6/97.6/97.6|99.6/99.6/99.6|
|-|28.3/85.7/53.4|91.6/92.8/92.8|93.6/96.8/96.7|
|*|8.4/**98.3**/47.8|37.2/**39.3**/39.3|49.8/**49.8**/49.8|
|/|18.6/**91.3**/50.4|69.3/**69.3**/69.3|81.8/**82.2**/82.2|
|1-2 digit|41.2/89.5/62.4|86.0/87.0/87.0|94.0/95.5/95.5|
|3-5 digit|25.7/**87.2**/52.6|66.3/**67.0**/67.0|73.2/**73.7**/73.7|
|all|31.9/**88.1**/56.5|74.2/**75.0**/75.0|81.5/**82.4**/82.4|

We further tested the reliability of GPT4-o on arithmetic datasets. We found that even GPT4-o remains unreliable and lacks the ability to reject out-of-knowledge questions. From Llama2 to ChatGPT to GPT4-o, they all use a large amount of industrial-grade generic instruction-tuning data, but they still lack good reliability. We believe that, on the one hand, these generic preference data may lack appropriate rejection data (rejecting only when the model lacks relevant knowledge, otherwise it needs to answer questions). On the other hand, the RLHF training process constructs preference pairs based on the model's own sampling results, making it difficult to generate appropriate rejection behavior during the sampling process. Therefore, we believe that it is necessary to synthesize reliable preference pairs through RLKF to make the model more reliable.

## 3. Re: Comment#4

Sorry for any trouble caused by the writing. Your understanding is correct; our method does not differ from PPO during training. The key lies in how we construct preference pairs, which differs from RLHF. We construct preference pairs based on feedback from knowledge, specifically whether the model has relevant knowledge to answer the question. The core objective is to make the model more reliable, teaching it to explicitly reject when necessary, rather than improving the model's response score based on human preferences.

## 4. Re: Comment#5
We apologize for the inaccuracies in the writing of our paper. We will correct these inaccurate claims in our paper. What we meant to convey is that 1) on the one hand, existing generic preference pair data [2-4] is often fixed and lacks the ability to dynamically adjust for each model's capabilities and knowledge. For the same question, we expect the corresponding preference pair to be different for models capable of answering and those incapable. 2) on the other hand, existing LLMs such as Llama-2 initialize their reward models for RLHF from these generic preference pairs. Although subsequent optimization is based on human feedback and model sampling results, poor initialization can significantly affect the model's sampling space, making it difficult for the final model to have good reliability.

[1] Joshi, Mandar, et al. "Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension." arXiv preprint arXiv:1705.03551 (2017).

[2] Bai, Yuntao, et al. "Training a helpful and harmless assistant with reinforcement learning from human feedback." arXiv preprint arXiv:2204.05862 (2022).

[3] Cui, Ganqu, et al. "Ultrafeedback: Boosting language models with high-quality feedback." arXiv preprint arXiv:2310.01377 (2023).

[4] Stiennon, Nisan, et al. "Learning to summarize with human feedback." Advances in Neural Information Processing Systems 33 (2020): 3008-3021.

We hope the above responses can address your concerns.
I appreciate the authors' additional experiments. I will summarize my original reasons for rejection, and whether they have been addressed.

1. **Re: Calibration** — the authors compare to a large number of post-hoc calibration methods, which they agree to include in the final paper. While they find that their method underperforms a majority vote approach, I agree that there are merits to reliability baked into the model. The weaker performance compared to majority vote limits their empirical claims, but is not an independent reason for rejection.
2. **The scope of experiments is too limited.** This is unaddressed. The authors evaluate on arithmetic benchmarks, one of which is synthetic. It would benefit the paper greatly to show that if reward models are trained on generic preference data using their method, it will lead to higher performance on a broad swath of tasks.
3. **Large amounts of preference data will naturally contain examples where refusal to answer is preferred to incorrect answers.** This is not directly addressed. The authors do provide results showing that `gpt-3.5-turbo` is still not-that-reliable. However, it's not clear whether this is because "refusal > incorrect" pairs are not present in preference data, or because they are not enough to lead to reliable behavior.
4. **The main contribution, which is about how to create preference data for the reward model, is not clear.** As far as I can tell, this concern is not addressed.
5. **Previous PPO approaches (e.g., Llama2-chat) do use model-specific reward models, contradicting what the authors claim.** As far as I can see, this concern is not addressed. The (incorrect) argument that current reward models are model-agnostic is used to motivate the authors' approach.

To summarize, the paper introduces a simple approach for training better reward models — introduce preference data where rejections are preferred to incorrect answers. There are merits to an approach to bake reliability into the model, compared to posthoc calibration methods, which the authors will add to the experiments.

However, I wish the writing were more clear about the simplicity of their approach (#4), which is an **advantage** rather than a downside. Moreover, the paper's experiments are in a relatively narrow test case, and it's not clear whether the method will be effective more generally (concern #2 and #3).

If the other reviews are willing to champion the paper, I will not argue against acceptance.
Hi tasN, can you check the authors' response and update your review if it addressed your concern (or participate in discussion with the authors if it did not)?
Hi J8K1, can you check the authors' response and update your review if it addressed your concern (or participate in discussion with the authors if it did not)?
Yes, it can achieve an accuracy of around 95 for both llama and llama + rlkf. In Appendix D of our paper, we provide templates for synthesized arithmetic problems, such as "Compute the result of {input}." Extracting the arithmetic part "{input}" using an  LLM is very simple, and since the calculator itself does not make errors, the accuracy is very high.
> directly converting the question into an equation and sending it to a calculator achieves very high accuracy

how much accuracy does this achieve? higher than 85.2?

ty
Hi reviewers, please take a look at the author's rebuttals and the other reviews for this paper!

If the rebuttals addressed your concerns, please let the authors know about this and update your review. If not, please continue to engage with the authors and the other reviewers in the discussion forum.

Reviewer tasN, you have concerns about missing related work, a limited scope of experiments relying on synthetic benchmarks, and inaccuracies / opacity in the writing. Does the author's response address these concerns? 

qwqv and TGqk are both positive about this paper's acceptance. Would you like to argue for its acceptance, or do you share concerns with the other reviewers?
Thanks for the clarification. It does seem like the paper would be strengthened by showing that the method worked on a different data set, e.g. knowledge-based QA. Currently, we see that the method works on a synthetic arithmetic task, but shows a very limited range of improvement on a complex and realistic arithmetic task.
We further compare our method with calibration-based methods, and provide the results and our analysis below:

|method|inference cost(dialogue turns * sampling num)|arithmetic(acc/truth/rely) |gsm8k(acc/truth/rely)|
|---|---|---|---|
|rlkf|1|31.9/88.1/56.5|17.0/59.6/41.5|
|Raw logits(thresh-arith)[1]|1 |37.7/55.4/52.3|23.7/24.9/24.9|
|Raw logits(thresh-gsm8k)[1]|1 |21.2/85.9/44.0|13.9/72.2/38.2|
|P(True) [3]| 1|17.6/71.3/42.5|19.5/39.3/35.4|
|verb. 1S top-1 [4]|1|16.7/50.2/39.0|4.8/11.6/11.1|
|verb. 2S top-1 [4]|2|14.7/87.8/34.4|4.4/20.7/18.0|
|agreement(consistency)[1]|10 |37.9/79.9/62.3|20.1/77.1/44.6|

- **Unable to Reject Explicitly**: Calibration-based methods need to search and determine the best threshold for rejection and provide human-crafted rejection templates as responses. As we shown in the table, we search the threshold for arithmetic and gsm8k separately (on 100 validation cases from each dataset). However, the thresholds are quite different for different datasets which results in significant performance degradation with different thresholds.  In contrast, our method can enable the model to reject out-of-knowledge questions with personalized responses for different prompts automatically. 

- **High Inference Cost**: Consistency-based methods, on the one hand, require multiple samplings to obtain results, and on the other hand, may necessitate the use of additional models to extract answers for voting (we use ChatGPT to extract answers because rule-based methods may result in inaccurate extraction). This results in 5(sampling num) * 2(1 for answer generation, 1 for answer extraction) = 10 times (or at least 5 times) the inference cost than other methods. Some Verbalized-based methods (verb. 2S) also require the model to generate confidence through an additional round of response after generating the answer.

- **High Calibration Variance**: Utilizing calibration methods to determine the accuracy of answers is not stable. For instance, logit-based methods are not quite reasonable when the model generates longer responses, and Verbalized-based methods result in significant fluctuations in confidence scores and even prediction results (as shown in the gsm8k results of verb. methods) due to the variability in prompts.

In summary,  calibration methods are more suitable for analyzing the uncertainty of model responses or constructing training data (such as the self-consistency[5] introduced in our paper). However, our alignment research on reliability aims to enable the model to acquire self-knowledge and explicitly refuse out-of-knowledge questions. Experimental results show our method can enable the model to reject automatically without additional inference costs and improve the accuracy of rejections compared to most calibration methods.

[1] Calibrating Large Language Models with Sample Consistency

[2] Calibration-Tuning: Teaching Large Language Models to Know What They Don’t Know

[3] Language Models (Mostly) Know What They Know

[4] Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

[5] Self-consistency improves chain of thought reasoning in language models.
|subsets| llama2+rlkf(acc/truth/rely) |chatgpt(acc/truth/rely)|
|----|----|----|
|+|72.8/77.2/74.7|97.6/97.6/97.6|
|-|28.3/85.7/53.4|91.6/92.8/92.8|
|*|8.4/**98.3**/47.8|37.2/**39.3**/39.3|
|/|18.6/**91.3**/50.4|69.3/**69.3**/69.3|
|1-2 digit|41.2/89.5/62.4|86.0/87.0/87.0|
|3-5 digit|25.7/**87.2**/52.6|66.3/**67.0**/67.0|
|all|31.9/**88.1**/56.5|74.2/**75.0**/75.0|

The table above shows the experimental results of ChatGPT (gpt-3.5-turbo-0125) on different arithmetic tasks. As illustrated, although ChatGPT has a higher accuracy, its reliability remains low, and it lacks the ability to refuse out-of-knowledge questions. In different sub-tasks, the truthfulness of ChatGPT is generally on par with its accuracy, which indicates that it does not refuse arithmetic problems (even though we used a prudent system prompt to instruct ChatGPT to refuse questions it cannot answer). Moreover, in more challenging tasks such as multiplication, division, and problems involving 3-5 digit numbers, ChatGPT exhibits strong hallucinations, with truthfulness significantly lower than llama2+RLKF. Therefore, we do not believe that the issue of reliability can be resolved through broader generic instruction-tuning.