Refine Knowledge of Large Language Models via Adaptive Contrastive Learning
Authors: Yinghui Li
Haojing Huang
Jiayi Kuang
Yangning Li
Shu-Yu Guo
Chao Qu
Xiaoyu Tan
Hai-Tao Zheng
Ying Shen
Philip S. Yu
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Model Hallucination
Model Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: This paper introduces an Adaptive Contrastive Learning strategy that mimics human learning to reduce hallucinations in LLMs.

Abstract: How to alleviate the hallucinations of Large Language Models (LLMs) has always been the fundamental goal pursued by the LLMs research community. Looking through numerous hallucination-related studies, a mainstream category of methods is to reduce hallucinations by optimizing the knowledge representation of LLMs to change their output. Considering that the core focus of these works is the knowledge acquired by models, and knowledge has long been a central theme in human societal progress, we believe that the process of models refining knowledge can greatly benefit from the way humans learn. In our work, by imitating the human learning process, we design an Adaptive Contrastive Learning strategy. Our method flexibly constructs different positive and negative samples for contrastive learning based on LLMs' actual mastery of knowledge. This strategy helps LLMs consolidate the correct knowledge they already possess, deepen their understanding of the correct knowledge they have encountered but not fully grasped, forget the incorrect knowledge they previously learned, and honestly acknowledge the knowledge they lack. Extensive experiments and detailed analyses on widely used datasets demonstrate the effectiveness and competitiveness of our method.

Published as a conference paper at ICLR 2025
REFINE KNOWLEDGE OF LARGE LANGUAGE MODELS VIA ADAPTIVE CONTRASTIVE LEARNING
Yinghui Li Haojing Huang" Jiayi Kuang Yangning Li', Shu-Yu Guo
Chao Qu Xiaoyu Tan* Hai-Tas Tao Zheng'.4., Ying Shen Philip S. Yu
'Tsinghua University, SSnn Yat Sen University
IIFFY TECH (Shanghai) Co., Ltd., *Peng Cheng Laboratory Universiity of Illinois Chicago (liyinghu20, hhj23] @mails tsinghua. edu. en
ABSTRACT
How to alleviate the hallucinations of Large Language Models (LLMs) has always been the fundamental goal pursued by the LLMs research community. Looking through numerous hallucinatioo-rrlatee studies, mainstream category of meth- ods is to reduce hallucinations by optimizing the knowledge representation LLMs change their output. Considering that the core focus of these works is the knowledge acquired by models, and knowledge has long been a central theme in human societal progress, we believe that the process of models refining knowledge can greatly benefit from the way humans learn. In our work, by im- itating the human learning process, we design an Adaptive Contrastive Learning strategy. Our method flexibly constructs different positive and negative samples for contrastive learning based on LLMs' actual mastery of knowledge. This strat- egy helps LLMSs consolidate the correct knowledge they already possess, deepen their understanding of the correct knowledge they have encountered but not fully grasped, forget the incorrect knowledge they previously learned, and honestly ac- knowledge the knowledge they lack. Extensive experiments and detailed analyses widely used datasets demonstrate the effectiveness of our method.
INTRODUCTION
Based on the massive training corpora and a large number of training resources from the real knowl- edge of the human world, Large Language Models (LLMs) store many general and domain-spectin knowledge and show excellent performance in many natural language processing tasks (Wang al., 2024 Zhang et al. 2024a). LLMs are equipped with excellent command comprehension, logical reasoning and text generation capabilities. As chat assistants, LLMs help users realize many tasks in their work, study, and daily life, greatly enriching the user experience and satisfying their needs (Bansal al.) 2024 Asadi et al.) 2024] Guo et al. (024). However, LLMs-based chat assistants, although greatly helping users, show hallucination problems in more and more tasks (Chen et al.) 2024 Zhang et al.) 2024e d). They encounter factual errors in their answers, sometimes even fab- ricate non-existent knowledge, or contradict the user's original intention when entering instructions (Mu & Lim] 2024 Lv al. 2024). Some these hallucinatory errors, such as fabricating false doc- umentaries, difficult for the user recognize directly, but they have serious consequences when used and are likely to reduce the user's trust in the AI assistant.
When the LLM hallucinates in its reply, provides factual errors, or f'ariccttes false knowledge, can assume that the LLM lacks the relevant knowledge for the problem and does not have the ability to respond accurately (Li et al. 2024 Cheng al.) 2024 Zhhang et al. 2024c). From the perspective of human knowledge needs, for questions beyond the scope of LLM's 's knowledge, we receive an honest response like "I don't know" rather than a fabricated, plausible-sounding answer. This will be more conducive to utilizing Al to get the valid information that users need, as users can actually
"indicates equal contribution.
1 "Correspnnding authors: Hai- Zheng, Ying Shen, and Xiaoyu Tan

---

Published as a conference paper at ICLR 2025
Model knows (Known)
Original
How from sonterhing
furtiture?
the fartiire??
Ground buti:: Use suratcing soot and fumiitur with double
+
A Use srrviiingg post cover fumiture
#MNegathe instancs
Q How ioos my from suaadhinng the turn ture? A: ddoort brow.
Sincee isspoess
LUM
don't know that
Regatte Instane
cniv reet!
Q can soop my or from sentrring Speny
Model (Unknown)
Bifld
Ground truth: Fifh Tower, Murnum,
Museum, Notre-Juree Some Oafherml
Notts Dame Catheokal al.
Negeniee
Paris?
Acc IDK
ure "Pank Cantle"" ferros witter attactionn
Protive incrance
G: how car you ball sccmocon turyyly rapery or just pretending ldo''t know.
Unbnews
Original Instance
Negative intance
Now can you tell it somsone truy pouyhoun happy
just precending din to be?
Ground touth: hapsinsss shows everaal pkmmaaoor including face and body lang-age
You the brightnisss
enile.
Figure 1: The illustration of our adaptive contrastive learning. Based on the multiple sampling responses of the model, we represent the knowledge regions with different mastery capabilities, and construct different positive and negative examples of adaptive contrastive learning help the model better consolidate what is known.
achieve what they want by providing more knowledge about the initial question, or by asking for additional help.
Therefore, to get an honest LLM assistant with higher creditworthiness, the LLM must have two abilities: the LLM not only has to be able to choose refuse answer when faced with an un- known question but also has to acquire the ability to correctly distinguish whether it really knows the knowledge related it knows and what it does not (Zhang et al. 2024c). The LLM's ability to perceive knowledge can be represented by the knowledge quadrants Yin et al. 2023b). The knowledge quadrant is divided into four quadrant regions shown in Figure Known Knowns, Known Unknowns, Unknown Knowns, and Unknown Unknowns, where the vertical axis represents the model's 's perception of it- self (i.e., the model's thought of what it knows does not know). The horizontal axis represents the model's 's actual knowledge mastery (i.e., whether or not the model actually masters particular piece of knowledge). Thus, knowledge in the first quadrant "Know Knowns", represents that the AT knows that it knows, while knowledge in the forth quadrant "Unknown knowns", represents that the AI does not know that knows, and the same can be inferred for the other quadrants.
a question or not, which means the model needs to be aware of what
Ideally, the LLM gives the correct response to the question, which falls into "Known Knowns", whereas when the LLM faced with the hallucination problem, it does not realize that it does not know the relevant knowledge and gives the wrong response, which falls into the "Unknown Un- knowns" quadrant. The horizontal axis represents the LLM's specific knowledge mastery, which is limited by the LLM's own training process and knowledge capability. Aiming to mitigate the hallucination problem and meet the needs of the honest L.M system, we pay more attention to the vertical "Known" ability. We believe that providing correct response to known questions and fusing to respond to unknown questions, as outlined in the two quadrants above, are both acceptable behaviors for honest LLM. Thus, our goal in mitigating the LLM hallucination problem is have more input instructions and responses that can be moved from the lower two quadrants, into the upper two quadrants.
Research on methods to mitigate the LLM hallucination problem and improve its honesty based on knowledge representation faces the following problems:
2

---

Published as a conference paper at ICLR 2025
1. The design of knowledge boundaries is relatively simple and much knowledge is not utilized. In order to help LLMs better represent the knowledge known to the model and the knowledge unknown to the model, early studies such as Zhang et al. (2024c) and Cheng et al. (2024) would design knowledge delineation methods to divide relevant questions into known and unknown, and then further teach the model to answer honestly through fine- tuning. However, these methods of knowledge boundary delineation are simplistic and ignore much of the critical knowledge that the model is uncertain about and does not utilize well. As a result, while the model may refuse to answer questions it does not know, this reduces the model's confidence and causes which greatly affects the validity of its responses.
respond only when it is very certain,
2. Lack of learning from the model's incorrect replies. In the early work, most of the focus was on the model's known knowledge and its correct responses, as well as the fine- tuning effect the model, but it did not consider the use of the model's incorrect responses (Cheng et al.) 2024). Inspired by the process of human knowledge learning and thinking, we believe that the model not only needs learn correct knowledge, should also recog- nize its own mistakes from incorrect replies. By learning from this incorrect feedback, the model can enhance its understanding and representation of knowledge by avoiding incor- rect reasoning and forgetting related incorrect knowledge in subsequent learning.
Therefore, we propose Adaptive Contrastive Learning, which refine the boundaries of an LLM's knowledge representation align with real-world human knowledge needs. This approach en- ables the model to maintain known knowledge, consolidate uncertain knowledge, and forget unknown knowledge. We introduce an innovative knowledge boundary design scheme featuring an upper threshold, Know Rate (TK), and a lower threshold, Don't 't Know Rate (TDK), to delin- eate knowledge representation. Using well-annotated Question Answering pairs, we input these into the LLM, sample responses multiple times, and calculate the response accuracy. Responses with ac- curacies greater than or equal to IK indicate the model knows what it knows, those between IK and IDK suggest uncertainty. and those less than IDK imply unknown knowledge. Based these knowledge quadrants, we construct comparative learning data, designing adaptive positive and nega- tive samples for each representation. This includes learning correct boundary knowledge, addressing incorrect responses, and reinforcing rejected answers. To optimize the LLM using these samples, implement an adaptive contrastive learning strategy with tailored loss functions for different knowledge representations, while fine-tuning the model's generation capabilities. By maximizing the distance between negative samples and minimizing the distance between positive samples, the model learns accurate knowledge while discarding inaccuracies, thereby enhancing its confidence and reliability. Our contributions are as follows:
We propose a new knowledge representation method for LLMs, which assists the model in better refining its own knowledge scope, enhances the model's honesty and alleviates the model's hallucination problem through a new knowledge boundary division.
We design an Adaptive Contrastive Learning strategy, through which the model can main- tain its known knowledge, consolidate the known but uncertain knowledge, and forget the unknown knowledge, which improves the validity and honesty of the model's responses.
We conduct experiments and analyses various advanced LLMs and test them on both in- distribution and out-of-distributtin data. The experimental results show that our approach achieves the highest Truthful rate, verifying the effectiveness of proposed Adaptive Contrastive Learning strategy.
2
RELATED WORK
2.1 FINE-TUNING LLMS WITH HUMAN KNOWLEDGE
Large language models learn extensive knowledge from human society and carry out various tasks based on this knowledge (Burns et al., 2003 Ouyang et al. 2022). Among mainstream techniques, supervised fine-tuning (SFT) aligns LLM with real-world human knowledge, playing critical role in fulfilling the tasks expected of LLM by humans. After gathering a large annotated corpus for natural language processing, researchers fine-tune pre-trained language models, enabling L.LMs to
3

---

Published as a conference paper at ICLR 2025
learn from the carefully selected and annotated human knowledge and apply it to unseen tasks (Wei jtt al., 2022: Sanh et al. 2022). During the SFT process, enhancing model performance can be achieved by expanding the size of fine-tuning data (Chung et al., (022) and gathering complex data across various domains and tasks (Shen et al. 2023 Longpre et al. 2023), however, it may sacrifice generality by focusing on specific domain fine-tuning. Beyond different data, instruction tuning methods design various instruction-responss pairs and enable models to learn response patterns and content more in line with human instruction expectations, making LLMs more truthful, credible, and helpful (Liu et al. 2024b; Muennighoff et al. 2024). However, LLMs in instruction tuning are not robust to changes instruction texts. often generating inconsistent answers slightly rephrased instructions (Liu et al. 2024a). Yan et al. (2024) propose contrastive instruction tuning approach that maximizes similarity between semantically equivalent instruction pairs and minimizes similarity between semantically distinct pairs, significantly enhancing robustness in instruction fine-tuning.
2.2 MITIGATING HALLUCINATION WITH LLM'S KNOWLEDGE
LLMs accumulate substantial knowledge during training and fine fine-tuning, yet unavoidably encounter data compression challenges during training, where the training data size is several times larger than the model's parameters (Zhu et al. 2023). This makes it challenging for LLMs to fully restore the original knowledge, leading to hallucinations (Gekhman et al., 2024). Moreover, when the model lacks sufficient information to respond to questions, it tends to guess, producing inaccurate outputs (Mündler et al. 2024 Zhang al.) (024d). Hallucinations typically involve logical fallacies (errors in reasoning by the model) (Mu & Lim, 2024), factual errors (when the model confidently asserts non-existent facts) (Lv et al. 2024 Li et al. (2024), and data-driven biases (when certain data pre- vails, potentially skewing the model's 's output in certain directions) (Sarkar et al. 2024 Zhang et al.) 2024b). Numerous studies focus on enhancing the models' utilization of knowledge representation
to mitigate hallucination issues. Asai et al. 20044) Niu et al.] (2024) employ the RAG approach, bolstering knowledge retrieval and feld-feed thereby alleviating the model's hallucinations. Chuang et al.) (2024) propose a decoding strategy, which contrasts the differences in tokens' logits obtained from different layers for reducing hallucinations. Chiiiionnn,CCCnggggg et al. (2024): /Zhang et al. (2024c)
are also exploring the issue of model honesty, asserting that when the model can honestly respond "I don't 't know" to unknown knowledge, it enhances the model's credibility.
METHODOLOGY
3.1 KNOWLEDGE BOUNDARIES FOR MODELS
Different question-ansswrring datasets encompass various types of knowledge and expressions, mak- ing challenging to assess whether models truly possess specific knowledge due to varying con- fidence levels. To enable models better identify their knowledge boundaries and determine their ability to answer related questions, we establish two thresholds: the upper threshold IK (I Know rate) and the lower threshold TDK (I Don't Know rate). These correspond to three knowl- edge types: the model knows it knows (knowns-known knowledge), the model does not realize it knows mankknowws-knww//nneeen (unknowns-unke) unknown knowledge).
knowledge), and the model does not know it does not know
Following previous studies Cheng et al. 20024) Zhang et al. (2024c), we filter questions from public datasets and query LLMs multiple times to sample responses, calculating accuracy as a measure of the model's confidence in each question. The number of sampled responses and the thresholds IK and IDK serve as hyperparameters. As illustrated in Figure 2] when confidence exceeds IK, the model deemed to fully master that knowledge. If confidence is between IK and IDK, the model has the knowledge but may not recognize it, leading to occasional incorrect answers. Confidence below IDK indicates a lack of knowledge, resulting in low correct response rates. By defining these knowledge boundaries, we will construct contrastive learning data and adaptively tune the model based on these knowledge types. This process aims to help the model retain known knowledge, reinforce uncertain knowledge, and forget unknown knowledge. We aspire for adaptive contrastive learning instruction tuning to shift more knowledge from the third and fourth quadrants to the first and second quadrants, enhancing model honesty while ensuring helpful responses.
4

---

Published as a conference paper at ICLR 2025
Sauple cmspusse
Knonledge Regina
ACC
bonccdly vediane pa Morroace produst?
Stotel knewe known
frosh Earth?
DE SADE
Medal doeunt know labber
Q Which Sureused three- lank wood iotuunttt dieetivent
Mode dosor?
Vnow
Knostedge Regresmbatinn
Frome
Model cosun't know doesn't know
Model docort knowlt bnows
Medel B knows
Figure 2: The illustration of the knowledge boundaries and the sample response.
We choose the widely used mainstream knowledge-sensitive open-domain question-answering dataset, TriviaQA, based on which we collect model responses. To determine whether the model's responses are correct or not, we select lexical matching as an automatic evaluation metric assess whether the model's output is correct not. By calculating the consistency rate between the model's responses and the human-labeled responses in the dataset, we consider that the model correctly an- swers the question when it reaches approximately 90%.
3.2 CONSTRUCTION OF CONTRASTIVE LEARNING DATA
Based on the knowledge boundary delineation in Section 3.1 we have collected knowledge data of what the model knows and what it does not know, which corresponds to the right and the left quadrants of the knowledge, respectively. We further perform contrastive learning data construction for the knowledge, as shown in Figure
3.2. QUESTIONS THAT THE MODEL KNOWS
In Section 3.1 the accuracy of the question greater than the upper threshold IK falls in the first quadrant of the knowledge of "what the model knows that it knows". For given question, the correct response of the corresponding model is designed as a positive instance. There are two types of negative instances: one is the model's response "I don't know", which can be pushed farther away in the contrastive learning instruction tuning to enhance the model's confidence and avoid the situation where the model knows the knowledge but thinks it doesn't know. Another negative instance is given question with incorrect answers generated by the L.M,, applied to help the LLM forget the incorrect knowledge during training. It is worth noting that the selection of instances is related to the specific parameters of our threshold. When the I K is chosen as 1.0 for the upper known question the model answers
threshold, it means that the question will only be treated all of them correctly during all sampling instances. At this point in the construction of the negative instances, there will be no corresponding incorrect answer negative instance, because questions with incorrect answers are already categorized in questions that the model does not know.
3.2.2 QUESTIONS THAT THE MODEL DOES NOT KNOW
Questions that are answered with correctness rate less than the upper threshold IK, including "the model does not know that knows" and "the model does not know that it does not know", fall into the two quadrants on the bottom of the knowledge quadrants, and are therefore considered as "model doesn't know". The negative instance the two kinds questions a given question with incorrect answers generated by the LLM. In the construction of the positive instances, for questions with an accuracy greater than the lower threshold IDK, we consider that the model actually has the relevant knowledge, but is just not sure whether it knows it not, which belongs the knowledge "the model does not know that it knows". We set the questions with correct responses as positive instances, hoping to enhance the model's mastery of uncertain knowledge by introducing such knowledge.
5

---

Published as a conference paper at ICLR 2025
Model knows Linte
6: pontauy herano bemwar, produces what onfformane prerent Seato Scound tuth
0: senear compuny bimoaaly high oerfo performance productl Lddott
Hoget exlleee
ferano dynovall produces what cerfonrance
Quadrant 1
Quadiant
Mour Cepooo
Model doeat' de
What hobs prepreee of bo with
Fraske rutence
Contirtio
Lstt
dsenn't know doesn't Aome
JCe cerctriienne grott
whic men pccene-arnee
6 Reaatten indance ni Whthot grottet valuree then eater wattron-arit? Co
rellnccee dieet. nesciom
Quadrant a
Alt kiree
Negate UC kitter band counel retucchre va diveet Liayeein Sap 2015? Authes Ditee stteed)
Quadrant 3
Figure 3: The examples of the contrastive learning data. Quadrant 2 should contain the knowl- edge that models know that they don't know. Unfortunately, the initial model without our optimiza- tion does not have the ability to distinguish this kind of knowledge.
For questions with a correct response rate less than the lower threshold IDK, we consider that the model does not have the relevant knowledge, which belongs to the knowledge that "the model does not know that it does not know". We design the corresponding question and the "I don't 't know" answer positive instance. By narrowing the distance between this positive sample, we can better encourage the model to admit "I don't know" when facing the knowledge that the model doesn't have and enhance the honesty and creditability of the LLM responses.
To compute the adaptive contrastive loss in Section 3.3 we need to construct different contrastive data for various quadrants. Specifically, construct different instruction data / (x. y), where & is the input question and is the answer. the contrastive setup, besides the anchor 7, there are positive instances I+ and negative instances J- The differences primarily focus the selection of which will be detailed in the following sections.
3.3 ADAPTIVE CONTRASTIVE LEARNING STRATEGY
In Section 3.2 obtain the constructed instruction pairs for correlated positive and negative in- stances of model-knowns versus model-unknowns knowledge from the three quadrants. We then perform adaptive contrastive instruction tuning. Compared the traditional SFT, the contrastive instruction tuning approach can better enhance the model's mastery of knowledge by pulling the positive instances closer together and pushing the negative instances away Yan al.] (2024). In our adaptive contrastive learning strategy, for the positive and negative instances corresponding to knowledge different quadrants, we design different data using strategy, which can adaptively per- form the calculation of positive and negative samples in different quadrants. So that the model can be more targeted to think about different knowledge, and better enhance the model's ability to maintain what it knows, consolidate what it knows but is uncertain about, and forget unknown knowledge.
For the knowledge that the "models know that they know" in Quadrant (as shown in Figure 3 we have the contrastive data IQ., T and T,,- The i-th sample in training batch includes the original input 12, (its yf is the golden answer), positive instance ri.t (its y is the previous correct answer of I.LM), and negative instance n (its y² is "I don't know" or previous wrong answers of LLM).. We bring the positive instance [2 closer and push the negative instance The farther away.

---

Published as a conference paper at ICLR 2025
To calculate the contrastive loss, we obtain the hidden states h of I following the method of Yan et al. (2024) and have designed a contrastive loss function:
cos /2T
Ch log (oo 2 )/T + t coolbb, ua, 1/7
where cos(b),, h2.)) is the cosine similarity, and T is temperature hyperparameter.
In order to maintain the generative capability of the LLM, we follow the method of et al. (2022) and design the standard cross-entropy loss for each instruction pair:
cingen Q=
Σ
log p(vi 1 v<ck),
k=1
where is the length of the desired output for the input, v is the k-th token of y², v is the previous tokens before z.z The overall learning objective function was obtained by combining the above two loss functions:
cigen 1)
Cigen + max(x, X, detach( Ai = LQ =
-Q1
C.etr
where detach denotes that the loss value is detached from the computation graph and thus is treated only as a scalar, and À is the upper bound of the weight.
For the knowledge that the model does not know that it does not know in Quadrant - 3 and the model does not know that it knows in Quadrant - 4 (as shown Figure 3), we design similar objective functions, as described in detail in the Appendix, A.I
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
4.1.1 TRAINING DETAILS METHOΟSS
In order to verify the superiority of our proposed contrastive learning fine-tuning strategy, we design different LLM-based methods as baselines, and implement the baselines with the same details for fair comparison. Their training details are presented here.
IDK Prompting. In order to equip the model with the initial honesty of refusing answer questions that it does not know, we designed the Prompting scheme directly for the baseline large model. In the Prompt, the model is told to reply "I don't know" when it encounters guestion that it cannot answer. This approach requires the model not only to have good command following ability, which correctly follow the instructions provided by us, but also to have the ability to distinguish its own knowledge boundaries, i.e., to distinguish what is question that it cannot answer. In IDK Prompting, the model is not additionally trained or fine-tuned, and the model's own ability to follow instructions and distinguish between knowledge boundaries is completely examined.
IDK SFT. Supervised fine-tuning (SFT) very important technique for fine-tuning LLMs and can be very useful to help models quickly learn new knowledge patterns from constructed high- quality datasets. We fine-tune the model based on the problems that the model knows versus the problems that the model does not know, from the contrastive learning data we construct. In the input data, each pair of instances consists of a question and response, where the questions that the model knows correspond to correct responses and the questions that the model does not know correspond to don't know" responses. By calculating the standard sequence-to-sene to-sequenc loss of the model-generatee responses with respect to the standard responses, train the model.
4.1.2 IMPLEMENTATION DETAILS
We select LLaMA-2-7B-chhe and Cestral-BBIInstruce as base models for testing on TriviaQA Joshi al. 2017) and Natural Questions (Kwiatkowski al. 2019), respectively. During the
7

---

Published as a conference paper at ICLR 2025
LLMMA-2-BB-caat
Method
TriviaQA
Natural Questions
IK-IK
IK-IDK
TRUTHFUL
IK-IK
IK-IDK
TRUTHFUL
IDK- Proopropp
37.4
29.6
66.9
19.7
41,4
61.1
IDK-SFT
28.0
45.2
73.2
32.8
27.5
60.3
IDK SFT-Adpt-Ctr
37.3
40.9
78.2
21.8
45.4
67.2
Mistral-7B IB-nIstrucc-toOO..
Method
TriviaQA
Natural Questions
IK-IK
IK-IDK
TRUTHFUL
IK-IK
IK-IDK
TRUTHFUL
IDK Prompting 50.8
5.9
56.7
23.6
20.8
44.4
IDK-SFT
31.4
40.8
72.2
4.8
65.4
70.2
IDK-SFT-Adpt-Ctr
24.4
49.0
73.5
9.3
67.7
77.0
Table 1: Overall results the test of the IDK dataset constructed based on TriviaQA and out- of-distribution test sets.
training of the LLaMA model, we used a batch size of 16, learning rate of 5e-5, a context length of 1024, and trained for 2 epochs. For the Mistral model, we used a batch size of 16, learning rate of
le-5, context length of 1024, and also trained for 2 epochs. The T set to 0.01 and the À is set to 1. All experiments are conducted Nvidia A100 80GB GPUs. During inference, we utilize the vllm framework to accelerate the process and employ a greedy search strategy to generate responses.
4.2 DATASETS
TriviaQA (Joshi al. 2017) is a reading comprehension dataset with question-answering pairs widely used in open domain quizzing, which contains 87,622 pairs in the training set. Referring to past work Cheng et (2024), we use 90% of the training set to construct training set for compar- ative learning data and 10% validation set. Since there is no standard answer in TriviaQA's test set, we select 11,313 Q&A pairs from the development set to build our final test set.
Natural Questions (K wiitkoowsii et al. 2019) is fine-tuned by constructing comparative learning dataset on TriviaQA, and we perform tests with the same data distribution. In order to validate the performance of our fine-tuning method on OOD data, we select Natural Questions our test dataset. Natural questions are real Q&A pairs from the Google search engine, where the development set containing 3,610 instances is used to build our test set. Same as when building the data in TriviaQA, we also use lexical matching as a metric for automatic evaluation when testing on Natural Questions.
4.3 EVALUATION METRICS
We employ the following evaluation metrics: IK-IK Rate: This metric reflects the model's "I know what I know" capability, which is calculated the percentage of correct answers relative to the total number of questions. 2) IK-IDK Rate: This represents the model's know what don't know" ability, which calculates the ratio of those questions that correctly refuse answering to the total number of questions. 3) Truthful Rate: Both correct answers and correct refusals to answer outside knowledge boundaries considered reliable, the Truthful Rate is computed as the sum of the IK-IK and IK-IDK rates, and regarded as a comprehensive measure of its knowledge refinement capabilities. We introduce the evaluation metries in detail in Appendix B.1
4.4
MAIN RESULTS
From the Table T we can observe that in the TriviaQA dataset using the LLaMA model, the model fine-tuned with SFT shows a 6.3% improvement in the Truthful Rate compared to directly using IDK-Prompting. After incorporating our Adaptive Loss, there is an additional 5.0% improvement over the SFT model. Notably, the IK-IK rate shows a significant increase of 9.3% compared to the SFT model, indicating that Adaptive Loss helps mitigate the loss of knowledge that the model unaware it possesses due to SFT. On the OOD dataset, Natural Question, our model achieves improvements of 6.1 % and 6.9% over Iroopptng and IDK-SFT, respectively.
8

---

Published as a conference paper at ICLR 2025
In the of the Mistral model the TriviaQA dataset, the model fine-tuned with SFT exhibits 15.5% improvement in the Truthful Rate compared to directly using IDK- -romppppp. Building on this significant improvement, the addition of Adaptive Loss further enhances performance by 1.3%. On the OOD dataset, Natural Question, our model achieves improvements 15.9% and 6.8% over IDK- Prompropg and IDK-SFT, respectively.
LLaMA-2-7B-chat
Knowing Rate
TriviaQA
Natural Questions
IK-IK
IK-IDK
TRUTHFUL
IK-IK
IK-IDK
TRUTHFUL
0.5
42.7
34.5
77.2
29.8
32.4
62.2
0.7
37.3
40.9
78.2
21.8
45.4
67.2
0.9
15.7
57.9
73.6
13.5
50.3
63.8
Table 2: The impact of different IDK values the performance of LLaMA-2-7B-chatt
DISCUSSION
5.1 IMPACT DIFFERENT IDK VALUES
In Section 3.1 and Section 3.2 we utilize the IDK Rate threshold to distinguish between the model's uncertain knowledge (the model does not know that it knows) and unknown knowledge (the model does not know that it does not know). In this process, aim refine the model's correct knowledge and reduce its uncertainty regarding the former, while encouraging to forget incorrect knowledge regarding the latter. We conducted experiments with different IDK Rates evaluate model perfor- mance under varying thresholds. As shown in Table the model achieves optimal performance at an IDK Rate of 0.7, followed by 0.5, with 0.9 being the lowest. We believe that categorizing many uncertain responses as "the model does not know that it knows" can harm performance, as forcing the model assert knowledge in low-certainty situations can have negative effects. Conversely, high certainty threshold of 0.9 may prevent the model from reaching its full potential.
TriviaQA
Loss Combination
IK-IK
IK-IDK
TRUTHFUL
Co, (Model knows it knows)
28.7
46.0
74.7
Las (Model doesn't I know it doesn't know)
30.0
43.8
73.8
Lo, (Model doesn't 't know it knows)
36.5
39.3
75.8
Lq, + Las
26.5
47.9
74.4
Lq + Los
32.9
78.0
Los + Lo+
29.4
46.7
76.1
Total
37.3
40.9
78.2
Table 3: Results of different loss combinations. Experiments are conducted on LLaMA-2-7B-chat.
5.2 IMPACT OF DIFFERENT CONTRASTIVE Loss COMBINATIONS
In designing the Adaptive Contrastive Learning strategy, we developed different contrastive learning strategies based on the model's certainty regarding various issues. Here, we isolate each type of contrastive learning loss by setting the others zero to evaluate the impact of each specific loss type. From Table 3, we can see that:
With the Lo, (i.e., "Model knows it knows" in Quadrant - 1), the model's responses are conservative, resulting in lower IK-IK rate and higher IK-IDK rate.
The Los (i.e., "Model doesn't 't know it doesn't know" in Quadrand model to be conservative and forget incorrect knowledge, resulting in a lower IK-IK rate.
encourages the
Using the Lo, (i.e., "Model doesn't 't know it knows" in Quadrant - 4) allows the model to correctly answer more uncertain questions, leading to higher IK-IK rate.

---

Published as a conference paper at ICLR 2025
For Lq, + Lqs the model has higher IK-IDK rate due to cautious responses, but lower IK- IK than Total. Missing Lq, limits the use of underlying knowledge for confident answers.
With Lo, + Lo.. the model sees strong Truthful rate and decent IK-IK results. However, without Lqs. it's less cautious about unknowns, affecting the IK-IDK rate.
The Lqs + Lq. provides balanced performance but lower IK-IK rate comparing Total. This suggests that without Lq the model struggles with certainty in known answers.
Combining these three components forms our method, which achieves superior overall performance.
5.3 REPEATED SAMPLING DISTRIBUTION
To more intuitively demonstrate the improvements of our method over IDK-SFT, we conduct ten repeated samplings of questions from the TriviaQA test set on I.LaMA2--BB-hha categorizing them into "Unknown Questions" and "Known Questions." When calculating aceuracy, the ground truth for "Known Questions" is the correct answer, while for "Unknown Questions", it is a refusal to respond. The distribution of our sampling results is shown in Figure In this figure, the vertical axis represents the proportion of samples within the entire dataset that fall into specific accuracy range. We employ repeated sampling to evaluate both models by having them respond to each question in the TriviaQA test set 10 times. The number of correct responses out of these 10 attempts (i.e., accuracy) is used as measure of the model's performance and confidence. To explain Figure 4 in the "Unknown Questions" sub-figure of Figure 4 the bars corresponding to an accuracy of I indicate the proportion of questions from the dataset that the models consistently declined to answer across all 10 attempts. Similarly, in the "Known Questions" sub-figure, the bars corresponding to an accuracy of 0.8 represent the proportion of questions for which the models provided 8 correct answers out of 10 attempts, relative to the total number of questions in the dataset.
From Figure 4 we observe that our method achieves a higher number of instances with high accuracy (greater than 0.7) in both Unknown and Known questions compared to Idk-SFT. This indicates that the model fine-tuned with our method is more likely to produce correct results in a single response.
Unknown Questions
Known Questions
1: 13k SET
midi: SFT SPT- Cir
1: SPT
ti SPT SPT-AAde-CU
0.35
1.155
0.00
0
0.1
4.2
03
044
6.5
0.6
0.7
0.8
0.9
0.1
0.2
6.
04
0.5
0.6
0.7
08
0.9
Figure 4: Accuracy Distribution for Known and Unknown Questions in TriviaQA.
6 CONCLUSION
In this paper, we focus on the problem of hallucination in LLM responses. We believe that this hallucination arises from the LLM's choice reply despite being confronted with a question that exceeds its own knowledge. This motivates our exploration of the LLM's mastery of its own in- ternal knowledge and its knowledge boundaries. We design new knowledge boundary delineation method, which helps the model better represent and refine its own knowledge quadrants. We propose the strategy of Adaptive Contrastive Learning, by targeting different knowledge mastery abilities in different knowledge quadrants, we design different positive and negative samples with adaptive loss functions to help the model maintain known knowledge, consolidate uncertain knowledge, and forget the wrong knowledge. We conduct experiments in-distribution and out-of-distributtin datasets, and the results show that this proposed contrastive learning strategy well improves the Truthful rate of the models. We further provide presentations threshold analysis, loss function ablation experiments, and visualization of results with knowledge quadrants, which not only further demonstrates the validity of our results, but also provides valuable inspiration for future work.
10

---

Published as a conference paper at ICLR 2025
REGNOWLEDGEEMENT
This research is supported by the National Natural Science Foundation of China (Grant No. 62276154), the Natural Science Foundation of Guangdong Province (Grant No. Shenzhen Science and Technology Program
02231111512911 and 4403002410031000001777 (Grant No. WDZC20231 128091437002), Basic Research Fund of Shenzhen City (Grant No. GJHZ202402183000
and JCYJ202408 131 12009013), the Major Key Project of PCL for Experiments and Applications (PCL2022205 and PCI.2023A09, This work is also supported in part by NSF under grants II1-1066558, and POSF-2346158.
REFERENCES
Amir-Reza Reza Asadi, Joel Appiah, Siddique Abubakr Muntaka, and Jess Kropczynski. Actions, not apps: Toward using IIms to reshape context aware interactions in mixed reality systems. Con- stantine Stephanidis, Margherita Antona, Stavroula Ntoa, and Gavriel Salvendy (eds.), HCI In- ternational 2024 Posters 26th International Conference on Human- Computee Interaction, HCII 2024, Washington, DC, USA, June 29 -July 4. 2024, Proceedings, Part VII, volume 2120 of Communications in Computer and Information Science, pp. 166-176. Springer, 2024.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self self-reflection. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.ne,, 2024.
Rachit Bansal, Bidisha Samanta, Siddharth Dalmia, Nitish Gupta, Sriram Ganapathy, Abhishek Bapna, Prateek Jain, and Partha Talukdar. LLM augmented lmss: Expanding capabilities through composition. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy I Liang. Semantic parsing on freebase from question-answe answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Wash- ington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL., pp. 1533-1544. ACL, 2013. URL http://///aahatthhoaoaa wttpo/oo..o///1111111100
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net 2023.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. IN- SIDE: 11ms' internal states retain the power of hallucination detection. In The Twelfth Interna- tional Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu. Can Al assistants know what they don't 't know? In Forty-firs: International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21- 27, 2024. OpenReview.net, 2024.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery. Sharan Narang, Gaurav Mishra, Adams Yu. Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts. Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-fieetue language models. CoRR, abs/2210.11416 2022.

---

Published as a conference paper at ICLR 2025
Feiteng Fang, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, and Ruifeng Xu. Enhancing noise robustness of strieval-augmeente language models with adaptive adversarial training. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 10028- Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024. 1124.ac1-long.544 URL
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning Iims new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024.
Zishan Guo, Yufei Huang, and Deyi Xiong. Ctooleval: chinese benchmark for Ilm-powered agent evaluation in real-world API interactions. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 1571 1-5124, Association for Computational Lin- guisties, 2024.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver. Canada, July 30 August 4, Volume 1: Long Papers, pp. 1601-1611. Association for Computational Linguistics. 2017.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wol Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452-466, 2019.
Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Xin Zhao, Jian- Yun Nie, and Ji-Rong Wen. The dawn after the dark: An empirical study on factuality hallucination in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 10879-10899, Association for Computational Lin- guistics, 2024.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang. Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for align- ment? A comprehensive study of automatic data selection in instruction uning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-1 1, 2024. OpenReview.net, 2024b.
Yixin Liu, Pengfei Liu, Dragomir R. Radev, and Graham Neubig. BRIO: bringing order to ab- stractive summarization. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 2890-2903. Association for Computational Linguistics, 2022.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp.
PMLR, 2023.
Qitan Lv. Jie Wang, Hanzhu Chen, Bin Li, Yongdong Zhang, and Feng Wu. Coarse-to-fine high- lighting: Reducing knowledge hallucination in large language models. In Forty-first Interna- tional Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. Open- Review.net, 2024.
12

---

Published as a conference paper at ICLR 2025
Wenchuan Mu and Kwan Hui Lim. Explicitly stating assumptions reduces hallucinations in natural language inference. In The Second Tiny Papers Track at ICLR 2024, Tiny Papers @ ICLR 2024, Vienna, Austria, May 11, 2024. OpenReview.net, 2024.
Niklas Muennighoff, Qian Liu, Armel Randy Zehaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruc- tion tuning code large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, 2024.
Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin T. Vechev. Self-contradictrry hallucina- tions of large language models: Evaluation, detection and mitigation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024.
Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACL. 2024, Bangkok, Thailand, August 11-16, 2024, pp. 10862-10878 Association for Computational Linguistics, 2024.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28. December 9, 2022, 2022.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö. Arik, and Tomas Pfister. Mitigating object hallucination via data augmented contrastive tuning. CoRR, abs/2405.18654, 2024.
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Bar- ret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Y. Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. Flan-moe: Scaling cnatuuttionfinetuua language models with sparse mixture of experts. CoRR, abs/2305. 147055 2023.
Xingyao Wang. Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: evaluating Ilms in multi-turn interaction with tools and language feedback. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
13

---

Published as a conference paper at ICLR 2025
Tianyi Yan, Fei Wang. James Y. Huang. Wenxuan Zhou, Fan Yin, Aram Galstyan, Wenpeng Yin, and Muhao Chen. Contrastive instruction tuning. In Lun-Wet Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August I 1-16, 2024, pp. 10288-10302. Association for Computa- tional Linguistics, 2024.
Xunjian Yin, Baizhou Huang, and Xiaojun Wan. ALCUNA: Large language models meet new knowledge. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pp. 1397-1414, Singapore, De- cember 2023a. Association for Computational Linguistics. doi: 10.18653/v 110.6////w/2223.mm-mmmmm 87. URL
org. emnip main.87 87
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACI 2023, Toronto, Canada, July 9-14, 2023, pp. 8653-8665. Association for Computational Linguistics, 2023b.
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets LLM finetuning: The effect of data, model and finetuning method. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a.
Dongxu Zhang, Varun Gangal, Barrett Martin Lattimer, and Yi Yang. Enhancing hallucination detection through perturbation-based synthetic data generation in system responses. In Lun-Wa Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, T'aalland and virtual meeting, August 11-16, 2024, pp. 13321- 13332. Association for Computational Linguistics, 2024b.
Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say 'i don't t know'. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Pupers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, Association for Computational Linguistics, 2024c.
pp.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. How language model hallucinations can snowball. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024d.
Xiaoying Zhang, Baolin Peng. Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song. Haitao Mi, and He- len Meng. Self-alignment for factuality: Mitigating hallucinations in lims via self-evaluation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 1946-1965. Association for Computational Linguistics, 2024e.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. CoRR, abs/2308.0 076633,.773....8000 2023.
14

---

Published as a conference paper at ICLR 2025
A
ADDITIONAL DETAILS OF METHODOLOGY
A.1 OBJECTIVE FUNCTION
The model does not know knows. For the knowledge that the "model doesn't t know that it knows" in Quadrant 4 (as shown in Figure 3 we have the contrastive data lq., 12. and T0. The i- th sample in a training batch includes the original input fQ. (its yf the golden answer), positive + (its y the golden answer), and negative instance f². (its y² is previous incorrect instance 1Q.
answers of LLM). We bring the positive instance QQ closer and push the negative instance fo. farther away. We design the loss function:
cos(big, 12/)/T
C.
log
wo h 1/7
1/7
where cos(h h2.) is the cosine similarity, and T is temperature hyperparameter. Again, there is cross-entropy loss with the final objective function:
LQ
log Mogp(yklle Plg. verr)
(5)
sigen
Qu
Adod
pigen QQ
+ max(), detach Q4
)) L
(6)
Ch
where detach denotes that the loss value is detached from the computation graph and thus is treated only as a scalar, and À is the upper bound of the weight.
The model does not know that it does not know. For the knowledge that the "models do not know that they do not know" in Quadrant 3 (as shown in Figure 3 we have the contrastive data IQ Ie and Io.- The i-th sample in a training batch includes the original input/2, (its y is don't know"), positive instance 12t (its y is "Inn''s 't know"), and negative instance I2 (its y previous incorrect answers of LLM). We bring the positive instance I closer and push the negative instance
1o farther away. We design the loss function:
cos cos 1o2 1/T
L
log
coslt Mos 1/7 coolttiss 1155 1/7
B ADDITIONAL DETAILS OF EXPERIMENT
B. EVALUATION METRICS
We use the following evaluation metrics, all of the evaluation scores show that the higher the score, the better the model effect:
IK-IK rate:
This means that the model "I know what I know", which means the model does not refuse to answer the question, but gives a correct answer, and the question labeled as "model knows". We calculate the number of correct answers given by the model percentage of the total number questions in the dataset as fraction of the IK-IDK rate.
IK-IDK rate: This represents the model's 's "I know what I don't know", meaning that the model refuses to answer question that is labeled as "model-don's t-kwow question" in the test set. When we detect the presence our predetermined "I don't know", we calculate the ratio of the number of questions that the model refuses to answer to the total number of questions the IK-IDK rate.
15

---

Published as a conference paper at ICLR 2025
Truthful Rate: This represents the overall percentage of reliable In the use of large models, for the model giving correct responses, versus refusing to respond to responses that outside the knowledge boundaries, both responses be considered reliable, so the Truthful rate is calculated as the sum of IK-IK and IK-IDK. Since all data are categorized into "model knowns" and "model unknowns", although the upper bounds for the separate scores for IK- IK and IK-IDK may vary depending on the data categorization, ideally the Truthful Rate can reach 100. Therefore, we believe that the Truthful Rate more comprchensive criterion for assessing the intellectual honesty of model, as the upper bound is not affected by data partitioning, and is more comprehensive measure of the model's mastery of the capability to refine knowledge.
given by the model.
C ADDITIONAL EXPERIMENT
This section is entirely new, highlighting is omitted.
C.I MORE DATASETS
Experimental Setup To evaluate the model's ability to respond appropriately to completely un- known questions, we selected the ALCUNA(Yin al.) (223a) dataset for testing. This dataset is designed by creating artificial entities through the alteration of existing entity attributes, leading the generation of questions about these novel entities. Since these entities are artificially con- structed, it is nearly impossible for the model to possess prior knowledge about them, making this dataset ideal for testing the model's ability to refuse to answer inconceivable queries. We randomly sampled 1000 instances from the ALCUNA dataset to serve our out-of-domain test set.
ALCUNA
Method
IK-IK
IK-IDK
TRUTHFUL
ODK-Prompting IDK-SFT
1.6
90.3
91.9
1.2
96.6
97.8
IDK-SFT-Adp--Ctr
1.0
97.3
98.3
Table Performance of different methods on the ALCUNA dataset.
Results Analysis As shown in Table our proposed methods perform better in the context of unknown queries that require refusal to respond, compared to models that have only finetuned by IDK-SFT. This underlines the effectiveness of our approach in reinforcing the model's ability to manage queries about unknown entities.
TriviaQA
Method
IK-IK
IK-IDK
TRUTHFUL
IDK-Prompting
-with LLaMA-2-7B-chan
37.4
29.6
66.9
wwtt LLaMA-2 2- 2-13B-chal
37.7
31.6
69.3
IDK-SFT
with -LLaMA-2-7B-cl
28.0
45.2
73.2
with LLaMA-2 - 2-1BB-chat
32.8
41.3
74.1
IDK-SFT-Adpt-
with 2-----hh--BBhat
37.3
40.9 41.1
78.2
with Aa-uMA---13B----
37.8
78.9
Table 5: Performance of different model sizes on TriviaQA dataset.
16

---

Published as a conference paper at ICLR 2025
C.2 IMPACT OF MODEL SIZE ON PERFORMANCE
Experimental Setup To investigate the effect of different model sizes on our proposed method, conducted experiments using the ALaMA-2-13B-che model. The experimental setup was con- sistent with the previous sections. By comparing with the LLaMA-2-7B-char model, we aimed to understand how scaling the model impacts performance across various metrics.
Results Analysis The results, as presented in Table 5 demonstrate nuanced changes in perfor- mance the model size increases from 7B 13B parameters. For all method, using the larger 1EMMA-2--13-ch-- model resulted in slight improvements across IK-IK, IK-IDK, and Truthful rate, indicating that a larger model can enhance the ability to differentiate known and unknown knowledge and present truthful responses. Moreover, the proposed method consistently gained performance im- provements across all metrics with the larger LLaMA-2-1 1-----2--B---- 13B-chat model, thereby underscoring the robustness and scalability of our approach.
C.3 COMPARISON WITH RAG INTEGRATION
Experimental Setup assess the adaptability of our method, we integrated the Retrieval- Augmented Generation (RAG) technique. We conduct experiments using the LLaMA-2-7B 7B-chat model, and we employe the RAG-Bench (Fang et al. (024) dataset, constructed from three open- domain question answering datasets: NQ (Cheng
al. 2024). TriviaQA (Joshi et al. 2017). and WebQ (Berant et al. 2013}. For each dataset, a retrieval model sourced relevant paragraphs from Wikipedia for each query side the query into the model (denoted as "with RAG" in the table) and compared the results with those obtained without using context. We randomly selected 1000 samples from RAG-Bench to use as our out-of-doman test test.
context. In this experimental setup, input the context along-
RAG-Bench
Method
IK-IK
IK-IDK
TRUTHFUL.
IDK- OKK-Prompiin
47.4
11.3
58.7
with RAG
62.5
5.8
68.3
IDK-SFT
60.3
1.9
62.2
with RAG
66.1
3.3
69.4
IDK-SFT-Adpt-Ctr
44.5
22.4
66.9
with RAG
58.4
12.8
71.2
Table 6: Performance comparison using LLAMA-2-7B-chat on the RAG-Bench dataset with and without RAG integration.
Results Analysis As illustrated in Table integrating the RAG technique positively impacted the Truthful Rate of all three methods. Additionally, it can be seen that IDK- and IDK-SFT- Adpt-Ctr show a decreasing trend IK-IDK after combining with RAG, while IDK-SFT shows an increasing trend in IK-IDK after combining with RAG. The reasons for this phenomenon as follows:
1. is intuitive and reasonable that IDK- PeK-Propp and LDD-SFF-Adpt--OE exhibit decreas- ing trend in IK-IDK when combined with RAG, as the model naturally tends to answer questions more directly and reduce refusal-to-answer after being input with more addi- tional contextual information from RAG. Therefore, from the perspective of metrics, after adding RAG these two methods, IK-IDK decreased.
2.
The phenomenon of the increase in IK-IDK after adding RAG to IDK-SFT can only in- dicate an increase in correct refusal-to-answerl but this does not conflict with the above analysis, as the overall refusal-pe toraalltt--nnne to-answer include both correct and wrong ones. After care- ful observation, we found that the overall rejection rate of IDK-SFTis is about 10%, while the overall rejection rate of DD--SFT-RAG is about 4%. This indicates that the overall rejec- tion rate of IDK-SFT still decreases after adding RAG, and the main reason for the increase in IK-IDK is that considerable portion of refusal-to -oanswerr in IDK-SFT' is wrong.
17


Reviews:
[{"review_id": "CqzTYnj795", "reviewer": "Reviewer_tNNV", "summary": "In this paper, the authors tackle the issue of hallucination, specifically factual errors, in Large Language Models (LLMs) by introducing a novel knowledge boundary delineation method and an Adaptive Contrastive Learning strategy. This approach aims to enhance the model's ability to accurately represent and refine its knowledge, maintain known facts, consolidate uncertain knowledge, and forget incorrect information. Through experiments on both in-distribution and out-of-distribution datasets, the authors demonstrate a significant improvement in the models' Truthful rate, bolstering the validity of their proposed methods and offering valuable insights for future research in enhancing the reliability and honesty of LLMs.", "questions": "Since the use of the proposed method enables the model to distinguish knowledge boundaries effectively, it would be interesting to study the performance of the model after using the RAG technique. For example, comparison experiments with paper [Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training](https://aclanthology.org/2024.acl-long.540) (Fang et al., ACL 2024) could be added.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors propose a new knowledge representation method for LLMs, which assists the model in better refining its own knowledge scope, enhances the model’s honesty and alleviates the model’s hallucination problem through a new knowledge boundary division.\n\nThe  authors design an Adaptive Contrastive Learning strategy, through which the model can maintain its known knowledge, consolidate the known but uncertain knowledge, and forget the unknown knowledge, which improves the validity and honesty of the model’s responses.", "weaknesses": "The experiments in the paper are not solid enough, only on LLAMA2-7B, and more different models such as Qwen2.5 series should be considered to enhance the said reliability of the experimental results. And the effect on different size models should also be considered and analyzed, such as 13B and above.  \n\nThe baseline in the paper is too simple and lacks comparison with other up-to-date approaches to knowledge boundary identification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IKLq1BMoRJ", "reviewer": "Reviewer_BPiT", "summary": "This paper tries to mitigate the hallucination problem of large language models by proposing an Adaptive Contrastive Learning strategy. The method curates data that represent the zones of \"model knows it knows\", \"model doesn't know it knows\", \"model doesn't know it doesn't know\", and designs contrastive learning loss to strengthen the model's desired behavior to be more trustworthy when answering questions. Experimental results show that the proposed method outperforms \"I don't know\" prompting and \"I don't know\" SFT.", "questions": "I think the paper can benefit from a careful round of proofreading. Here are some nits I discovered when reviewing the paper:\n1. In Table 1, it's better to use \"IDK\" instead of \"idk\" to be accord with the main paper.\n2. I suggest rewriting Section 5.2 as the current writing is a bit confusing, even though I can get the meaning by reading Table 3. For example, for the first bullet point, \"With\" should be \"Without\" based on my understanding.\n3. I am wondering if the notations of the paper can be improved. For example, using $IDKIK$ as subscript in equations looks weird.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The results of the paper are solid, as the proposed method outperforms IDK-Prompting and IDK-SFT by a large margin. Contrastive learning is not a new method, but the paper provides a good example of how to use it to improve large language models with careful contrastive pairs and loss design.\n2. The paper is well-motivated with well-made figures illustrating the data construction idea. The hallucination issue is a major block of using LLMs in high-stake scenarios, and many papers have thought of tuning the models to output \"I don't know\"[1]. However, the existing papers usually only consider the dichotomy of answerable and unanswerable questions without having a 2-D division of the space. It's interesting to see this paper brings out this concept as it's not only useful for helping model learning but could also help human learning [2].\n\n[1] R-Tuning: Instructing Large Language Models to Say \"I Don't Know\", Zhang et al., NAACL 2024.\n\n[2] Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations, Jiang et al., EMNLP 2024.", "weaknesses": "1. Even though the results look strong by looking at the Truthful Rate, I am not sure whether the proposed Truthful Rate is well-defined. Given the denominators of IK-IK Rate and IK-IDK Rate are different, I don't think it's rigorous to add them together as the final score. An alternative could be treating answering \"I don't know\" to those answerable questions as correct and computing the overall accuracy on the full dataset. Also, why the IK-IK rate for Mistral-7B-Instruct-v0.1 drops drastically compared to IDK-prompting? This is a bit concerning as it indicates after Adaptive Contrastive Learning, the model may not be useful anymore.\n2. i am unsure what's the major takeaway of Section 5.3. Does the accuracy under repeated samplings represent the concept of \"confidence\"? Actually, the paper mentions the term of \"confidence\" when introducing the motivation of the proposed method. It would be interesting to see whether the proposed method can draw a more rigorous connection of model's confidence in its generation.\n3. The presentation for the paper can be further improved. See my detailed comments below.\n\n\nI am happy to increase my score if these issues can be addressed.\n\n**Update: I updated my score to indicate support of acceptance since many points were addressed by the revised version.**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UfGfSirTRH", "reviewer": "Reviewer_5ff6", "summary": "This paper primarily addresses the issue of hallucination in large language models (LLMs), particularly when models provide incorrect or fabricated answers to questions they cannot answer, severely impacting model trustworthiness.", "questions": "I don't have questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is well-structured, especially in its quadrant-based classification of knowledge and its innovative use of an adaptive contrastive learning strategy, demonstrating a rigorous design approach. By combining knowledge classification with an adaptive loss function, it presents a novel solution to effectively address the hallucination problem in LLMs.", "weaknesses": "First, the experiment coverage is relatively narrow, with results demonstrated on only two datasets. Although the method shows promising results, its broad applicability across more tasks and domains is uncertain. Expanding the evaluation to more diverse datasets, such as dialogue or fact-checking datasets, would help validate the robustness and applicability of the method. \nSecond, although the paper proposes using a combination of three loss functions for different types of knowledge, it does not delve into the interactions and balance between these loss functions. In different scenarios, each loss may contribute differently; exploring the interactions and balance of these loss functions across various tasks would further strengthen the theoretical foundation of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4j5ia1p1a1", "reviewer": "Reviewer_ebFf", "summary": "In this paper, the authors emulate the human learning process by proposing an **Adaptive Contrastive Learning** strategy. This approach dynamically constructs positive and negative samples based on the actual mastery of knowledge in Large Language Models (LLMs). It enhances LLM's understanding of the correct knowledge they have encountered but not fully grasped, allows them to discard incorrect knowledge they previously learned, and candidly recognizes the knowledge they lack. Extensive experiments and detailed analyses demonstrate the effectiveness of this method.", "questions": "- How does varying the inference temperature affect the model confidence in your setting?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Innovative Knowledge Representation Method:** This is a well-written paper. The knowledge quadrant refines the boundaries of an LLM’s knowledge representation, aligning it more closely with real-world human knowledge needs.\n\n- **Well-Motivated Adaptive Contrastive Learning Strategy:** By maximizing the distance between negative samples and minimizing the distance between positive samples, the model learns to preserve its existing knowledge, consolidate known but uncertain information, and forget unknown knowledge. This approach enhances the validity and honesty of the model’s responses.\n\n- **Effective Approach:** Experimental results demonstrate that the strategy achieves the highest truthful rate across various advanced LLMs with both in-distribution and out-of-distribution data, thereby confirming the effectiveness of the proposed strategy.", "weaknesses": "- In this work, the discussion surrounding the concept of \"I know that I don't know\" (Quadrant 2) is lacking. Could you provide an explanation of how to differentiate between the knowledge categories of \"I know that I know\" and \"I know that I don't know\"? Specifically, would you categorize information as \"I know that I don't know\" if the model consistently responds with \"I don't know\" across all sampling responses?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)