The Internal State of an LLM Knows When It's Lying
Authors: Amos Azaria
Tom Mitchell
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
hallucination in LLM
LLM veracity
LLM activations
TL;DR: This paper demonstrates a novel approach to determining the truthfulness of statements using the internal state of LLMs, enhancing the reliability of AI-generated content.

Abstract: While Large Language Models (LLMs) have shown exceptional performance in various tasks, one of their most prominent drawbacks is generating inaccurate or false information with a confident tone.  In this paper, we provide evidence that the LLM's internal state can be used to reveal the truthfulness of statements. This includes both statements provided to the LLM, and statements that the LLM itself generates.
Our approach is to train a classifier that outputs the probability that a statement is truthful, based on the hidden layer activations of the LLM as it reads or generates the statement.  Experiments demonstrate that given a set of test sentences, of which half are true and half false, our trained classifier achieves an average of 71\% to 83\% accuracy labeling which sentences are true versus false, depending on the LLM base model.   
Furthermore, we explore the relationship between our classifier's performance and approaches based on the probability assigned to the sentence by the LLM.  We show that while LLM-assigned sentence probability is related to sentence truthfulness, this probability is also dependent on sentence length and the frequencies of words in the sentence, resulting in our trained classifier providing a more reliable approach to detecting truthfulness, highlighting its potential to enhance the reliability of LLM-generated content and its practical applicability in real-world scenarios.

The Internal State of an LLM Knows When It's Lying
Amos Azaria
Tom Mitchell
School of Computer Science, Ariel University, Israel
Machine Learning Dept.,
Carnegie Mellon University, Pittsburgh, PA
Abstract
Pluto the
While Large Language Models (LLMs) have shown exceptional performance in various tasks, one of their most prominent drawbacks is generating inaccurate false information with a confident tone. In this paper, we provide evidence that the LLM's internal state can be used to reveal the truthfulness of statements. This includes both statements provided LLM, and statements that the LLM itself gen- erates. Our approach is to train a classifier that outputs the probability that a statement truth- ful, based on the hidden layer activations of the LLM as it reads or generates the statement. Experiments demonstrate that given set of test sentences, of which half are true and half false, our trained classifier achieves an average of 71% 5 83% accuracy labeling which sen- tences are true versus false, depending on the LLM base model. Furthermore, we explore the relationship between our classifier's 's perfor- mance and approaches based on the probability assigned the sentence by the LLM. We show that while L.M L.M-assigee sentence probability is related to sentence truthfulness, this prob- ability is also dependent on sentence length and the frequencies of words in the sentence, resulting in our trained classifier providing more reliable approach detecting truthful- ness, highlighting its potential to enhance the reliability ALM--geerated content and its practical applicability in real-world scenarios.
second
largest
smallest
only
mosl
celestial body the solar systemi overr been clascified planct
éwarf solur
sysiem.
Figure tree diagram that demonstrates how gener- ating words one at time and committing to them may result in generating inaccurate information.
of confident but incorrect information, may cause significant harm, as people may accept the LLM as a knowledgeable source, and fall for its confident and compelling language, even when providing false information.
We believe that in order to perform well, LLM must have some internal notion as whether sentence is true or false, as this information is required for generating (or predicting) following tokens. For example, consider an LLM generating the following false information "TThe sun orbits the Earth." After stating this incorrect fact, the LLM is more likely to attempt to correct itself by saying that this is misconception from the past. But after stating
fact, for example "The Earth orbits the sun," it is more likely focus on other planets that orbit the sun. Therefore, we hypothesize that the truth or falsehood of a statement should be represented by, and therefore extractable from, the LLM's s internal state.
1 Introduction
Large Language Models (LLMs) have recently demonstrated remarkable success in a broad range of tasks (Brown et al., 2020; Bommarito II and Katz, 2022; Driess et al., 2023; Bubeck et al., 2023). However, when composing response, LLMs tend hallucinate facts and provide inaccurate infor- mation (Ji al., 2023). Furthermore, they seem to provide this incorrect information using confident and compelling language. The combination a broad body of knowledge, along with the provision
Interestingly, retrospectively "understanding" that statement that an LLM has just generated false does not entail that the LLM will not gener- ate it in the first place. We identify three reasons for such behavior. The first reason that an LLM gen- erates a token at a time, and it "commits" to each

---

token generated. Therefore, even if maximizing the likelihood of each token given the previous tokens, the overall likelihood of the complete statement may be low. For example, consider a statement about Pluto. The statement begins with the com- mon words "Pluto is the", then, since Pluto used to be the smallest planet the word "smallest" may be a very plausible choice. Once the sentence is "Pluto is the smallest", completing it correctly is very challenging, and when prompted to complete the sentence, GPT-4 (March 23rd version) com- pletes it incorrectly: "Pluto is the smallest dwarf planet in our solar system." In fact, Pluto is the sec- ond largest dwarf planet in our solar system (after Eris). One plausible completion of the sentence correctly is "Pluto is the smallest celestial body in the solar system that has ever been classified planet." (see Figure 1). Consider the follow- ing additional example: "Tiztoutine is a town in Africa located in the republic of Niger." Indeed, Tiztoutine is a town in Africa, and many countries' names in Africa begin with "the republic of How- ever, Tiztoutine is located in Morocco, which is not a republic, so once the LLM commits to "the republic of", it cannot complete the sentence using "Morocco", but completes it with "Niger". In addi- tion, committing to word at a time may lead the LLM to be required to complete a sentence that simply does not know how to complete. For exam- ple, when describing a city, it may predict that the next words should describe the city's population. Therefore, it may include in a sentence "Tiztou- tine has population of", but the population size is not present in the dataset, so it must complete the sentence with a pure guess.
by an LLM is truthful or not. Namely, we build classifier that receives as input the activation values of the hidden layers of an LLM. The classifier de- termines for each statement generated by the LLM if it is true or false. Importantly, the classifier is trained on out-of-distributio data, which allows to focus specifically whether the LLM has an internal representation of statement being true or false, regardless of the statement's s topic.
In order to train SAPLMA we created a dataset of true and false statements from 6 different top- ics. Each statement is fed to the LLM, and its hidden layers' values are recorded. The classifier is then trained to predict whether statement is true or false only based on the hidden layer's val- ues. Importantly, our classifier is not tested the topics is trained, but on a held-out topic. We believe that this is an important measure, as requires SAPLMA to extract the LLM's inter- nal belief, rather than learning how information must be aligned to be classified as true. We show that SAPLMA, which leverages the LLM's internal states, results in a better performance than prompt- ing the LLM to explicitly state whether a statement is true or false. Specifically, SAPLMA reaches ac- curacy levels between 60% to 80% on specific topics, while few-shot prompting achieves only slightly above random performance, with no more than a 56% accuracy level.
Of course there will be some relationship be- tween the truth/falsehood of sentence, and the probability assigned to that sentence by well- trained LLM. But the probability assigned by an LLM to a given statement depends heavily on the frequency of the tokens in the statement as well as its length. Therefore, sentence probabilities pro- vide only a weak signal of the truth/falschood of the sentence. At minimum, they must be normalized to become useful signal of the veracity of state- ment. As we discuss later, SAPLMA classifications of truth/falschood significantly outperform simple sentence probability. one test described later in more detail, we show that SAPLMA performs well on a set of L.LM-generatdd sentences that contain 50% truc, and 50% false statements. We further dis- cuss the relationship between statement probability and veracity in the discussion section.
The second reason for an LLM to provide false information is that at times, there may be many ways to complete a sentence correctly, but fewer ways to complete it incorrectly. Therefore, it might be that a single incorrect completion may have a higher likelihood than any of the correct comple- tions (when considered separately).
Finally, since common for an LLM to not use the maximal probability for the next word, but to sample according to the distribution over the words, it may sample words that result in false information.
In this paper we present our Statement Accuracy Prediction, based on Language Model Activations (SAPLMA). SAPLMA is a simple yet powerful method for detecting whether a statement generated
SAPLMA employs a simple and relatively shal- low feedforward neural network as its classifier, which requires very little computational power at inference time. Therefore, it can be computed

---

alongside the LLM's output. We propose for SAPLMA supplement an LLM presenting in- formation to users. If SAPLMA detects that the LLM "believes" that a statement that it has just generated false, the LLM can mark it as such. This could raise human trust in the LLM responses, Alternatively, the LLM may merely delete the in- correct statement and generate new one instead.
text summarization. Their benchmark is developed by gathering summaries from several summariza- tion methods and requested humans to annotate their errors. The authors analyze these anotation and the proportion of the different factual error of the summarization methods. We note that most works that consider hallucination do so with rela- tion to given input (c.g., a passage) that the model operates For example, summarization model that outputs information that does not appear in the provided article, considered hallucinate it, regard- less if the information factually truc. However, in this work we consider a different problem -the veracity of the output of an LLM, without respect to specific input.
To summarize, the contribution of this paper twofold.
The release of a dataset of true-false state- ments along with method for generating such information.
Demonstrating that an LLM might "know" when a statement that it has just generated is false, and proposing SAPLMA, a method for extracting this information.
Some methods for reducing hallucination as- sume that the LLM is black box (Peng et al., 2023). This approach uses different methods for prompting the LLM,, possibly by posting multiple queries for achieving better performance. Some methods that can be used for detecting false state- ments may include repeated queries and measuring the discrepancy among them. We note that instead of asking the LLM to answer the same query mul- tiple times, it is possible to request the LLM to rephrase the query (without changing its meaning or any possible answer) and then asking it to an- swer each rephrased question.
2 Related Work
this section we provide an overview of previ- ous research on LLM hallucination, accuracy, and methods for detecting false information, and we discuss datasets used to that end.
Many works have focused on hallucination in machine translation (Dale al., 2022; et al., 2023). For example, Dale et al. (Dale et al., 2022) consider hallucinations as translations that are detached from the source, hence they propose a method that evaluates the percentage of the source contribution to generated translation. If this con- tribution is low, they assume that the translation is detached from the source and is thus considered to be hallucinated. Their method improves detection accuracy hallucinations. The authors also propose to use multilingual embeddings, and compare the similarity between the embeddings of the source sentence and the target sentence. If this similarity is low, the target sentence is considered to be hal- lucinated. The authors show that the latter method works better. However, their approach is very dif- ferent than ours, as we do not assume any pair of source and target sentences. In addition, while we also use the internal states of the model, we do so by using the hidden states to descriminate between statements that the LLM "believes" are true and those that are false. Furtheremore, we focus on detecting false statements rather than hallucination, defined by their work.
Other methods finetune the LLM, using human feedback, reinforcement learning, both (Bakker et al., 2022; Ouyang et al., 2022). Ouyang et al. propose method to improve L.M--genertted con- tent using reinforcement learning from human feed- back. Their approach focuses on fine-tuning the LLM with a reward model based on human judg- ments, aiming to encourage the generation of bet- ter content. However, fine tuning, in general, may cause model to not perform well other tasks (Kirkpatrick et al., 2017). , In this paper, we take an intermediate approach, that is, we assume access the model parameters, but do not fine-tune or mod- ify them. Another approach that can be applied to our settings is presented by (Burns et al., 2022), named
Search (CCS). How- ever, CCS requires rephrasing statement into a question, evaluating the LLM on two different ver- sion of the prompt, and requires training data from the same dataset (topic) as the test set. These limi- tations render unsuitable for running in practice on statements generated by an LLM. In addition, CCS increases the accuracy by only approximately
Other works have focused on hallucination in text summarization (Pagnoni et al., 2021). Pagnoni et al. propose a benchmark for factuality metrics of

---

4% over the O-shot LLM query, while our approach demonstrates a nearly 20% increase over the O-shot LLM
For the first 5 topics, we used the following method to compose the dataset. We used a reliable source' that included table with several properties for each instance. For example, for the "chemical el- ements" we used a table that included, for each element, its name, atomic number, symbol, stan- dard state, group block, and a unique property (e.g.. Hydrogen, 1, H. Gas, Nonmetal, the most abun- dant element in the universe). For each element we composed true statement using the element name and one of its properties (e.g.. "The atomic number of Hydrogen is 1*). Then, we randomly selected a different row for composing a false statement (e.g., "The atomic number of Hydrogen is 34"). If the value in the different row is identical to the value in the current row, we resample a different row until obtain value that is different. This process was repeated for the all topics except the "Scien- tific Facts". For the "Scientific Facts" topic, we asked ChatGPT (Feb 13 version) to provide "sci- entific facts that are well known to humans" (e.g. "The sky is often cloudy when it's going to rain"). We then asked ChatGPT to provide the opposite of each statement such that it becomes a false state- ment (c.g.. "The sky is often clear when it's going to rain"). The statements provided by ChatGPT were manually curated, and verified by two hu- man annotators. The classification of 48 facts were questioned by at least one of the annotators; these facts were removed from the dataset. The true- false dataset comprises 6,084 sentences, including 1,458 sentences for "Cities", 876 for "Inventions", 930 for "Chemical Elements". 1,008 for "Animals", 1,200 for "Companies", and 612 for "Scientific Facts". The following are some examples of true statements from the dataset:
A dataset commonly used for training and fine- tuning LLMs is the Wizard-of-Wikipediia (Dinan et al., 2018). The Wizard-of-Wikiediia cludes interactions between a human apprentice and human wizard. The human wizard receives relevant Wikipedia articles, which should be used to select a relevant sentence and compose the re- sponse. The goal is to replace the wizard with a learned agent (such as an LLM). Another highly relevant dataset is FEVER (Thorne et al., 201 8, 2019). The FEVER dataset is designed for devel- oping models that receive input claim and passage, and must determine whether the passage supports the claim, refutes it, or does not provide enough information to support or refute it. While the FEVER dataset is highly relevant, it does not provide simple sentence that are clearly true or false independently of a provided passage. In ad- dition, the FEVER dataset is not partitioned into different topics as the truc-false dataset provided in this paper.
Wikipedia dataset in-
In conclusion, while several approaches have been proposed to address the problem of halluci- nation and inaccuracy in automatically generated content, our work is unique in its focus on utilizing the LLM's hidden layer activations to determine the veracity of generated statements. Our method offers the potential for more general applicability in real-world scenarios, operating alongside an LLM, without the need for fine-tuning or task-specific modifications.
3 The True-False Dataset
Cities: "Oranjestad is city in Aruba"
The work presented in this paper requires a dataset of true and false statements. These statements must have a clear true or false label, and must be based information present in the LLM's training data. Furthermore, since our approach intends to reveal that the hidden states of an LLM have notion of a statement being true or false, the dataset must cover several disjoint topics, such that a classifier can be trained on the LLM's activations of some topics while being tested on another. Unfortunately, we could not find any such dataset and therefore, compose the true-false dataset.
Inventions: "Grace Hopper invented the COBOL programming language"
Animals: "The Ilama has a diet of herbivore"
Companies: "Meta Platforms has headquar- ters in United States"
Scientific Facts: "TTe Earth's tides are pri- marily caused by the gravitational pull of the moon"
The following are some examples of false state- ments from the dataset:
'Cities Downloaded from simplemaps. Inventions: OB-
Our true-false dataset covers the following top- ics: "Cities", "Inventions", "Chemical Elements", "Animals", "Companies", and "Scientific Facts".
tained from Wikipedia list of inventors. Chemical Elements: Downloaded from pubchem ncbi nim nih gov periodic-table. Animals: Ohtained from kids national gengraphics. Compa- nies: Forbes Global 2000 List 2022: The Top 200.

---

"Indium is in the Lan- thanide group"
Chemical Elements:
are true and which it "believes" are false, in a gen- eral setting, and not specifically with respect to the topic being tested. To obtain more reliable results, we train each classifier three times with different initial random weights. This process is repeated for cach topic, and we report the accuracy mean over these three runs.
Animals: "The whale has a long, tubular snout, large ears, and a powerful digging abil- ity to locate and consume termites and ants." Scientific Facts: "Ice sinks in water due to its higher density"
Other candidates for topics to be added the dataset include sports, celebrities, and movies. The true-false dataset is available at: azariaa. Content/Datasetss faleee walsseddatset.2222
5 Results
We compare the performance of SAPLMA against three different baselines. The first is BERT, for which we train a classifier (with an identical archi- tecture to the one used by SAPLMA) on the BERT embeddings of each sentence. Our second baseline is few shot-learner using Ob..6.7b. This base- line an attempt reveal whether the LLM itself "knows" whether statement is true or false. Un- fortunately, any attempts to explicitly prompt the LLM in a 'zero-shot" manner to determine whether a statement is true or false completely failed with accuracy levels not going beyond 52%. Therefore, we use few shot-query instead, which provided the LLM with truth values from the same topic it is being tested on. Note that this is very dif- ferent from our methodology, but was necessary for obtaining some results. Namely, we provided few statements along with the ground-trum label, and then the statement in question. We recorded the probability that the LLM assigned to the token "true" and to the token "false". Unfortunately, the LLM had tendency to assign higher values to the "true" token; therefore, we divided the probability assigned the "true" token by the one assigned to the "false" token. Finally, we considered the LLM's prediction "true" if the value was greater than the average, and "false" otherwise. We tested a 3-shot and a 5-shot version. The third baseline, given a statement 'X', we mcasure the probabilities (using OPT-6.7) of the sentences "It is true that X", and "It is false that X", and pick the higher proba- bility (considering only the probabilities for X, not the added words). This normalizes the length and frequency factors.
4 SAPLMA
In this section, we present our Statement Accuracy Prediction, based on Language Model Activations (SAPLMA), method designed to determine the truthfulness of statements generated by L.M using the values in its hidden layers. Our general hypothesis is that the values in the hidden layers of an LLM contain information on whether the LLM "believes" that a statement is true or false. However, it is unclear which layer should be the best candi- date for retaining such information. While the last hidden layer should contain such information, it is primarily focused generating the next token. Conversely, layers closer to the input are likely fo- cused on extracting lower-level information from the input. Therefore, we use several hidden layers as candidates. We use two different LLMs: Face- book OPT-6.7b (Zhang et al., 2022) and LLAMA2- 7b (Roumeliotis et al., 2023); both composed of 32 layers. For each LLM, we compose five differ- ent models, each using activations from different layer. Namely, we use the last hidden layer, the 28th layer (which is the 4th before last), the 24th layer (which is the 8th before last), the 20th layer (which is the 12th before last), and the middle layer (which is the 16th layer). We note that each layer is composed of 4096 hidden units.
SAPLMA's classifier employs feedforward neural network, featuring three hidden layers with decreasing numbers of hidden units (256, 128, 64), all utilizing ReLU activations. The final layer is a sigmoid output. We use the Adam optimizer. We do not fine-tune any of these hyper-parameters for this task. The classifier is trained for 5 epochs.
Table and Figure 2 present the accuracy of all the models tested using the OPT-6.7b LLM, for each of the topics, along with the average aceu- racy. As depicted by the table and figure, SAPLMA clearly outperforms BERT and Few-shot learning. with BERT, 3-shot, and 5-shot learning achiev- ing only slightly above a random guess (0.50). It can also be observed that SAPLMA for OPT-6.7b
For each topic in the true-false dataset, we train the classifier using only the activation values ob- tained from all other topics and test its accuracy on the current topic. This way, the classifier is required to determine which sentences the LLM "believes"

---

Model
Cities
Invent.
Rlements
Amimus
Cump.
Facts
Average
Model
Gities
Invent
Floments
Animuls
Comp.
Farts
Average
laa--ayer
0.7796
0.5696
0.5760
0.6022
0.6925
0.6498
0.6449
Ias-layer
0,7574
0.6755
0.8814
0.7338
0,6736
0.7444
0.7107
28th-ayeer
0,7732
0.5761
0.5007
0.5777
0.7247
0,6618
0.6807
28th-aaeer
0.8146
0.7207
0.6767
0.724)
0.6892
0.7662
0.7321
240h-1yerr
0.7963
0.5800
0.775
0.6368
24th
08722
0.7816
0.6849
0.7944
0.7094
0.7858
0.7622
20th-ayeer
6.8125
0L7268
06697
08122
0.6819
0,70Ms
20th-lyer
0.8820
0.8459
0.8650
0.7758
0.8119
0.8053
0.8060
midde layer
0,7435
[U6400
0,5645
0,5800
0,7570
0.6237
0,6115
16th 16th-ayer
0.9223
0.8938
0.6839
0.7774
0,9658
0.5254
0.8298
BERT
0,5357
0.3537
0,5645
0.5228
0.5535
0,5302
0,5434
3 3-ttot
0,5110
044999
0.5650
0.538
0.164
0,5334
Table 2: Accuracy classifying truthfulness of externally generated sentences using SAPLMA with LLAMA2-7h. The table shows accuracy of all the models tested for each of the topies, and the average accuracy.
5-dhut
It true
0.525
0.5088
0,5085
0.4851
0.6385
0
0,5593
Table 1: Accuracy classifying truthfulness of externally generated sentences during reading. The table shows accuracy of all the models tested for each of the topics, and average accuracy using OPT-6.7b 7. as the LLM.
As for the differences between the topics, we believe that these values depend very much on the training data of the LLM. That is, we believe that the data used for training OPT-6.7b and LLAMA2- 7b includes information or stories about many cities and companies, and not as much on chemical ele- ments and animals (other than the very common ones). Therefore, we conjecture that is the reason SAPLMA achieves high accuracy for the "cities" topic (while trained on all the rest) and the "com- panies" topic, but achieves much lower accuracy when tested on the "animals" and "clements" top- ics.
0.7
0.4
Aweag
DESAPCMAB12ds euelt RERT Es
Figure 2: A bar-chart comparing the accuracy of SAPLMA (20th-layer), BERT, 3-shot, 5-shot, and It- is-true on the 6 topics, and the average. SAPLMA con- sistently outperforms other models across all categories. Since the data is balanced, a random classifier should obtain an accuracy of 0.5.
In addition to the topics from the true-false dataset, also created a second data set of state- ments generated by the LLM itself (the OPT-6.7b model). For generating statements, we provided a true statement not present in the dataset, and al- lowed the LLM to generate a following statement. We first filtered out any statements that were not factual statements (e.g., "I' m not familiar with the Japanese versions of the games."). ")). All statements were generated using the most probable next word at each step, i.e., we did not use sampling. This resulted in 245 labeled statements. The statements were fact-checked and manually labeled by three human judges based on web-searches. The human judges had a very high average observed agree- ment of 97.82%, and an average Cohen's 's Kappa of 0.9566. The majority determined the ground- ground-truth label for each statement. 48.6% of the statements were labeled as true, resulting in a balanced dataset. Each of the models was trained 14 times using the same classifier described in Section 4. The models were trained on the entire true-false dataset (i.c., all topics, but not the generated sentences) and tested on the generated sentences.
seems to work best when using the 20th layer (out of 32). Recall that each model was trained three times. We note that the standard deviation between the accuracy among the different runs was very small; therefore. the differences in performance between the different layers seem consistent. How- ever, the optimal layer to be used for SAPLMA is very likely to depend on the I.L.M
We note that the average training accuracy for SAPLMA (using OPT-6.7b's 20th layer) is 86.4%. We believe that this relatively high value may indi- cate once again that the veracity of a statement is in- herit to the LLM. To demonstrate this, we run test where the true/false labels are randomly permuted. The average accuracy on the random training data is only 62.5%. This indicates that our model does not have the capacity to completely over-fit the training data, and thus, must exploit structure and patterns that appear in the data.
Additionally, Table 2 presents the performance of SAPLMA using LLAMA2-7b. As expected, SAPLMA using LLAMA2-7b 7b achieves much bet- ter performance. Interestingly, for LLAMA2-7b, the middle layer performs best.
Table 3 presents the average accuracy of all mod- els on the sentences generated by the LLM. As anticipated, SAPLMA (using OPT-6.7b) clearly outperforms the baselines, which appear to be en- tirely ineffectual in this task, achieving an accuracy

---

Model
Accuracy
AUC
number of positive predictions would match the number of negative predictions, which matches the distribution in of the data.
last-layer
0.6187
0.7587
28th-layer
0.6362
0.7614
24th-layer
0.6134
0.7435
Model
Avg Threshold
Accuracy
20th-layer
0.6029
0.7182
last-layer
0.8687
0.7052
middle-layer
0.5566
0.6610
28th-layer
0.8838
0.7134
BERT
0.5115
0.5989
24th-layer
0,8801
0.6988
3-shot
0.5041
0.4845
20th-layer
0.9063
0.6587
5-shot
0.5125
0.4822
middle-layer
0.8123
0.650
Table Accuracy classifying truthfulness of sentences generated by the LLM (OPT-6.7b) itself.
BERT
0.9403
0.5705
Table 4: Accuracy classifying truthfulness of sentences generated by the LLM itself. Unlike the data in Table 3. where threshold of 0.5 the classifier output was used to classify sentences as true or false, in this ta- ble the results were obtained by estimating the optimal threshold from held-out validation data set (30% of the original test-set).
near 50%. However, the accuracy of SAPLMA on these sentences is not as promising as the accuracy achieved when tested on some of the topies in the true-false dataset (i.e.. the cities and companies). Since we expected the LLM to generate sentences that are more aligned with the data it was trained on, we did expect SAPLMA's performance the generated sentences to be closer to its performance on topics such as cities and companies, which are likely aligned with the data the LLM was trained on. However, there may be also a counter-effect in play: the sentences in the true-false dataset were mostly generated using specific pattern (except the scientific facts topic), such that each sentence is clearly either true false. However, the sentences generated by the LLM where much more open, and their truth value may be less clearly defined (despite being agreed upon by the human judges). For example, one of the sentences classified by all human judges as false is "Lima gets an average of I hour of sunshine per day." However, this sen- tence is true during the winter. Another example is "Brink is a river," which was also classified as false by all three human judges; however, brink refers to river bank (but is not name of a specifie river, and does not mean river). Indeed, SAPLMA classified approximately 70% the sentences as true, and the AUC values seem more promising. This may hint that any sentence that seems plausible is classified as true. Therefore, we evaluate the models using 30% of the generated sentences for determining which threshold to use, i.c., any prediction above the threshold is considered true. Importantly, do not use this validation set for any other goal. We test the models on the remaining 70% of the gen- crated sentences. We do not evaluate the few shot models again, as our evaluation guaranteed that the
Table 4 presents the accuracy of all models when using the optimal threshold from the validation set. Clearly, SAPLMA performs better with higher threshold. This somewhat confirms our assumption that the truth value of the sentences generated by the LLM is more subjective than those that appear in the true-false dataset. We note that also the BERT embeddings perform better with higher threshold. The use of higher threshold can also be justified by the notion that it is better to delete or mark as unsure sentence that is actually true, than to promote false information.
Another interesting observation is that the 20th- layer no longer performs best for the statements generated by the LLM, but the 28th layer seems to perform best. This is somewhat perplexing and we do not have a good guess as to why this might be happening. Nevertheless, we stress that the differ- ences between the accuracy levels of the 28th-layer and the others are statistically significant (using a two-tailed student t-test; p < 0.05). future work we will consider fusing multiple layers together, and using the intermediate activation values out- putted by the LLM for all the words appearing in the statement (rather than using only the LLM's output for the final word).
We also ran the statements generated by the OPT 6.7b model GPT-4 (March 23rd version), prompting
determine whether each statement was true or false. Specifically, we provided the following prompt "Copy the following statements and for each statement write true if it is true and

---

false if it is false:", and fed it 30 statements at a time. It achieved an accuracy level of 84.4%, and a Cohen's s Kappa agreement level of 0.6883 with the true label.
bility. However, while SAPLMA still classifies the statement "Kevin Durant is basketeer" as false, it's still much more confident that Kevin Durant is not a baseball player, in contrast to the probabilities. Since the statement "Kevin Duarnt is basketball player" has a typo, its probability is extremely low, but SAPLMA still classifies the statement as true.
6 Discussion
In this work we explicitly do not consider models that were trained or fine-tuned on data from the same topic of the test-set. This is particularly im- portant for the sentences generated by the LLM, as training on held-out set from them would allow the classifier to learn which type of sentences gen- erated by the LLM are generally true, and which are false. While this information may be useful in practice, and its usage is likely to yield much higher accuracy, it deviates from this paper's focus.
"Jennifer Aniston is female person" is im- plicit truth, a fact that is universally acknowledged without needing to be explicitly stated; thus, it is unlikely to be mentioned in the LLM's training data. Therefore, its probability is very now-much lower than "Jennifer Aniston is not an actress" despite having the same number of words. Nevertheless, SAPLMA classifies it correctly, albeit not with very high confidence.
While we show that the probabilities cannot be used alone determine the veracity of statement, they are not useless and do convey important infor- mation. Therefore, in future work we will consider providing SAPLMA with the probabilities of the generated words; however, this information may be redundant, especially if SAPLMA uses the interme- diate activation values for all the words appearing in the statement.
We note that the probability of the entire sen- tence (computed by multiplying the conditional probabilities of each word, given the previous words) cannot be directly translated to a truth value for the sentence, as many words are more common than others. Therefore, while sentence probabilities may be useful to determine which of two similar sentences is true, they cannot be used alone for the general purpose of determining the truthfulness of a given sentence.
Statement
Label Probubility
SAPLMA
(ayt-layer
In Table 5 we compare the probability assigned by the LLM and the sigmoid output from SAPLMA on 14 statements, which do not appear in the true- false dataset. We use the 28-layer, as it proved to perform best on the statements generated by the LLM, but we note that other layers provide very similar results on this set of statements. As depicted by the table, the probabilities provided by the LLM are highly susceptible to the syntax, i.e., the exact words and the statement's length. The first two sets of examples in the table illustrate how sentence length highly affects the probabilities, but not SPLMA. In the following examples the false statements not necessarily shorter than the true statements, yet the probabilities remain highly unreliable, while SPLMA generally succeeds in making accurate predictions.
1120 water, essential for
Tru
6614--16
0.9032
Humans don't need water
False
2.650-10
0.0282
The sun is hot and rodiates its heat to Earth protects Earth hast
True
101E-17
0.9620
Folar
2.036-14
0.3751
The Earth flat The world is round and rotones flat like pomezake
False
5:276-07
00342
True
2966-11
0.6191
Folle
3.88-10
0.0097
Kevin Durant a basketball playe
True
2
0.9883
Kevin Durant a buschall player Kevin Dumat icsi brodetioon
Falie
4 4.56E-12
0
5.78F-16
0.469
Kevin Duarst is basketbull playe
True
1.526-21
0.7105
Jennifer Aniston actress
True
1 10
0U9R5
Jennifer Aniston Jennifer Aniston is female person
Fille
1.14F-11
0
Trme
2.78E-
0.6433
Hurry Pottrr real
Falie
9.46E-09
0.0016
Hurry Potter fickknal
1.535-09
0.9266
Harry Potee in imaginary figure
Trme
6.31E 14
0.8354
Table 5: Comparison of the probability assigned by the LLM and the sigmoid output from SAPLMA's 28th layer (using OPT-6.7b), color-coded for clarity. SAPLMA'S values are much better aligned with the truth value.
The statement "The Earth is flat" as well as "The Earth is flat like pancake" probably appear several times in the LLM's training data, therefore, it has relatively high probability; however, SAPLMA is not baffled by it and is almost certain that both sen- tences are false. basketeer a rare word meaning a basketball player. It seems that the LLM is not familiar with the phrase and assigns it a low proba-
Conclusions & Future Work
In this paper, we tackle fundamental problem associated with LLMs, i.e., the generation of in- correct and false information. We have introduced SAPLMA, a method that leverages the hidden layer activations of an LLM to predict the truthfulness

---

of generated statements. We demonstrated that SAPLMA outperforms few-shot prompting in de- tecting whether a statement is true or false, achiev- ing accuracy levels between 60% to 80% on spe- cific topics when using OPT-6.7b, and between 70% to 90% when using LLAMA2-7b. This is significant improvement over the maximum 56% accuracy level achieved by few-shot prompting (for OPT-6.7b).
and Yi, 2021). The overall system can benefit from multiple outcomes, such that if the LLM is not pos- itive whether the statement is true or false, it can be marked for the user to treat with caution. However, if a statement is classified as false, the LLM can delete it and generate a different statement instead. To avoid regenerating the same statement again, the probability of sampling the words that appear in the current statement should be adjusted downward. Our work was only tested in English. However, we believe that a multilingual LLM can be trained on one language and applied on statements in an- other language. We will test this hypothesis in future work.
Our findings suggest that LLMs possess an in- ternal representation of statement accuracy, which can be harnessed by SAPLMA to filter out incor- rect information before it reaches the user, and furthermore that this representation of accuracy is very different from the probability assigned to the sentence by the LLM. Using SAPLMA as a supplement to LLMs may increase human trust the generated responses and mitigate the risks associated with the dissemination of false informa- tion. Furthermore, we have released the true-false dataset and proposed a method for generating such data. This dataset, along with our methodology. provides valuable resources for future research in improving LLMs' abilities generate accurate and reliable information.
In work, collected the activation val- ues when each sentence was generated separately. However, in practice, in an LLM generating longer responses the activation values develop over time, so they may process both correct and incorrect in- formation. Therefore, the activation values would need to be decoupled so that they can be tested whether the most recent statement was true or false. One approach might be to subtract the value of the activations obtained after the previous statement from the current activation values (a discrete deriva- tive). Clearly, training must be performed using the same approach.
In future work we intend to apply our method to larger LLMs, and run experiments with humans, such that control group will interact with unfil- tered LLM, and the experimental group will inter- act with system that augments SAPLMA and an LLM. We hope to demonstrate that humans trust and better understand the limitations of system that is able to review itself and mark statements that it is unsure about. We also intend to study how the activations develop over time as additional words are generated, and consider multilingual input.
Ethical Impact
One of the primary ethical concerns of LLMs is the generation of false information; yet, we believe that SAPLMA could potentially reduce this issue. On the other hand, it is important to acknowledge that certain ethical issues, such bias, may per- sist, potentially being transferred from the LLM SAPLMA. Specifically, if the LLM exhibits bias towards certain ethnic groups, SAPLMA may like- wise classify statements as true or false based on these inherited biases from the original LLM. Nev- ertheless, it may be possible to adapt the approach presented in this paper to bias mitigation.
8 Limitations
This paper focuses on detecting whether a state- ment is true or false. However, practice, it may be more beneficial to detect the LLM is positive that a statement is correct or it is unsure. The most simple adjustment to the proposed method in this paper is to lift the required threshold for classi- fying a statement as true above 0.5; however, the exact value would require some form of calibration of the model (Bella et al., 2010). Another option is use multiple classifiers and require all (or a vast majority of) classifiers to output "true", for the statement to be considered true. Alternatively, dropout layers can be used for the same goal (Chen
10 Acknowledgments
This work was supported, in part, by the Ministry of Science and Technology, Israel, and by the Israel Innovation Authority.
References
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Camphell-Gilinggann, Jan

---

Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. 2022. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Infor- mation Processing Systems, 35:381 15:38176-3811 17-388889.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka
al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national acudemy of sciences, 114(13)3521-3526
Antonio Bella, César Ferri José Hernández-Oraee and María José Ceanirez-Quintane 2010. Calibration of machine learning models. In Handbook Research on Machine Learning Applications and Trends: Al- gorithms, Methods, and Techniques, pages 128-146. IGI Global.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744
Michael Bommarito II and Daniel Martin Katz. 2022. Gpt takes the bar exam. arXiv preprint arXiv:2212.14402.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with frank: A benchmark for factuality metrics. arXiv preprint arXiv:2104. 13346.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:18 1377-9919
Baolin Peng, Michel Galley, Pengeheng He, Hao Cheng, Yujia Xie. Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302. 12813.
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lec, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303.1270
Konstantinos I Roumeliotis, Nikolaos D Tselikas, and Dimitrios K Nasiopoulos. 2023. Llama 2: Early adopters' utilization meta's new open-source pre- trained model.
James Thorne Andreas Vlachos, Christos Christoddoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.053:
Collin Burns, Haotian Ye, Dan Klein, and Jacob Stein- hardt, 2022, Discovering latent knowledge in lan- guage models without supervision. arXiv preprint arXiv:2212.03827.
James Thore, Andreas Vlachos, Oana Cocarascu. Christos Christodoulopoulos, and Arpit Mittal. 2019.
Yuanyuan Chen and Zhang Yi. 2021. Adaptive sparse dropout: Learning the certainty and uncertainty in deep neural networks. Neurocomputing, 450:354 361.
The fever2. 0 shared task. In Proceedings of the Sec- ond Workshop on Fact Extraction and VERification (FEVER), pages 1-6.
David Dale, Elena Voita, Loic Barrault, and Marta R Costa-jussà. 2022. Detecting and mitigating halluci- nations in machine translation: Model internal work- ings alone do well, sentence similarity even better. arXiv preprint 272007221
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.
Opt: Open pre-trained transformer language models. arXiv preprint arx:::2205.00000
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powee powered conversational agents. arXiv preprint 1811:::211121244
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. arXiv preprint Niv.23003.03378.
Ziwei Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung, 2023. Survey of halluci- nation natural language generation, ACM Comput- ing Surveys, 55 35(12):1-38


Reviews:
[{"review_id": "TANNXBr2Bw", "reviewer": "Reviewer_72bu", "summary": "", "questions": "I have a few questions to the authors:\n\nA. Regarding BERT classifiers and the strategy of using generation probabilities, what would the results be, if they are posed a simplified task of comparing pairs of true / false statements? (Given the construction method of the dataset, the true and false statements should be in pairs) This would serve as a measure to the information these perplexity/pseudo-perplexity values have on the veracity of statements, if the length/frequency factors etc. could be marginalized.\n\nB. Given the difference in accuracy values with the change in thresholds, it feels it would ultimately be better to report AUC values instead of these accuracy values, for the AUC values are a more comprehensive measure of discriminative power along the spectrum of thresholds. Thoughts?\n\nC. Considering future work, is it possible for the LM to tell at which tokens did its own outputs begin to derail?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The reviewer raises several specific questions for the authors – we address these here:\n\nA. As another new baseline, given a statement X, we measured the probabilities (using OPT-6.7) of the sentences “It is true that X”, and “It is false that X”, and picked the higher probability (considering only the probabilities for X, not the added words). This normalizes the length/frequency factors, as suggested by the reviewer. This results in an improvement over the zero-shot or even few-shot approach, with an accuracy of between 52.3% to 60.4%, depending on the topic, but is still way below SAPLMA’s performance. These results will be added to the paper as another baseline. Note that, if deployed in practice, each method will only receive a single sentence at a time. While it is possible to automatically generate two competing sentences (e.g., it is true that X, and it is false that X), it is not possible (in practice) to obtain a complementing sentence from a dataset.\n\nB. We agree that AUC captures the performance of the methods under multiple thresholds and thus we also report AUC. However, if our system were to be used in practice, one must first determine the threshold based on some data and then use that threshold for inference. We are interested to know how well such a system would perform in practice.\n\nC. The reviewer raises an interesting idea for future work. Our current attempt is to detect which sentence generated by an LLM contains false information, so that it can be marked or deleted and a new sentence generated instead. We agree that a more interesting (and challenging) task would be to detect which token caused the sentence to begin leaning toward a false statement. One can envision a system that first attempts to correct a single word, then, if it fails a few times it corrects a full sentence, and then if it fails a few times, it regenerates a paragraph, and so on. This might be somewhat similar to how humans generate text. We note that while humans many times delete and rewrite sentences, this data is usually not recorded, and thus, current LLMs never predict a ‘delete’ token.\n\nWe thank the reviewer for the pointer to relevant work, which will be added to the paper.\n\nRegarding reproducibility of our results, we note that the code and dataset were submitted as the supplementary material and will be made public if accepted. We could not make the data public due to the anonymity period of EMNLP. Furthermore, this work was already reproduced by multiple scholars worldwide, who contacted us to obtain the data and code, which we happily provided.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem targeted by this paper, the hallucination issue and improving trust in LLMs, is one of great importance to the deployment of LLMs. The proposed method shows promising performance with lightweight architecture, experiments are thorough in the different conditions, generalization from artificial training data to actual model generations is also covered.\n\nIn the analysis of why a model would generate false claims even when they are able to discern them from true ones, the first reason discussed is also inspiring.", "reasons_to_reject": "The method is only evaluated on a dataset that the authors created with the paper, where the exact sources of information for these datasets, as well as the construction details (e.g. how the tables are turned into sentences, etc.), are not mentioned (lines 324-325)\n\nWhile the authors compared with a BERT classifier and few-shot self-correction, the pool of baselines feels a bit thin, with a lack of external baselines tackling the same problem (e.g. the missing reference listed below).\n\nThe scope of the paper is limited to simple factoid sentences, where the veracity of factoid claims embedded in more complex utterances, or the veracity of the more complex utterances themselves are not covered.\n\nA number of presentation improvements could be made, for instance, the second reason-for-generation failure in lines 105-111 is not obvious, need further support.", "questions_to_authors": "I have a few questions to the authors:\n\nA. Regarding BERT classifiers and the strategy of using generation probabilities, what would the results be, if they are posed a simplified task of comparing pairs of true / false statements? (Given the construction method of the dataset, the true and false statements should be in pairs) This would serve as a measure to the information these perplexity/pseudo-perplexity values have on the veracity of statements, if the length/frequency factors etc. could be marginalized.\n\nB. Given the difference in accuracy values with the change in thresholds, it feels it would ultimately be better to report AUC values instead of these accuracy values, for the AUC values are a more comprehensive measure of discriminative power along the spectrum of thresholds. Thoughts?\n\nC. Considering future work, is it possible for the LM to tell at which tokens did its own outputs begin to derail?", "paper_topic_and_main_contributions": "The paper describes a method to get a model to identify the veracity of its own generations at a low additional cost. It shows that with a simple classifier on top of certain layers of a model's output hidden states, the veracity prediction of model generations can be learned with a modest amount of training data, and is generalizable to unseen domains.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "lines 018-020: repetitive to lines 008-010;\nline 118: capital M in Model;\nline 205: multilingual\nline 222: text\nline 267: LM/LLM, not LMM"}, "extra_text": {"Missing_References": "The following paper also tackles the problem of having a LM critique on its own predictions: Language Models (Mostly) Know What They Know (https://arxiv.org/abs/2207.05221). While it mostly concerns the problem of calibration of logit likelihoods, the two papers are of similar spirits and the methods may serve as additional baselines."}}, {"review_id": "lC8li2PUPs", "reviewer": "Reviewer_DLxR", "summary": "", "questions": "* The methods section reads \"We used a reliable source that included a table with several properties for each instance\". What is this source?\n\n* Did you verify the scientific-facts portion of the data, generated by ChatGPT?\n\n* The paper reports that trying to prompt GPT directly to decide whether the statement is true of false yielded random accuracy. This is a very surprising - is that true for all categories in the data? Did you try to look at the probabilities of the actual true and false answers, rather than at the probabilities of the tokens \"true\" and \"false\"?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The reviewer raises several issues and questions, which we address below.\n\nThe dataset we provide is indeed completely balanced. \n\nFollowing the reviewer’s comment we will rename the title to “An LLM’s Internal State Can Indicate When False Information Is Generated” (and remove the use of the controversial word “knows”).  \n\nOur true-false dataset is composed of 6 very different topics, and we consider sentences generated by the LLM as another dataset. The paper focuses on the OPT-6.7B model, but we also ran experiments on OPT-2.3B, OPT-1.3B, OPT-350M, and OPT-125M. We will briefly report these results. In addition, we will provide results for the LLAMA 2 model in the final version of the paper.\n\nThe tables for the topics were obtained from the following sources:\nCities: Downloaded from simplemaps.\nInventions: Obtained from Wikipedia list of inventors.\nChemical Elements: Downloaded from pubchem ncbi nlm nih gov periodic-table.\nAnimals: Obtained from kids national geographics.\nCompanies: Forbes Global 2000 List 2022: The Top 200.\n\nThe scientific facts, which were obtained from ChatGPT, were verified by two human annotators. Indeed, the classification of 48 facts were questioned by at least one of the annotators. These facts were removed from the dataset. We will add this information to the paper.\n\nAs another new baseline, given a statement X, we measured the probabilities (using OPT-6.7) of the sentences “It is true that X”, and “It is false that X”, and picked the higher probability (considering only the probabilities for X, not the added words). This normalizes the length/frequency factors. This results in an improvement over the zero-shot or even few-shot approach, with an accuracy of between 52.3% to 60.4%, depending on the topic, but is still way below SAPLMA’s performance. These results will be added to the paper as another baseline. We ran a logistic regression, which resulted in a drop of 2.5% in the accuracy, on average. We will add this to the paper.\n\nRegarding reproducibility of our results, we note that the code and dataset were submitted as the supplementary material and will be made public if accepted. We could not make the data public due to the anonymity period of EMNLP. Furthermore, this work was already reproduced by multiple scholars worldwide, who contacted us for obtaining the data and code.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses an important question -- hallucinations in LM generation and ways to identify them. The key result is nontrivial, showing that models encode the truthfulness of an answer regardless of the text they actually generate as a continuation of the prompt. The leave-on-out design of the probe is nice and is showing good generalization to different domains (although it would be nice to *also* report numbers on the same domain). ", "reasons_to_reject": "* Testing on a single dataset and a single model. At the very least, the results should be replicated on different OPT models and on different factual datasets.\n\n* Several design decisions are not clearly motivated - see questions for the authors. Particularly, it is not clear whether the GPT-generated portion of the data was validated.\n\n* Assuming the data is balanced, the accuracy scores reported are low. This puts into questions the claims that models \"know\" when the statement is true.\n\n* It is not clear whether the information extracted by the probe is being used by the model (in the causal sense). It would be nice to include linear probes and perform linear erasure / amnesic probing experiments.", "questions_to_authors": "* The methods section reads \"We used a reliable source that included a table with several properties for each instance\". What is this source?\n\n* Did you verify the scientific-facts portion of the data, generated by ChatGPT?\n\n* The paper reports that trying to prompt GPT directly to decide whether the statement is true of false yielded random accuracy. This is a very surprising - is that true for all categories in the data? Did you try to look at the probabilities of the actual true and false answers, rather than at the probabilities of the tokens \"true\" and \"false\"?", "paper_topic_and_main_contributions": "The paper shows evidence for the encoding of the factuality-related information in the latent representations of LMs, even when they generate nonfactual text, providing evidence for a disparity between the content of the representation space and the nature of the generated text. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "teW6R1nwYi", "reviewer": "Reviewer_UwiC", "summary": "", "questions": "- A. Can you explain why you \"expect that the BERT embeddings will perform very poorly, as they focus on the semantics of the statement rather than on whether a statement is true or false.\" ? It's unclear for me.\n- B. Can you provide the experiment results for larger models like OPT-175B?\n- C. Can you provide some ablation study or analysis to show that your classifier architecture is optimal?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "As the reviewer points out, we do not know whether the gap in SAPLMA’s performance is due to its inability to detect if a statement is true or not, or whether statements similar to the current statements appear often in the data used to train the LLM; this is a keen observation. We show that the performance of SAPLMA varies significantly between the different topics of the dataset. This is very likely due to the distribution of the data used for training the LLM .  However, even if an exact statement appears in the dataset,the LLM is not guaranteed to predict it correctly. We note that the inventions topic is based on a Wikipedia page, which is part of the OPT training dataset; however, the performance on this topic is relatively poor. In addition, due to the nature of the true-false dataset, it is quite unlikely that any significant portion of it has changed in the last 3 years.\n\nBERT’s objective is to predict the value of masked or altered words. Therefore, its embeddings capture the semantics of the sentence. However, the veracity of a statement seems irrelevant to the task it tries to accomplish (except in very specific cases). We claim that detecting the veracity of a statement is more appropriate for generative LLMs, whose objective is to predict the next word in a sequence. We note that SAPLMA using OPT-125MB, which has fewer parameters than BERT, outperforms it by 2.4% on the true-false dataset. Nevertheless, following the reviewer’s recommendation, we will move the discussion of this hypothesis to after the results.\n\nWe will provide results for a larger language model in the final version of the paper based on LLAMA 2; however, it is unlikely that we will achieve the resources required to run SAPLMA with a huge model such as OPT-175B .\n\nWe do not claim that the proposed neural network architecture is optimal. In fact, we intentionally did not consider multiple architectures since we believe that the correct way to consider multiple architectures requires an additional train test validation split to avoid the winner’s curse (i.e., the selected architecture would be likely to not perform as well in practice as it did on the test set). We ran a logistic regression, which resulted in a drop of 2.5% in accuracy. We will add this to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Addresses an important issue - false information generated confidently by large LLMs. Proposes a practical method to detect such false statements.\n- Thorough experiments with different LLM layer activations, different topics, and human-labeled LLM-generated statements. Compares well to baselines.", "reasons_to_reject": "- I don't know why the author claimed that \"we expect that the BERT embeddings will perform very poorly, as they focus on the semantics of the statement rather than on whether a statement is true or false. \" In my opinion, the classifier trained on OPT-175B and the classifier trained on BERT should be serving exactly the same function. The only difference is the model size, BERT is relatively small, while OPT-6.7B is more powerful. It's weird to say that \"BERT embedding focus on the semantics, not true/falsehood\". This should be your conclusion/explanation after you see the experiment results, not your claim before doing the experiment. Please explain if I misunderstand your claim here.\n- As you didn't consider inputting any grounded documents when examining the truthfulness of the model, a better way is that you should only construct the examples using the pretraining data of OPT-6.7B. You showed that the final accuracy is around 70%, but we cannot know whether the remaining 30% accuracy gap is caused by #1) OPT-6.7B has not been trained on some of the new knowledge existing in your new dataset, so it's unsure (lying) about its prediction, even if the input sentence is truthful #2) the classifier is just not good enough. If you can construct a dataset in this way, we can leave out reason #1 and only focus on reason #2.\n- No experiments on larger models like GPT-175B or LLaMA-65B. Unclear if findings transfer to more capable LLMs.\n- The classifier architecture used seems arbitrary. No tuning of hyperparameters. More analysis is needed on optimal architectures.", "questions_to_authors": "- A. Can you explain why you \"expect that the BERT embeddings will perform very poorly, as they focus on the semantics of the statement rather than on whether a statement is true or false.\" ? It's unclear for me.\n- B. Can you provide the experiment results for larger models like OPT-175B?\n- C. Can you provide some ablation study or analysis to show that your classifier architecture is optimal?", "paper_topic_and_main_contributions": "This paper proposes a method called SAPLMA to detect whether statements generated by large language models (LLMs) like GPT are true or false. The key idea is to use the hidden layer activations of an LLM and classify whether the LLM \"believes\" the statement is true/false. The authors create a dataset of true/false factual statements across 6 topics, train a classifier on 5 sets and test on the hold-out set. This ensures the classifier learns to extract the LLM's internal signal of truth/falsehood rather than topic-specific knowledge.\n\nExperiments with OPT-6.7B showed that SAPLMA achieves ~70% accuracy on held-out topics, significantly outperforming 1) few-shot prompting of the LLM itself 2) classifiers trained on BERT embeddings. The 20th hidden layer works best. SAPLMA also gets 63% accuracy on statements generated by the LLM, again outperforming baselines.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "8Bg7T1CXO6", "reviewer": "Reviewer_ayRU", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We took the reviewer’s suggestion to run a test where the true/false labels were randomly permuted.  As expected, the test accuracy in this case came out to be only 50.1% (compared to test accuracy of 70.9% for SAPLMA). This is totally expected, since the model wasn't exposed to these labels during training, making its predictions purely arbitrary. However, even the accuracy on the random training data was only 62.5% (compared to training accuracy of 86.4% for SAPLMA). This indicates that our model does not have the capacity to completely over-fit the training data, and thus, must exploit structure and patterns that appear in the data. We will add a discussion of this point to the paper.\n\nFollowing the reviewer’s comment we will rename the title to “An LLM’s Internal State Can Indicate When False Information Is Generated” (and remove the use of the controversial word “lying”).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a very interesting and important question. The experiments are generally well-designed and informative.\nThe organization and narrative of the paper is excellent, making very clear what was done and why, with good examples.", "reasons_to_reject": "There are significant numbers of small writing errors that do not impede comprehension or readability. A careful rewrite by an author who did not do much of the initial writing would fix this. These are obvious errors that would be caught by an on-line style and grammar checker.\n\nThe authors test five layers. This is a multiple compatsions setting, so statistics aiming to show that the result is real should use some kind of correction for multiple comparisons. \n\nIt is also worth, for safety, running a simple control in which the true/false labels are randomly permuted, so that each sentence is associated with the truth value of a randomly chosen counterpart. If all is well, we will find that the model is unable to predict the permuted labels with any accuracy.  This will help to establish beyond a reasonable doubt that the observed performance is due to the ability to detect truth, not to some strange effect of the behavior of large capacity models capable of learning almost anything. Possibly, doing this test is overly paranoid, but since it is cheap to do and the result is high-impact, why not?\n\nI don't like the use of the anthropomorphic term \"lying\" in the title and intro. It's just about OK, because of the very good discussion of why the model might produce false statements even though both another LLM and SAPLMA can tell that they are false. But I would still prefer a more moderate phrasing in the title.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores the hypothesis that there exists, somewhere in a large language model, a generalizable representation of the truth or falsity of a statement. The authors use a small special-purpose network to train a classifier that can convert the outputs of any hidden layer into an estimate of the truthfulness of the statement. The performance of classifiers of this type leads the authors to conclude that the generalized representation is indeed present.\n\nIn service of the hypothesis, the authors create a small corpus of examples, spanning several domains. This will be a resource for replications and future studies.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "see commenta bove"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received slightly disparate scores: 3,2,3,4 (soundness) and 4,3,4,4 (excitement). Three of the reviewers have rated the soundness & excitement as Good or Strong, whereas one reviewer (R2 / DLxR) raises objections and provides lower scores.

Condensing the reviews, the following major strengths and weaknesses were noted:

Strengths:
- everyone agreed that the paper targets a key issue in LLMs
- the reviewers specifically praised various aspects of the paper, such that nontriviality of the result (R2), thorough experiments (R3), and good presentation (R4)

Weaknesses:

- testing on a single dataset (R1,R2) and a single model (R2,R3), with some concerns about lacking details on the newly constructed dataset (R1)
- a limited set of baselines (R1)
- scope is limited to simple factoid sentences (R1)
- a range of unclear design decisions were mentioned (R2,R3)
- concerns about the link between extractability with a probe and causality on the model behavior (R2)

R2 provided a low soundness score. However, the concerns they mentioned were overall similar to those mentioned by the other reviewers, and the authors addressed them to some extent. I thus believe that these soundness concerns do not warrant rejecting the paper.
Decision: Accept-Findings