Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability
Authors: Jiri Hron
Laura A Culp
Gamaleldin Fathy Elsayed
Rosanne Liu
Jasper Snoek
Simon Kornblith
Alex Rizkowsky
Isabelle Simpson
Jascha Sohl-Dickstein
Noah Fiedel
Aaron T Parisi
Alexander A Alemi
Azade Nova
Ben Adlam
Bernd Bohnet
Gaurav Mishra
Hanie Sedghi
Izzeddin Gur
Jaehoon Lee
John D Co-Reyes
Kathleen Kenealy
Kelvin Xu
Kevin Swersky
Igor Mordatch
Lechao Xiao
Maxwell Bileschi
Peter J Liu
Roman Novak
Sharad Vikram
Tris Warkentin
Jeffrey Pennington
Venue: COLM (2024)
Keywords: Language models, Knowledge graph, Hallucination, Scaling
TL;DR: We trained language models on the Knowledge Graph to investigate models' hallucination behavior and detectability with regards to scale.

Abstract: While many capabilities of language models (LMs) improve with increased training budget, the influence of scale on hallucinations is not yet fully understood. Hallucinations come in many forms, and there is no universally accepted definition. We thus focus on studying only those hallucinations where a correct answer appears verbatim in the training set. To fully control the training data content, we construct a knowledge graph (KG)-based dataset, and use it to train a set of increasingly large LMs. We find that fora fixed dataset, larger and longer-trained LMs hallucinate less. However, hallucinating on≤5% of the training data requires an order of magnitude larger model, and thus an order of magnitude more compute, than Hoffmann et al. (2022) reported was optimal. Given this costliness, we study how hallucination detectors depend on scale. While we see detector size improves performance on fixed LM’s outputs, we find an inverse relationship between the scale of the LM and the detectability of its hallucinations.

Published as a conference paper at COLM 2024
Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability
Jiri Hron't
Laura Culp"
Gamaleldin Elsayed"
Rosanne Liu"
Ben Adlam
Maxwell Bileschi Bernd Bohnet JD Co-Reyes Noah Fiedel
C. Daniel Freeman
Izzeddin Gur Kathleen Kenealy Jaehoon Lee
Peter J. Liu
Gaurav Mishra Igor Mordatch Azade Nova
Roman Novak
Aaron Parisi Jeffrey Pennington Alex Rizkowsky Isabelle Simpson
Hanie Sedghi
Jascha Sohl-dickstein Kevin Swersky Sharad Vikram
Tris Warkentin Lechao Xiao Kelvin Xu
Jasper Snoek"
Simon Kornblith
Google DeepMind
Abstract
While many capabilities of language models (LMs) improve with increased training budget, the influence of scale on hallucinations is not yet fully un- derstood. Hallucinations come in many forms, and there no universally accepted definition. We thus focus on studying only those hallucinations where correct answer appears verbatim in the training set. To fully con- trol the training data content, we construct knowledge graph (KG)-based dataset, and use it to train set of increasingly large LMs. We find that for a fixed dataset, larger and longer-trained LMs hallucinate less. However, hallucinating on S. 5% of the training data requires an order of magnitude larger model, and thus an order of magnitude more compute, than Hoff- mann et al. (2022) reported was optimal. Given this costliness, we study how hallucination detectors depend on scale. While we see detector size im- proves performance fixed LM's outputs, find an inverse relationship between the scale of the LM and the detectability of its hallucinations.
1 Introduction
Despite rapid progress in generative and predictive capabilities, hallucinations remain
significant challenge for large language models (Gemini 2023 OpenAl 2023). Although researchers have carefully studied "scaling laws" (Kaplan et al. 2020 Hotimann et al.)
2022 an empirical phenomenon where LM performance improves as dataset and model size increase-litle is known about how hallucinations depend scale. To fill this gap, the first issue at hand is to precisely define and quantify hallucinations. However, in natural language setting this is very hard as language expressions can be ambitious, and the exact knowledge content in training data is notoriously unclear.
Knowledge graph (KG), the other hand, offers full controllablility of its factual content: is ttraightfowward to query whether a generated fact from an LM indeed exists in the dataset or not, hence offering quantifiable measure of hallucination. Training LMs on KG allows us to study the extent to which LMs misrepresent their training data, and how this phenomenon depends on scale.
"Core contributors. Correspond to jiihron@google.com *Equal contribution.
Work done while Google DeepMind.

---

Published as a conference paper at COLM 2024
(a) Original data (KG);
(b) Formatted data to pretrain LMs:
TTNN Subject 0 <P_TKNS Predicate 0 -O_TRN- Object _
"Tensortoow" "Software of"
"Google"
("Tensodlow")
("Software in gonee") ("Machine learning))
Founded by"
"Founded by"
<8_ S_TKN> Subject_ I <P_TKN> Predicate 1 <0_ -OTTKN> Object_
"Software genre"
"Sengey Brin"
"Larry Page"
(c) Evaluating pretrained LMs:
Inventor
Inventor"
TKN Subject_0 0 <P_ P_TNN> Predicate_0 <0_ TKN>
Object D
"Machine leammin"
Tagge Rank"
<
IKN> Subject <p TKN Predicate <O_IKN>
LM
Object 1
(d) Hallucination detector:
TKN= Subject_ i <P <P_TKN= Predicate_ <0__Ko_TKTA
TKN>
LM
Detector
Hallucination!
Figure 1: Data and the training pipeline. <S_TKN>, <P_TKN> and <0_TKN> are special tokens indicating subject, predicate, and object, respectively. (a) The original data exist in the form of Knowledge Graph (KG), where nodes representing subjects and objects are connected by predicates (arrows). (b) The KG then formatted into triplets: subject, predicate, object, and further prefixed with special tokens indicating their identity. Such formatted data are used to pretrain autoregressive LMs with the common next-token-rrediction loss. (c) Pretrained LMs are evaluated by prefixing with subject and predicate alongside special tokens to predict objects. (d) On top of pretrained LMs, detectors are trained to detect the presence of hallucinations during generation.
We therefore train a number of increasingly large LMs from scratch on data extracted from KG. A KG contains knowledge about the world in the form of [subject, predicate, object] triplets (Figure aa)). We concatenate the triplets into strings, and use these to train the LMs by optimising the auto-regressive log-likelihood [Tigure[[b)) When evaluating, we prompt the model with a subject and predicate, and let it complete the object (Figure I (c)) generation is considered a hallucination if the prediction does not match any object that appears with given subject-predicate pair in the training set.
The speciality to this setup is that each unique piece of information only seen once per epoch: each triplet contains unique piece of information that is distinctive from any other triplet. This unlike any traditional LMS trained over natural language, where the same information content may appear multiple times even in deduplicated datasets, due to the flexibility of natural language. This difference engenders three major observations that deviate from our current understanding of LM (pre)training text.
First, multi-epoch training is necessary, as one might expect in this setting, for the training loss to converge and hallucination rate to go down to an acceptable level. Secondly, the scaling between the model size and the number of training tokens is different from what is currently considered optimal (Hoffmann et al. 2022;; LMs several times larger are needed to eliminate training set hallucinations. What's more, counter to standard scaling laws (Kaplan) et 2020 Hoffmann et al., 2022), the non-duplication of triplets also means that increasing the training set size (adding more pieces of information) leads to worse performance for any fixed LM size and training length. To decrease the training set hallucination rate, one can train for longer and set the generation temperature to zero, but this often comes at the cost of worse generalisation to data not seen during training. Thus, there is trade-off between training set hallucination rate and other LM capabilities.
Equipped with a range of LMs trained from KG with various hallucination rates, we then investigate methods that prevent hallucinations by detecting- and optionally also correcting-therr post-hoc, as well as the scaling behaviour of such methods (Figure [(d). While we do find that larger detectors perform better, and that detection accuracy improves with the LM size and length of pretraining, there is a caveat: the improvement in accuracy with LM scale is confounded by the already lower hallucination rate of the more powerful LMs. Examining the detectors' precision-recall (PR) curves, we find an inverse relationship between the LM size and the detector's 's area under the PR curve (AUC-PR). In other words,
2

---

Published as a conference paper at COLM 2024
Language Model
Detector
Dataset
Size (total triplets)
Trained on?
Eval-ed on?
Trained on?
Eval-ed on?
FVS, 1%
6,505,590 65,150,414
FVS, 10%
FVS, 100%
/
/
X
555,969,00
PVS, 10%
X
159,502 1,606,058
PVS, 100%
/
X
IVS
x
X
19,311
Table 1: Datasets created with different levels of visibility and scale. The fully visible set (FVS) is used for training of both the LMs and detectors (except for the 100% version). The partially visible set (PVS) is only used for the training of LMs, not the detectors. The invisible set (IVS) is fully held-out test used only for some of the evaluations. For each visibility level, different dataset sizes are created by sampling 1% or 10% of the respective full set. The total number of triplets contained in dataset version is shown. We always evaluate LMs on both training and test sets, while only evaluate detectors on held-out test sets.
Training epochs
Dataset
2
10
20
100
200
FVS PVS, 1% FVS + PVS, 10% /
/
/
/
X
X
FVS + PVS, 100%
/
X
X
Table 2: Length of training for different dataset size. We train a maximum of 200 epochs, on the smallest subset, and as the training set grows, we reduce the training length.
the more powerful the LM, the harder to detect its hallucinations. This despite the fact that we have control over what the model is exposed to, which has been thought to be key to making hallucination detectors work well (Schulman) 2023
The rest of the paper is structured as follows. We describe the dataset and training of LMs in Section 2 investigate the interplay between hallucinations and LM scale Section 3 and study the relation between LM scale and hallucination detectability in Section The remaining sections are devoted to the discussion of limitations and takeaways.
Controlling What an LM Knows
core challenge in studying LM hallucinations that we typically do not know what infor- mation the model exposed to during training. Without this knowledge, we cannot deter- mine whether the model output is wrong because (a) it has never been trained on a given fact, (b) it trained on but did not memorize it, or (c) the model memorized factually incor- rect information that did appear in the training set. To avoid these issues, which are further confounded by various pretraining and finetuning strategies used in state-of-the-ar LMs, we train LMs from scratch (Section 2.2) on data specifically constructed to give us perfect control over the information a model sees (Section 2.1) This will later enable us to investigate how model and dataset scale affects hallucinations (Section 3), and their detectability (Section
2.1 The Knowledge Graph dataset
We propose using Knowledge Graph (KG) as a way of controlling the information a model sees. KGs structured, factual data, that are often used within organisations to feed knowledge into various applications. We use KG as it provides a repository of information which is self-consistent, and mirrors the structure of information in the real-world; the hope
3

---

Published as a conference paper at COLM 2024
is that this mirrored structure will ensure that the character of any hallucinations we see is somewhat similar to hallucinations we would see from models trained on data more typical for LM training. The main benefit to using KG, however, is that we know exactly what a model has seen, and since it is structured data, we can easily query the data to see if its predictions are correct.
The KG we use (Google, 2012) contains semantic triples: a Subject, Predicate, and Object. We further insert special tokens before each of the Subject, Predicate, and Object, as indicated in Figure 1 and use the concatenated strings to train our LMs. The processing removes the ambiguity of natural language, which makes the task both easier and harder for an LM. The task is easier because the samples are now structured: LMs no longer need to pick up the intricacies of grammar and distinguish different phrasings, and can instead just focus on learning facts. It is, however, also harder, because there is very little correlation between data samples, unless they share items, and thus very little positive transfer between learning one fact to the other.
In later sections (Sections and 4), we will be training and evaluating LMs and hallucination detectors. We therefore need to carefully design data splits to fully understand the impact of data. We design datasets that reflect three levels of visibility: 1) a fully visible set (FVS) that both the LMs and detectors are trained on, 2) partially visible set (PVS) that only the LMs are trained on, and finally 3) an invisible set (TVS) that neither the LM or the detector have seen. We then vary the sizes of FVS and PVS to study the effect of scale.
To construct these three sets of triplets, we perform an i.i.d split at the subject level. Some subject-predicate pairs are associated with multiple objects (e-g., names of tracks on an album). In these cases, we need to ensure that all objects associated with a given subject- subject-predicate belong to the same set, as otherwise we might label correctly deduced object predictions as hallucinations. A similar issue can exist at the subject level, e.g., age can be deduced from date of birth. Several subject-predicate pairs are associated with hundreds of objects. To simplify evaluations, we remove all subject-predicates linked with more than 20 objects. This eliminates extreme long-tails that would be hard meaningfully analyse. reports the resulting dataset size.
2.2 Training LMs the Knowledge Graph
We trained decoder-only transformer LMs (Vaswani et al. 2017) with varying number of non-embedding parameters (3.15M-1.61BB) on various sizes of the KG dataset (1%-100%). The parameters are optimized with respect to the autoregressive cross-entropy loss over formatted strings created from triplets with special tokens (Section 2.1).
Where a single triplet does not fill the context window (256 tokens), we used packing (on average, ~0 triplets fit into the context window). For optimization, we used Adam (Kingma & Ba 2014) with linear learning rate warmup from 0 to our base learning rate (4K steps), followed by cosine decay down to 5% of the base rate. We varied the total number of steps to study the effect of multi-epoch training (see Tables 2 and 4 for details). The base learning rate is set to a constant divided by the square root of the number of LM's non-embedding parameters. The constant was determined by hyperparameter search over 2.5, 5, and 10 (due to compute limitations this exploration was not done for all models). The exact learning rates we used can be found in Table in Appendix.
Hallucination Rate and How It Scales
Scaling laws are an empirical phenomenon where the cross-entropy loss of LMs decay as power law of the model and training set size (Kaplan et al. 2020 Hoffmann et al. (022). Since cross-entropy is related to the accuracy of model predictions, one can wonder whether hallucination rates follow similar trend. Figure 2 shows this is not the case. While for a fixed dataset size, larger, longer-trained models fend to hallucinate less, increasing the dataset size yields higher, rather than a lower, hallucination rate (top left vs. top right); a similar trend is observed for the cross-entropy loss-evaluaterd on full triplets, not just the object-in Figure This is because of two factors: (1) many triplets in the KG require
4

---

Published as a conference paper at COLM 2024
Data (train FVS PVS)
10% Data (train FVS PVS)
60
% Hallucinated
40
20
Model Size
0
3.15M
1% Data (test
10% Data (test
Hallucinated
60
403M
1.618
40
20
0
10ts
1036
10
10a8
10th
1020
10
10
1017
1008
10tH
1020
FLOPs
FLOPs
Figure 2: Hallucination rate per LM training FLOPs on examples seen (top) and unseen (bottom) during training, on small (left) and large (right) size of data. Each dot an independent training run with learning rate schedule adjusted to the training length (Section 2.2 Dots correspond [1, 10, 20, 100, 200] epochs for 1% Data, and [1, 10, 20J epochs for 10% Data. For fixed dataset, the more FLOPs, the lower the hallucination. In contrast to established scaling laws for loss on text (Kpllan al. 2020 Hoffmann al., 2022), performance actually worsens with dataset size (top left vs. top right), as the larger dataset requires learning more facts. Training for 20+ epochs is necessary to minimise hallu- cinations
data (top), but can lead to overfitting a trade-off between fact recall and ability to generalize. This is even more pronounced at temp 0.0 (Figure 8). The hallucination rate upticks for 113M and 404M LMs 1% Data are not mirrored by the training loss (Figure 3,, i.e., they are not due to loss divergence.
unseen data (bottom), presenting
1% Data
10% Data
10
3x10
Loss
10"
10a
10
10ta
1017
10t8
10
1000
10th
10th
1077
102h
10
10
FLOPs
FLOPs
Model
3.15M
113M
1.618
25.2M
403M
Figure 3: Training loss per LM training FLOPs. The y-axis shows per-token average autoregressive cross-entropy loss over all tokens (i.e., subject, predicate, and object). Complementing Figure 2 larger models attain smaller loss a fixed dataset size, but the loss increases when training set size grows.

---

Published as a conference paper at COLM 2024
1.0
1.0
Epoch
2
0.9
20
0.8
0.8
Precision
0.6
0.7
0.4
0.6
0.2
0.5
0.0
0.5
0.6
0.7
0.8
0.9
1.0
Recall
Figure 4: Precision and recall as function of temperature on the 1% Data. Marker size represents the number of non-embedding parameters. Marker type the number of epochs for which the LM was trained. For each temperature in [0.0, 0.2, 0.4, 0.6, 0.8, 1.0], we generate 16 object predictions for every subject-predicat and evaluate precision and recall against the valid completions. The reported numbers are average over all examples. Lower temperatures yield higher precision, higher temperatures yield higher recall.
memorization (i.e., correct answer cannot be inferred from other data points); (2) each triplet appears in the training set only once. The bottom two plots in Figure 2 demonstrate the necessity of memorization; no model attains less than 50% hallucination rate on data not seen at training time. Examples of facts requiring memorization are names of tracks on an album, or dates of birth.
Another consequence of the necessity memorize and of the lack of repetition is that 20+ epochs are needed to achieve close to minimal hallucination rate for given LM size. This stands in sharp contrast to the current practice training LMs for only one or a few epochs (Chowdhery et al. 2023 Hoffmann et al. 2022 Touvron et al. 2023) we show for models trained for fewer (1 or epochs in the beginning of each line in Figure
An unfortunate side-effect of training for 20+ epochs seems to be decreasing ability to generalise unseen data (note the eventually upwards slope the bottom plots). This trend is even more pronounced at temp 0.0 (Figure (), presenting trade-off between hallucination rates and other model performance measures. While even the most capable modern LMs hallucinate (Gemini 2023 OpenAl 2023), repetition of facts in their training data likely alleviates the issue, as also suggested in Kandpal et al. (2023). Excessive repetition might however be harmful (Hernandez et al., 2022; Xue et al. 2023 Muennighoff et al., 2023), presenting a challenge for curation of training data.
It is striking that, in our case, even when training 1% of the full KG, the hallucination rate on data seen during training remains ~5% for the largest and longest-trained LM. This hallucination rate can be pushed down to ~1.5% by lowering the temperature from 1.0 to 0.0. However, the KG contains large number of subject-predicate pairs with multiple associated objects (e.g., names of all authors of a particular paper), and reducing sampling temperature leads the model to generate fewer of these objects. In Figure 4, we show how varying temperature affects both precision, defined as 1-hallucination rate, and recall, defined as the average proportion of objects in the original training data that the model generates at least once when sampling 16 completions for each subject-predicate pair. This highlights an issue with focusing hallucination rate only: easy way to hallucinate less is to make fewer claims. A related issue is that finetuning LMs to refuse to answer when uncertain may potentially lead to overly conservative behaviour.

---

Published as a conference paper at COLM 2024
Our best-performing LMs are order of magnitude larger than the predicted Chinchilla- optimal size for the number of tokens on which we train (Hoffmann et al. 2022), and therefore even larger than the even smaller models geared towards efficient inference (Touvron et al. 2023). This suggests that even if we just wanted LMs to rarely hallucinate the data that they have seen during pretraining, we would need computational budget several times higher than what is currently considered optimal. A more efficient alternative may be aatrivaal-ugmeetttion (Lewis et al. 2020 Borgeaud et al. 2022), or methods for expressing uncertainty or self-correcting (e.g., Dhuliawala et al. 2023 OpenAl 2023 Li et al.
2023). In Section 4 we study influence of scale on methods that belong to the latter category.
Hallucination Detectability and How It Scales
4.1 Setup
In Section 3 we have seen hallucination rates typically decay with LM size and training length. However, even LMs much larger than currently considered optimal-for given training set size-continue to hallucinate ~5% of the time on data seen and ~50% on data unseen during training (Figure 2 Our experiments also exhibit a trade-off between in- distribution and out-of-distribution hallucination rates (Figures and (). We therefore need to understand whether it is possible to further reduce hallucination rates by other means.
There are many types of alternative interventions, ranging from retrieval to model self-correction (see Appendix(A) One promising direction hallucination delectors which try to identify hallucinations either from the LM output itself, or from the LM's internal representations (e.g,, Kadavath et al. 2022 Dhuliawala et al.] 2023 OpenAl 2023 Li et al. 2023). Our aim is to understand (i) how the eftectiveness of hallucination detectors depends on the scale and training length of the LM they are judging, (ii) what types of detectors perform better, and (iii) if there is evidence other interventions beyond detectors are needed.
We explore two types of detection tasks:
sentence:
The detector ingests both original subject-predicat and the predicted object tokens, and judges if the object is hallucinated
The detector takes embedding of a token from a given layer of the trained LM, and asked say if is hallucinated. For a given subject- a predicted object token is labelled as hallucinated if it and the tokens preceding it do not match the token prefix of any entry the KG. The goal is to predict the first hallucinated token; any tokens that come after are discarded from the detector's 's training and evaluation data.
token:
input,
For each of the tasks, we experiment with two types of hallucination detectors:
head: Adds new readout head on top of the pretrained LM that produced the output (remaining LM weights are frozen). For the sentence task, this is equivalent adding two new tokens to the dictionary (hallucination & mon-hallucination), and finetuning the LM readout layer to predict whether the preceding triplet is hallucinated. The same holds for the token task, but the point of prediction changes; we also experiment with training detectors based token representations from after each transformer block (Figure 10)
full: Takes the pretrained LM as a starting point, and finetunes all its weights. For this setup, we only consider the case where the readout head is attached to the top layer embeddings in the token task.
In preliminary experiments, we tested three different setups: (a) predict 'ahead-of-timee i.e. straight after the subject-predicat (related to Kadavath et al. 2022); (b) predict straight after the last object token; and (c) predict after the end-of-sentence (EOS) special token. While (a) is more efficient in terms of inference costs, it lags behind the other two in terms of hallucination detection accuracy. From (b) and (c), the latter was more stable to train, presumably because it clearly signals to the detector that the LM output is complete. Since our main interest lies in accurate detection of hallucinations, we used setup (c) in all the presented experiments.
7

---

Published as a conference paper at COLM 2024
The combination of token task and head detector applied to the top layer embeddings is similar to the approach taken in Kadavath et al. (2022). ) For token and full, the results should provide an upper bound on the performance achievable by finetuning the whole LM to say 'I don't know' (IDK), as we do not force the detector to preserve any of the other LM capabilities For the sentence task, the setup is related to techniques based on post-hoc verification, including self-critique (e.g./ OpenAl 2023). The combination with full detectors in particular should provide an upper bound on performance achievable by self-critique, as we force the preserve any of the other LM capabilities 3
We trained distinct detector (of each type) for every combination of a pretrained LM and detection task. Training and evaluation data are obtained by generating 5 object predictions for every subject-predical
the LM training set. This data is split into a detector training (90%), validation (5%), and test sets (5%). The validation set is used for hyperparameter tuning and early stopping, the test set for measuring performance.
For the token experiments, we use the Adam optimizer (Kingma & Ba, 2014), with 1K warm-up steps and cosine decay (5K steps for head, 20K steps for full). Peak learning rates le-4 was used for head, and 5e-5 for full detectors. Training length and learning rates were determined using the validation set, optimizing for high AUC-PR. We did not finetune every possible combination of LM, detector task, and algorithm due to the large size of the Cartesian product (over 300). Visually, the above choices were optimal (or close) for most setups, but may not be optimal.
For the sentence experiments, the Adam optimizer alone underperformed for the full detectors, so we switched to linear probing then full finetuning (LPFT; Kumar et al. 2022). LPFT works in two stages: (i) only the readout layer is optimized, with all other weights frozen; (ii) both readout and other layer weights are optimized. We used the Adafactor optimizer (Shazeer & Stern 2018) for both stages. In the first, we used a cosine decay schedule with peak learning rate 1e-2 for 10K steps. In the second, we used linear warmup (10K steps) combined with cosine decay (250K steps) with peak learning rate 1e-3. For head detectors, only the first stage was used. The number of training steps and learning rates were determined using the validation set, aiming for high AUC-PR. Unlike in the token setup, found it harder to find a common set of hyperparameters that would work for all detectors.
4.2 Results
Figure 5 shows how the pretrained LM size affects overall accuracy across tasks and approaches (Section 4.1). As expected, the full detectors outperform head, as they tend to be more flexible. Since this is also the case for other metrics, we focus on full detectors in the rest of this section. The token task formulation generally yielded better detector accuracy than the sentence task. Detector accuracy also tends to grow with the size of the underlying LM. However, these results are confounded by sensitivity of the accuracy metric to the underlying LM hallucination rate (see Figure 2). For example, if the rate is only 5%, even a naive detector which catches all hallucinations attains 95% accuracy.
We use AUC-PR to assess how well our detectors identify hallucinations. In Figure 6 we observe that: (a) sentence task formulation is superior in terms of AUC-PR, and (b) the lower the LM's hallucination rate, the worse the detector's 's AUC-PR. Per Figure 2 the lowest hallucination rates are achieved by the largest longest trained models. Thus, for fixed training set size, there is an inverse relationship between the FLOPs spend on the LM pretraining and the detectability of its hallucinations.
Figure emphasizes the inverse relationship between LM scale and hallucination detectability, showing the PR curves corresponding to the sentence task AUC-PR values in Figure 6 Note the general ordering of the curves, with those corresponding to detectors for the largest LMs being the lowest, and the ones for the smallest LMs being the highest.
Upper bound does not hold if there is a synergy between the original LMM capabilities and learning to say IDK.
3 "Again, this does not hold if there is synergy.
8

---

Published as a conference paper at COLM 2024
LM trained on 1% Data for 100 Epochs
LM trained on 10% Data for 20 Epochs
Detector Accuracy
0.95
0.85
0.90
0.85
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
LM Size
1eg
LM Size
l09
Detection Task
Detector Type
sonconcce
ful
toleen
head
Figure 5: Hallucination detection accuracy as function of the LM size for various task formulations and detector types. Detectors were trained and evaluated on distinct splits of data obtained by having a given pretrained LM generate 5 completions for every subject- predicate in its training set (using temp = 1.0). The accuracy of all the trained hallucination detectors is generally high, especially for outputs of the larger LMs. Larger (full) detectors work better than smaller ones (head). The token-level detection task formulation seems to provide higher detection accuracy, although not in all cases. The results here are confounded by the varying hallucination rates of the underlying LM (e.g., if the LM hallucinates only 5% of the time, a detector which finds no hallucinations achieves 95% accuracy).
1.00
Detection Task
0.95
sentence
token LM Size
Detector AUC-PR
0.90
X
3.15M
2M
0.80
113M
403M
0.75
1.61B
0.70
LM % Data
*
10
0.0
0.1
0.2
0.3
0.5 0.6
LM Hallucination Rate
Figure 6: AUC-PR as function of LM hallucination rate for the full detectors. Same setup as in Figure 5 except LM size is now represented by the marker size. Showing results for data generated by LMs trained for 100 (resp. 20) epochs on 1% (resp. 10%) of the data. AUC-PR does not depend on the proportion of hallucinations in the evaluation data (i.e., the LM's hallucination rate), thus providing providing a better measure of the detector's 's ability to catch hallucinations. Unlike for accuracy (Figure 5), the sentence task is clearly superior in AUC-PR terms (can also be seen in Figure 9,, although better token performance can be attained by attaching the detector to different LM layer (Figure TO). More importantly, the detectability of hallucinations is inversely proportional
the LM size (largest dots/ dots/LMs LMMs in bottom left, smallest in top right). Larger LMs have lower hallucination rates, but it's also the harder detect their hallucinations. This can be seen even more clearly in Figure 7
Limitations
Several factors may limit correspondence of our results behaviour of state-of-the-art (SOTA) LMs. Firstly, the KG differs from the data normally used for LM training (e.g., no repetition of facts in the corpora, no semantic ambiguity, simple sentence structure). Secondly, the LMs we train are significantly smaller than SOTA LMs, for which we adjust by using proportionally smaller dataset. While we believe the qualitative interpretation of our results would generalize, we cannot rule out appearance of emergent capabilities (Wei et al. 2022) which may, e.g., significantly reduce the hallucination rate on unseen data. Thirdly, we only study hallucinations that result in not remembering something that

---

Published as a conference paper at COLM 2024
LM trained on 1% Data for 100 Epochs
LM trained on 10% Data for 20 Epochs
Detector Precision
0.75
0.75
0.50
0.50
0.25
Detector Recall
Detector Recall
LM Size
25 2M
403M
Figure 7: Precision-Recall curves for the full detectors trained on the sentence task. These curves correspond the relevant subset of the AUC-PR values from Figure The larger the LM, the harder it is to detect its hallucinations.
appeared verbatim in the training data. While this has enabled most of our analysis, some of our conclusions may not translate other types of hallucinations.
For hallucination detectors, we focused on general trends rather than exhaustive coverage of existing methods, and it is possible that alternative methods could have yielded better results. However, the qualitative consistency of the results relating to dependence of detector performance on LM scale suggests these are general properties of hallucination detectors. Finally, dataset imbalance driven by varying hallucination rates between our LMs had strong effect on performance of the hallucination detectors. We did not experiment with methods to adjust for such imbalance, beyond implicitly measuring performance at different score cut-offs via AUC-PR and the PR curves.
Conclusion
We explored the relationship between LM hallucinations and their scale. In particular, our paper sheds light on what it would take to eliminate factual hallucinations in the case where the correct answers-an and no wrong answers answers-appen appear verbatim in the training set. Our study also be seen studying how much computation is needed for LM to memorize all the knowledge content of a dataset. For a fixed dataset size, larger longer trained LMs hallucinate less. However, we found increasing dataset size also increased hallucination rate, given a fixed number of training epochs and LM size. This is because larger dataset means more facts to learn. Similar to Kandpal et al. (2023), we hypothesise that standard LM training data contains repeated facts-ard and other repetitive text structures that do not require hard memorization- makes the effect of dataset size on loss positive.
We found achieving a <5% hallucination rate on training data requires LMs much larger than currently considered optimal (Hoffmann al.) 2022), trained for much longer (20+ epochs). Moreover, the longer training can hurt LM generalisation, presenting trade-off between hallucination rate and other performance metrics.
Given the difficulty of eliminating hallucinations outright, we explored how detectability of LM hallucinations depends on scale. We found an inverse relationship between LM scale and the detectability of its hallucinations. That is, while larger, longer-trained LMs hallucinate less, detecting their remaining hallucinations becomes increasingly hard.
References
Ayush Agrawal, Lester Mackey, and Adam Kalai. Do language models know when they're 're hallucinating references? ar aiv, abs/ /ts//23000033441, 2023.
10

---

Published as a conference paper at COLM 2024
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Lespiau, Bogdan Damoc, Aidan
Millican, George Bm Van Den Driessche, Clark, et al. Improving language models by retrieving from trillions of tokens. In ICML, 2022.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In ICLR, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. JMLR, 2023.
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celiky- ilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models, 2023.
Nouha Dziri, Andrea Madotto, Osmar Zaïane, and Avishek Joey Bose. Neural path hunter: Reducing hallucination in dialogue systems via path grounding, In EMNLP, 2021.
Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy, On the origin of hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022.
Team Gemini. Gemini: family of highly capable multimodal models. arXiv, 2023.
Google. Introducing the knowledge graph: things, not strings, 2012. URL http:::/tpssltt hooog////rouuccts/eeeee////mnirr htw.:ulleeeee-rra----i-n----eo-o Accessed: 2024-
01-15.
Roger Baker Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamil.e Lukovsiute, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Sam Bowman. Studying large language model generalization with influence functions. arXiv, 2023.
Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training runs. Xrv, 2023.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El- Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. ar arii 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training oompule-optimal large language models. arv, 2022.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Sure., 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know, 2022.
N Kandpal, H Deng, A Roberts, E Wallace, and C Raffel. Large language models struggle to learn long-tail knowledge. In ICMI,, 2023.
11

---

Published as a conference paper at COLM 2024
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. Xrv, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXio, 2014.
Ananya Kumar, Aditi Raghunathan, RobbieJones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv, 2022.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensive NLP tasks. NeurIPS, 2020.
Kenneth Li, Oam Patel, Fernanda Vi Vi'egas, Hans-Rüdiger Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. ar arxio, abs/2306.0334 2023.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. ar Arviv, 2023.
OpenAL. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and D. Reitter. Measuring attribution in nat- ural language generation models. arXiv, abs/2112.12870, 2021. URL https://api http://o......rreorrooooo::::::0
John Schulman. Reinforcement learning from human feedback: Progress and challenges, 2023. URL /tLL/htps::///// httpp::/www.yyouuubbbe.. http://wwwyotuuteeeom/////////// wom//iie//iiiiws///_ 2tr.000002222xxx1/oooo/////11110
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML, 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arv,v, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NewrlPS, 2017.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv, 2022.
Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling LLM under token-crisis. arxiv, 2023.
12

---

Published as a conference paper at COLM 2024
Appendix
A Related Work
Hallucination in the context of natural language processing (NLP) usually defined as the phenomenon of model-generated text that is seemingly fluent and coherent, but is actually nonsensical or unfaithful the provided source content (li et al. 2023). Although the study of hallucinations in LMs is nascent, there is significant interest due to the challenges and risks they pose. Here we discuss variety of relevant work, but direct the reader (Ji al., 2023
for more comprehensive survey.
Kandpal al. (2023) found that LMs struggee to learn long-tail knowledge that is not seen by the model multiple times during training. This is consistent with our finding that multiple passes over triplets are needed for the model to memorize facts. Kadavath et al. (2022) performed a large scale study to determine if models can give an indication of whether they will give a correct response to a multiple choice question. They found that for the majority of questions, the model was well calibrated in predicting correctness, with the exception of long-tail or dustof-distribution data.
Carlini et al. 2023) study memorization in LMs and, similarly to this work, find that models memorize more with respect to scale and the number of times concept is seen during training. We find that even in larger models, the hallucination rate remains above 5% the training set even after 100 passes over the data. Dziri et al. (2022) found that the phenomenon of hallucinations is compounded by datasets and benchmarks which are themselves rife with hallucinated data. AAgrwaal et al. (2023) study hallucinations of references and find that inconsistency of multiple sampled references strong indicator of hallucinations.
Some recent work (Guu et al., 2023: Grosse et al., 2023 has developed methods to
attribute predictions to individual training examples using influence functions. However, computing influence functions exhaustively over the training data remains prohibitively expensive. Therefore, we study hallucinations in the setting where we can know and control relationships between facts contained in the data that is exposed to the model.
Mitigation techniques range from automated model rewrites (Dhuliawala et al., 2023 OpenAl. 2023), attribution and grounding (Rashkin et al., 2021), to classification (probing) based on model internals. Dziri et al. (2021) use a critic and retrieval augmented dialogue model to retrieve facts from KG. Li et al. (2023) use linear probes to detect internal model attention heads that are more likely to indicate truthfulness, and use this to push the model more towards truthful responses. In this work, we try to assess, in best case scenario, how well either works to mitigate hallucinations.
B Learning rates, number of training steps
As mentioned in Section 2.2 the base learning rate for each model size is set to be a constant (a hyperparameter searched over 2.5, 5, and 10) divided by the square root of the number of LM's non-embedding parameters. The resulting exact learning rates are listed in Table 3
Model size
Learning rate
3.15M
0.002885
25.2M
0.001
113M
0.00047
403M
0.00025
1.61B
0.00011
Table Base learning rates used for various model sizes.
13

---

Published as a conference paper at COLM 2024
Data subset
2.6K
5.2K
26K
52K
260K
520K
FVS PVS, 1% FVS PVS, 10% FVS PVS, 100%
/
V
X
/
/
X
X
/
Table Number of training steps for different data sizes.
C
Additional plots
1% Data (train FVS PVS)
10% Data (train FVS + PVS)
60
% &Hllluciated
Model Size
3.15M
25.2M
113M
% Hallusinated
403M
1.618
20
1015
1014
1032
10:4
10
1024
1015
1014
10:2
10.4
109
1023
Figure 8: Hallucination rate per LM training FLOPs on examples seen (top) and not seen (bottom) during training. Same as Figure except for using temp 0 instead of temp to generate the samples. Note the more pronounced decay in out-of-distribution (IVS) performance with length of training, emphasising the possible trade-off between training set hallucination rate and
14

---

Published as a conference paper at COLM 2024
LM trained on 1% Data for 100 Epochs
Detector AUC-PR
0.8
0.6
0.4
LM trained on 10% Data for 20 Epochs
1.0
Detector AUC-PE C-PR
0.8
0.6
0.4
0.5
LM Size
1e9
Detection Task
Detector Type
sentence
full
token
head
Figure 9: AUC-PR as function of LM size for various task formulations and detector architectures. Same setup in 5 Complementing Figure shows the sentence task is superior in terms of AUC-PR. However, the token task performance could be improved by attaching the detector to different layer of the LM (Figure 10).
15

---

Published as a conference paper at COLM 2024
LM trained on 1% Data for 100 Epochs
Detector AUC-PR
0.6
0.4
LM trained on 10% Data for 20 Epochs
Detector AUC-PR
0.6
0.4
0.0
0.2
0.4
0.6
0.8
1.0
Relative Depth of Source LM Layer
LM Size
3.15M
113M
1.61B
25.2M
403M
Figure 10: AUC-PR of head detectors in the token setup as function of LM layer depth. We fitted a distinct head detector using token representations from different layers of the LM, specifically from after each transformer block. The AUC-PR is the lowest near LM input, then grows, before often dipping again near LM output. This is interesting given that head detectors are typically used with top layer embeddings (e.g., Kadavath et al., 2022; Li et al. 2023). We thus only report the top layer embedding performance of head detectors throughout Section 4.2 but the reader should remember these results are typically few percent lower than the best possible.
Unseen Data Halluci nation Rate vs Training Data Hallucination on Rate
Epochs
2 Epochs
10 Epochs
20 Epochs
100 Epocis
200 Epocls
Helbcchull Iute
5.6
2300 85
955
0.1
0.2
0.2
0.2
0.2
2.4
0.5
0.2
0.0
0.0
0/
0.5
0.6
5.4
2.5
0.6
0.00
0.25
0.50
0.00
0.25
0.5
0.00
0.25
3.50
0.00
0.25
0.50
Ialiciaation Rate
2.5
0.4
3m
Trenie
0.4
02
0.2
403m
2b
Training
Holl.dmation Rate
Holludiaaton Rate
Hollucination Rate
Hollucintioon Rate
Figure 11: Hallucination rate for held-out unseen data vs. the hallucination rate for train- ing data. Models always hallucinate more on data has seen before. This especially true after several training epochs, when the LMs presumably start to memorize more.
16


Reviews:
[{"review_id": "icmS7MhNT7", "reviewer": "Reviewer_o4Hv", "summary": "The paper explores the relationship between LM hallucinations and their scale, focusing on eliminating hallucinations where correct answers are in the training set. To this end, the authors train a set of LMs on a KG-based dataset. The major conclusions include: 1) Larger and longer trained LMs hallucinate less on a fixed dataset, but increasing dataset size can increase hallucination rate. 2) Achieving a low hallucination rate requires LMs much larger than currently considered optimal trained for longer periods, with a trade-off between hallucination rate and other performance metrics. 3) Detectability of LM hallucinations decreases as LM size increases.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that our manuscript addresses an important topic, is well written, and could provide insights for practical solutions. \n\nWe further appreciate the reviewer's concerns about the generalizability of our conclusions, and agree that the data used for training LMs in this paper is different from the usual natural language data. Our training data is entirely extracted from a KG, a structured representation of factual knowledge, and formatted to suit LM training. We purposefully adopted this setting, because it allows for a fully controlled data environment where we can control what knowledge the LM was exposed to during training, and thus make a clear separation of knowledge pieces between training and test. This hard separation is difficult, to the point of impossible, to ensure with natural language data, where the “unit” of knowledge does not have a clear definition. To exclude a piece of knowledge entirely from training means that every possible linguistic variation of such knowledge needs to be excluded. It thus poses a great challenge to fully control the factual content in natural language data. We therefore resort to KG-based LM training to seek a deeper understanding of the hallucination effect.\n\nFurthermore, training LMs on KG is fundamentally not that different from training on language. In the end, we are training LMs to perform next-token-prediction and study its generalizability, on in-distribution and out-of-distribution data. Our setting simplifies the process by having a clear definition of the two distributions, removes the complexities that come with linguistic data, and comes up with a simple format to handle structured data.\n\nAt the moment, training on KG triplets may seem to go against the current standard of LM training, however, the field of LM training is evolving fast, and since KGs are already widely used in a variety of applications alongside natural-language LMs, including question answering, information retrieval, and knowledge discovery, it may not be long before we witness alternative ways of training LMs employing a wide range of data modalities. In one of the many possible futures of LM training, this setting will not seem so odd.\n\nTherefore, we believe that our conclusions are generalizable to current and future LMs. Please let us know if you have any more questions or comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is generally well-organized and clearly written.\n* The paper explores an important topic with reasonable experiments and analysis.\n* The conclusions of the paper provide potentially useful knowledge regarding developing practical solutions to reduce LM hallucinations.", "reasons_to_reject": "* My major concern is that the data used for training LMs in this paper is quite different from the data used for LM training in practice. While the paper has a short discussion about it in the limitations section, it would make the paper stronger if the authors could provide empirical evidence to justify the conclusions' generalizability.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IPrzkFVvgE", "reviewer": "Reviewer_tRfu", "summary": "This paper investigates the scaling law of language models (LMs) and hallucination. The authors control the train/test facts by constructing a knowledge graph (KG) based dataset and train language models from scratch. They ablate LMs of varying sizes and evaluate their hallucination rates on seen and unseen data. They find that while increasing the LM size and training compute reduces hallucinations on the training set, eliminating hallucinations entirely requires an order of magnitude more compute than currently considered optimal. They also study hallucination detectors and find an inverse relationship between LM scale and the detectability of hallucinations by these detectors.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for pointing out our comprehensive analysis of scaling of LMs and hallucination detectors, the clear writing and visualization, and that our setup enables quantifying hallucinations in a previously impossible precise manner. \n\nFirst, we answer your questions:\n\n## Q1\nThis is an intriguing question. Could the reviewer please clarify what they mean by “advanced language understanding capabilities” in this case? Perhaps we can better answer this question given an example of such capability. \n\n## Q2\nWe appreciate this thought-provoking question! Logical/Counterfactual hallucinations could also be studied if we extend the KG-based approach by introducing conflicting <subject>,<predicate> pairs during inference. One could also add contamination to the dataset with generated untruths to investigate their impact on hallucinations. One could also add contradictions and various kinds of structured noise one might find in natural language online. \n\nNext, we hope to address your reasons for rejection.\n \n## Reason 1\nThis study focuses on hallucination instances where a model is violating the facts it is trained on, because it is easy to measure, and because understanding this type of hallucination is a crucial first step to understanding many other types of hallucinations. We are open to extending other types of hallucinations as mentioned above in answer to Q2.\n\n## Reason 2\nTraining LMs on KG is fundamentally not that different from training on language; both aim for next-token prediction and generalization to OOD data. Our setting just simplifies it: there is a  clear definition of the two distributions, and complexities arising from linguistic data are removed by using structured data. \n\n## Reason 3\nWe found that the LMs we trained hallucinate a lot even on these small datasets, even when trained for much longer than usual. Further, if we take the ~20 params/token Chinchilla optimal ratio, the 1.6B model trained on 10% of the data would have been roughly optimal (see Table 1), but gets ~50% hallucination rate on the data seen in training (Figure 2, top right), which is too high to be the only object of our investigation. Extrapolating the pattern from Figure 2, we realized that to train a proportionately larger LM on a larger dataset to the point where it doesn't hallucinate a lot is going to be too costly, and such compute budget could be better spent elsewhere.\n\nPlease let us know if you have any further questions or comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The use of a controlled KG-based dataset is a novel approach that enables quantifying hallucinations in a precise manner not applicable to previous data. This controlled setup reveals interesting insights.\n2. Comprehensive analysis of how hallucination rates depend on LM scale, training compute, and dataset size.\n3. Investigation into the scaling properties of hallucination detectors and their ability to catch hallucinations from LMs of different scales. The inverse scaling of hallucination detectability with LM scale is an interesting result with potential ramifications for detector-based mitigation strategies.\n4. Clear writing and good visualization of key results.", "reasons_to_reject": "1. Limited to studying hallucinations on factual knowledge present verbatim in the training data (which is not natural language sentences). Other types of hallucinations are not explored.\n2. Some discussion on similarities/differences compared to natural language could have strengthened the connection to real-world LM applications.\n3. LM and dataset sizes studied are still significantly smaller than state-of-the-art LLMs, raising potential generalizability concerns.", "questions_to_authors": "1. Can the authors comment on how the findings may change when considering more advanced language understanding capabilities needed for hallucinations in natural language?\n2. What are some ways the KG-based approach could be extended to study other types of hallucinations beyond factual knowledge verbatim in the training set?", "extra_scores": {}}, {"review_id": "DJTl6cGIPs", "reviewer": "Reviewer_MRVp", "summary": "The paper examines hallucinations in language models trained on knowledge graphs, focusing on cases where correct answers are present in the training data. It reveals that larger models have lower hallucination rates but require more computational power. The study also finds that the detectability of hallucinations decreases as model size increases, suggesting a trade-off between model scale and generalization capabilities.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the novelty and significance of our work. We agree that our work has limitations, but we believe that it still makes a significant and novel contribution. First, we answer your questions:\n\n## Q1\nThe first and last tokens are just like BOS and EOS. P_TKN and O_TKN (Fig 1) delineate individual triplet parts, preventing ambiguities that could confound evaluation (e.g. in case of prefix overlap). These tokens may simplify the task, similar to eg, arXiv:2301.13195, arXiv:2310.02226, arXiv:2309.16588, but we think the task is still difficult given the need to train larger than Chinchilla optimal models for many epochs. For a discussion of generalizability of our conclusions to natural language, please see our reply to Reviewer o4Hv and tRfu (Reason 2).\n\n## Q2\nIn Fig 4, there are many variables in play: LM size (marker size), training length (dot vs cross), temperature (color), but the overall trend is consistent: the longer the training, the better the PR. The larger the LM size, the better the PR. Lower temperature -> higher precision. Higher temperature -> higher recall. We will make those observations clearer.\n\n## Q3\nLearning rates are lowered as model size increases for training stability. We use this formula stated in Section 3.2: learning rate = constant / sqrt(number of non-embedding parameters).\n\nNext, we hope to address your reasons for rejection.\n\n## Reason 1\nYou’re right that we focus only on one hallucination type at the moment, in a controlled environment with special data. You can consider this a “lab” setting to gain insights and understanding. We think that thoroughly understanding even a single type of hallucinations in a simplified setting is a crucial first step to understanding many other types of hallucinations in other real-world settings. \n\n## Reason 2\nYou’re correct, we will highlight more data requires more compute! The data and model sizes we used are due to the rather challenging nature of the KG dataset (see our reply to Reason 3 of Reviewer tRfu). In short, achieving non-trivial hallucination rate requires an order of magnitude bigger models trained for much longer than Chinchilla optimal. With limited resources, we studied how different ways of \"overscaling\" affect hallucination rates, illuminating the scale of the challenge.\n\n## Reason 3\nThanks, we will try to improve our presentation. For example, see answer to Q2 above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The approach of training LMs from scratch using knowledge graphs is innovative and brings a fresh perspective to the study.\n2. The paper presents intriguing conclusions that hold significant implications for future research in the field.", "reasons_to_reject": "1. This paper studies the phenomenon of hallucinations, particularly focusing on the task of predicting the object entity in a triplet. However, the research scenario presented in this work is relatively singular, primarily concentrating on the direct prediction of facts, which seems limited compared to the broader types of hallucinations observed in real-world tasks, such as text generation, dialogue systems, summarization, translation, etc. As a result, the conclusions drawn from this study may lack universality and may not adequately reflect the performance of language models across a diverse array of tasks.\n\n2. Some findings in this paper are interesting but potentially insufficient and possibly incorrect. For example, the analysis presented in Figures 2 and 3 indicates that an increase in dataset size leads to a higher rate of hallucinations. However, this might be due to the models not being sufficiently converged when the same amounts of FLOPs are applied across different dataset sizes. To achieve more reliable conclusions, models should be allowed more FLOPs for larger datasets to ensure adequate training and convergence.\n\n3. The manuscript's complexity and lack of clarity in certain experimental descriptions make it difficult to follow. The figures, with their numerous variables, are not clearly articulated, hindering the extraction of useful conclusions. The paper would benefit from a more concise presentation and clearer figure design to enhance readability and comprehension.", "questions_to_authors": "1. Could the authors please elaborate on the necessity of incorporating special tokens during the training phase, and discuss whether the deviation from the natural language text format could potentially influence the conclusions drawn from the study?\n\n2. The results presented in some of the figures, such as Figures 4 and 7, are not as intuitive as they could be, making it challenging to derive clear conclusions. With the presence of multiple variables in these charts, it seems difficult to reach a valid conclusion. The authors should consider revising the presentation to allow for a more straightforward interpretation of the results.\n\n3. In Appendix B, I notice that the authors used different learning rates for models of different sizes. But apparently, the values of these learning rates are very strange, how did the authors get these particular learning rate values (e.g., 0.00047 for the 113M model)?", "extra_scores": {}}, {"review_id": "ErwQvANDwm", "reviewer": "Reviewer_cjCt", "summary": "This paper presents a comprehensive study on the relationship between LM hallucinations and their scale. The authors construct a knowledge graph (KG) based dataset to train LMs, creating a controllable experimental environment to study hallucinations. The authors also design several hallucination detectors to study how hallucination detectors depend on scale (e.g., LM size, training epochs, datasets).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for calling our findings insightful and interesting, and appreciating the comprehensiveness of our investigation. We agree that our work has limitations, but we believe that it still makes a significant and novel contribution to the field of LM research.\n\nWe hope to address your reasons for rejection below.\n\n## Reason 1\nWe experimented with creating a taxonomy of hallucination causes during our experiments, but found the categories did not transfer well between model sizes and datasets (likely because of the rather larger performance differences). We are not aware of noise issues in the KG, but it would be an exciting direction of future research to try to procedurally generate triplets/sentences with incorrect objects, and measure the impact of such contamination on model performance. Another direction would be to contaminate the hallucination labels the detectors are trained on, and measure its impact. Our work trains LMs on KG entirely, but we can imagine a future where KG is combined with linguistic data; for more discussions of generalizability of our conclusions to natural language, please see our reply to Reviewer o4Hv.\n\n## Reason 2\nThanks for a great suggestion! We agree that providing some case studies of hallucination examples will really help drive our points home. Here are two examples, note that we removed the identifying information for simplicity.\n\nInput: '<S_TKN> A specific unnamed language <P_TKN> Main places spoken <O_TKN>' \nGround truth answer: ['United States Regions'] \nOutput by model: ['Italy', 'Papua New Guinea']\n\nInput: '<S_TKN> A specific album name <P_TKN> Genre <O_TKN>' \nGround truth answer: ['Indie rock', 'Indie pop'] \nOutput by model: ['Indie game']\n\nAs you can see that while the model does hallucinate (make up wrong answers), the output is still semantically reasonably adjacent to the ground truth answer.\n\n## Reason 3\nThank you for your valuable suggestion. We agree that Related Work should be in the main text. If accepted, we will use the additional page to include related work in the camera ready version. Comparing to existing methods (e.g. methods for mitigating hallucinations) across model and training settings will be challenging, but it is a valuable direction worth thinking about.\n\nThank you for your constructive suggestions, and please let us know if we can answer any more questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The presentation of the methods and experiments is clear and methodical. And this paper provides a comprehensive understanding of the hallucination problem with the scale. The results and findings presented in the paper are insightful and interesting, for example, this paper reveals a trade-off between the model's ability to generalize and its tendency to hallucinate.", "reasons_to_reject": "1.It will be better if the authors could provide more detail analysis about the causes of hallucinations, such as noise in the dataset (training on KG may limit this setting), etc., may provide richer insights. I also wonder what will happens if training LMs on KG with incorporating textual context. \n\n2.Providing some case studies of hallucination examples on the scale of LMs can help readers to understand the nature of hallucination and the working mechanism of the detector more intuitively.\n\n3.The Related Work section should be included in the main tex, rather than in the Appendix. And this paper mentions some correlations with existing methods but does not provide direct comparisons.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Metareview:
The paper investigates the relationship between LM hallucinations and their scale, focusing on eliminating hallucinations where correct answers are in the training set. The authors train a set of LMs on a KG-based dataset, concluding that larger and longer-trained LMs hallucinate less on a fixed dataset, but increasing dataset size can increase hallucination rates.
Reasons To Accept:
1. The paper is well-organized and clearly written.
2. It explores an important topic with reasonable experiments and analysis.
3. The conclusions provide potentially useful insights for developing practical solutions to reduce LM hallucinations.
Reasons To Reject:
1. The data used for training LMs in this paper is different from the data used for LM training in practice, raising concerns about the generalizability of the conclusions.
2. The research scenario is relatively singular, focusing on the direct prediction of facts, which seems limited compared to the broader types of hallucinations observed in real-world tasks.
3. The analysis indicates that increasing dataset size leads to higher hallucination rates, but this might be due to models not being sufficiently converged.
Decision: Accept