Statistical Knowledge Assessment for Large Language Models
Authors: Qingxiu Dong
Jingjing Xu
Lingpeng Kong
Zhifang Sui
Lei Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Models
Knowledge Assessment
Evaluation
TL;DR: We introduce a statistical knowledge assessment framework to quantifies a large language model's knowledge reliability.

Abstract: Given varying prompts regarding a factoid question, can a large language model (LLM) reliably generate factually correct answers? Existing LLMs may generate distinct responses for different prompts. In this paper, we study the problem of quantifying knowledge contained in an LLM regarding a given set of facts. We propose KaRR, a statistical approach to assess factual knowledge for LLMs. The main idea is to estimate the ratio of LLM generating text corresponding to the answer entity given diverse prompts of the subject and the querying relation, versus it generating by random chances. Our assessment suite contains a comprehensive set of 994,123 entities and 600 relations, with 1,395,905 text aliases. We use our method to evaluate 20 LLMs of various sizes, including LLaMA, Alpaca, OPT, etc. Experiments show that our results have a strong correlation (0.43 Kendall's $\tau$) with the results of human assessment on LLMs. Our results reveal that the knowledge in LLMs with the same backbone architecture adheres to the scaling law, while tuning on instruction-following data sometimes compromises the model's capability to generate factually correct text reliably.

Statistical Knowledge Assessment for Large Language Models
Qingxiu Dong', Jiigjing Xu², Lingpeng Kong", Zhifang Sui! and Lei Li* National Key Laboratory for Multimedia Information Processing, School Computer Science, Peking University
2 Shanghai Al Lab 'Tee University of Hong Kong Carnegie Mellon University dqx@stu. petssttu.kku..eo edu. en, {jingjingxu, szf (eryyepkued..... edu. en, 1pk0cs. hku.hk, lei deili@cs.cmm cmu.
Abstract
Given varying prompts regarding a factoid question, can a large language model (LLM) reliably generate factually correct answers? Existing LLMs may generate distinct responses for different prompts. this paper, we study the problem quantifying knowledge contained in an LLM regarding given set of facts. We propose KaRR, statistical approach to assess factual knowledge for LLMs. The main idea is to estimate the ratio of LLM generating text corresponding the answer entity given diverse prompts of the subject and the querying relation, versus it generating by random chances. Our assessment suite contains comprehensive set of 994, 123 entities and 600 relations, with 1,395,900 905 text aliases. We use our method evaluate 20 LLMs of various sizes, including LLaMA, Alpaca, OPT, etc. Experiments show that our results have a strong correlation (0.43 Kendall's r) with the results of human assessment LLMs. Our results reveal that the knowledge LLMs with the same backbone architecture adheres to the scaling law, while tuning instruction-followe data sometimes compromises the model's 's capability generate factually correct text reliably.
1
Introduction
Large language models (LLMMs) have achieved impressive performance on many tasks including text generation, question answering, and dialog generation [Brown et al., 2020, Thoppilan et al., 2022, Bubeck al., 2023]. Previous studies have shown that pretrained language models store factual knowledge in parameters [Roberts et al., 2020, Dai et al., 2022a,b, Singhal et al., 2023]. demonstrating considerable potential as assistants for responding to user questions based on their stored knowledge.
Despite the remarkable success of LLMs, critical concerns arise - LLMs often generate unreliable answers given varying prompts [Ji al., 2023, Maharana al., 2023, Chang and Bergen, 2023, Chen et al., 2023]. As shown in Fig. 1, reliability refers to the ability to consistently generate knowledge- сorrect text, posing higher standard than accuracy. For example, although Alpaca-7B [Taori et al., 2023] generates accurately (playwright) given prompt of "William Shakespeare' 's job", it generates incorrect text (boatman) given a prompt of "the job of Swan of Avon is"" ChatGPT [OpenAl, 2022] correctly generates playwright and teacher. However, it answers "No" for the prompt "Is William Shakespeare a leucher?" In this paper, we assess the knowledge contained an LLM and investigate whether an LIM is able to consistently generate factually correct answers given varying prompts. Knowledge assessment is critical problem for I.LMs. The assessment results directly affect the people's trust in the LLM generated content. Once we identify inconsistency of LLM generation, we could potentially correct such knowledge in LLMs [De Cao et al., 2021, Zhao et al., 2021, Dong et al., 2022].
Swan of Avon is a nickname of Shakespeare.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

William Shakespeure's 's job
Large
playwright.
William Shakespeure's job
Large
playright ...
Language
Language
1st the jobs of Swan of Avon:
Model
bostuen, and...
List the jobs of Swan of Avon:
Model
playright, and..
Accurate but Unreliable
Reliable
Figure 1: Accuracy pertains to generating the correct knowledge in response to a specific prompt, evaluated binary value (correct or incorrect). However, accuracy alone does not ensure reliability, as responses may vary with different prompt expressions. Reliability requires consistently generating correct knowledge for semantically similar prompts and is measured as a continuous value. Reliability sets a higher standard than accuracy by maintaining consistent accuracy across related prompts.
Important steps have been taken in the automatic knowledge evaluation for pre-trained language models [Petroni et al., 2019, Poerner et al., 2020, Jiang et al., 2020]. However, two issues still remain: 1) Prior methods focus on assessing the accuracy but not the reliability. Consequently, their probing results are prone to inflation or bias since models exhibit spurious correlations towards specific prompts [Poerner et al., 2020]. Most prior methods target masked language models (MLMs) and don't 't provide a universal solution for assessing the knowledge in generative LLMs. MLMs are evaluated by probing whether they predict the masked object in response to a given prompt in cloze form, while the freely generated results of generative LLMs might be irrelevant to factual knowledge. Therefore, there a need for method that assesses both the accuracy and consistency, i.e. reliability, of knowledge in LLMs.
In this paper, introduce KaRR, statistical approach assess whether large language model contains reliable factual knowledge. We use triplets to represent the probing facts (e.g. welliamm_Saaaeaae
occupation, Playwright>). Our main idea is to estimate the ratio of the likelihood of LLM generating correct surface text for ground-truth object entities (Playwright given varying prompts with subject entity (William Salllimm_hhalee aaaaaaassseeeearreeee and relation (occupation and the likelihood of generating entity text by pure chances. We distinguish the entities versus their surface text since each entity and relation may contain several text aliases. We introduce three latent variables to represent subject, object, and relation, and a Bayesian network to represent the causal dependency among them and generated text. Our proposed KaRR quantifies the ratio of generating text with and without specified relation/subject. This metric effectively captures the combined impact of both the subject and the relation in determining the model generation probability for the plausible text of the object (e.g., "playwright", "dramatist").
We develop a large-scale assessment suite with 994, entities, 600 relations, and their text aliases. Our suite is orders of magnitude larger than prior studies which only cover very few relations. We evaluate a comprehensive set of 20 LLMs using our method, including LLaMA [TTuvron al., 2023a], OPT [Zhang et al.. 2022], and others. Moreover, we also invite human experts to assess knowledge in these LLMs. Results reveal a strong correlation (0.43 Kendall's T) between KaRR and human evaluation. Our experimental results show that knowledge of LLMs with the same backbone architecture conforms the scaling law.
In summary, our contributions are: (1) We propose KaRR, a statistical method to assess the reliable knowledge contained in LLMs. We present probabilistic graphical model to tackle varying text aliases regarding fact. (2) We develop a large-scale assessment suite with 994,123 entities and 600 relations, which will be released to the community for further study. (3) We conduct a comprehensive evaluation of 20 LLMs. The experiments show that KaRR exhibits a strong correlation (0.43 Kendall's 7) with human assessment, and achieves a low variance to varying prompts, with a standard deviation of 0.82. Furthermore, our evaluation results are seldom affected by spurious correlations. Our code and data are available at https: //gi thub. com/ uKAss/sssss.
2 Related Work
Recent studies have demonstrated that language models possess factual knowledge, which can be utilized for complex reasoning, question answering, and performing various tasks[Roberts et al., 2020, Choudhary and Reddy, 2023]. To evaluate the knowledge in language models, previous works implement knowledge probing methods on MLMs. Petroni et al. [2019] introduces LAMA, studying whether MLMs correctly predict masked object entities in cloze-style prompt. However, Elazar et al. [2021] shows that probing outcomes are inconsistent with various prompts, resulting in contradictory or unreliable results. Shin et al. [2020] show that
Knowledge Probing

---

Relaton
Subject given Shutespeare)
(r, occusadinn
subject
worked occepetoz WaS
Object lo)
Stakespare
Reladon
Oblettol
P)
Swazn of Avon
p()
Swan of Avon
Stakesgerre
not oiven mamedd warkel as diderded a
Alaymggi dramskast
LLM
En [P(cks Rij
Sudject not given (S)
Shakmpper:
's job
workee as
Paywrihtt dramalist.
LLM
Be (P1
Plols.)) 3a. (Ploh, RJ)
Edisoo
(o.tR
3. [P(o8.r)]
Plola.r)
(a)
(b)
Figure 2: Illustration of KaRR using the fact <William, Shakespeare ceapeeeppare.cccuattiioon occupation,playwrighee playwright>
<william_Shakespear
as an example. S and R are latent variables representing the subject and relation, respectively; s, and o refer to specific values of subject, relation and object, respectively. (a) illustrates KaRR, which measures the impact of specifying r or not on LLM generating o given 8. (b) illustrates KaRR.. measuring the impact of specifying or not on LLM generating given
well-engineered automatic prompts result in much higher results than handcrafted LAMA prompts. Therefore, ParaRel [Elazar et al., 2021] and LPAQA [Jiang et al., 2020] proposed for MLMs. ParaRel measures the knowledge consistency of MLMs through the use of several paraphrased prompts. Subsequently, Hase et al. [2023] adopt their metric for paraphrase consistency measure of belief, and Raj et al. [2022] extend the metric to measure the general semantic consistency of OPT [Zhang al., 2022]. LPAQA focuses prompts selection and ensembling improve the prediction accuracy, by automatically discovering better prompts to query. However, it has not been comprehensively assessed to what extent LLMs contain knowledge about fact.
Spurious Correlations in Probing Recent literature reveals that empirical probing results for masked entities may be affected by spurious correlations within the text [Poerner al., 2020, Dong et al., 2022]. Spurious correlations occur when model takes shortcuts to generate answers based on superficial information in surface forms, even without possessing relevant knowledge. For instance, model with limited knowledge can correctly answer "French" when queried about the native language of actor "Jean Marais", simply because "Jean Marais" is a typically French name. This phenomenon inflates probing results. On the one hand, Zhao et al. [2021] find that probing results are influenced by contextual biases, and Dong et al. [2022] also note frequency biases in rank-based probing. On the other hand, the subject's 's class has been shown to introduce biases in predictions [Poerner et al., 2020]. To address these issues, Poerner al. [2020] eliminate vulnerable relation classes, however, this approach restricts coverage and does not provide comprehensive solution. In our statistical knowledge assessment, we account for both the subject and relation by decomposing their influence and using multiple aliases to minimize spurious correlations.
3 Statistical Knowledge Assessment
In this section, we present a statistical method for assessing knowledge in LLMs. We begin by introducing the graphical model for knowledge assessment, which serves the foundation of our approach. Subsequently, we introduce the Knowledge Assessment Risk Ratio (KaRR), which we then convert into computable form based the graphical model.
3.1 Graphical Model for Knowledge Assessment
Fig. 3 shows the graphical model that describes the knowl- edge symbols and their corresponding text forms. Here, we introduce latent variables S, R, and O to represent the sym- bolic subject, relation, and object, respectively. We use s,r, and o to refer to specific values of these variables, and the triplet (s, (s) to denote simple fact/knowledge, such as <William_Shakespee Shakespeare occupation, playwright>. We intro- duce a. B, 7 to denote the random variables for the textual forms (textual aliases²) of (S), (S, R), (S, R, O), respectively. Here, a, 3. and 7 refer specific values of a, B, and y, such being expressed as "Shakespeare".
S
α Text (S)
R
B
(S,
0
(S,
Figure 3: Graphical model for knowledge assessment.
"Alliases' are alternative names for entities relations, defined in Wikidata (https: wwwp::wwwwwwiaaa..a. org/ wiki wiki/Help:Allase
3

---

In general, fully observing the LLM probability on the latent variables of symbolic knowledge is infeasible. Although we can calculate the probability of the model generating "Shakespeare" we cannot calculate the probability of it generating the symbol <william <willlam_Shakepeeeeeeeee wh..hspeaaae...... This is because L.MM pretraining focuses on textual input rather than symbolie representations, and each symbol may encompass a multitude of textual variations (paraphrased expressions) that signify the same entity or relation. Fortunately, we can establish a connection between symbols and text forms, enabling the calculation of the model's probability for specific textual forms. As demonstrated in the graphical model, a hollow circle indicates the variable is latent, and a shaded circle indicates the variable is observed. The parent node of a is S, while S and R are parents to B. Additionally, y has three parent nodes, namely S, R. and O. This structure enables the calculation of model probability symbols through their corresponding text forms. Therefore, the goal of consistent L.M knowledge assessment is to estimate the model knowledge on symbols through the observable model probability across diverse corresponding textual forms.
3.2 Knowledge Assessment Risk Ratio
For a given LLM M, we define its factual knowledge correctness regarding a simple fact (s,, r. o) as the extent to which it generates knowledge-correct answers to various textual prompts. Specifically, we assess the joint impact of subject and relation symbols on the LLM's ability to generate the object symbol. This joint impact comprises two components: (1) the impact of specifying or not on M generating given s and (2) the impact of specifying s or not on M generating o given r.
Risk Ratio (RR) is statistical measure employed to compare the likelihood of specific event occurring in an exposed group versus an unexposed group [Sistrom and Garvan, 2004]. To quantify the impact of specifying symbols (or not) on the LLM generation accuracy, we adapt RR into new metric for knowledge assessment called KaRR. For component (1), we consider the exposed group as the case where relation symbol r is specified, and the unexposed group cases where the relation is not specified. The outcome is the probability of model generating the object symbol o, representing any possible textual representation of We denote the RR of specifying relation r as KaRR,:
r(o KaRR, ER [P)
P() P(o]s)
(1)
As the example illustrated in Fig. 2 (a), for the fact <william <william_Shakespaal Shakespeare, occupation, playwright>, KaRR, compare the probability of model generating any alias of the object entity playwright when the relation is specified as occupation (e.g., "Shakespeare works as a". "William Shakespeare's 's occupation is a a"), with that when the relation is not specified, the expectation of model probability on generating an object alias under various possible relations such as "Shakespeare married a". Similarly, the Risk Ratio (RR) for a specified subject s, denoted as KaRR,, is defined as,
r( Es [P(oSS))
p(o P(olr)
KaRR.
While the numerator for KaRR, is the same as that for KaRR,, it compares the numerator with the model's probability when the subject not specified. As shown in Fig. (b), KaRR, calculates the expected model probability of generating an object alias across multiple possible subjects (e.g.. "Dante worked as a" and "Thomas Edison worked as a"). To effectively represent the combined impact of KaRR, and KaRR,, calculate the joint influence the geometric mean of them, and name it Knowledge Assessment Risk Ratio (KaRR). Formally, we define KaRR on (s, 0) as,
KaRR(s, (s, 0) o) = /KaRR, (5, r, (s.
Intuitively, there are two components in KaRR: (1) the ratio of the chance of LLM generating o given s and r, and that of LLM generating o only given s; (2) the ratio of the chance of M generating o given 8 and r, and that of LLM generating only given r.
3.3 Computing KaRR using Graphical Model
The Knowledge Assessment Risk Ratio (KaRR), as indicated in Eq. (3), is formulated based on knowledge symbols, while directly computing the KaRR score using the definition (on symbols) is un- feasible. In light of this, our graphical model for knowledge assessment facilitates the implementation of KaRR by employing model probabilities on the text.

---

Based on Fig. 3, we can use the graphical model to evaluate the numerator of KaRR, and KaRR, in Eq. (1) and Eq. (2). To do so, we can use the following formula:
[8]
p(ols,r) E P(o, (el,,))
Σ P(Buls,r) P(ols,r,3u
(4)
The probability is computed by summing over all possible values of 3, namely the of aliases for r). For each specific value of B. 3g. we apply the product rule of probability and decompose P(o, By |s, r) into the probability of observing the relation 5% given s and r. denoted by P(8x r). and the probability of observing the object given the subject s, relation r, and the specific relation type 3k, denoted by P(o|s, r, Bt). We use PM denote the generation probability of model M. According to the law of total probability for marginal distribution, P(ols, (x) can be expanded as,
P(o|s,r, v. Ba)
E Pío, r,ss,r
E P,
(5)
In the context of calculating the denominator of KaRR,, it is important to note that each subject symbol may possess multiple aliases. For instance, the subject symbol 8, <William_Shaeleee Shakespeare>, may be expressed with varying probabilities ai ("Shakespeare" 02 ("Swan of Avon "), 0g
("William Shakespeare "), and so forth. To account for this variability, we expand the denominator as demonstrated in Eq. (6) by leveraging principles of marginal distribution and Bayes Rule.
P(o]s) - P(o, asl)
L P(as P(olss
(6)
P(ols, ox) is obtained by summing the joint probabilities of and 7j given 8, oz under all possible 75- Mathematically, this can be expressed as:
17|
bx]
P(o|s, a.) a.) = - P(o.n,ls, = Σ Pultibs.a.)Poosss....)
(7)
1-1
Subsequently, we obtain (s, as follows,
(s, P(ols.r)
Em P(Bels, En P.
(8)
P(o]s) E.S P(a.(s) E|T PM P()os a..
For the denominator of KaRR,, in the above example, P(olr) represents the expected probability of generating an alias for playwright ight when presented with a sentence that semantically conveys the occupation relation, over all the subject symbols. Consequently, we can expand P(o[r) as
(ull P(sulr)P(a) su,r). The computation of P(o) 8u, r) is identical to that in Eq. (4). Combining these results, we can determine the value of KaRR, (s, r, o) as Eq. (9).
D P(6. 1s, E,,,
KaRR = ElPu P(s. r) D P(B (s.. E PM r, r, P())7))))))
(9)
The KaRR score is calculated as the geometric mean of the results obtained from Eq. (8) and (9), and its detailed implementation is shown in Appendix A.
Experiments
We implement KaRR to examine knowledge in 14 LLMs and analyze its correlation with human assessment on model knowledge. Compared to large language models, knowledge graphs store a vast amount of structured information explicitly, which is ensured to be correct through manual construction and verification. Therefore, we implement KaRR based on existing large knowledge graphs, including T-REx [Elsahar et al., 2018] and Wikidata [Vrandečić and Krötzsch, 2014].
4.1
Data and Settings
In the following paragraphs, we provide details on the data, models and settings used in our experi- ments, with additional information provided in Appendix B and C.
Symbolic Facts We utilize T-REx knowledge graph [Elsahar al., 2018) 8] our primary source of symbolic knowledge. T-REx comprises 11 million triples that are aligned with 3.09 million Wikipedia abstracts; its quality validated by extensive crowdsourcing evaluation. In our main experiments, we consider all 600 English relations available in T-REx and sample a maximum of 20 facts per relation, resulting in total of 10,691 facts for knowledge assessment.
5

---

Subj.
Obj-
Rel.
Rel.
Method
Recall
Kendall's r p-value
Method
Alias
Alias
Alias
Cvg.
LAMA @1 1
83.25%
0.17
0.10
LAMA @1
X
x
6.83%
LAMA 10
65.81%
0.08
0.23
LAMA @ 10
X
X
X
6.83%
ParaRel
69.15%
0.22
0.02
ParaRel
X
X
6.33%
K-Prompts
78.00
0.32
0.03
KaRR
100%
KaRR
95.18%
0.43
0.03
(a) Basic information.
(b) Human evaluation results.
Table Basic information and human evaluation results. The abbreviations Subj., Rel., Obj.. and Rel. Cvg. represent Subject, Relation, Object, and the coverage of English relations in T-REx, respectively. The methods evaluated through human evaluation are assessed using the same set of 410 randomly sampled facts. K- K-Prompts is implemented using shared basic information with KaRR.
Multiple Entity Aliases For the text forms of subjects and objects involved in calculating KaRR, and KaRR, we search the entity aliases from Wikidata with Wikidata Integrator'. Ultimately, we obtain 1.349,474 aliases for 968,426 subjects and 368,511 aliases for 207,985 objects.
Multiple Relation Templates Previously, Elazar et al. [2021] manually wrote paraphrased relation templates for 41 relations, covering 6% of the relation classes in T-REx. To expand the coverage and diversity of templates for R. we incorporated placeholders "[X]" and "[Y]" into the relation aliases in Wikidata. Subsequently, we utilized a fine-tuned Flan-T5 model* for grammar correction to generate corresponding template candidates. We then filtered the templates with missing subjects or objects, and the remaining template candidates were all validated and corrected manually by two human annotators. Finally, we obtained 3,488 templates for 600 relations, with an average of 5.82 paraphrased templates per relation, achieving full coverage of the English relation classes in T-REx.
Overall KaRR Score The KaRR score assigned each fact is continuous value. If the KaRR score exceeds a predefined threshold, the fact is considered consistently known by the LLM. When applying method to of facts for the general knowledge reliability of LLMs, the overall KaRR score is calculated as the proportion of facts with a KaRR score exceeding the threshold.
As the statistical information shown in Tab. 1 (a), our method exhibits a broader coverage of relations and entity aliases compared to previous studies. To implement KaRR, we initially sample the knowledge triplet from T-REx, which comprises the symbolic subject relation and object By leveraging entity aliases and relation templates that we have devised, we can generate multiple expressions for each element in the KaRR metric. For instance, is implemented as the set of all searched subject aliases from our aliases datastore, and 3 represents the combination of all possible relation templates and subject aliases (by replacing the placeholder with the subject aliases).
4.2 Models
We explore the following models for knowledge assessment: (1) traditional size pretrained models
(< 5B), including GPT [Radford et al., 2018], XLNet [Yang et al., 2019], GPT2-XL [Radford et al., 2019]. GPT-NEO [Black et al., 2021], Ta-large and T5-3B [Raffel et al., 2020] and Phi-1.5 [Li et al., 2023] (2) medium size LLMSs (2 5B, < 50B), including GLM [Du et al., 2022], Falcon [Penedo et al., 2023], Dolly', Moss LLaMA [TTuurron et al., 2023a] and its variants Alpaca [Taori et al., 2023]. Vicuna [Chiang al., 2023] and WizardLM [Xu et al., 2023]; (3) large LLMs (2 50B), including the 65B LLaMAA [Touvron et al., 2023a], the 65B LLaMA2 [Touvron et al., 2023b] and the 175B OPT [Zhang et al., 2022].
4.3 Results of Knowledge Assessment
We present the results of our statistical knowledge assessment on various LLMs in Tab. 4 and more detailed results in Appendix D. Among the small and the medium-sized LLMs, most models' KaRR scores are between 3 to 13, indicating that they often struggle to generate factually accurate sentences consistently. However, it surprising to note that the 2.65B GPT-NEO outperforms the T5-3B with a 3.92 KaRR score difference. We hypothesize that this difference in performance may be attributed to the pretraining corpora. Specifically, GPT-NEO (trained on the Pile dataset [Gao et al., 2021])
hhttps: http://w.w.......a..aiaaaaattere
htatps://hugginngfaee wttpaaap./pppseeaaaaaaaatt----/e htt.hh/...ccm/taaaarraaaaaa///l/

---

Model
Size
KaRR Score
Model
Size
KaRR Score
GPT
0.12B
9.57
GLM
10B
5.59
XLNet
0.12B
5.86
Dolly
12B
15.60
T5-large
0.74B
3.22
LLaMA
13B
13.86
Phi-1.5
1.3B
10.58
Alpaca
13B
8.24
GPT2-XL
1.56B
12.27
Vicuna
13B
19.50
GPT-NEO
2.65B
13.44
WizardLM
13B
16.90
T5-3B
3B
9.52
Moss
16B
11.20
Falcon
7B
7.97
LLaMA
65B
14.56
BLOOM
7B
7.72
LLaMA2
65B
19.71
LLaMA
7B
12.37
OPT
175B
23.06
Table 2: Statistical knowledge assessment results on 20 LLMs. KaRR scores indicate that fine-tuning
on catructioo-following data (Alpaca) jeopardizes the reliability of knowledge-correct generation of the original LLaMA, whereas utilizing data from a more knowledgeable model enhances model knowledge (Vicuna). Additionally, the same backbone architecture adheres the scaling law.
primarily utilizes high-quality data, while T5 (pretrained on combination of corpora uses more varied set data sources. For larger LLMs, a comparison between the original LLaMA and the Alpaca model reveals that instruction-tuning might influence the model's ability to generate consistent and correct knowledge. Compared to the KaRR score the original LLaMA, the KaRR score of Vicuna indicates that fine-tuning LLM on data collected from more knowledgeable model could augment its knowledge. This finding is not commonly observed in previous studies but is consistent with our study presented in Appendix E.
Fig. demonstrates that our statistical evaluation approach supports the law of model scaling, where larger models tend possess more factual knowledge. This observation is consistent with our intuition about the relationship be- tween model size and knowledge representation. However, the benefits of scaling vary across models. For instance, T5 shows significant improvement in KaRR score when scaled from T5-small to T5-3B, while GPT2 and OPT also benefit from scaling, albeit to lesser extent. Notably, observe that when the model size is smaller than IB, GPT2 and OPT significantly outperform T5 in generating correct knowledge. This finding highlights the importance of selecting an appropriate model size for a given task and dataset, as larger models may always provide the best performance on generation factuality. Overall, the results of our statistical evaluation method consistent with our intuition, and highlight the importance of large- scale pretraining on knowledge-riee corpora for building figh-performing language models.
20.0
17.5
15.0
KaRR Score
12.5
+ T5
5.0
GPT2
2.5
Lama
Alpaca
0.0
#Modee Parameters (B)
Figure 4: Knowledge in models with the same backbone architecture follows the scaling law.
4.4 Human Evaluation for Knowledge Assessment
To further validate our approach, we conduct the human evaluation for model knowledge assessment. Utilizing the mean score from human evaluation as the benchmark for knowledge assessment on individual facts, we examine the validity of our knowledge assessment outcomes from two perspectives: 1) the [Kendall, 1938] correlation between the KaRR score on each fact and the human-annotated score on each fact and 2) the Recall for identifying knowledge that is not consistently known (facts with human average scores below 0.5) when employing the GPT2-XL model.
Human Evaluation Our human evaluation for the reliable knowledge generation ability of GPT2- XL contains two procedures: 1) Annotating: We have three annotation volunteers with NLP back- grounds. Each annotator asked to write three prompts to probe the model knowledge from multiple aspects for each fact. They are instructed to refine the prompts based on model generations until the generations are of the same type as the target answer (but not necessarily the target entity). To ensure fair comparison with LAMA [Petroni et al., 2019]. which is only applicable to 41 nigh-frequency

---

Method
Var (L)
Std (1)
Method
SP (J)
AP (J)
LAMA@1
1.90
1.37
LAMA@1
3.81
0.00
LAMA@N 1AMA@1O 10
5.14
2.27
LAMA@I 10
64.29
47.31
ParaRel
0.77
0.94
ParaRel
2.66
-0.51
K-Prompts
2.34
5.47
K-Prompts
0.00
-7.54
KaRR
0.67
0.82
KaRR
1.94
-14.94
(a) Evaluation variance towards varied prompts.
(b) Spurious correlation of knowledge assessment.
Table 3: Evaluation variance and spurious correlation analysis. Compared LAMA and ParaRel, our statistical knowledge assessment is more robust and less influenced by the spurious correlation.
relations, we restricted the relation classes to those included in LAMA. In total, we obtained 4,140 valid manually annotated prompts for 410 facts in T-REx. 2) Rating: We then asked three other annotators to rate the knowledge (0 or 1) in GPT2-XL according to the model generations. We used the average score of their ratings on all the prompts as the gold standard for knowledge assessment on each fact. Detailed annotation procedures and scoring criteria are shown in Appendix F.
Baselines As a comparison, we also calculated the correlation between three baseline methods and human evaluation results. The baseline methods, LAMA [Petroni et al., 2019] and ParaRel [Flazar al., 2021], were designed for MIMs, we modified them for LLMs by deleting words following the objects in the templates. We implement the Consistent-Ace score [Elazar et al., 2021] for ParaRel, which combines knowledge and consistency during evaluation. LAMA @ and LAMA@ 10 refer to the results obtained by checking whether the top-1 top- 10 generations contain the object, respectively. Furthermore, we propose K-Prompts baseline that computes the average probability of k randomly sampled prompts. It can be considered adapted version of LPAQA [Shin et al., 2020] for LLMs, utilizing a collection of ensembled high-quality prompts identical to those used in KaRR. Here, we adopt K- K-Prompts simple baseline to evaluate the knowledge assessment in the absence of latent variables. All baseline methods are executed on the same set facts as KaRR. LAMA and ParaRel are implemented as per the original papers [Petroni al., 2019, Elazar al., 2021], and K-Prompts is implemented using shared aliases and templates with KaRR. We set 22 as the threshold of KaRR, which is chosen by aligning the proportion of GPT2-XL known facts distinguished by humans on a sampled set of 200 facts. Similarly, the threshold for K-Prompts has been set to 0.13, 11,, as result of aligning with the proportion recognized by humans.
Results Tab. 1 (b) demonstrate that KaRR performs exceptionally well in knowledge detection, achieving Recall of 98. This highlights the potential of our statistical knowledge assessment to identify unreliable knowledge in LLMs, which is essential for downstream knowledge updat- ing [De Cao et al., 2021]. Furthermore, the evaluation results of KaRR show strong correlation (Kendall's 0.43) with human evaluation results. However, averaging the results of paraphrased prompts (ParaRel and K-Prompts) not effective due to varying prompt frequencies and diverse impacts of different prompts on the model's probability of generating an "object". Since human anno- tation for model knowledge assessment is demanding and time-consuming, our statistical evaluation method provides fast and accurate solution for the automatic knowledge assessment of LLMs.
5 Variance toward Prompts
Since the aim of the evaluation is to reflect the correctness of an LLM's knowledge towards various prompts, the evaluation result should be stable and not easily affected by the text forms used. In this section, we investigate the evaluation variance of our method. In contrast to ParaRel [Blazar al., 2021], J. which studies the consistency of models under paraphrased inputs, we investigate the variance of the knowledge assessment method. Specifically, we evaluate the evaluation variance, which measures how much the same evaluation results vary when different prompts are used.
Following the same procedures as described in Sec. 4.4, we sampled 4,100 facts and employed the paraphrased relation templates from ParaRel [Elazar et al., 2021] to replace the relation templates in the baseline methods and KaRR, resulting in three distinct relation templates for each fact. We evaluated the effectiveness of the different evaluation approaches each of the three paraphrased relation templates separately, the model is GPT2-XL. Additionally, we measured the evaluation variance by caleulating the variance and standard deviation of the evaluation scores.
Tab. (a) illustrates that the results of LAMA and K-Prompts exhibit significant variations when the probing prompt is changed to a paraphrased one, despite the overall prompt similarity in ParaRel
8

---

3.0
1.0
2.3
2.0
Std
Correlation
0.8
Kendall's
0.6
1.5
9 1.0
0.4
0.3
0.2
0.0
0.0
16
32
2
16
32
Figure 5: Ablation study on the sampling parameter K. With the increase in K, the variability in knowledge assessment decreases, while its correlation with human evaluation improves.
being 0.96. In contrast, our statistical knowledge assessment approach demonstrates robustness to prompt variation, with a 0.67 score variance and a 0.82 standard deviation.
6 Spurious Correlation in Knowledge Assessment
Both LAMA and KaRR use text as proxy estimate unobservable knowledge. However, spurious correlations can lead to inflated probing results where LLMs "guess" the probing answer based on these correlations. For example, when probed with the relation template "[X]'s birthplace is [Y]",
LLMs assign "America" the highest probability. regardless of the subject entity [Zhao et al., 2021]. To address this issue, we investigate spurious correlations in knowledge assessment, particularly focusing on the correlation between relation templates and high- figh-frequency objects.
We synthesize false facts by replacing objects with high-frequency but factually incorrect ones. They are non-existent knowledge in LLMs as they are not part of the pretraining corpus. Ideally, the scores on synthetic facts should be 0 if the assessment is not influenced by spurious correlations. We obtain high-frequency objects by identifying the top-5 GPT2-XL predictions on 3 ParaRel relation templates without subjects, yielding 14 relations with valid high-frequency objects relation, we randomly sample 100 facts and replace the original object with the high-frequency object (if the original object is not the high-frequene" object). We employ two metries: Spurious Positive (SP) and Positive Gap (AP). SP quantifies the proportion of false synthetic facts that are incorrectly recognized as known by the LLMs, and AP is calculated by SP minus model's true positive rate on the actual facts. Ideally, SP and (AP) should be close to 0 and (AP) should be negative; otherwise, it indicates significant bias in the evaluation result due to spurious correlations.
of 41 relations. For each
Tab. (b) presents results on GPT2-XL. LAMA scores are highly correlated to the prompt, leading to the high AP and SP. For example, 3.81 % and 64.29% false facts are estimated learned facts by LAMA @1 and LAMA @ 10, respectively. In contrast, K-Prompts and KaRR are less influenced by spurious correlations, with an SP score lower than 2% and a negative AP. By assessing the model knowledge from various perspectives, the ultimate scores of KaRR indicate the comprehensive knowledge mastery of LLMs. Therefore, our results are less influenced by co-occurrence shortcuts.
7 Discussion
In this section, discuss the causal explanation of KaRR and explore the effects of the sampling parameter, as well as the influence of model size.
Causal Explanation for KaRR From causal perspective, problem can be defined as: "given a subject 8, the causal effect of given the relation r or not for the LLM generating the object o". Assuming there are no confounders, given fixed the treatment (T) is whether a specific is provided or not, the outcome (Y) is the probability of the model generating o, then the Average Treatment Effect (ATE) [Holland, 1986] is defined as:
(1) B (0) B| T B|Y T 0J P(ols, r) ER [P(oss, R)
(10)
Note that in ATE, the minuend and subtrahend correspond to the denominators and numerator of KaRR, in Eq. (1), respectively. Assuming confounders are present, the only distinction lies in the mathematical operation utilized (subtraction or division). Both subtraction and division can be used compare the difference between the exposed group and the unexposed group, with the only distinction being the difference in the threshold for the resulting value. As a result, ATE helps clarify and validate KaRR from causal perspective.

---

0PT-350M
OPT-2.78
OPT-175B
Pizea
Flcar
PiSSI
P2578
P1378
919
200
P211
P2SAT
P57D1
PIM2
KaRR scares on relations OPT-350M
1.0
0.8
OPT-2.78
OPT-1758
0.
M874
$210
42744
PLASt
Pinne
1841
P2370
PI3M
Pazzo
Plapa
Pazza
92597
KaRR scores the 30 best known relations OPT-350M
Figure 6: KaRR scores on different relations when scaling up OPT from 350M to 2.7B and 175B.
Influence of Sampling Parameter When calculating KaRR, the sampling number K makes trade-off between the evaluation accuracy and efficiency. We perform an ablation study on the results of KaRR implemented with different K. Following the same settings of Sec. 5 and Sec. 4.4, we analyze the evaluation variance and evaluation aceuracy of the overall KaRR scores of GPT2-XI.. Fig. 5 demonstrates that increasing K decreases variance in knowledge assessment, but the impact is minor since evaluation variance already low. The sampling parameter has more significant effect on evaluation accuracy, with higher accuracy correlating with higher K values. When K is less than 4, the evaluation results are unreliable due to weak correlation with human evaluation results.
Impact of Scaling on LLM Knowledge To further study the impact of scaling up on LLM knowledge, we compare the KaRR scores of OPT-350M, OPT-2.7B, and OPT-175B on 30 best- known and 30 worst-known relations of OPT-350M. Fig. shows that larger models exhibit better and more consistent knowledge-corme correct generation ability on the relations where small models also generation ability. Moreover, larger models surpass small models in terms of knowledge on wider range of relations. Therefore, the scaling law of LLM knowledge is attributed to the broader and more extensive knowledge of larger I.LMs.
show some knowledge-corre
8 Limitations and Future Work
While KaRR demonstrates promising effectiveness for LLM knowledge assessment, there few aspects that require further investigation. First, KaRR scores are calculated for each fact using LLM inference on various prompts, which can be time-consuming and may limit the scalability of the evaluation. Second, our assessment is currently limited to scenarios where the logits of LLMs are available. This excludes many real-world applications where the models' internal representations may not be accessible or may be difficult to obtain. Third, the generality of KaRR can be further expanded. The current implementation focuses on atomic knowledge, characterized by triplets in the format of <subject, relation, object>. Nevertheless, real-world scenarios encompass significant portion of complex knowledge, necessitating multi-hop reasoning or the comprehension of intricate relationships among multiple entities [Choudhary and Reddy, 2023]. Evaluating such knowledge in L.LMs continues to be a challenging and unresolved issue. Overall, these areas for improvement emphasize the opportunities for continued research and development of knowledge assessment approaches for LLMs, taking into account the complexities of roal-world applications and the diverse types of knowledge to be assessed.
Conclusion
We investigate the consistent knowledge-correct generation ability of LLMs and propose a statistical knowledge assessment approach, called KaRR. By implementing our approach on 20 LLMs, we carry out an extensive assessment of the reliability of factual knowledge generation in these models. Our experiments reveal that our method yields high correlation with human evaluation, and addresses the issues of variance and spurious correlation during the knowledge assessment of generative LLMs. Additionally, our assessment suite includes millions of entity and relation aliases, enabling large-scale knowledge evaluation. Overall, our method evaluates the reliable knowledge generation ability of I.L.s, providing robust foundation for future developments knowledge refinement, augmentation, and beyond.
10

---

Acknowledgement
We thank all the anonymous reviewers for their constructive comments, Yuxuan Fan, Lei Li and Ce Zheng for their valuable suggestions data collection. Zhifang Sui is the corresponding author. This paper is supported by the National Key Research and Development Program China 2020AAA0106700 and NSFC project U19A2065.
References
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss: Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balean, and Hsuan- Tien Lin, editors, Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, ArXiv preprint, 2023.
Tyler A Chang and Benjamin K Bergen. Language model behavior: A comprehensive survey. ArXiv preprint, 2023.
Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. Say what you mean! large language models speak too positively about negative commonsense knowledge. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
Nurendra Choudhary and Chandan K Reddy. Complex logical reasoning over knowledge graphs using large language models. ArXiv preprint, 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computationall Linguistics (Volume Long Papers), 2022a.
Damai Dai, Wenbin Jiang, Qingxiu Dong, Yajuan Lyu, Qiaoqiao She, and Zhifang Sui. Neural knowledge bank for pretrained transformers. ArXiv preprint, 2022b.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, 2021.
Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. Calibrating factual knowledge pretrained language models. Findings of the Association for Computational Linguistics: EMNLP 2022, 2022.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), 2022.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Trans- actions of the Association for Computational Linguistics, 1.
11

---

Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. ArXiv preprint, 2021.
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Methods for measuring, updating, and visualizing factual beliefs in language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023.
Paul W Holland. Statistics and causal inference. Journal of the American statistical Association, 1986.
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. Survey hallucination in natural language generation. ACM Computing Surveys, 2023.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How know what language models know? Transactions of the Association for Computational Linguistics, 2020.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 1938.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.0544 054663, 2023,
Adyasha Maharana, Amita Kamath, Christopher Clark, Mohit Bansal, and Aniruddha Kembhavi. Exposing and addressing cross-task inconsistency in unified vision-language models. ArXiv preprint, 2023.
OpenAl.
2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 2002.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouci, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 11,, 2023.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. E-BERT: Efficient- eeffefett entity embeddings for BERT. In Findings of the Association for Computational Linguistics: EMNIP 2020, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training 2018
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 2020.
Harsh Raj, Domenic Rosati, and Subhabrata Majumdar. Measuring reliability of large language models through semantic consistency. ArXiv preprint, 2022.
12

---

Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 2023.
Christopher L Sistrom and Cynthia W Garvan. Proportions, odds, and risk. Radiology, 2004.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follovii Ilama model. 2023.
htttps://gtt thutthh.ccco. cww/ccoo///a//o catts-laabsttaforddd
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tzi Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Shung-Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Kathleen S. Meier-Hellseen Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. Lamda: Language models for dialog applications. ArXiv preprint, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023b.
Denny Vrandečié and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Communica- tions of the ACM, 2014.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint ar 23v4: 1..304..2244, 12244, 2023.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe Le. Xlnet: Generalized autoregressive pretraining for language understanding, In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Aché-Bucc Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. ArXiv preprint, 2022.
Zihao Zhao, Eric Wallace. Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, Proceedings of Machine Learning Research, 2021.
13

---

Appendix
A KaRR Implementation
To implement KaRR, we define a binary function òtezt (symbols)) that takes the symbols as input and returns value of either 0 1 depending whether any alias of all the symbols appears in the text. By applying this function, we can transform P(3x|s, r) and P(ag(s) into a format that compatible with the model probability.
P(Bu(s,r) P(8K,8,T) PM((u)) (s.r) P(s,r)
(11)
P(s,r)
P(as.)) P(s)
Puí(a)) . 5a, (8)
P(ouls)
(12)
P(s)
If the aliases Bg contain any alias of both and then (M r, 3x) is equal to (M (ri) (8)) Oth- erwise, PM (7g ss, (B,) is zero. Similarly, if the aliases contain any alias of s, then PM a.) equal PM (7y) (a)); otherwise. PM a4) is zero.
Furthermore, the probabilities P3))n;) and P(o s, ag, 75) are both equal to 6,, (o). where 8n the binary function that checks whether any alias of the object appears in the text Ty- By substituting the transformed formulas into the equation Eq. (9), we obtain an implementation of KaRR:
KaRR, = P(s) D (M)) S0 (8, Σ Ih PM((7 13)6,, E Pm(as) Sm, (8) Σ, PM(
(13)
Based on the graphical model depicted above, S and R conditionally independent when O is unknown. Consequently, we can simplify the expression P(a.r)
P(s) to P()) Therefore,
1 18l PM((B)) 6p, (s, Σ j-= PM P(r) E Pm((os) 60, (8) Σ 5=1 Pac(sslo)65 (o)
(s,R,(s,T,,))
(14)
The calculation of subject replacement is analogous. When O is unknown, P(aulr)-P(arr) prnser) equals to P(aj then we have,
Σ PM (Bx) 63 (8, E/T
CaRR,(s,r,,0
Σ (uul P(sulr) P(s r) E Par(3p Plan.rj 5% (su. r) EJTL Parl1 (o) uumi
Σ [ PM (3x) 8pg (8, E17l 1=1 PM (7 16,, (0)
P(s)
(3. 85 (S r) E| PM (3))
(15)
In practical scenarios, computing all possible values of a or 3 for the KaRR metric can be pro- make the assumption that the symbols S
hibitively expensive. To address this challenge, and R follow a unified distribution, and then use Markov Chain Monte Carlo (MCMC) sampling to estimate the denominators by sampling K subjects or objects. This results in P(a.) or P(r. being added to the denominators, offset by the P(r and P(s) terms in Eq. (14) and (15). By employing this approach, are able improve the accuracy of the KaRR metric within a limited sampling budget and derive a new score based on the sampled subset. Our default sampling parameter, K, is set to 4.
The KaRR score is calculated as the geometric mean of the results obtained from Eq. (14) and (15).
B Experimental Details
In this section, we provide further information the experimental details.
14

---

B.1 Settings for Different LLMs
In this paper, we load the models (except the OPT-175B) from the Huggingface Models' and implement the OPT-175B according to Metaseg'. The specifie version of Vicuna employed in this paper is musys/vieuma-7bdeelaae
B.2 OOV Objects
We collect a vast range aliases for objects, certain aliases may contain out-of-vocabulary (OOV) words. Consequently, the model generation probability for these aliases cannot be computed. To address this issue, we follow the approach suggested in LAMA [Petroni et al., 2019] and exclude OOV object aliases when implementing KaRR on each LLM.
B.3 Scoring Details
According the KaRR implementation above, we can obtain the KaRR score on each fact. the KaRR is higher than a threshold, the fact is regarded as a consistently known one by the LLM; otherwise not. When implementing the statistical framework on large set of facts for the general knowledge correctness of LLMs, we get the proportion of known facts over the 10,691 symbolic facts as the (KaRR higher than the threshold) as the overall KaRR score. We set 22 the threshold for KaRR, and the threshold is chosen by aligning the proportion of GPT2-XL known facts distinguished by humans on sampled set of facts. Also result of alignment with human-recognized proportion, the threshold for the K-Prompts baseline implemented in this paper is set to 0.13.
C Data Distribution
In this section, we provide detailed information on alias distribution and relation distribution.
C.1 Alias Distribution
Type
Top-k
Avg. Aliases
Proportion of Related Facts
50
7.62
12.10%
100
6.73
16. 13%
subject
500
4.68
28.34%
1000
3.70
33.70%
50
5.06
39.84%
100
4.79
47.35%
object
500
3.80
65.44%
1000
3.48
71.93%
Table 4: Average number of aliases and proportion of related facts for top-k frequent entities.
In Tab. 4, we present the average number of aliases and the proportion of related facts for the top-k frequent entities. The most frequent entities typically possess more than five aliases, related to large proportion of facts. This long-tailed tailed pattern is consistent with real-world alias distribution, where substantial portion of entities have only one alias.
C.2 Relation Distribution
As shown Tab. 5, the distribution of relations in our dataset exhibits a long-tail phenomenon, which is consistent with the distribution of relations in real-world knowledge and the distribution of relations within knowledge graphs [Elsahar et al., 2018].
hnttps: co/models
15

---

Top-k
Avg. Aliases
# Rellated Facts
Proportion of Related Facts
50
6.96
12,638,356
90.05%
100
5.97
13,538,967
96.47%
150
5.54
98.35% 13,802,467
200
5.22
99.20% 13,921,742
250
4.84
13,977,899
99.60%
Table Distributions of top-k relations.
Model Size
KaRR Score
KaRR, Score
KaRR, Score
Best-known Relations
Worst-known
Model
Relations
GPT
0.12B
9.57
4.30
31.9
P2822, P1537, P3095
P1427, P1951, P3091
XLNet
0.12B
5,86
0.25
37.78
P1537, P1264, P2922
P664, P31, P1951
TS-large
0.74B
3.22
4.61
6.82
P872, P560, P2860
P664, P1951, P3091
Phi-1.5
1.3B
10.58
6.30
36.67
P3403, P538, P2500
P1542, P184, P926
GPT2-XL
1.56B
12.27
12.12
30.38
P2922, P2499, P397
P3091, P2159, P210
GPT-NEO
2.65B
13.44
12.97
29.15
P1537, P2499, P3403
P210, P2746, P681
TS-3B
3B
9.52
3.59
23.04
P3095, P3033, P1537
P1427, P674, P1951
Falcon
7B
7.97
10.44
23.73
P3403, P1376, P1165
P805, P1542, P184
BLOOM
7B
7.72
6.58
33.42
P3403, P2922, P2289
P263, P1542, P184
LLaMA
7B
12.37
5.30
44.67
P1264, P3403, P3095
P1542, P926, P3275
GLM
10B
5.59
5.12
24.05
P2922, P2499, P397
P3091, P681, P210
Dolly
12B
15,60
11.35
39.88
P1264, P3403, P2822
P1542, P926, P3275
LLaMA
13B
13.86
6.02
45.36
P1264, P3403, P3095
P1542, P926, P3275
Alpaca
13B
8.24
3.77
31.5
P770, P1906, P2974
P1427, P674, P1951
Vicuna
13B
19.50
8.34
49.83
P410, P974, P2159
P263, P1542, P1906
WizardLM
13B
16.90
1.13
50.24
P2839, P521, P3085
P926, P1427, P922
Moss
16B
11.20
12.83
27.01
P1264, P3403, P1537
P1542, P184, P926
ILaMA
65B
14.56
6.98
44.80
P3261, P1576, P2551
P263, P1542, P1906
LLaMA2
65B
19.71
10.13
45.78
P2839, P974, P3261
P1542, P1906, P926
OPT
175B
23.06
19.89
42.31
P3403, P1264, P2922
P2321, P2746, P538
Table 6: Detailed results of statistical knowledge assessment 20 LLMs.
D Detailed Results
Tab. provides detailed results for statistical knowledge assessment on LLMs.
E Case Study on LLM Generation
Tab. provides case study on LLM generation.
Human Annotation Specification
In this section, we provide further details on our human annotation.
F1 Main Procedures
In our Human Annotation Assessment (HKA), there are two primary procedures: generation-guided prompting and knowledge mastery scoring. Each procedure is carried out by a group of three volunteer annotators who have backgrounds in NLP.
F.1.1 Generation-Guider Prompting
In our HKA, a critical procedure is generation-guided prompting. That is, annotators continually modify their prompts according to both manual and automatic feedback from the model generations.
16

---

Model
Question
Generation
The occupation of Swan of Avon
The Swan of Avon British cargo ship...
William Shakespeare worked
playwright and actor in London
LLaMA
Joseph Armstrong worked for:
The Daily Telegraph (London)
The companies where Joseph Armstrong worked in are:
1. The Armstrong Company, 1900-1901
2. The Armstrong Company, 1901-1902...
The occupation of Swan of Avon was:
that of a boatman...
William Shakespeare worked as a
playwright, poet, and actor...
Alpaca
Joseph Armstrong is former employee of the following companies:
Joseph Armstrong worked for:
Joseph Armstrong currently works for:...
The companies where Joseph Armstrong worked in are:
Joseph Armstrong's average salary was $100,000...
The occupation of Swan of Avon was:
The Swan of Avon was a paddle steamer..
William Shakespeare worked as
playwright and actor...
Vicuna
Joseph Armstrong was member of the following organisations...
Joseph Armstrong worked for:
The companies where Joseph Armstrong worked in are:
Armstrong, Whitworth & Co...
Table The generation cases of the original 7B LLaMA model and its instruction tunne variants Alpaca and Vicuna. For each model, we provide the same Knowledge-related prompts.
This approach differs from previous scenarios where annotators merely wrote relation templates for each fact create prompts [Petroni et al., 2019]. Instead, ask each annotator to write prompts from various perspectives, including prompting for entities, relations, or the entire fact. Moreover, we perform manual answer-type checking throughout the annotation process. Adjusting prompts according to the model's generation is essential for accurately assessing the model's knowledge. For example, because of the prompt template "[X] is [Y]" in LAMA, 100% of the top-1 prediction for facts of relation 'P31' is a definite article or a predicate. This induces the underestimation of model knowledge while implementing LAMA @ And for 'P19', although the prompt in LAMA, "[X] was born in [Y]." is plausible one, most of the top- generations are dates or months. This indicates the significance of model-generation www.ggennnnnnnuddeeeee guidedd prompt refining.
Manual Answer-type Checking To ensure that the prompt accurately prompts the model to generate the relevant text of the current fact in the next word, instead of freely describing other content, we conduct manual answer-type checking. Only when the model generations are mostly of the same type target answers, the prompts are deemed valid ones; otherwise, they will be further refined to a valid one.
Automatic Diversity Checking To explore the model generations towards different prompts, implement an automatic diversity check for the prompts. Regarding the other valid prompts as references and the new prompt the candidate, and set a similarity threshold to 0.8 to filter similar prompts. Besides, annotates are asked to not only write prompts probe the object entity of each fact but also write prompts to probe the subject, the relation, or the whole fact.
calculate the BLEU score [Papineni et al., 2002]
E.1.2 Knowledge Mastery Scoring
In the knowledge mastery scoring procedure, we enlist an additional three annotators to evaluate the model's knowledge. Provided with prompt and the corresponding GPT2-XL model generation (top-1 generation using beam search), annotators are asked to score the model's 's knowledge mastery on scale between 0 (unknown) and (known). Ultimately, the scores for prompts related to the
17

---

same fact are averaged to determine the model's knowledge mastery score for that particular piece of factual knowledge.
F.2
Annotation Specification
As introduced above, our annotation mainly contains two parts: A. prompting, B. scoring. All the annotators volunteer students majoring in Computational Linguistics in China. We told them the annotation intention and explained how the data would be used.
E2.1 Prompting
Given triplet fact, annotators are instructed to do their best to determine whether the knowledge is stored in the LLM and compose three prompts (from various perspectives) for each fact to probe
During the prompting process, annotators should create prompts that not only probe the object entity but also examine the subject, the relation, or the entire fact. For instance, for the fact <Obama, birthplace, Hawaii>, possible prompts could include:
One U.S. president was born in Hawaii, and his was subject)
(This prompt probes for the
Obama was born in (this prompt probes for the object).
Hawaii is Obama's
(this prompt probes for the relation).
The state where Barack Obama born is (this prompt probes for the object).
President Obama was born Honolulu,
(this prompt probes for the object).
To ensure the diversity of prompts, we employ the BLEU metric [Papineni et al., 2002] for automatic diversity checks. Specifically, we consider all valid prompts references and the prompt the candidate, setting threshold of 0.5 to filter out prompts that are similar to existing ones in terms of surface forms.
Annotators assess whether a given prompt effectively elicits knowledge based on the model-generated results. If the generated content is primarily related the target aspect, the prompt is considered valid, For instance, when prompting for object entities, if the model's output consistently includes a of entities (both correct and incorrect ones acceptable, long the type broadly relevant), the prompt is deemed valid. However, if the generated text or tokens consist of stopwords or unrelated entity types, the prompt is classified capital city of China is to", "is now a". "is described as" or "is the birthplace". While The capital city of China is called is a valid prompt as the model generations are city names like: "Shanghai", "Beijing" or "Hangzhou".
invalid. For example, the fact <China, capital, Beijing>, The is not a valid prompt for GPT2 as the generations are: "inching", "home
F.2.2 Scoring
The scoring criteria follows:
"Invalid":
Factually unrelated generations or improper prompts.
1: The output produced during the greedy decoding process contains at least alias of the correct answer, without any factually incorrect answers.
0: The generated content does not include any related entities or relations, nor are there any correct answers expressed in paraphrased words or phrases. In other words, the correct answer is not present in any of the generated responses.
E3 Annotation Results
We ultimately obtained 4,140 manually annotated prompts for 410 valid facts. The overall sentence similarity among the prompts for the same fact is 0.78, which demonstrates good diversity compared to the 0.95 overall similarity of prompts in ParaRel [Elazar et al., 2021]. The Kappa score between human raters is 0.4.
While HKA provides relatively trustworthy and reliable results for assessing model knowledge correctness, it places high demands on human annotators and is time-consuming to implement for
18

---

each fact in the model-specific annotation process. In our experience, annotating 100 facts on GPT2- XL takes an average of hours per annotator. Consequently, HKA more suitable as a solution for obtaining gold-standard results for knowledge assessment on smaller scale. Kendall's 7 correlation between the results of a knowledge-assessing approach and HKA reflects the evaluation's 's accuracy.
19


Reviews:
[{"review_id": "ntf0YTILoa", "reviewer": "Reviewer_bA4R", "summary": "The paper presents a framework for the quantitative assessment of the knowledge captured by large language models, which includes a proposed metric and a large set of relations. Given subject-relation-object triplets, a basic approach to quantify the model's knowledge would be to use the probability of the object entity being generated, given the subject and the relation. This naive approach has shortcomings, which the authors address in the following way:\n1. Entities can have multiple synonyms, which are important to consider to accurately assess consistency. The authors therefore augment their proposed evaluation suite with a set of aliases for each entity, extracted from Wikidata, and adapt the metric to take these into account.\n2. As this kind of probing often suffers from spurious correlations, the authors propose the \"knowledge assessment risk ratio\" metric, which  also takes into account the expected generation probability when either the subject or the relation entity are not specified.\n\nThe authors thoroughly evaluate the proposed approach on 14 generative model. They additionally measure the effectiveness of their approach compared to other pre-existing metrics, showing strong correlation with human judgement and robustness of the metric towards prompt variation.", "questions": "* Do you think your proposed approach could be effective as a validation metric for the training of LLMs?", "limitations": "The authors did a decent job of addressing limitations, and I don't expect any potential negative societal impact from this work.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* This is a sound and extensively evaluated approach towards the automatic quantitative assessment of the knowledge learned by LLMs. This method displays strong correlation with human judgements. While it cannot replace human annotation entirely, it should be extremely useful to the community as a cheaper, faster automatic way of performing knowledge assessment, much like how BLEU can be used during model development to validate the performance of translation models.\n* The authors should be commended for calling out robustness (consistency of generation given similar prompts). This aspect is crucial for real-world applications of such models, and is often ignored in similar works.", "weaknesses": "* The proposed framework is limited to assessing knowledge in a fairly simplistic way, via the prediction of entities in subject-relation-object triplets. It would have been interesting to hear more about the limitations of such an approach, taking into account e.g. slightly more complex type of queries (see e.g. arXiv:2305.01157).\n\nVery minor:\n* lines 141-144, 148, 202, 291, 313-315: straight quotes should be turned into curly quotes", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer bA4R for the positive feedback and we are grateful for the time you spent on our submission. We are also glad for the acknowledgment that the problem we are working on is realistic and that the method we propose is sound. We would like to provide comprehensive responses to your comments and questions.\n\n**Q1 \"The proposed framework is limited to assessing knowledge in a fairly simplistic way, via the prediction of entities in subject-relation-object triplets. It would have been interesting to hear more about the limitations of such an approach, taking into account e.g. slightly more complex types of queries (see e.g. arXiv:2305.01157).\"**\n\nMany thanks for your constructive comments! Yes, our knowledge assessment focuses on atomic knowledge--each piece of knowledge consists of a triplet. \n\nAs suggested, we outline several limitations of the current method as follows: \n1. struggle to evaluate the knowledge that requires multi-hop reasoning or the understanding of complex relationships between multiple entities. \n2.  struggle to evaluate model knowledge mastery with context-dependent or time-varying information, which is often crucial for correct knowledge understanding and representation.\n\nThanks for your advice again! We've incorporated them into our limitation section, and we're interested in eliminating the limitations in future work.\n\n**Q2 \"Do you think your proposed approach could be effective as a validation metric for the training of LLMs?\"**\n\nThank you for making a great point. Our approach can be well-suited as a knowledge evaluation metric during LLM training, as our testing approach aligns with the model's next token prediction objective. It can be tested across various base models, such as the LLaMA, GPT2-XL, and T5-large reported in Table 2. \n\nIt's noteworthy that the goals of the language model training phase go beyond just model knowledge - they might also include semantic understanding, among others. These goals and the learning of knowledge may involve certain trade-offs. If we directly apply KaRR as a validation metric, it could lead to new issues. Nevertheless, through the implementation of reasonable improvements to our metrics, we can overcome these issues.\n\nWe appreciate your constructive feedback and are keen to explore this perspective further.\n\n**Q3. Typo and format.**\n\nThanks for kindly pointing out our typos and format problems. We have revised them all in the latest version.\n\nThank you very much for the constructive comments, which really help us further improve our work. We hope our answers have addressed your concerns. If you have any further questions, we are happy to address them."}, {"review_id": "s4I32YWmd7", "reviewer": "Reviewer_e9uD", "summary": "this paper proposes a statistical method to probe the knowledge in generative language models, which aims at connecting symbolic knowledge and GLM's text format generation. More specifically, the KaRR comprises two components with regard to specifying relation and subject entity. The authors also present the graphical model for model implementation on the text. The results show that the knowledge in GLMs follows the scaling law, but when the model is finetuned on instruction-following data, it may compromise the model's ability to consistently generate factually correct text.", "questions": "1. I believe it would be interesting to see the model's knowledge across different domains, such as encyclopedia, biomedical, etc.\n2. Since high-quality aliases are often difficult to obtain, it would be interesting to see the impact of alias, such as alias count, etc.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. very important and interesting problem, assessing the knowledge stored in generative language models is challenging and worth studying\n2. the proposed KaRR method shows strong robustness to prompt variance\n3. Interesting findings: instruction-following data compromises the model’s capability to generate factually correct answer.", "weaknesses": "1. This work only focuses on generating the object, I believe the author can assess the model's ability to generate the subject (by reversing the triples/facts) to gain a more comprehensive assessment of knowledge stored in GLMs.\n2. The average alias count for each subject is approx 1.39, and for each object it is about 1.78, which I believe could not support the \"diverse\" claim.\n3. The proposed KaRR’s design to measure reliability is not clear enough.\n4. In line 38, the authors claim \"Prior methods are designed for masked language models (MLMs) and are incapable of measuring GLMs.\". However, to the best of my knowledge, there are several works about assessing the knowledge in GLMs, such as [1, 2].\n \n[1] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.\n[2] Dhingra, B., Cole, J. R., Eisenschlos, J. M., Gillick, D., Eisenstein, J., & Cohen, W. W. (2022). Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10, 257-273.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer e9uD for your review and are grateful for the time you spent on our submission. We are also glad you think our research problem is important and our findings are interesting. Below we would like to give detailed responses to each of your comments.\n\n**Q1 “This work only focuses on generating the object, I believe the author can assess the model's ability to generate the subject (by reversing the triples/facts) to gain a more comprehensive assessment of knowledge stored in GLMs.”**\n\nThank you for your comments. It is worth mentioning that the reversed facts have already been included  in our knowledge base, T-REx. For example, both <Barack Obama (Q76), spouse, Michelle Obama (Q13133)> and <Michelle Obama (Q13133), spouse, Barack Obama (Q76)> are covered. Please note that not all facts can be reversed, so the final number of subject and object entities we obtain are slightly different.\n\nMoreover, our approach is intrinsically not restricted to particular entity types, as long as corresponding triplets are constructed. Formally, our method incorporates the probabilities $P(e_2|e_1, r)$, $P(e_2|r)$, and $P(e_2|e_1)$, where $e_1$ or $e_2$ can serve as either subject or object and maintain symmetric properties. We implement this method by employing naturally existing triplets within the current knowledge base, which covers a wide range of knowledge.\n\n**Q2 \"The average alias count for each subject is approx 1.39, and for each object it is about 1.78, which I believe could not support the \"diverse\" claim.”**\n\nThank you for highlighting the potential confusion. We'd like to clarify that real-world alias distribution exhibits a long-tailed pattern, with a substantial portion of entities having only one alias. This significantly affects the average number of aliases.\n\nHowever, the most frequent entities typically possess more than five aliases, related to a large proportion of facts (details shown in the table below).\n\n|\\# Top-k  | Type | \\# Avg. aliases | Proportion of related facts |\n|------------- | ------------- |  ------------- | ------------- |\n|50  | subject   | 7.62  | 12.10\\% |\n|100  | subject    | 6.73  | 16.13\\% |\n|500 | subject   | 4.68  | 28.34\\% |\n|1000  |subject   | 3.70  | 33.70\\% |\n|50  | object  | 5.06 | 39.84\\% |\n|100  | object  | 4.79  |  47.35\\% |\n|500 | object  | 3.80  | 65.44\\% |\n|1000  | object | 3.48 | 71.93\\% |\n|\n\n**Q3 “KaRR’s design to measure reliability is not clear enough.”**\n\nAs mentioned in Lines 24-29 and illustrated in Fig. 1, reliability refers to the ability to consistently generate knowledge-correct text towards various possible prompts with the same semantics.\n\nTo measure reliability, we take multiple text forms for the same entity or relation into knowledge assessment and build the graphical model of text forms and symbolic knowledge (triplets). As mentioned in Sec. 3.3, we expand the KaRR metrics based on the graphical model (Eq. 4-9), so as to evaluate GLM on a simple knowledge with various possible prompts.\n\n**Q4 “There are several works about assessing the knowledge in GLMs, such as [1, 2].”**\n\nThank you for mentioning related works. We'd like to clarify that existing methods, both open-form and closed-form, are tailored for specific models or tasks, but not well-suited for a comprehensive knowledge assessment of most GLMs.\n\nOpen-form methods like TEMPLAMA [1] probe factual knowledge in MLMs and an exceptional GLM, T5, using a cloze-test format. However, they're not applicable to most GLMs without masked token/span prediction objectives (see Lines 35-42).\n\nClosed-form methods, such as multiple-choice questions [2], evaluate domain-specific knowledge-utilization and problem-solving abilities rather than assessing knowledge itself. These methods may also be biased toward specific option numbers.\n\nTo avoid confusion, we've revised the statement to: \"Prior methods target MLMs and don't provide a universal solution for assessing GLMs' knowledge.\"\n\n**Q5 \"I believe it would be interesting to see the model's knowledge across different domains, such as encyclopedia, biomedical, etc.\"**\n\nThank you for your constructive comments. We agree that it is valuable to assess and analyze model knowledge across different domains. Due to the lack of domain-specific entity aliases, we focus on encyclopedia knowledge using T-REx and Wikidata.\n\nNonetheless, our proposed graphical model and KaRR metric provide versatile solutions for knowledge assessment, adaptable to diverse domains by replacing the knowledge base with relevant domain-specific data. This flexible approach sets the stage for future research in domain-specific knowledge evaluation.\n\n**Q6 \"Since high-quality aliases are often difficult to obtain, it would be interesting to see the impact of alias, such as alias count, etc.\"**\n\nThank you for making a great point. As suggested, we add the experiments using different numbers of aliases. The results (listed in the following table) show that a larger number of aliases decrease the variance of knowledge assessment, which is consistent with our intuition and the analysis of the sampling number  K in Sec. 7.\n\n\\# Avg. aliases  | KaRR Score | Variance |\n------------- | ------------- |  ------------- |\n1  | 19.67  | 0.96   | \n2  | 18.98  | 0.72  |\n4  | 18.78  | 0.55  |\n8  | 18.82  | 0.51  |\n|\n\n*(GPT2-XL on 500\\*20 facts of 500 frequent entities.)*\n\nOverall, we greatly appreciate your efforts for your thoughtful comments on our paper. We hope our answers have addressed your concerns. We have revised the paper to clear the issues you mentioned in the comments in our latest version.\n\n**Reference:**\n\n[1] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding.\n\n[2] Dhingra, B., Cole, J. R., Eisenschlos, J. M., Gillick, D., Eisenstein, J., & Cohen, W. W. (2022). Time-aware language models as temporal knowledge bases."}, {"review_id": "59cTYqQR8M", "reviewer": "Reviewer_adWh", "summary": "The paper proposes KaRR, a statistical approach to assess factual knowledge for generative language models based on graphical models. An assessment suite is also proposed for future research. Experiments are conducted with 14 popular large language models and comprehensive analyses are also conducted to reveal related properties.", "questions": "-  The conclusion of \"instruction tuning impairs knowledge reliability\" should be considered more carefully (line 236). There are at least two factors contributing to the KaRR difference between Alpaca and Vicuna, data quality and tuning style. You do not know exactly whether data or tuning style is the major reason.\n- Minor comments: \n  - line 228, repeated \"KaRR score\".\n  - line 230, should be \"with a 3.92 KaRR score difference\"\n  - GLM in Table 2 and GLM across the paper are a little bit confusing.\n- What are the distributions of relations in the dataset?\n", "limitations": "Refer to previous weaknesses and questions.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper released a dataset for assessing factual knowledge in generative language models, which is large-scale (millions of entities and text aliases) and could be used in future works.\n- This paper proposed a statistical score KaRR based on graphical models and KaRR aligns well with human preferences as shown with human evaluation.\n- Lots of popular large language models are experimented with and the analyses are interesting, e.g., Table 2(b).", "weaknesses": "- The knowledge assessment focuses on entity-aware knowledge, which could be a relatively limited knowledge form. The current LLMs are good at identifying entity-aware knowledge. The major hallucinations actually come from numbers, dates, etc. Not a strict weakness, just wondering if KaRR could be extended to these aspects.\n- There are in total 600 relations in the assessment suite. Does that mean the suite can only be employed for the specific 600 relations? What if there are new relations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe sincerely thank Reviewer adWh for the positive comments on our method and analyses as well as the valuable suggestions. We would like to give detailed responses to each of your comments.\n\n**Q1. “The knowledge assessment focuses on entity-aware knowledge, which could be a relatively limited knowledge form. The current LLMs are good at identifying entity-aware knowledge. The major hallucinations actually come from numbers, dates, etc. Not a strict weakness, just wondering if KaRR could be extended to these aspects.”**\n\nThank you for this interesting point. Yes, we do rely on entity-aware knowledge; however, our definition of \"entity\" encompasses a wide range of categories, including dates, natural features, numbers, laws, and more. As a result, KaRR already incorporates the evaluation of facts associated with such diverse entities.\n\nIn fact, dates can have aliases as well. For example, the fact <Barack Obama, date_of_birth, 4 August 1961 (Q69285218)> can be expressed in various ways within the prompt, including \"August 4, 1961,\" \"4 August 1961,\" and \"1961-08-04.\" Similarly, for numbers, the number 7 (Q23350) can be represented as \"the number 7,\" \"seven,\" \"number seven,\" \"number 7,\" and so on.\n\n**Q2. “There are in total 600 relations in the assessment suite. Does that mean the suite can only be employed for the specific 600 relations? What if there are new relations?”**\n\nThank you for highlighting the potential confusion. We would like to clarify that our statistical knowledge assessment approach is not limited to the 600 relations used in our experiments. The graphical model for knowledge assessment and the KaRR metric, as described in Section 3, can be implemented with various entities or relation types. For instance, researchers focusing on medical relations can easily substitute relations with predicates from a medical knowledge graph, such as RepoDB[1] and SemMedDB[2].\n\nIf new relations emerge, we can generate relation templates for these new relations following the same process outlined in Section 4.1, and subsequently incorporate them into the relation and fact sets. As our method is designed to serve as a general framework for GLM knowledge assessment, it can be flexibly adapted to different relation types based on specific requirements.\n\n\n**Q3. “The conclusion of \"instruction tuning impairs knowledge reliability\" should be considered more carefully (line 236). There are at least two factors contributing to the KaRR difference between Alpaca and Vicuna, data quality and tuning style. You do not know exactly whether data or tuning style is the major reason.”**\n\nThanks for the great suggestion and sorry for the confusion. The conclusion is derived from the comparison of **the original LLaMA and the Alpaca**, as the Alpaca is finetuned on the LLaMA with instruction-following data.\n\nWe agree that there are other factors contributing to the KaRR difference between Alpaca and Vicuna and we have utilized the phrase \"could lead to\" (Line 234) to convey our hypothesis regarding the potential cause.\nTo be more rigorous, we have modified the claims accordingly in the revised version (i.e., \"For larger GLMs, a comparison between the original LLaMA and the Alpaca model reveals that instruction-tuning might influence the model's ability to generate consistent and correct knowledge\" ).\n\n**Q4 \"What are the distributions of relations in the dataset?\"**\n\nThanks for this great point! We've incorporated the following table into our Appendix 7. The distribution of relations exhibits a long-tail phenomenon, which is consistent with the distribution of relations in real-world knowledge and the distribution of relations within knowledge graphs[3].\n\nTop-k  | \\# Avg. aliases| \\# Related facts | Proportion of related facts |\n------------- | ------------- |  ------------- | ------------- |\n50  | 6.96  | 12638356   | 90.05\\% |\n100  | 5.97  | 13538967   | 96.47\\% |\n150  | 5.54  | 13802467  | 98.35\\% |\n200  | 5.22  | 13921742  | 99.20\\% |\n250  | 4.84  | 13977899   | 99.60\\% |\n|\n\n*(Distributions of top-k relations in the dataset.)*\n\nMoreover, Figure 2 in the T-REx paper[1] demonstrates the distribution of the number of alignments created for each relation within the T-REx dataset. T-REx notably surpasses other datasets in terms of the number of examples provided, not only for the most prevalent predicates but also for those found in the long tail.\n\n**Q5. Typo and writing**\n\nThank you for your gracious help in identifying typos and writing problems in our work. We have implemented all required changes in the newest version.\n\n**Reference:**\n\n[1] Brown, Adam S., and Chirag J. Patel. A standard database for drug repositioning.\n\n[2] Kilicoglu, Halil, et al. SemMedDB: a PubMed-scale repository of biomedical semantic predications\n\n[3] Elsahar, Hady, et al. T-REx: A large scale alignment of natural language with knowledge base triples"}, {"review_id": "krGyyLXCV7", "reviewer": "Reviewer_zX87", "summary": "The paper introduces an automatic evaluation metric to assess the amount of factual knowledge kept by large language models (LLMs). This proposed metric considers various surface forms of factual knowledge presentation, allowing for an evaluation that not only measures the accuracy of the models in terms of factual knowledge but also considers the prediction robustness of the knowledge. Later, the authors demonstrated that this metric has a higher correlation with human annotations than previously proposed metrics. Additionally, the paper includes several robustness analyses for the metric, such as examining the impact of the prompting format and its relationship to co-occurrence statistics. And it shows the new metric's effectiveness compared to previous metrics.", "questions": "1. In The KaRR scoring details section in Appendix, you mentioned the selection of a threshold based on human judgment alignment. It would be valuable to explore the impact of different threshold values on the metric's performance and investigate the generalizability of the chosen threshold on other sets of knowledge. Does certain threshold value also result in high correlation in other datasets?\n2. Minor suggestion: compress Figure 5 by representing the average performance of the best and worst relations. This modification could improve the readability of the figure and make it easier to interpret the performance trends.\n3. Adding the best and worst relations to the appendix would help understand the model performance. I would be curious to see it personally.\n4. There is a missing number in the equation mentioned in line 43 of the appendix. ", "limitations": "No, the limitation of the work is not discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed metric is commendable as it goes beyond capturing mere accuracy and considers model performance consistency. This aspect is crucial in evaluating the factual knowledge of large language models, as it reflects their ability to provide correct information consistently.\n2. The strong invariance results achieved by the automatic evaluation metric compared to other baselines are a notable strength.\n3. The paper's comprehensive analysis of the proposed metric is great. By covering multiple aspects, the authors thoroughly evaluate the metric's performance. This level of analysis contributes to a better understanding of the metric's strengths, limitations, and overall effectiveness in assessing the factual knowledge of large language models.", "weaknesses": "1. The human correlation score of 0.43 for the automatic evaluation metric may be considered relatively low. \n2. The proposed metric is not interpretable. To improve the interpretability of the score, calculating an oracle score and then comparing the current score to the oracle score ratio could be a useful approach. This ratio can aid in better understanding the effectiveness of the metric and provide a clearer picture of its performance relative to the best achievable outcome.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer zX87 for the positive recommendation as well as the valuable suggestions.\nWe really appreciate your kind words that our metric is commendable and our analysis is comprehensive. Below we would like to give detailed responses to each of your comments.\n\n**Q1 \"The human correlation score of 0.43 for the automatic evaluation metric may be considered relatively low.\"**\n\nWe'd like to clarify that 0.43 Kendall's tau correlation is a relatively high score for an automatic metric. To resolve possible misunderstandings, it's worth mentioning Kendall's tau correlation scores in [1][2]. \n\nAs our evaluation of the KaRR metric is similar to the segment-level metric evaluation for machine translation, below we list Kendall's tau scores for some widely accepted metrics for machine translation evaluation for comparison. As shown, Kendall's tau is a relatively strict metric compared to Pearson’s correlation, and 0.43 is a relatively high score for automatic metrics.\n\nMetric  | Pearson’s correlation |  Kendall's tau |\n------------- | ------------- |  ------------- |\nMETEOR  | 0.484  | 0.324 |\nNLEPOR | 0.483  | 0.281  |\nSENTBLEU-MOSES | 0.465  | 0.266  |\nDEP-REF-EX  | 0.453  | 0.307   |\n|\n\n*(Pearson’s correlation and Kendall’s τ between WMT-13 segment-level metrics and human assessment for Spanish-to-English. Please refer to Table 2, Sec 4.2 in [1] for the whole table.)*\nMetric  | Kendall's tau |\n------------- | ------------- |\nHTER  | 0.4324 |\n HMEANT gold - monolinguals \\* |0.4324  |\nHMEANT auto - monolinguals \\* | 0.3964  | \nBLEU / METEOR / TER / PER | 0.1982  | \n|\n\n*(Sentence-level correlation with human adequacy judgments. The weights for individual roles in the metric are tuned by optimizing the correlation.  Please refer to Table 8, Sec. 8.2 in [2] for the whole table.)*  \n\nIn addition, we compared the Kendall-tau correlation of KaRR with humans and the baseline metrics for model knowledge evaluation such as LAMA with humans in Table 1 (b). KaRR shows a much stronger correlation (KaRR: 0.43  versus LAMA@1: 0.17, K-prompts: 0.32). Besides, the Recall of finding human-detected false knowledge in  Table 1 (b) (KaRR: 95.18% versus LAMA@1: 83.25%, K-prompts: 78.00%) further support KaRR‘s correlation with human evaluation. \n\n**Q2 \"To improve the interpretability of the score, calculating an oracle score and then comparing the current score to the oracle score ratio could be a useful approach. \"**\n\nMany thanks for your insightful suggestion! \nYes, we agree that calculating an oracle score and then comparing the current score to the oracle score ratio would be more interpretable. However, it is worth noting that obtaining the oracle score for each GLM on every piece of knowledge is hardly feasible. We can only sample a portion of knowledge and obtain the human evaluation results (Sec. 4.4), but the cost of the human evaluation is substantial.\nTo improve the interpretability of the score, we have updated our draft. In the latest version, we have included the quantitative results for both $KaRR_r$ and $KaRR_s$. \n\n**Q3 \"In the KaRR scoring details section in Appendix, you mentioned the selection of a threshold based on human judgment alignment. It would be valuable to explore the impact of different threshold values on the metric's performance and investigate the generalizability of the chosen threshold on other sets of knowledge. Does certain threshold value also result in high correlation in other datasets?\"**\n\nThank you for making a great point. As suggested, we add the experiments on different threshold values and the generalizability of the chosen threshold. \n\nThe results below highlight the significance of selecting a human-aligned threshold for evaluation accuracy, as minimal human input enhances the correlation with human assessments. If the threshold is too low, the criteria become lenient, and variance slightly increases. Conversely, an excessively high threshold results in near-100% Recall for false knowledge detection due to strictness but slightly reduces the correlation with human judgment. \n\nEncouragingly, our approach displays commendable generalization for the chosen threshold on other sets of knowledge. This implies that once we select a threshold in alignment with human judgment in a specific knowledge base, it can be directly applied to other knowledge bases.\n\n Threshold  | KaRR Score | Variance | Recall | Kendall's \\tau |\n------------- | ------------- |  ------------- |------------- | ------------- | \n22 (\\*human-aligned)  | 12.27  | 0.67  | 95.18  | 0.43  |\n8  | 33.05  | 0.87  |  84.20  | 0.28  |\n16  | 17.20 | 0.69  | 90.05  | 0.36  |\n32 |  7.93  | 0.64  | 99.53  | 0.32  |\n|\n\n*(Impact of different threshold values.)*\n\n Threshold  | KaRR Score | Variance | Recall | Kendall's \\tau |\n------------- | ------------- |  ------------- | ------------- |  ------------- |\nT-REx  | 12.27  | 0.67  | 0.67  | 95.18 | 0.43  |\nGoogle-RE  |  9.76| 0.54  | 91.78  | 0.39   |\nConceptNet | 10.02  | 0.68  | 87.06  | 0.39    |\nSQuAD  | 9.98  | 0.80  | 85.21  |  0.34  |\n|\n\n*(Generalizability of the chosen threshold on other sets of knowledge. For the three new databases, facts are randomly sampled and manually aligned since some of the knowledge bases do not correspond with entity ids in Wikidata.)*\n\nMany thanks for your constructive suggestions on Figure 5 and the appendix. As suggested, we've added the best and worst relations for each GLM in Table 2(a) to Appendix 6,  and added the equation number \"(8)\" in line 43 of the appendix. \n\nThanks again for your detailed and constructive comments! We hope our answers have addressed your concerns. \n\n**Reference:**\n\n[1] Graham, Yvette, Timothy Baldwin, and Nitika Mathur. Accurate evaluation of segment-level machine translation metrics.\n\n[2] Lo, Chi-kiu, and Dekai Wu. MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles."}, {"review_id": "7MsAYrJlQQ", "reviewer": "Reviewer_4epC", "summary": "This paper proposes a statistical approach called KaRR to assess the factual knowledge contained in Generative Language Models (GLMs). The authors use a large-scale assessment suite with 994,123 entities and 600 relations to evaluate 14 GLMs of various sizes. The results show that KaRR exhibits a strong correlation with human assessment and achieves a lower variance to varying prompts. The experiments also reveal interesting insights into the scaling law of GLMs and the impact of tuning on instruction-following data. ", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed statistical approach is novel and effective in assessing the factual knowledge contained in GLMs. The proposed method effectively considers different surface forms of subject, object, and relation.\n2. The large-scale assessment suite used in the experiments is comprehensive and orders of magnitude larger than prior studies.1. Some of the paper's findings, such as scaling laws of knowledge and fine-tuning instructions to improve consistency, have been found in previous work.\n2. Previous work has systematically assessed the consistency of the same facts under multiple prompts, but the facts considered in this paper are larger in scale and prompts are more diverse.", "weaknesses": "1. Some of the paper's findings, such as scaling laws of knowledge and fine-tuning instructions to improve consistency, have been found in previous work.\n2. Previous work has systematically assessed the consistency of the same facts under multiple prompts, but the facts considered in this paper are larger in scale and prompts are more diverse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your effort in reviewing our paper and your acknowledgment of our paper’s contribution. We are very glad that you liked our statistical approach, the large-scale assessment suite, and the insightful findings.\n\nOur response to your further comments is as follows:\n\n**Q1 \"Some of the paper's findings, such as scaling laws of knowledge and fine-tuning instructions to improve consistency, have been found in previous work.\"**\n\nWe want to clarify our motivation for scaling the model size and highlight that our results are different from the findings in previous work. \n\n**Scaling laws of knowledge:** \nThere may be some misunderstanding. Our primary objective in examining the evaluation results of scaling model size is to demonstrate that the knowledge assessment findings align with previous studies on scaling laws, thereby validating the performance of our approach. \n\nIn addition, our research offers some unique insights. Figure in Table 2(b) first shows the scaling law of model knowledge of the different curves of scaling laws of different large generative language models. Besides, Figure 5 illustrates the detailed differences in model knowledge across different relations, providing an interpretation of the disparities among models of 350M, 2.7B, and 175B.\n\n**Fine-tuning instructions:** \nYes, previous studies have mentioned fine-tuning instructions to consistency but our finding is novel that tuning on instruction-following data may compromise the model's capability to generate factually correct text consistently. However, our novel finding suggests that tuning based on instruction-following data might actually impede the model's ability to consistently generate factually accurate text. \n\nWhile instruction-tuning does enhance the understanding of instructions, it is not surprising that it also improves the general consistency of LLMs in response to prompts, as evidenced by earlier research. Our study, conversely, examines the consistency of knowledge, specifically in generating factually accurate answers consistently. Contrary to general consistency, our results reveal potential trade-offs between instruction understanding ability and consistent mastery of knowledge. We further present intriguing examples of this point in Table 1, located in Appendix 4.\n\n**Other findings:** \nIn addition to these two findings, we have also uncovered many other findings that have not been previously investigated. For example, the spurious correlation in knowledge assessment (Table 3 (b)), the knowledge evaluation variance towards different prompts (Table 3 (a)), etc. We hope that these findings will enhance the research community's understanding of the knowledge assessment of GLMs and encourage further investigation.\n\nAgain, we thank you for acknowledging that our approach is novel and effective, and experimental results are insightful. If you have any additional suggestions or concerns, we would be happy to discuss them further with you.\n\n**Reference:**\n\n[1] Petroni, Fabio, et al. Language models as knowledge bases?\n\n[2] Dhingra, Bhuwan, et al. Time-aware language models as temporal knowledge bases.\n\n[3] Roberts, Adam, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model?"}]

Author rebuttal:

Meta-review: This paper presents a statistical approach to determine whether generative language models (GLMs) can produce factual knowledge. It concentrates on examining knowledge expressed as subject-verb-object triplets, and also provides a dataset for testing such knowledge. The paper also provides an extensive evaluation of recent models, as well as insights into how instruction tuning affects the performance of models in terms of producing factual knowledge. This evaluation and the proposed dataset can be used to inform future research on the factuality of GLMs. I encourage the authors to update the paper to clarify the points discussed with the reviewers, and to also elaborate on the generality of the proposed method, as it focuses on evaluating knowledge that can be expressed in a triplet format.
Decision: Accept (poster)