SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qingxiu Dong $^{1}$ , Jingjing Xu $^{2}$ , Lingpeng Kong $^{3}$ , Zhifang Sui $^{1}$  and Lei Li $^{4}$
$^{1}$  National Key Laboratory for Multimedia Information Processing,
$^{2}$  Shanghai AI Lab  $^{3}$  The University of Hong Kong  $^{4}$  Carnegie Mellon University
dqx@stu.pku.edu.cn, {jingjingxu, szf}@pku.edu.cn, lpk@cs.hku.hk, leili@cs.cmu.edu
Given varying prompts regarding a factoid question, can a large language model (LLM) reliably generate factually correct answers? Existing LLMs may generate distinct responses for different prompts. In this paper, we study the problem of quantifying knowledge contained in an LLM regarding a given set of facts. We propose KaRR, a statistical approach to assess factual knowledge for LLMs. The main idea is to estimate the ratio of LLM generating text corresponding to the answer entity given diverse prompts of the subject and the querying relation, versus it generating by random chances. Our assessment suite contains a comprehensive set of 994,123 entities and 600 relations, with 1,395,905 text aliases. We use our method to evaluate 20 LLMs of various sizes, including LLaMA, Alpaca, OPT, etc. Experiments show that our results have a strong correlation (0.43 Kendall's  $\tau$ ) with the results of human assessment on LLMs. Our results reveal that the knowledge in LLMs with the same backbone architecture adheres to the scaling law, while tuning on instruction-following data sometimes compromises the model's capability to generate factually correct text reliably.
Large language models (LLMs) have achieved impressive performance on many tasks including text generation, question answering, and dialog generation [Brown et al., 2020, Thoppilan et al., 2022, Bubeck et al., 2023]. Previous studies have shown that pretrained language models store factual knowledge in parameters [Roberts et al., 2020, Dai et al., 2022a,b, Singhal et al., 2023], demonstrating considerable potential as assistants for responding to user questions based on their stored knowledge.
Despite the remarkable success of LLMs, critical concerns arise — LLMs often generate unreliable answers given varying prompts [Ji et al., 2023, Maharana et al., 2023, Chang and Bergen, 2023, Chen et al., 2023]. As shown in Fig. 1, reliability refers to the ability to consistently generate knowledge-correct text, posing a higher standard than accuracy. For example, although Alpaca-7B [Taori et al., 2023] generates accurately (playwright) given a prompt of “William Shakespeare’s job”, it generates incorrect text (boatman) given a prompt of “the job of Swan of Avon is”<sup>1</sup>. ChatGPT [OpenAI, 2022] correctly generates playwright and teacher. However, it answers “No” for the prompt “Is William Shakespeare a teacher?” In this paper, we assess the knowledge contained in an LLM and investigate whether an LLM is able to consistently generate factually correct answers given varying prompts. Knowledge assessment is a critical problem for LLMs. The assessment results directly affect the people’s trust in the LLM generated content. Once we identify inconsistency of LLM generation, we could potentially correct such knowledge in LLMs [De Cao et al., 2021, Zhao et al., 2021, Dong et al., 2022].
1 Swan of Avon is a nickname of Shakespeare.
Figure 1: Accuracy pertains to generating the correct knowledge in response to a specific prompt, evaluated as a binary value (correct or incorrect). However, accuracy alone does not ensure reliability, as responses may vary with different prompt expressions. Reliability requires consistently generating correct knowledge for semantically similar prompts and is measured as a continuous value. Reliability sets a higher standard than accuracy by maintaining consistent accuracy across related prompts.
Important steps have been taken in the automatic knowledge evaluation for pre-trained language models [Petroni et al., 2019, Poerner et al., 2020, Jiang et al., 2020]. However, two issues still remain: 1) Prior methods focus on assessing the accuracy but not the reliability. Consequently, their probing results are prone to inflation or bias since models exhibit spurious correlations towards specific prompts [Poerner et al., 2020]. 2) Most prior methods target masked language models (MLMs) and don't provide a universal solution for assessing the knowledge in generative LLMs. MLMs are evaluated by probing whether they predict the masked object in response to a given prompt in cloze form, while the freely generated results of generative LLMs might be irrelevant to factual knowledge. Therefore, there is a need for a method that assesses both the accuracy and consistency, i.e. reliability, of knowledge in LLMs.
In this paper, we introduce KaRR, a statistical approach to assess whether a large language model contains reliable factual knowledge. We use triplets to represent the probing facts (e.g. <William_Shakespeare, occupation, Playwright>). Our main idea is to estimate the ratio of the likelihood of LLM generating correct surface text for ground-truth object entities (Playwright) given varying prompts with subject entity (William_Shakespeare) and relation (occupation and the likelihood of generating entity text by pure chances. We distinguish the entities versus their surface text since each entity and relation may contain several text aliases. We introduce three latent variables to represent subject, object, and relation, and a Bayesian network to represent the causal dependency among them and generated text. Our proposed KaRR quantifies the ratio of generating text with and without a specified relation/subject. This metric effectively captures the combined impact of both the subject and the relation in determining the model generation probability for the plausible text of the object (e.g., "playwright", "dramatist").
We develop a large-scale assessment suite with 994,123 entities, 600 relations, and their text aliases. Our suite is orders of magnitude larger than prior studies which only cover very few relations. We evaluate a comprehensive set of 20 LLMs using our method, including LLaMA [Touvron et al., 2023a], OPT [Zhang et al., 2022], and others. Moreover, we also invite human experts to assess knowledge in these LLMs. Results reveal a strong correlation (0.43 Kendall's  $\tau$ ) between KaRR and human evaluation. Our experimental results show that knowledge of LLMs with the same backbone architecture conforms to the scaling law.
In summary, our contributions are: (1) We propose KaRR, a statistical method to assess the reliable knowledge contained in LLMs. We present a probabilistic graphical model to tackle varying text aliases regarding a fact. (2) We develop a large-scale assessment suite with 994,123 entities and 600 relations, which will be released to the community for further study. (3) We conduct a comprehensive evaluation of 20 LLMs. The experiments show that KaRR exhibits a strong correlation (0.43 Kendall's  $\tau$ ) with human assessment, and achieves a low variance to varying prompts, with a standard deviation of 0.82. Furthermore, our evaluation results are seldom affected by spurious correlations. Our code and data are available at https://github.com/dqxiu/KAssess.
Knowledge Probing Recent studies have demonstrated that language models possess factual knowledge, which can be utilized for complex reasoning, question answering, and performing various tasks[Roberts et al., 2020, Choudhary and Reddy, 2023]. To evaluate the knowledge in language models, previous works implement knowledge probing methods on MLMs. Petroni et al. [2019] introduces LAMA, studying whether MLMs correctly predict masked object entities in a cloze-style prompt. However, Elazar et al. [2021] shows that probing outcomes are inconsistent with various prompts, resulting in contradictory or unreliable results. Shin et al. [2020] show that
(b)
Figure 2: Illustration of KaRR using the fact <William_Shakespeare,occupation,playwright> as an example. S and R are latent variables representing the subject and relation, respectively;  $s$ ,  $r$  and  $o$  refer to specific values of a subject, relation and object, respectively. (a) illustrates  $\mathrm{KaRR}_r$ , which measures the impact of specifying  $r$  or not on LLM generating  $o$  given  $s$ . (b) illustrates  $\mathrm{KaRR}_s$ , measuring the impact of specifying  $s$  or not on LLM generating  $o$  given  $r$ .
well-engineered automatic prompts result in much higher results than handcrafted LAMA prompts. Therefore, ParaRel [Elazar et al., 2021] and LPAQA [Jiang et al., 2020] are proposed for MLMs. ParaRel measures the knowledge consistency of MLMs through the use of several paraphrased prompts. Subsequently, Hase et al. [2023] adopt their metric for paraphrase consistency as a measure of belief, and Raj et al. [2022] extend the metric to measure the general semantic consistency of OPT [Zhang et al., 2022]. LPAQA focuses on prompts selection and ensembling to improve the prediction accuracy, by automatically discovering better prompts to query. However, it has not been comprehensively assessed to what extent LLMs contain knowledge about a fact.
Spurious Correlations in Probing Recent literature reveals that empirical probing results for masked entities may be affected by spurious correlations within the text [Poerner et al., 2020, Dong et al., 2022]. Spurious correlations occur when a model takes shortcuts to generate answers based on superficial information in surface forms, even without possessing relevant knowledge. For instance, a model with limited knowledge can correctly answer "French" when queried about the native language of actor "Jean Marais", simply because "Jean Marais" is a typically French name. This phenomenon inflates probing results. On the one hand, Zhao et al. [2021] find that probing results are influenced by contextual biases, and Dong et al. [2022] also note frequency biases in rank-based probing. On the other hand, the subject's class has been shown to introduce biases in predictions [Poerner et al., 2020]. To address these issues, Poerner et al. [2020] eliminate vulnerable relation classes, however, this approach restricts coverage and does not provide a comprehensive solution. In our statistical knowledge assessment, we account for both the subject and relation by decomposing their influence and using multiple aliases to minimize spurious correlations.
3 Statistical Knowledge Assessment
In this section, we present a statistical method for assessing knowledge in LLMs. We begin by introducing the graphical model for knowledge assessment, which serves as the foundation of our approach. Subsequently, we introduce the Knowledge Assessment Risk Ratio (KaRR), which we then convert into a computable form based on the graphical model.
3.1 Graphical Model for Knowledge Assessment
Figure 3: Graphical model for knowledge assessment.
Fig. 3 shows the graphical model that describes the knowledge symbols and their corresponding text forms. Here, we introduce latent variables  $\mathbf{S},\mathbf{R}$ , and  $\mathbf{O}$  to represent the symbolic subject, relation, and object, respectively. We use  $s,r$ , and  $o$  to refer to specific values of these variables, and the triplet  $(s,r,o)$  to denote a simple fact/knowledge, such as <William_Shakespeare, occupation, playwright>. We introduce  $\alpha,\beta,\gamma$  to denote the random variables for the textual forms (textual aliases²) of  $(\mathbf{S}),(\mathbf{S},\mathbf{R}),(\mathbf{S},\mathbf{R},\mathbf{O})$ , respectively. Here,  $\alpha,\beta$ , and  $\gamma$  refer to specific values of  $\alpha,\beta$ , and  $\gamma$ , such as  $\alpha$  being expressed as "Shakespeare".
In general, fully observing the LLM probability on the latent variables of symbolic knowledge is infeasible. Although we can calculate the probability of the model generating "Shakespeare", we cannot calculate the probability of it generating the symbol  $<$  William_Shakespeare>. This is because LLM pretraining focuses on textual input rather than symbolic representations, and each symbol may encompass a multitude of textual variations (paraphrased expressions) that signify the same entity or relation. Fortunately, we can establish a connection between symbols and text forms, enabling the calculation of the model's probability for specific textual forms. As demonstrated in the graphical model, a hollow circle indicates the variable is latent, and a shaded circle indicates the variable is observed. The parent node of  $\alpha$  is S, while S and R are parents to  $\beta$ . Additionally,  $\gamma$  has three parent nodes, namely S, R, and O. This structure enables the calculation of model probability on symbols through their corresponding text forms. Therefore, the goal of consistent LLM knowledge assessment is to estimate the model knowledge on symbols through the observable model probability across diverse corresponding textual forms.
3.2 Knowledge Assessment Risk Ratio
For a given LLM  $\mathcal{M}$ , we define its factual knowledge correctness regarding a simple fact  $(s, r, o)$  as the extent to which it generates knowledge-correct answers to various textual prompts. Specifically, we assess the joint impact of subject and relation symbols on the LLM's ability to generate the object symbol. This joint impact comprises two components: (1) the impact of specifying  $r$  or not on  $\mathcal{M}$  generating  $o$  given  $s$  and (2) the impact of specifying  $s$  or not on  $\mathcal{M}$  generating  $o$  given  $r$ .
Risk Ratio (RR) is a statistical measure employed to compare the likelihood of a specific event occurring in an exposed group versus an unexposed group [Sistrom and Garvan, 2004]. To quantify the impact of specifying symbols (or not) on the LLM generation accuracy, we adapt RR into a new metric for knowledge assessment called KaRR. For component (1), we consider the exposed group as the case where a relation symbol  $r$  is specified, and the unexposed group as cases where the relation is not specified. The outcome is the probability of model generating the object symbol  $o$ , representing any possible textual representation of  $o$ . We denote the RR of specifying relation  $r$  as  $\mathrm{KaRR}_r$ :
$$
\mathrm {K a R R} _ {r} (s, r, o) = \frac {P (o | s , r)}{\mathbb {E} _ {\mathbf {R}} [ P (o | s , \mathbf {R}) ]} = \frac {P (o | s , r)}{P (o | s)} \tag {1}
$$
As the example illustrated in Fig. 2 (a), for the fact  $\langle William\_Shakespeare, occupation, playwright \rangle$ ,  $\mathrm{KaRR}_r$  compare the probability of model generating any alias of the object entity playwright when the relation is specified as occupation (e.g., "Shakespeare works as a", "William Shakespeare's occupation is a"), with that when the relation is not specified, the expectation of model probability on generating an object alias under various possible relations such as "Shakespeare married a". Similarly, the Risk Ratio (RR) for a specified subject  $s$ , denoted as  $\mathrm{KaRR}_s$ , is defined as,
$$
\operatorname {K a R R} _ {s} (s, r, o) = \frac {P (o | s , r)}{\mathbb {E} _ {\mathbf {S}} [ P (o | \mathbf {S} , r) ]} = \frac {P (o | s , r)}{P (o | r)} \tag {2}
$$
While the numerator for  $\mathrm{KaRR}_s$  is the same as that for  $\mathrm{KaRR}_r$ , it compares the numerator with the model's probability when the subject is not specified. As shown in Fig. 2 (b),  $\mathrm{KaRR}_s$  calculates the expected model probability of generating an object alias across multiple possible subjects (e.g., "Dante worked as a" and "Thomas Edison worked as a"). To effectively represent the combined impact of  $\mathrm{KaRR}_s$  and  $\mathrm{KaRR}_r$ , we calculate the joint influence as the geometric mean of them, and name it Knowledge Assessment Risk Ratio (KaRR). Formally, we define KaRR on  $(s,r,o)$  as,
$$
\operatorname {K a R R} (s, r, o) = \sqrt {\operatorname {K a R R} _ {r} (s , r , o) \cdot \operatorname {K a R R} _ {s} (s , r , o)} \tag {3}
$$
Intuitively, there are two components in KaRR: (1) the ratio of the chance of LLM generating  $o$  given  $s$  and  $r$ , and that of LLM generating  $o$  only given  $s$ ; (2) the ratio of the chance of M generating  $o$  given  $s$  and  $r$ , and that of LLM generating  $o$  only given  $r$ .
3.3 Computing KaRR using Graphical Model
The Knowledge Assessment Risk Ratio (KaRR), as indicated in Eq. (3), is formulated based on knowledge symbols, while directly computing the KaRR score using the definition (on symbols) is unfeasible. In light of this, our graphical model for knowledge assessment facilitates the implementation of KaRR by employing model probabilities on the text.
Based on Fig. 3, we can use the graphical model to evaluate the numerator of  $\mathrm{KaRR}_s$  and  $\mathrm{KaRR}_r$  in Eq. (1) and Eq. (2). To do so, we can use the following formula:
$$
P (o | s, r) = \sum_ {k = 1} ^ {| \beta |} P (o, \beta_ {k} | s, r) = \sum_ {k = 1} ^ {| \beta |} P \left(\beta_ {k} | s, r\right) \cdot P (o | s, r, \beta_ {k}) \tag {4}
$$
The probability is computed by summing over all possible values of  $\beta$ , namely the set of aliases for  $(s, r)$ . For each specific value of  $\beta$ ,  $\beta_k$ , we apply the product rule of probability and decompose  $P(o, \beta_k | s, r)$  into the probability of observing the relation  $\beta_k$  given  $s$  and  $r$ , denoted by  $P(\beta_k | s, r)$ , and the probability of observing the object  $o$  given the subject  $s$ , relation  $r$ , and the specific relation type  $\beta_k$ , denoted by  $P(o | s, r, \beta_k)$ . We use  $P_{\mathcal{M}}$  to denote the generation probability of model  $\mathcal{M}$ . According to the law of total probability for marginal distribution,  $P(o | s, r, \beta_k)$  can be expanded as,
$$
P (o | s, r, \beta_ {k}) = \sum_ {j = 1} ^ {| \gamma |} P (o, \gamma_ {j} | s, r, \beta_ {k}) = \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} (\gamma_ {j} | s, r, \beta_ {k}) P (o | \gamma_ {j}) \tag {5}
$$
In the context of calculating the denominator of  $\mathrm{KaRR}_r$ , it is important to note that each subject symbol may possess multiple aliases. For instance, the subject symbol  $s$ , <William_Shakespeare>, may be expressed with varying probabilities as  $\alpha_{1}$  ("Shakespeare"),  $\alpha_{2}$  ("Swan of Avon"),  $\alpha_{3}$  ("William Shakespeare"), and so forth. To account for this variability, we expand the denominator as demonstrated in Eq. (6) by leveraging principles of marginal distribution and Bayes Rule.
$$
P (o | s) = \sum_ {i = 1} ^ {| \alpha |} P (o, \alpha_ {i} | s) = \sum_ {i = 1} ^ {| \alpha |} P (\alpha_ {i} | s) P (o | s, \alpha_ {i}) \tag {6}
$$
$P(o|s,\alpha_i)$  is obtained by summing the joint probabilities of  $o$  and  $\gamma_{j}$  given  $s,\alpha_{i}$  under all possible  $\gamma_{j}$ . Mathematically, this can be expressed as:
$$
P (o | s, \alpha_ {i}) = \sum_ {j = 1} ^ {| \gamma |} P (o, \gamma_ {j} | s, \alpha_ {i}) = \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} (\gamma_ {j} | s, \alpha_ {i}) P (o | s, \alpha_ {i}, \gamma_ {j}) \tag {7}
$$
Subsequently, we obtain  $RR_{r}(s,r,o)$  as follows,
$$
\operatorname {K a R R} _ {r} (s, r, o) = \frac {P (o | s , r)}{P (o | s)} = \frac {\sum_ {k = 1} ^ {| \beta |} P \left(\beta_ {k} | s , r\right) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} \left(\gamma_ {j} | s , r , \beta_ {k}\right) P (o | \gamma_ {j})}{\sum_ {i = 1} ^ {| \alpha |} P \left(\alpha_ {i} | s\right) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} \left(\gamma_ {j} | s , \alpha_ {i}\right) P (o | s , \alpha_ {i} , \gamma_ {j})} \tag {8}
$$
For the denominator of  $\mathrm{KaRR}_s$ , in the above example,  $P(o|r)$  represents the expected probability of generating an alias for playwright when presented with a sentence that semantically conveys the occupation relation, over all the subject symbols. Consequently, we can expand  $P(o|r)$  as  $\sum_{u=1}^{|s_u|} P(s_u|r)P(o|s_u,r)$ . The computation of  $P(o|s_u,r)$  is identical to that in Eq. (4). Combining these results, we can determine the value of  $\mathrm{KaRR}_s(s,r,o)$  as Eq. (9).
$$
\mathrm {K a R R} _ {s} (s, r, o) = \frac {\sum_ {k = 1} ^ {| \beta |} P \left(\beta_ {k} \mid s , r\right) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} \left(\gamma_ {j} \mid s , r , \beta_ {k}\right) P \left(o \mid \gamma_ {j}\right)}{\sum_ {u = 1} ^ {| s _ {u} |} P \left(s _ {u} \mid r\right) \cdot \sum_ {k = 1} ^ {| \beta |} P \left(\beta_ {k} \mid s _ {u} , r\right) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} \left(\gamma_ {j} \mid s _ {u} , r, \beta_ {k}\right) P \left(o \mid \gamma_ {j}\right)} \tag {9}
$$
The KaRR score is calculated as the geometric mean of the results obtained from Eq. (8) and (9), and its detailed implementation is shown in Appendix A.
We implement KaRR to examine knowledge in 14 LLMs and analyze its correlation with human assessment on model knowledge. Compared to large language models, knowledge graphs store a vast amount of structured information explicitly, which is ensured to be correct through manual construction and verification. Therefore, we implement KaRR based on existing large knowledge graphs, including T-REx [Elsahar et al., 2018] and Wikidata [Vrandecic and Krötzsch, 2014].
4.1 Data and Settings
In the following paragraphs, we provide details on the data, models and settings used in our experiments, with additional information provided in Appendix B and C.
**Symbolic Facts** We utilize T-REx knowledge graph [Elsahar et al., 2018] as our primary source of symbolic knowledge. T-REx comprises 11 million triples that are aligned with 3.09 million Wikipedia abstracts; its quality is validated by extensive crowdsourcing evaluation. In our main experiments, we consider all 600 English relations available in T-REx and sample a maximum of 20 facts per relation, resulting in a total of 10,691 facts for knowledge assessment.
Method	Subj. Alias	Obj. Alias	Rel. Alias	Rel. Cvg.
LAMA@1	X	X	X	6.83%
LAMA@10	X	X	X	6.83%
ParaRel	X	X	✓	6.33%
KaRR	✓	✓	✓	100%
(a) Basic information.
Method	Recall	Kendall's τ	p-value
LAMA@1	83.25%	0.17	0.10
LAMA@10	65.81%	0.08	0.23
ParaRel	69.15%	0.22	0.02
K-Prompts	78.00 %	0.32	0.03
KaRR	95.18%	0.43	0.03
(b) Human evaluation results.
Table 1: Basic information and human evaluation results. The abbreviations Subj., Rel., Obj., and Rel. Cvg. represent Subject, Relation, Object, and the coverage of English relations in T-REx, respectively. The methods evaluated through human evaluation are assessed using the same set of 410 randomly sampled facts. K-Prompts is implemented using shared basic information with KaRR.
Multiple Entity Aliases For the text forms of subjects and objects involved in calculating  $\mathrm{KaRR_s}$  and  $\mathrm{KaRR_r}$ , we search the entity aliases from Wikidata with Wikidata Integrator<sup>3</sup>. Ultimately, we obtain 1,349,474 aliases for 968,426 subjects and 368,511 aliases for 207,985 objects.
Multiple Relation Templates Previously, Elazar et al. [2021] manually wrote paraphrased relation templates for 41 relations, covering  $6\%$  of the relation classes in T-REx. To expand the coverage and diversity of templates for  $\mathbf{R}$ , we incorporated placeholders “[X]” and “[Y]” into the relation aliases in Wikidata. Subsequently, we utilized a fine-tuned Flan-T5 model<sup>4</sup> for grammar correction to generate corresponding template candidates. We then filtered the templates with missing subjects or objects, and the remaining template candidates were all validated and corrected manually by two human annotators. Finally, we obtained 3,488 templates for 600 relations, with an average of 5.82 paraphrased templates per relation, achieving full coverage of the English relation classes in T-REx.
Overall KaRR Score The KaRR score assigned to each fact is a continuous value. If the KaRR score exceeds a predefined threshold, the fact is considered consistently known by the LLM. When applying our method to a set of facts for the general knowledge reliability of LLMs, the overall KaRR score is calculated as the proportion of facts with a KaRR score exceeding the threshold.