SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As the statistical information shown in Tab. 1 (a), our method exhibits a broader coverage of relations and entity aliases compared to previous studies. To implement KaRR, we initially sample the knowledge triplet from T-REx, which comprises the symbolic subject  $s$ , relation  $r$ , and object  $o$ . By leveraging entity aliases and relation templates that we have devised, we can generate multiple expressions for each element in the KaRR metric. For instance,  $\alpha$  is implemented as the set of all searched subject aliases from our aliases dataset, and  $\beta$  represents the combination of all possible relation templates and subject aliases (by replacing the placeholder with the subject aliases).
We explore the following models for knowledge assessment: (1) traditional size pretrained models  $(< 5\mathrm{B})$ , including GPT [Radford et al., 2018], XLNet [Yang et al., 2019], GPT2-XL [Radford et al., 2019], GPT-NEO [Black et al., 2021], T5-large and T5-3B [Raffel et al., 2020] and Phi-1.5 [Li et al., 2023]; (2) medium size LLMs  $(\geq 5\mathrm{B}, < 50\mathrm{B})$ , including GLM [Du et al., 2022], Falcon [Penedo et al., 2023], Dolly, Moss, LLaMA [Touvron et al., 2023a] and its variants Alpaca [Taori et al., 2023], Vicuna [Chiang et al., 2023] and WizardLM [Xu et al., 2023]; (3) large LLMs  $(\geq 50\mathrm{B})$ , including the 65B LLaMA [Touvron et al., 2023a], the 65B LLaMA2 [Touvron et al., 2023b] and the 175B OPT [Zhang et al., 2022].
4.3 Results of Knowledge Assessment
We present the results of our statistical knowledge assessment on various LLMs in Tab. 4 and more detailed results in Appendix D. Among the small and the medium-sized LLMs, most models' KaRR scores are between 3 to 13, indicating that they often struggle to generate factually accurate sentences consistently. However, it is surprising to note that the 2.65B GPT-NEO outperforms the T5-3B with a 3.92 KaRR score difference. We hypothesize that this difference in performance may be attributed to the pretraining corpora. Specifically, GPT-NEO (trained on the Pile dataset [Gao et al., 2021])
<sup>3</sup>https://github.com/SuLab/WikidataIntegrator
4https://huggingface.co/pszemraj/flan-t5-large-grammar-synthesis
<sup>5</sup>https://github.com/databrickslabs/dolly
$^{6}$ https://github.com/OpenLMLab/MOSS
Model	Size	KaRR Score	Model	Size	KaRR Score
GPT	0.12B	9.57	GLM	10B	5.59
XLNet	0.12B	5.86	Dolly	12B	15.60
T5-large	0.74B	3.22	LLaMA	13B	13.86
Phi-1.5	1.3B	10.58	Alpaca	13B	8.24
GPT2-XL	1.56B	12.27	Vicuna	13B	19.50
GPT-NEO	2.65B	13.44	WizardLM	13B	16.90
T5-3B	3B	9.52	Moss	16B	11.20
Falcon	7B	7.97	LLaMA	65B	14.56
BLOOM	7B	7.72	LLaMA2	65B	19.71
LLaMA	7B	12.37	OPT	175B	23.06
Table 2: Statistical knowledge assessment results on 20 LLMs. KaRR scores indicate that fine-tuning on instruction-following data (Alpaca) jeopardizes the reliability of knowledge-correct generation of the original LLaMA, whereas utilizing data from a more knowledgeable model enhances model knowledge (Vicuna). Additionally, the same backbone architecture adheres to the scaling law.
primarily utilizes high-quality data, while T5 (pretrained on a combination of corpora) uses a more varied set of data sources. For larger LLMs, a comparison between the original LLaMA and the Alpaca model reveals that instruction-tuning might influence the model's ability to generate consistent and correct knowledge. Compared to the KaRR score of the original LLaMA, the KaRR score of Vicuna indicates that fine-tuning LLM on data collected from a more knowledgeable model could augment its knowledge. This finding is not commonly observed in previous studies but is consistent with our case study presented in Appendix E.
Figure 4: Knowledge in models with the same backbone architecture follows the scaling law.
Fig. 4 demonstrates that our statistical evaluation approach supports the law of model scaling, where larger models tend to possess more factual knowledge. This observation is consistent with our intuition about the relationship between model size and knowledge representation. However, the benefits of scaling vary across models. For instance, T5 shows a significant improvement in KaRR score when scaled from T5-small to T5-3B, while GPT2 and OPT also benefit from scaling, albeit to a lesser extent. Notably, we observe that when the model size is smaller than 1B, GPT2 and OPT significantly outperform T5 in generating correct knowledge. This finding highlights the importance of selecting an appropriate model size for a given task and dataset, as larger models may not always provide the best performance on generation factuality. Overall, the results of our statistical evaluation method are consistent with our intuition, and highlight the importance of large-scale pretraining on knowledge-rich corpora for building high-performing language models.
4.4 Human Evaluation for Knowledge Assessment
To further validate our approach, we conduct the human evaluation for model knowledge assessment. Utilizing the mean score from human evaluation as the benchmark for knowledge assessment on individual facts, we examine the validity of our knowledge assessment outcomes from two perspectives: 1) the  $\tau$  [Kendall, 1938] correlation between the KaRR score on each fact and the human-annotated score on each fact and 2) the Recall for identifying knowledge that is not consistently known (facts with human average scores below 0.5) when employing the GPT2-XL model.
Human Evaluation Our human evaluation for the reliable knowledge generation ability of GPT2-XL contains two procedures: 1) Annotating: We have three annotation volunteers with NLP backgrounds. Each annotator is asked to write three prompts to probe the model knowledge from multiple aspects for each fact. They are instructed to refine the prompts based on model generations until the generations are of the same type as the target answer (but not necessarily the target entity). To ensure a fair comparison with LAMA [Petroni et al., 2019], which is only applicable to 41 high-frequency
Method	Var (↓)	Std (↓)
LAMA@1	1.90	1.37
LAMA@10	5.14	2.27
ParaRel	0.77	0.94
K-Prompts	2.34	5.47
KaRR	0.67	0.82
(a) Evaluation variance towards varied prompts.
Method	SP (↓)	ΔP (↓)
LAMA@1	3.81	0.00
LAMA@10	64.29	47.31
ParaRel	2.66	-0.51
K-Prompts	0.00	-7.54
KaRR	1.94	-14.94
(b) Spurious correlation of knowledge assessment.
Table 3: Evaluation variance and spurious correlation analysis. Compared to LAMA and ParaRel, our statistical knowledge assessment is more robust and less influenced by the spurious correlation.
relations, we restricted the relation classes to those included in LAMA. In total, we obtained 4,140 valid manually annotated prompts for 410 facts in T-REx. 2) Rating: We then asked three other annotators to rate the knowledge (0 or 1) in GPT2-XL according to the model generations. We used the average score of their ratings on all the prompts as the gold standard for knowledge assessment on each fact. Detailed annotation procedures and scoring criteria are shown in Appendix F.
Baselines As a comparison, we also calculated the correlation between three baseline methods and human evaluation results. The baseline methods, LAMA [Petroni et al., 2019] and ParaRel [Elazar et al., 2021], were designed for MLMs, we modified them for LLMs by deleting words following the objects in the templates. We implement the Consistent-Acc score [Elazar et al., 2021] for ParaRel, which combines knowledge and consistency during evaluation. LAMA@1 and LAMA@10 refer to the results obtained by checking whether the top-1 or top-10 generations contain the object, respectively. Furthermore, we propose a K-Prompts baseline that computes the average probability of k randomly sampled prompts. It can be considered an adapted version of LPAQA [Shin et al., 2020] for LLMs, utilizing a collection of ensembled high-quality prompts identical to those used in KaRR. Here, we adopt K-Prompts as a simple baseline to evaluate the knowledge assessment in the absence of latent variables. All baseline methods are executed on the same set of facts as KaRR. LAMA and ParaRel are implemented as per the original papers [Petroni et al., 2019, Elazar et al., 2021], and K-Prompts is implemented using shared aliases and templates with KaRR. We set 22 as the threshold of KaRR, which is chosen by aligning the proportion of GPT2-XL known facts distinguished by humans on a sampled set of 200 facts. Similarly, the threshold for K-Prompts has been set to 0.13, as a result of aligning with the proportion recognized by humans.
Results Tab. 1 (b) demonstrate that KaRR performs exceptionally well in knowledge detection, achieving a Recall of  $95.18\%$ . This highlights the potential of our statistical knowledge assessment to identify unreliable knowledge in LLMs, which is essential for downstream knowledge updating [De Cao et al., 2021]. Furthermore, the evaluation results of KaRR show a strong correlation (Kendall's 0.43) with human evaluation results. However, averaging the results of paraphrased prompts (ParaRel and K-Prompts) is not effective due to varying prompt frequencies and diverse impacts of different prompts on the model's probability of generating an "object". Since human annotation for model knowledge assessment is demanding and time-consuming, our statistical evaluation method provides a fast and accurate solution for the automatic knowledge assessment of LLMs.
5 Variance toward Prompts
Since the aim of the evaluation is to reflect the correctness of an LLM's knowledge towards various prompts, the evaluation result should be stable and not easily affected by the text forms used. In this section, we investigate the evaluation variance of our method. In contrast to ParaRel [Elazar et al., 2021], which studies the consistency of models under paraphrased inputs, we investigate the variance of the knowledge assessment method. Specifically, we evaluate the evaluation variance, which measures how much the same evaluation results vary when different prompts are used.
Following the same procedures as described in Sec. 4.4, we sampled 4,100 facts and employed the paraphrased relation templates from ParaRel [Elazar et al., 2021] to replace the relation templates in the baseline methods and KaRR, resulting in three distinct relation templates for each fact. We evaluated the effectiveness of the different evaluation approaches on each of the three paraphrased relation templates separately, the model is GPT2-XL. Additionally, we measured the evaluation variance by calculating the variance and standard deviation of the evaluation scores.
Tab. 3 (a) illustrates that the results of LAMA and K-Prompts exhibit significant variations when the probing prompt is changed to a paraphrased one, despite the overall prompt similarity in ParaRel
Figure 5: Ablation study on the sampling parameter K. With the increase in K, the variability in knowledge assessment decreases, while its correlation with human evaluation improves.
being 0.96. In contrast, our statistical knowledge assessment approach demonstrates robustness to prompt variation, with a 0.67 score variance and a 0.82 standard deviation.
6 Spurious Correlation in Knowledge Assessment
Both LAMA and KaRR use text as a proxy to estimate unobservable knowledge. However, spurious correlations can lead to inflated probing results where LLMs "guess" the probing answer based on these correlations. For example, when probed with the relation template "[X]'s birthplace is [Y]", LLMs assign "America" the highest probability, regardless of the subject entity [Zhao et al., 2021]. To address this issue, we investigate spurious correlations in knowledge assessment, particularly focusing on the correlation between relation templates and high-frequency objects.
We synthesize false facts by replacing objects with high-frequency but factually incorrect ones. They are non-existent knowledge in LLMs as they are not part of the pretraining corpus. Ideally, the scores on synthetic facts should be 0 if the assessment is not influenced by spurious correlations. We obtain high-frequency objects by identifying the top-5 GPT2-XL predictions on 3 ParaRel relation templates without subjects, yielding 14 relations with valid high-frequency objects out of 41 relations. For each relation, we randomly sample 100 facts and replace the original object with the high-frequency object (if the original object is not the high-frequency object). We employ two metrics: Spurious Positive (SP) and Positive Gap  $(\Delta P)$ . SP quantifies the proportion of false synthetic facts that are incorrectly recognized as known by the LLMs, and  $\Delta P$  is calculated by SP minus model's true positive rate on the actual facts. Ideally, SP and  $(\Delta P)$  should be close to 0 and  $(\Delta P)$  should be negative; otherwise, it indicates a significant bias in the evaluation result due to spurious correlations.
Tab. 3 (b) presents results on GPT2-XL. LAMA scores are highly correlated to the prompt, leading to the high  $\Delta P$  and SP. For example,  $3.81\%$  and  $64.29\%$  false facts are estimated as learned facts by LAMA@1 and LAMA@10, respectively. In contrast, K-Prompts and KaRR are less influenced by spurious correlations, with an SP score lower than  $2\%$  and a negative  $\Delta P$ . By assessing the model knowledge from various perspectives, the ultimate scores of KaRR indicate the comprehensive knowledge mastery of LLMs. Therefore, our results are less influenced by co-occurrence shortcuts.
In this section, we discuss the causal explanation of KaRR and explore the effects of the sampling parameter, as well as the influence of model size.
Causal Explanation for KaRR From a causal perspective, our problem can be defined as: "given a subject  $s$ , the causal effect of given the relation  $r$  or not for the LLM generating the object  $o$ ". Assuming there are no confounders, given a fixed  $s$ , the treatment (T) is whether a specific  $r$  is provided or not, the outcome (Y) is the probability of the model generating  $o$ , then the Average Treatment Effect (ATE) [Holland, 1986] is defined as:
$$
\mathbb {E} [ Y (1) ] - \mathbb {E} [ Y (0) ] = \mathbb {E} [ Y \mid T = 1 ] - \mathbb {E} [ Y \mid T = 0 ] = P (o | s, r) - \mathbb {E} _ {\mathbf {R}} \left[ P (o | s, \mathbf {R}) \right] \tag {10}
$$
Note that in ATE, the minuend and subtrahend correspond to the denominators and numerator of  $\mathrm{KaRR}_r$  in Eq. (1), respectively. Assuming no confounders are present, the only distinction lies in the mathematical operation utilized (subtraction or division). Both subtraction and division can be used to compare the difference between the exposed group and the unexposed group, with the only distinction being the difference in the threshold for the resulting value. As a result, ATE helps clarify and validate KaRR from a causal perspective.
Figure 6: KaRR scores on different relations when scaling up OPT from 350M to 2.7B and 175B.
Influence of Sampling Parameter When calculating KaRR, the sampling number K makes a trade-off between the evaluation accuracy and efficiency. We perform an ablation study on the results of KaRR implemented with different K. Following the same settings of Sec. 5 and Sec. 4.4, we analyze the evaluation variance and evaluation accuracy of the overall KaRR scores of GPT2-XL. Fig. 5 demonstrates that increasing K decreases variance in knowledge assessment, but the impact is minor since evaluation variance is already low. The sampling parameter has a more significant effect on evaluation accuracy, with higher accuracy correlating with higher K values. When K is less than 4, the evaluation results are unreliable due to weak correlation with human evaluation results.
Impact of Scaling on LLM Knowledge To further study the impact of scaling up on LLM knowledge, we compare the KaRR scores of OPT-350M, OPT-2.7B, and OPT-175B on 30 best-known and 30 worst-known relations of OPT-350M. Fig. 6 shows that larger models exhibit better and more consistent knowledge-correct generation ability on the relations where small models also show some knowledge-correct generation ability. Moreover, larger models surpass small models in terms of knowledge on a wider range of relations. Therefore, the scaling law of LLM knowledge is attributed to the broader and more extensive knowledge of larger LLMs.
While KaRR demonstrates promising effectiveness for LLM knowledge assessment, there are a few aspects that require further investigation. First, KaRR scores are calculated for each fact using LLM inference on various prompts, which can be time-consuming and may limit the scalability of the evaluation. Second, our assessment is currently limited to scenarios where the logits of LLMs are available. This excludes many real-world applications where the models' internal representations may not be accessible or may be difficult to obtain. Third, the generality of KaRR can be further expanded. The current implementation focuses on atomic knowledge, characterized by triplets in the format of <subject, relation, object>. Nevertheless, real-world scenarios encompass a significant portion of complex knowledge, necessitating multi-hop reasoning or the comprehension of intricate relationships among multiple entities [Choudhary and Reddy, 2023]. Evaluating such knowledge in LLMs continues to be a challenging and unresolved issue. Overall, these areas for improvement emphasize the opportunities for continued research and development of knowledge assessment approaches for LLMs, taking into account the complexities of real-world applications and the diverse types of knowledge to be assessed.
We investigate the consistent knowledge-correct generation ability of LLMs and propose a statistical knowledge assessment approach, called KaRR. By implementing our approach on 20 LLMs, we carry out an extensive assessment of the reliability of factual knowledge generation in these models. Our experiments reveal that our method yields a high correlation with human evaluation, and addresses the issues of variance and spurious correlation during the knowledge assessment of generative LLMs. Additionally, our assessment suite includes millions of entity and relation aliases, enabling large-scale knowledge evaluation. Overall, our method evaluates the reliable knowledge generation ability of LLMs, providing a robust foundation for future developments in knowledge refinement, augmentation, and beyond.
We thank all the anonymous reviewers for their constructive comments, Yuxuan Fan, Lei Li and Ce Zheng for their valuable suggestions on our data collection. Zhifang Sui is the corresponding author. This paper is supported by the National Key Research and Development Program of China 2020AAA0106700 and NSFC project U19A2065.
A KaRR Implementation
To implement KaRR, we define a binary function  $\delta_{text}(symbols)$  that takes the symbols as input and returns a value of either 0 or 1 depending on whether any alias of all the symbols appears in the text. By applying this function, we can transform  $P(\beta_k|s,r)$  and  $P(\alpha_i|s)$  into a format that is compatible with the model probability.
$$
P \left(\beta_ {k} \mid s, r\right) = \frac {P \left(\beta_ {k} , s , r\right)}{P (s , r)} = \frac {P _ {\mathcal {M}} \left(\beta_ {k}\right) \cdot \delta_ {\beta_ {k}} (s , r)}{P (s , r)} \tag {11}
$$
$$
P \left(\alpha_ {i} | s\right) = \frac {P \left(\alpha_ {i} , s\right)}{P (s)} = \frac {P _ {\mathcal {M}} \left(\alpha_ {i}\right) \cdot \delta_ {\alpha_ {i}} (s)}{P (s)} \tag {12}
$$
If the aliases  $\beta_{k}$  contain any alias of both  $s$  and  $r$ , then  $P_{\mathcal{M}}(\gamma_j|s,r,\beta_k)$  is equal to  $P_{\mathcal{M}}(\gamma_j|\beta_k)$ . Otherwise,  $P_{\mathcal{M}}(\gamma_j|s,r,\beta_k)$  is zero. Similarly, if the aliases  $\alpha_{i}$  contain any alias of  $s$ , then  $P_{\mathcal{M}}(\gamma_j|s,\alpha_i)$  is equal to  $P_{\mathcal{M}}(\gamma_j|\alpha_i)$ ; otherwise,  $P_{\mathcal{M}}(\gamma_j|s,\alpha_i)$  is zero.
Furthermore, the probabilities  $P(o|\gamma_j)$  and  $P(o|s,\alpha_i,\gamma_j)$  are both equal to  $\delta_{\gamma_j}(o)$ , where  $\delta_{\gamma_j}$  is the binary function that checks whether any alias of the object  $o$  appears in the text  $\gamma_j$ . By substituting the transformed formulas into the equation Eq. (9), we obtain an implementation of KaRR:
$$
\mathrm {K a R R} _ {r} (s, r, o) = \frac {P (s) \sum_ {k = 1} ^ {| \beta |} P _ {\mathcal {M}} (\beta_ {k}) \cdot \delta_ {\beta_ {k}} (s , r) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} \left(\gamma_ {j} \mid \beta_ {k}\right) \delta_ {\gamma_ {j}} (o)}{P (s , r) \sum_ {i = 1} ^ {| \alpha |} P _ {\mathcal {M}} \left(\alpha_ {i}\right) \cdot \delta_ {\alpha_ {i}} (s) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} \left(\gamma_ {j} \mid \alpha_ {i}\right) \delta_ {\gamma_ {j}} (o)} \tag {13}
$$
Based on the graphical model depicted above,  $\mathbf{S}$  and  $\mathbf{R}$  are conditionally independent when  $\mathbf{O}$  is unknown. Consequently, we can simplify the expression  $\frac{P(s)}{P(s,r)}$  to  $\frac{1}{P(r)}$ . Therefore,
$$
\mathrm {K a R R} _ {r} (s, r, o) = \frac {1}{P (r)} \cdot \frac {\sum_ {k = 1} ^ {| \beta |} P _ {\mathcal {M}} \left(\beta_ {k}\right) \cdot \delta_ {\beta_ {k}} (s , r) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} \left(\gamma_ {j} \mid \beta_ {k}\right) \delta_ {\gamma_ {j}} (o)}{\sum_ {i = 1} ^ {| \alpha |} P _ {\mathcal {M}} \left(\alpha_ {i}\right) \cdot \delta_ {\alpha_ {i}} (s) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} \left(\gamma_ {j} \mid \alpha_ {i}\right) \delta_ {\gamma_ {j}} (o)} \tag {14}
$$
The calculation of subject replacement is analogous. When  $\mathbf{O}$  is unknown,  $\frac{1}{P(s_u|r)\cdot P(s,r)\cdot\frac{1}{P(s_u,r)}}$  equals to  $\frac{1}{P(s)}$ , then we have,
$$
\begin{array}{l} \mathrm {K a R R} _ {s} (s, r, o) = \frac {\sum_ {k = 1} ^ {| \beta |} P _ {\mathcal {M}} (\beta_ {k}) \cdot \delta_ {\beta_ {k}} (s , r) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} (\gamma_ {j} | \beta_ {k}) \delta_ {\gamma_ {j}} (o)}{\sum_ {u = 1} ^ {| s _ {u} |} P (s _ {u} | r) \cdot P (s , r) \sum_ {k = 1} ^ {| \beta |} P _ {\mathcal {M}} (\beta_ {k}) \frac {1}{P (s _ {u} , r)} \cdot \delta_ {\beta_ {k}} (s _ {u} , r) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} (\gamma_ {j} | \beta_ {k}) \delta_ {\gamma_ {j}} (o)} \\ = \frac {1}{P (s)} \cdot \frac {\sum_ {k = 1} ^ {| \beta |} P _ {\mathcal {M}} \left(\beta_ {k}\right) \cdot \delta_ {\beta_ {k}} (s , r) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} \left(\gamma_ {j} \mid \beta_ {k}\right) \delta_ {\gamma_ {j}} (o)}{\sum_ {u = 1} ^ {| s _ {u} |} \sum_ {k = 1} ^ {| \beta |} P _ {\mathcal {M}} \left(\beta_ {k}\right) \cdot \delta_ {\beta_ {k}} \left(s _ {u} , r\right) \sum_ {j = 1} ^ {| \gamma |} P _ {\mathcal {M}} \left(\gamma_ {j} \mid \beta_ {k}\right) \delta_ {\gamma_ {j}} (o)} \tag {15} \\ \end{array}
$$
In practical scenarios, computing all possible values of  $\alpha$  or  $\beta$  for the KaRR metric can be prohibitively expensive. To address this challenge, we can make the assumption that the symbols S and R follow a unified distribution, and then use Markov Chain Monte Carlo (MCMC) sampling to estimate the denominators by sampling K subjects or objects. This results in  $\frac{1}{P(s_u)}$  or  $\frac{1}{P(r_u)}$  being added to the denominators, offset by the  $\frac{1}{P(r)}$  and  $\frac{1}{P(s)}$  terms in Eq. (14) and (15). By employing this approach, we are able to improve the accuracy of the KaRR metric within a limited sampling budget and derive a new score based on the sampled subset. Our default sampling parameter, K, is set to 4.
The KaRR score is calculated as the geometric mean of the results obtained from Eq. (14) and (15).
In this section, we provide further information on the experimental details.
B.1 Settings for Different LLMs
In this paper, we load the models (except the OPT-175B) from the Huggingface Models and implement the OPT-175B according to Metaseq. The specific version of Vicuna employed in this paper is lmsys/vicuna-7b-delta-v1.1.
B.2 OOV Objects