SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We collect a vast range of aliases for objects, certain aliases may contain out-of-vocabulary (OOV) words. Consequently, the model generation probability for these aliases cannot be computed. To address this issue, we follow the approach suggested in LAMA [Petroni et al., 2019] and exclude OOV object aliases when implementing KaRR on each LLM.
B.3 Scoring Details
According to the KaRR implementation above, we can obtain the KaRR score on each fact. If the KaRR is higher than a threshold, the fact is regarded as a consistently known one by the LLM; otherwise not. When implementing the statistical framework on a large set of facts for the general knowledge correctness of LLMs, we get the proportion of known facts over the 10,691 symbolic facts as the (KaRR higher than the threshold) as the overall KaRR score. We set 22 as the threshold for KaRR, and the threshold is chosen by aligning the proportion of GPT2-XL known facts distinguished by humans on a sampled set of facts. Also as a result of alignment with human-recognized proportion, the threshold for the K-Prompts baseline implemented in this paper is set to 0.13.
C Data Distribution
In this section, we provide detailed information on alias distribution and relation distribution.
C.1 Alias Distribution
Type	Top-k	Avg. Aliases	Proportion of Related Facts
subject	50	7.62	12.10%
100	6.73	16.13%
500	4.68	28.34%
1000	3.70	33.70%
object	50	5.06	39.84%
100	4.79	47.35%
500	3.80	65.44%
1000	3.48	71.93%
Table 4: Average number of aliases and proportion of related facts for top-k frequent entities.
In Tab. 4, we present the average number of aliases and the proportion of related facts for the top-k frequent entities. The most frequent entities typically possess more than five aliases, related to a large proportion of facts. This long-tailed pattern is consistent with real-world alias distribution, where a substantial portion of entities have only one alias.
C.2 Relation Distribution
As shown in Tab. 5, the distribution of relations in our dataset exhibits a long-tail phenomenon, which is consistent with the distribution of relations in real-world knowledge and the distribution of relations within knowledge graphs [Elsahar et al., 2018].
<sup>7</sup>https://huggingface.co/models
8https://github.com/facebookresearch/metaseq/tree/main
Top-k	Avg. Aliases	# Related Facts	Proportion of Related Facts
50	6.96	12,638,356	90.05%
100	5.97	13,538,967	96.47%
150	5.54	13,802,467	98.35%
200	5.22	13,921,742	99.20%
250	4.84	13,977,899	99.60%
Table 5: Distributions of top-k relations.
Model	Model Size	KaRR Score	KaRRs Score	KaRRr Score	Best-known Relations	Worst-known Relations
GPT	0.12B	9.57	4.30	31.9	P2822, P1537, P3095	P1427, P1951, P3091
XLNet	0.12B	5.86	0.25	37.78	P1537, P1264, P2922	P664, P31, P1951
T5-large	0.74B	3.22	4.61	6.82	P872, P560, P2860	P664, P1951, P3091
Phi-1.5	1.3B	10.58	6.30	36.67	P3403, P538, P2500	P1542, P184, P926
GPT2-XL	1.56B	12.27	12.12	30.38	P2922, P2499, P397	P3091, P2159, P210
GPT-NEO	2.65B	13.44	12.97	29.15	P1537, P2499, P3403	P210, P2746, P681
T5-3B	3B	9.52	3.59	23.04	P3095, P3033, P1537	P1427, P674, P1951
Falcon	7B	7.97	10.44	23.73	P3403, P1376, P1165	P805, P1542, P184
BLOOM	7B	7.72	6.58	33.42	P3403, P2922, P2289	P263, P1542, P184
LLaMA	7B	12.37	5.30	44.67	P1264, P3403, P3095	P1542, P926, P3275
GLM	10B	5.59	5.12	24.05	P2922, P2499, P397	P3091, P681, P210
Dolly	12B	15.60	11.35	39.88	P1264, P3403, P2822	P1542, P926, P3275
LLaMA	13B	13.86	6.02	45.36	P1264, P3403, P3095	P1542, P926, P3275
Alpaca	13B	8.24	3.77	31.5	P770, P1906, P2974	P1427, P674, P1951
Vicuna	13B	19.50	8.34	49.83	P410, P974, P2159	P263, P1542, P1906
WizardLM	13B	16.90	1.13	50.24	P2839, P521, P3085	P926, P1427, P922
Moss	16B	11.20	12.83	27.01	P1264, P3403, P1537	P1542, P184, P926
LLaMA	65B	14.56	6.98	44.80	P3261, P1576, P2551	P263, P1542, P1906
LLaMA2	65B	19.71	10.13	45.78	P2839, P974, P3261	P1542, P1906, P926
OPT	175B	23.06	19.89	42.31	P3403, P1264, P2922	P2321, P2746, P538
Table 6: Detailed results of statistical knowledge assessment on 20 LLMs.
Tab. 6 provides detailed results for statistical knowledge assessment on LLMs.
E Case Study on LLM Generation
Tab. 7 provides a case study on LLM generation.
F Human Annotation Specification
In this section, we provide further details on our human annotation.
F.1 Main Procedures
In our Human Annotation Assessment (HKA), there are two primary procedures: generation-guided prompting and knowledge mastery scoring. Each procedure is carried out by a group of three volunteer annotators who have backgrounds in NLP.
F.1.1 Generation-Guided Prompting
In our HKA, a critical procedure is generation-guided prompting. That is, annotators continually modify their prompts according to both manual and automatic feedback from the model generations.
Model	Question	Generation
LLaMA	The occupation of Swan of Avon was:	The Swan of Avon was a British cargo ship...
William Shakespeare worked as a	playwright and actor in London
Joseph Armstrong worked for:	The Daily Telegraph (London)
The companies where Joseph Armstrong worked in are:	1. The Armstrong Company, 1900-1901
2. The Armstrong Company, 1901-1902...
Alpaca	The occupation of Swan of Avon was:	that of a boatman...
William Shakespeare worked as a	a playwright, poet, and actor...
Joseph Armstrong worked for:	Joseph Armstrong is a former employee of the following companies:
Joseph Armstrong currently works for: ...
The companies where Joseph Armstrong worked in are:	Joseph Armstrong's average salary was $100,000...
Vicuna	The occupation of Swan of Avon was:	The Swan of Avon was a paddle steamer...
William Shakespeare worked as a	a playwright and actor...
Joseph Armstrong worked for:	Joseph Armstrong was a member of the following organisations: ...
The companies where Joseph Armstrong worked in are:	* Armstrong, Whitworth & Co....
Table 7: The generation cases of the original 7B LLaMA model and its instruction-tuned variants. Alpaca and Vicuna. For each model, we provide the same Knowledge-related prompts.
This approach differs from previous scenarios where annotators merely wrote relation templates for each fact to create prompts [Petroni et al., 2019]. Instead, we ask each annotator to write prompts from various perspectives, including prompting for entities, relations, or the entire fact. Moreover, we perform manual answer-type checking throughout the annotation process. Adjusting prompts according to the model's generation is essential for accurately assessing the model's knowledge. For example, because of the prompt template "[X] is a [Y]" in LAMA,  $100\%$  of the top-1 prediction for facts of relation 'P31' is a definite article or a predicate. This induces the underestimation of model knowledge while implementing LAMA@1. And for 'P19', although the prompt in LAMA, "[X] was born in [Y]." is a plausible one, most of the top-10 generations are dates or months. This indicates the significance of model-generation-guided prompt refining.
Manual Answer-type Checking To ensure that the prompt accurately prompts the model to generate the relevant text of the current fact in the next word, instead of freely describing other content, we conduct manual answer-type checking. Only when the model generations are mostly of the same type of target answers, the prompts are deemed valid ones; otherwise, they will be further refined to a valid one.
Automatic Diversity Checking To explore the model generations towards different prompts, we implement an automatic diversity check for the prompts. Regarding the other valid prompts as references and the new prompt as the candidate, we calculate the BLEU score [Papineni et al., 2002] and set a similarity threshold to 0.8 to filter similar prompts. Besides, annotates are asked to not only write prompts to probe the object entity of each fact but also write prompts to probe the subject, the relation, or the whole fact.
F.1.2 Knowledge Mastery Scoring
In the knowledge mastery scoring procedure, we enlist an additional three annotators to evaluate the model's knowledge. Provided with a prompt and the corresponding GPT2-XL model generation (top-1 generation using beam search), annotators are asked to score the model's knowledge mastery on a scale between 0 (unknown) and 1 (known). Ultimately, the scores for prompts related to the
same fact are averaged to determine the model's knowledge mastery score for that particular piece of factual knowledge.
F.2 Annotation Specification
As it introduced above, our annotation mainly contains two parts: A. prompting, B. scoring. All the annotators are volunteer students majoring in Computational Linguistics in China. We told them the annotation intention and explained how the data would be used.
F.2.1 Prompting
Given a triplet fact, annotators are instructed to do their best to determine whether the knowledge is stored in the LLM and compose three prompts (from various perspectives) for each fact to probe it.
During the prompting process, annotators should create prompts that not only probe the object entity but also examine the subject, the relation, or the entire fact. For instance, for the fact <Obama, birthplace, Hawai>, possible prompts could include:
- One U.S. president was born in Hawaii, and his name was ____ (This prompt probes for the subject)
- Obama was born in (this prompt probes for the object).
- Hawaii is Obama's ______ (this prompt probes for the relation).
- The state where Barack Obama born is (this prompt probes for the object).
- President Obama was born in Honolulu, ______ (this prompt probes for the object).
To ensure the diversity of prompts, we employ the BLEU metric [Papineni et al., 2002] for automatic diversity checks. Specifically, we consider all valid prompts as references and the new prompt as the candidate, setting a threshold of 0.5 to filter out prompts that are similar to existing ones in terms of surface forms.
Annotators assess whether a given prompt effectively elicits knowledge based on the model-generated results. If the generated content is primarily related to the target aspect, the prompt is considered valid. For instance, when prompting for object entities, if the model's output consistently includes a set of entities (both correct and incorrect ones are acceptable, as long as the type is broadly relevant), the prompt is deemed valid. However, if the generated text or tokens consist of stopwords or unrelated entity types, the prompt is classified as invalid. For example, the fact  $<\text{China}, \text{capital}, \text{Beijing}>$ , The capital city of China is ____ is not a valid prompt for GPT2 as the generations are: "inching", "home to", "is now a", "is described as" or "is the birthplace". While The capital city of China is called ____ is a valid prompt as the model generations are city names like: "Shanghai", "Beijing" or "Hangzhou".
F.2.2 Scoring
The scoring criteria are as follows:
- "Invalid": Factually unrelated generations or improper prompts.
- 1: The output produced during the greedy decoding process contains at least one alias of the correct answer, without any factually incorrect answers.
- 0: The generated content does not include any related entities or relations, nor are there any correct answers expressed in paraphrased words or phrases. In other words, the correct answer is not present in any of the generated responses.
F.3 Annotation Results
We ultimately obtained 4,140 manually annotated prompts for 410 valid facts. The overall sentence similarity among the prompts for the same fact is 0.78, which demonstrates good diversity compared to the 0.95 overall similarity of prompts in ParaRel [Elazar et al., 2021]. The Kappa score between human raters is 0.4.
While HKA provides relatively trustworthy and reliable results for assessing model knowledge correctness, it places high demands on human annotators and is time-consuming to implement for
each fact in the model-specific annotation process. In our experience, annotating 100 facts on GPT2-XL takes an average of 4 hours per annotator. Consequently, HKA is more suitable as a solution for obtaining gold-standard results for knowledge assessment on a smaller scale. Kendall's  $\tau$  correlation between the results of a knowledge-assessing approach and HKA reflects the evaluation's accuracy.