Do PLMs Know and Understand Ontological Knowledge?
Year: 2023

Abstract:
Ontological knowledge, which comprises classes and properties and their relationships, is integral to world knowledge. It is significant to explore whether Pretrained Language Models (PLMs) know and understand such knowledge. However, existing PLM-probing studies focus mainly on factual knowledge, lacking a systematic probing of ontological knowledge. In this paper, we focus on probing whether PLMs store ontological knowledge and have a semantic understanding of the knowledge rather than rote memorization of the surface form. To probe whether PLMs know ontological knowledge, we investigate how well PLMs memorize: (1) types of entities; (2) hierarchical relationships among classes and properties, e.g., Person is a subclass of Animal and Member of Sports Team is a subproperty of Member of ; (3) domain and range constraints of properties, e.g., the subject of Member of Sports Team should be a Person and the object should be a Sports Team. To further probe whether PLMs truly understand ontological knowledge beyond memorization, we comprehensively study whether they can reliably perform logical reasoning with given knowledge according to ontological entailment rules. Our probing results show that PLMs can memorize certain ontological knowledge and utilize implicit knowledge in reasoning. However, both the memorizing and reasoning performances are less than perfect, indicating incomplete knowledge and understanding.

heading: 1 Introduction
text: Pretrained Language Models (PLMs) have orchestrated impressive progress in NLP across a wide variety of downstream tasks, including knowledge-intensive tasks. Previous works propose that PLMs are capable of encoding a significant amount of knowledge from the pretraining corpora (AlKhamissi et al., 2022), and determine to explore the kinds of knowledge within PLMs.
∗Yong Jiang and Kewei Tu are corresponding authors.
Existing probing works mainly focus on factual knowledge associated with instances (Petroni et al., 2019; Jiang et al., 2020; Safavi and Koutra, 2021). Meanwhile, although classes (concepts) have raised some research interest (Bhatia and Richie, 2020; Peng et al., 2022; Lin and Ng, 2022), there is no systematic study of ontological knowledge.
Ontological knowledge models the world with a set of classes and properties and the relationships that hold between them (Nilsson, 2006; Kumar et al., 2019). It plays a vital role in many NLP tasks such as question answering by being injected into (Goodwin and Demner-Fushman, 2020) or embedded outside deep neural networks (Wang et al.,
3080
2017). Therefore, it is essential to explore whether PLMs can encode ontological knowledge and have a semantic understanding of the knowledge rather than rote memorizing its surface form.
In this paper, we first probe PLM’s memorization of ontological knowledge. Specifically, as shown in Figure 1(a), we construct memorization tests about (1) Types of entities. Entities can be categorized into classes, as Lionel Messi is a Person and Argentina National Football Team is a Sports Team. (2) Hierarchical relationships between classes, e.g., Person is a subclass of Animal. (3) Hierarchical relationships between properties, e.g., Member of Sports Team is a subproperty of Member of. (4) Domain constraints of properties. It specifies information about the subjects to which a property applies. For example, the subject of Member of Sports Team should be an instance of Person. (5) Range constraints of properties. Similar to domain, range specifies information about the object of a property, such as the object of Member of Sports Team should be an instance of Sports Team. Experiments prove that PLMs store a certain amount of ontological knowledge.
To further examine whether PLMs understand ontological knowledge, we investigate if PLMs can correctly perform logical reasoning that requires ontological knowledge. Illustrated in Figure 1(b), given the fact triple (Lionel Messi, Member of Sports Team, Argentina National Football Team) along with property constraints, we can perform type inferences to conclude that Lionel Messi is a Person, and Argentina National Football Team is a Sports Team. We comprehensively investigate the reasoning capability of PLMs over ontological knowledge following six entailment rules. Experiments show that PLMs can apply implicit ontological knowledge to draw conclusions through reasoning, but the accuracy of their reasoning falls short of perfection. This observation suggests that PLMs possess a limited understanding of ontological knowledge.
In summary, we systematically probe whether PLMs know and understand ontological knowledge. Our main contributions can be summarized as follows: (1) We construct a dataset that evaluates the ability of PLMs to memorize ontological knowledge and their capacity to draw inferences based on ontological entailment rules. (2) We comprehensively probe the reasoning ability of PLMs by carefully classifying how ontological knowledge
is given as a premise. (3) We find that PLMs can memorize certain ontological knowledge but have a limited understanding. We anticipate that our work will facilitate more in-depth research on ontological knowledge probing with PLMs. The code and dataset are released at https://github.com/ vickywu1022/OntoProbe-PLMs.

heading: 2 Benchmark Construction
text: In this section, we present our methodology for ontology construction and the process of generating memorizing and reasoning tasks based on the ontology for our probing analysis.

heading: 2.1 Ontology Building
text: Class We use DBpedia (Auer et al., 2007) to obtain classes and their instances. Specifically, we first retrieve all 783 classes in DBpedia, then use SPARQL (hommeaux, 2011) to query their instances using the type relation and superclasses using the subclass-of relation. We sample 20 instances for each class.
Property Properties are collected based on DBpedia and Wikidata (Vrandečić and Krötzsch, 2014) using the following pipeline: (1) Obtain properties from Wikidata and use subproperty of (P1647) in Wikidata to find their superproperties. (2) Query the domain and range constraints of the properties using property constraint (P2302) in Wikidata. (3) Align the Wikidata properties with DBpedia properties by equivalent property (P1628). (4) Query the domain and range constraints of the properties in DBpedia. (5) Cleanse the collected constraints using the above-collected class set as vocabulary. We choose 50 properties with sensible domain, range and superproperties.

heading: 2.2 Construction of Memorizing Task
text: The memorizing task consists of five subtasks, each probing the memorization of an ontological relationship: (1) TP: types of a given instance, (2) SCO: superclasses of a given class, (3) SPO: superproperties of a given property, (4) DM: domain constraint on a given property, and (5) RG: range constraint on a given property. Every subtask is formulated as a cloze-completion problem, as shown in Figure 1(b). Multiple correct answers exist for TP, SCO, and SPO, which form a chain of classes or properties. There is only one correct answer for DM and RG, as it is not sound to declare an expanded restriction on a property. For instance,
Task Ontological Rel. Candidate Train Dev Test
Animal is too broad as the domain constraint of the property Member of Sports Team (P54), hence applying Person as the domain.
We construct the dataset for each subtask using the ontology built in Sec. 2.1 and reserve 10 samples for training and 10 for validation to facilitate few-shot knowledge probing. The statistics of the dataset for each subtask are shown in Table 1.

heading: 2.3 Construction of Reasoning Task
text: We construct the reasoning task based on the entailment rules specified in the Resource Description Framework Schema (RDFS)1. We propose six subtasks, each probing the reasoning ability following a rule listed in Table 2. For rule rdfs2/3/7, we design a pattern for each property to be used between a pair of instances, e.g., "[X] is a player at [Y] ." for Member of Sports Team, where [X] and [Y] are the subject and object, respectively.
Each entailment rule describes a reasoning process: P1∧P2 |= H, where P1,P2 are the premises
1RDFS is an extension of RDF (Brickley and Guha, 2002; Gibbins and Shadbolt, 2009), a widely used and recognized data model. See https://www.w3.org/TR/rdf11-mt/ #rdfs-entailment for all the entailment rules.
and H is the hypothesis. Similar to the memorizing task, we formulate the reasoning task as cloze-completion by masking the hypothesis (see Figure 1(b)). Premises are also essential to the reasoning process and can be:
• Explicitly Given: The premise is explicitly included in the input of the model, and inferences are made with natural language statements.
• Implicitly Given: The premise is not explicitly given but memorized by the model as implicit knowledge. The model needs to utilize implicit knowledge to perform inferences, which relieves the effect of context and requires understanding the knowledge.
• Not Given: The premise is neither explicitly given nor memorized by the model. It serves as a baseline where the model makes no inference.
Hence, there exist 3× 3 different setups for two premises. It is a refinement of the experimental setup used by Talmor et al. (2020), which only distinguishes whether a premise is explicitly included in the input. We determine the memorization of a premise by the probing results of the memorizing task, which will be elaborated in Sec. 3.2.3.

heading: 3 Probing Methods
text: We investigate encoder-based PLMs (BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019)) that can be utilized as input encoders for various NLP tasks. Prompt is an intuitive method of our probing task as it matches the mask-filling nature
of BERT. We use OpenPrompt (Ding et al., 2022), an open-source framework for prompt learning that includes the mainstream prompt methods, to facilitate the experiments.

heading: 3.1 Probing Methods for Memorization

heading: 3.1.1 Prompt Templates
text: Manual Templates Manual prompts with human-designed templates written in discrete language phrases are widely used in zero-shot probing (Schick and Schütze, 2021) as PLMs can perform tasks without any training. Manual templates are designed for all the ontological relationships in our task, as shown in Table 3.
Soft Templates One of the disadvantages of manual prompts is that the performance can be significantly affected by perturbation to the prompt templates (Jiang et al., 2020). A common alternative is to use soft prompts that consist of learnable soft tokens (Liu et al., 2021; Li and Liang, 2021) instead of manually defined templates. The soft prompts we use for ontological relationships are also shown in Table 3. To probe using soft prompts, we tune randomly initialized soft tokens on the training set with the PLMs parameters being frozen. Detailed training setups are listed in Appendix A.

heading: 3.1.2 Candidates Scoring
text: Given a candidate c which can be tokenized into n tokens c1, c2, . . . , cn, such that ci ∈ V, i = {1, . . . , n}, n ≥ 1, where V is the vocabulary of the model, it is scored based on the log probability of predicting it in the masked prompt. We can either use n different [MASK] tokens or the same [MASK] token to obtain the log probability of each composing token ci, and then compute the
log probability of the candidate c. For simplicity, we use a single [MASK] token when illustrating our prompts.
Multiple Masks For a candidate c consisting of n tokens, we use n [MASK] tokens in the masked input, with the ith [MASK] token denoted as [MASK]i. The candidate probability can be computed by three different pooling methods: (1) mean: the average of log probabilities of composing tokens (Klein and Nabi, 2020), (2) max: the maximum log probability of all composing tokens, (3) first: the log probability of the first composing token. Formally, the score s of candidate c is computed as:
ŝi = log (p([MASK]i = ci))
s = Pooling(ŝ1, ŝ2, . . . , ŝn)
Single Mask We use one single [MASK] token to obtain an independent prediction of each token. The log probability of each composing token ci equals the log probability of recovering ci in the same [MASK], and the candidate is scored with the proposed pooling methods.
ŝi = log (p([MASK] = ci))

heading: 3.1.3 Metrics
text: We rank the candidates by their log probability scores and use the top K Recall (R@K) and Mean Reciprocal Rank (MRR) as our evaluation metrics. Since MRR only evaluates the ability to retrieve the first ground truth, we additionally take the average rank of all gold labels as the final rank when computing mean reciprocal rank to evaluate models’ ability to retrieve all the ground truths and denote
it as MRRa. Formally, MRRa is defined as:
MRRa = 1
n
n∑
i=1
1/( 1 |Gi| ∑
g∈Gi rank(g))
where n is the number of samples in the dataset and Gi is the gold label set of the ith sample.

heading: 3.2 Probing Methods for Reasoning
text: We explain how we concatenate the premises and hypothesis in the textual input, exclude the models’ memory of hypotheses and split a set of premises based on how well the knowledge they represent is memorized by the model. We follow the candidate scoring methods proposed in Sec. 3.1.2 and evaluation metrics in Sec. 3.1.3.

heading: 3.2.1 Prompt Templates
text: Apart from the prompt templates for our concerned ontological relationships introduced in Sec. 3.1.1, we further add conjunction tokens between the premises and hypothesis, which can be either manually designed or automatically tuned.
Manual Conj. As in Figure 1(b), we use a conjunctive adverb therefore between the premises and hypothesis. It is kept when there is no premise explicitly given in the input to exclude the effect of the template on probing results under different premise settings.
Soft Conj. We can also use soft conjunctions by adding a soft token between premises explicitly given in the input and a soft token between the premises and the hypothesis. Therefore, the input would be "P1 <s4> P2 <s5> H". The soft templates used in P1,P2 and H are loaded from the learned soft prompts in memorizing tasks and finetuned together with soft conjunctions.

heading: 3.2.2 Reasoning with Pseudowords
text: When testing the reasoning ability of PLMs, we replace the specific instances, classes, and properties in the hypothesis prompt with pseudowords to prevent probing the memorization of hypotheses. Pseudowords (Schütze, 1998; Zhang and Pei, 2022; Goodwin et al., 2020) are artificially constructed words without any specific lexical meaning. For example, the reasoning prompt for the transitivity of subclass (i.e., rule rdfs9) is "[X] is a person. Person is an animal. Therefore, [X] is a particular [MASK] .", where [X] is a pseudoword.
Inspired by (Karidi et al., 2021), we obtain pseudowords for PLMs by creating embeddings without
special semantics. Specifically, we sample embeddings at a given distance from the [MASK] token, as the [MASK] token can be used to predict all the words in the vocabulary and appear anywhere in the sentence. The sampling distance d is set to be smaller than the minimum L2 distance between [MASK] and any other tokens in the static embedding space. Formally:
d = α ·min t∈V ∥zt − z[MASK]∥2
where zt is the static embedding of token t and α ∈ (0, 1) is a coefficient. Moreover, we require that the distance between two pseudowords is at least the sampling distance d to ensure they can be distinguished from each other.

heading: 3.2.3 Classifying Premises: Memorized or not
text: To determine whether a premise is memorized by the model when it is not explicitly given in the input, we employ a classifying method based on the rank of the correct answer in the memorizing task to sort and divide the premise set. The first half of the premise set is regarded as memorized, and the second half is not.
Each rule consists of two premises and we classify them separately. For P1, which involves knowledge of subclass, subproperty, domain or range tested in the memorizing task, we can leverage previously calculated reciprocal rank during the evaluation. Premises are then sorted in descending order by the reciprocal rank. We conduct the same tests on P2, which involves knowledge of pseudowords, to examine model predispositions towards specific predictions and classify whether P2 is memorized or not. Finally, we form our test set by combining premises according to the entailment rule and how each premise is given.

heading: 4 Results and Findings
text: In this section, we introduce the performance of PLMs2 on the test sets of memorizing and reasoning tasks, and analyze the results to posit a series of findings. We then analyze the effectiveness of different prompts. Detailed experimental results can be found in Appendix C.

heading: 4.1 Memorizing Task
text: The baseline model used for the memorizing task is a frequency-based model which predicts a list
2We use variants of BERT and RoBERTa models from https://huggingface.co.
of gold labels in the training set based on the frequency at which they appear, followed by a random list of candidates that are not gold labels in the training set. It combines prior knowledge and random guesses and is stronger than a random baseline.
The experimental results of the memorizing task are summarized in Table 4, from which we can observe that: (1) The best performance of PLMs is better than the baseline on every task except for DM. On DM, the baseline achieves higher MRR. If taking all three metrics into account, the best performance of PLMs still surpasses the performance of the baseline. (2) Except for DM, BERT models achieve much better performance than the baseline in all subtasks and all metrics. Taking an average of the increase in each metric, they outperform the baseline by 43–198%. Only BERTbase-uncased and BERT-large-cased outperform the baseline in DM by a small margin of 1% and 7%. (3) RoBERTa models generally fall behind BERT, showing a 38–134% improvement compared with the baseline except for DM. (4) Despite a significant improvement from the baseline, the results are still not perfect in all subtasks.
PLMs can memorize certain ontological knowledge but not perfectly. Based on the above observation, we can conclude that PLMs have a certain memory of the concerned ontological rela-
tionships and the knowledge can be accessed via prompt, allowing them to outperform a strong baseline. It proves that during pretraining, language models learn not only facts about entities but also their ontological relationships, which is essential for a better organization of world knowledge. However, the memorization is not perfect, urging further efforts on ontology-aware pretraining.
Large models are not necessarily better at memorizing ontological knowledge. According to Petroni et al. (2019), models with larger sizes appear to store more knowledge and achieve better performance in both knowledge probing tasks and downstream NLP tasks. However, as shown in Table 4, BERT-large-uncased is worse than its smaller variant under most circumstances, and RoBERTalarge is worse than RoBERTa-base in TP and DM. It demonstrates that the scale of model parameters does not necessarily determine the storage of ontological knowledge.

heading: 4.2 Reasoning Task
text: We fix the usage of multiple masks and meanpooling in the reasoning experiments as they generally outperform other settings in the memorizing task (see Appendix B). We take an average of the MRR metrics using different templates and illustrate the results of BERT-base-cased and RoBERTa-
base in Figure 2. With neither premise given, the rank of the ground truth is usually low. It shows that models have little idea of the hypothesis, which is reasonable because the information of pseudowords is probed. With premises implicitly or explicitly given, especially P1, the MRR metrics improve in varying degrees. Moreover, results show that BERT-base-cased has better reasoning ability with our concerned ontological entailment rules than RoBERTa-base.
PLMs have a limited understanding of the semantics behind ontological knowledge. To reach a more general conclusion, we illustrate the overall reasoning performance in Figure 3 by averaging over all the entailment rules and PLMs, and find that: (1) When P1 is explicitly given in the input text, models are able to significantly improve the rank of gold labels. As P1 contains the ground truth in its context, it raises doubt about whether the improvement is obtained through logical reasoning or just priming (Misra et al., 2020). (2) Explicitly giving P2 introduces additional tokens that may not be present in gold labels, making P1/P2 = EX/EX worse than P1/P2 = EX/IM. (3) When premises are implicitly given, the MRR
metrics are higher than when they are not given. It implies that, to some extent, PLMs can utilize the implicit ontological knowledge and select the correct entailment rule to make inferences. (4) However, none of the premises combinations can give near-perfect reasoning performance (MRR metrics close to 1), suggesting that PLMs only have a weak understanding of ontological knowledge.
Paraphrased properties are a challenge for language models. In Figure 2(d), the premise P1 of rule rdfs7 contains a paraphrased version of the
ground truth, which is the manually-designed pattern of a particular property. Compared with rule rdfs5 shown in Figure 2(c), where P1 contains the surface form of the correct property, the MRR of BERT-base-cased of rdfs7 decreases by 23%, 49% and 29% when P1 is explicitly given and P2 is not, implicitly and explicitly given, respectively. Though the MRR of RoBERTa-base of rdfs7 increases when P2 is not given, it decreases by 40% and 15% when P2 is implicitly and explicitly given. This suggests that PLMs fail to understand the semantics of some properties, thus demonstrating a limited understanding of ontological knowledge.

heading: 4.3 Effectiveness of Prompts
text: In this section, we discuss how prompt templates affect performance. In the memorizing task, Table 4 shows that using soft templates generally improves the performance of memorizing tasks, in particular TP, SCO and SPO. It suggests that it is non-trivial to extract knowledge from PLMs.
Meanwhile, only a few models perform better with soft templates on DM and RG with a relatively marginal improvement. This could be explained by the fact that both the manual templates and semantics of domain and range constraints are more complex than those of other relationships. Therefore, it is difficult for models to capture with only three soft tokens. We also note that RoBERTa models appear to benefit more from soft templates than BERT models, probably due to their poor performance with manual templates.
Trained soft templates for each relation barely help with reasoning, though. In Figure 4, we summarize the performance by averaging across different models and reasoning tasks and find that it is the trained conjunction token which improves the performance of reasoning rather than the soft templates that describe ontological relationships. It might be inspiring that natural language inference with PLMs can be improved by adding trainable tokens as conjunctions instead of simply concatenating all the premises.

heading: 5 Preliminary Evaluation of ChatGPT
text: After we finished the majority of our probing experiments, ChatGPT, a decoder-only model, was publicly released and demonstrated remarkable capabilities in commonsense knowledge and reasoning. Therefore, we additionally perform a preliminary probe of the ability of ChatGPT to memorize and
understand ontological knowledge. Since ChatGPT is a decoder-only model, we employ a distinct probing method from what is expounded in Sec. 3. Instead of filling masks, we directly ask ChatGPT to answer multiple-choice questions with 20 candidate choices and evaluate the accuracy.

heading: 5.1 Probing for Memorization Ability
text: For memorization probing, we use the finestgrained gold label as the correct answer and randomly sample 19 negative candidates to form the choice set. Take the TP task as an example, we query the GPT-3.5-turbo API with the prompt "What is the type of Lionel Messi? (a) soccer player, (b) work, (c) ..." followed by remaining candidates. We sample 500 test cases for the TP and SCO tasks and use the complete test sets for the other tasks.
For comparison, we also conduct the experiments using BERT-base-uncased, a generally competitive PLM in memorizing and understanding ontological knowledge, with manual prompts and the identical candidate subset. The results presented in Table 5 indicate that ChatGPT outperforms BERT-
base-uncased significantly in most of the memorizing tasks associated with ontological knowledge.

heading: 5.2 Probing for Reasoning Ability
text: Since we cannot input embeddings in the GPT3.5-turbo API, we use X and Y to represent pseudowords as they are single letters that do not convey meanings. However, ChatGPT cannot generate any valid prediction without sufficient context regarding these pseudowords. Therefore, P2 needs to be explicitly provided to describe the characteristics or relations of the pseudowords. We then explore the ability of ChatGPT to select the correct answer from 20 candidates with different forms of P1. In this task, P1 is regarded as memorized if the model can correctly choose the gold answer from the given 20 candidates in the memorizing task.
Based on the results presented in Table 6, ChatGPT demonstrates high accuracy when P1 is either implicitly or explicitly given, suggesting its strong capacity to reason and understand ontological knowledge. Due to a substantial disparity in the knowledge memorized by ChatGPT compared to other models (as shown in section 5.1), their performance is not directly comparable when P1 is not given or implicitly given. Therefore, we only compare ChatGPT and BERT-base-uncased when P1 is explicitly given. Results show that ChatGPT significantly outperforms BERT-base-uncased in explicit reasoning (97.1% vs. 88.2%).

heading: 6 Related Work
text: Knowledge Probing Language models are shown to encode a wide variety of knowledge after being pretrained on a large-scale corpus. Recent studies probe PLMs for linguistic knowledge (Vulić et al., 2020; Hewitt and Manning, 2019), world knowledge (Petroni et al., 2019; Jiang et al., 2020; Safavi and Koutra, 2021), actionable knowledge (Huang et al., 2022), etc. via methods such as cloze prompts (Beloucif and Biemann, 2021; Petroni et al., 2020) and linear classifiers (Hewitt and Liang, 2019; Pimentel et al.,
2020). Although having explored extensive knowledge within PLMs, previous knowledge probing works have not studied ontological knowledge systematically. We cut through this gap to investigate how well PLMs know about ontological knowledge and the meaning behind the surface form.
Knowledge Reasoning Reasoning is the process of drawing new conclusions through the use of existing knowledge and rules. Progress has been reported in using PLMs to perform reasoning tasks, including arithmetic (Wang et al., 2022; Wei et al., 2022), commonsense (Talmor et al., 2019, 2020; Wei et al., 2022), logical (Creswell et al., 2022) and symbolic reasoning (Wei et al., 2022). These abilities can be unlocked by finetuning a classifier on downstream datasets (Talmor et al., 2020) or using proper prompting strategies (e.g., chain of thought (CoT) prompting (Wei et al., 2022) and generated knowledge prompting (Liu et al., 2022)). This suggests that despite their insensitivity to negation (Ettinger, 2020; Kassner and Schütze, 2020) and over-sensitivity to lexicon cues like priming words (Helwe et al., 2021; Misra et al., 2020), PLMs have the potential to make inferences over implicit knowledge and explicit natural language statements. In this work, we investigate the ability of PLMs to perform logical reasoning with implicit ontological knowledge to examine whether they understand the semantics beyond memorization.

heading: 7 Conclusion
text: In this work, we systematically probe whether PLMs encode ontological knowledge and understand its semantics beyond the surface form. Experiments show that PLMs can memorize some ontological knowledge and make inferences based on implicit knowledge following ontological entailment rules, suggesting that PLMs possess a certain level of awareness and understanding of ontological knowledge. However, it is important to note that both the accuracy of memorizing and reasoning is less than perfect, and the difficulty encountered by PLMs when processing paraphrased knowledge is confirmed. These observations indicate that their knowledge and understanding of ontology are limited. Therefore, enhancing the knowledge and understanding of ontology would be a worthy future research goal for language models. Our exploration into ChatGPT shows an improved performance in both memorizing and reasoning tasks, signifying the potential for further advancements.
Limitations
The purpose of our work is to evaluate the ontological knowledge of PLMs. However, a sea of classes and properties exist in the real world and we only cover a selective part of them. Consequently, the scope of our dataset for the experimental analysis is limited. The findings from our experiments demonstrate an imperfect knowledge and understanding obtained by the models, indicating a tangible room for enhancement in both ontological knowledge memorization and understanding and a need for a better ability to address paraphrasing. These observations lead us to contemplate refining the existing pretraining methods to help language models achieve better performance in related tasks.
Ethics Statement
We propose our ethics statement of the work in this section: (1) Dataset. Our data is obtained from DBpedia and Wikidata, two publicly available linked open data projects related to Wikipedia. Wikidata is under the Creative Commons CC0 License, and DBpedia is licensed under the terms of the Creative Commons Attribution-ShareAlike 3.0 license and the GNU Free Documentation License. We believe the privacy policies of DBpedia3 and Wikidata4 are well carried out. We inspect whether our dataset, especially instances collected, contains any unethical content. No private information or offensive topics are found during human inspection. (2) Labor considerations. During dataset construction, the authors voluntarily undertake works requiring human efforts, including data collection, cleansing, revision and design of property patterns. All the participants are well informed about how the dataset will be processed, used and released. (3) Probing results. As PLMs are pretrained on large corpora, they may give biased results when being probed. We randomly check some probing results and find no unethical content in these samples. Therefore, we believe that our study does not introduce additional risks.

heading: Acknowledgement
text: This work was supported by the National Natural Science Foundation of China (61976139) and by Alibaba Group through Alibaba Innovative Research Program.
3https://www.dbpedia.org/privacy/ 4https://foundation.wikimedia.org/wiki/
Privacy_policy

heading: A Experimental Setup
text: We train soft tokens for 100 epochs with AdamW optimizer. The learning rate is set to 0.5 and a linear warmup scheduler is used. Since both the memorizing and reasoning task can be formulated as a multi-label classification problem, we use BCEWithLogitsLoss or NLLLoss as our loss function in the memorizing task to report the better results given by one of these two and select a better training objective. Therefore, we fix the loss function to BCEWithLogitsLoss in the reasoning task.
For pseudowords, we set the coefficient α to 0.5 and sample 10 pairs of pseudowords for each entailment rule as we at most need two pseudowords to substitute the subject and object instances respectively, and report the averaged performance as the final result.

heading: B Multi-token Prompting Methods
text: In the main body of the paper, we discuss the impact of different prompts on the performance of knowledge probing and reasoning. In this section, we continuously discuss the impact of other prompt settings by comparing the averaged performance.
B.1 Number of [MASK] Tokens
To support multi-token candidate scoring, we use multiple [MASK] tokens or one single [MASK] token to predict with masked language models. The comparison between the two methods is shown in Figure 5, by averaging the performance of all the memorizing tasks and models. We can observe that single [MASK] prediction achieves better accuracy (R@1) with a negligible tiny margin but worse performance in other metrics. Therefore, using multiple [MASK] tokens to obtain prediction by forward pass inference is more sensible and achieves better results.
B.2 Pooling Methods
Three pooling methods are proposed when computing the probability of a candidate that can be tokenized into multiple subtokens. The mean-pooling method is usually used in multi-token probing. Furthermore, we introduce max-pooling and firstpooling, which retain the score of only one important token. They can exclude the influence of prepositions, e.g., by attending to mean or transportation when scoring the candidate mean of transportation, but at the cost of other useful information. We are interested in whether it is better to consider the whole word or focus on the important part.
Figure 6 shows that mean-pooling, as a classical method, is much better than the other two pooling methods. Besides, first-pooling gives clearly better results than max-pooling, which is possibly caused by the unique information contained in the headword (usually the first token). Consider candidates volleyball player, squash player and golf player, the conditional log probability of token player might be higher, but the candidates are distinguished by their headwords. In summary, mean-pooling obtains the best results with the most comprehensive information.
B.3 Loss Functions
As mentioned in Appendix A, we try two loss functions in the memorizing task. (1) The Binary Cross Entropy With Logits Loss (BCEWithLogitsLoss) is a common loss function for multi-label classification which numerically stably combines a Sigmoid layer and the Binary Cross Entropy Loss into one layer. All examples are given the same weight
0.5
Mean First Max
when calculating the loss. (2) The Negative Log Likelihood Loss (NLLLoss) is a loss function for multi-class classification. However, we can convert the original multi-label problem to a multi-class one by sampling one ground truth at a time to generate multiple single-label multi-class classification cases. As can be seen from Figure 7, using BCEWithLogitsLoss as the loss function achieves better results than using NLLLoss. Hence, in subsequent reasoning experiments, we stick to the classical loss for multi-label classification.
0.5
BCEWithLogitsLoss NLLLoss

heading: C Experimental Results
text: C.1 Task Examples
In order to enhance the clarity of the experiments, we have compiled a list in Table 7 that includes task
prompts as well as the top five predicted candidate words generated by BERT-base-cased. The table consists of examples with successful predictions for all correct answers (SPO, RG), examples with partial correct answers predicted (TP, SCO), and examples where the correct answer is not predicted within the top five candidates (DM).
C.2 Memorizing Results The complete results of the memorizing task are reported in Table 8, 9, 10, 11 and 12.
C.3 Reasoning Results We report the MRR Metric of BERT-baseuncased, BERT-large-cased, BERT-large-uncased and RoBERTa-large in Figure 8. It is generally consistent with the two models reported in the main body of the paper and the macro-averaged performance across different PLMs, so consistent conclusions can be drawn.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation section
3 A2. Did you discuss any potential risks of your work? Ethical section
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 2
3 B1. Did you cite the creators of artifacts you used? 2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Ethical section
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 2
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethical section
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 2
C 3 Did you run computational experiments? 4
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We focus on investigating whether PLMs know and understand ontological knowledge using models from the huggingface. We do not pay extra attention to the computational budget or computing infrastructure.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix A
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 2
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Ethical section
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Ethical section
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? 2, Ethical section
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 2
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? As the authors undertake the annotation work, reported demographic and geographic characteristics maybe violate the anonymous submission policy.

heading: Future Work
text: Therefore, enhancing the knowledge and understanding of ontology would be a worthy future research goal for language models. Our exploration into ChatGPT shows an improved performance in both memorizing and reasoning tasks, signifying the potential for further advancements. Limitations
The purpose of our work is to evaluate the ontological knowledge of PLMs. However, a sea of classes and properties exist in the real world and we only cover a selective part of them. Consequently, the scope of our dataset for the experimental analysis is limited. The findings from our experiments demonstrate an imperfect knowledge and understanding obtained by the models, indicating a tangible room for enhancement in both ontological knowledge memorization and understanding and a need for a better ability to address paraphrasing. These observations lead us to contemplate refining the existing pretraining methods to help language models achieve better performance in related tasks