Does the Correctness of Factual Knowledge Matter for Factual Knowledge-Enhanced Pre-trained Language Models?
Authors: Boxi Cao
Qiaoyu Tang
Hongyu Lin
Xianpei Han
Le Sun
Venue: EMNLP 2023 Main (2023)
Keywords: factual knowledge
language model
knowledge-enhanced

Abstract: In recent years, the injection of factual knowledge has been observed to have a significant positive correlation to the downstream task performance of pre-trained language models.
However, existing work neither demonstrates that pre-trained models successfully learn the injected factual knowledge nor proves that there is a causal relation between injected factual knowledge and downstream performance improvements.
In this paper, we introduce a counterfactual-based analysis framework to explore the causal effects of factual knowledge injection on the performance of language models within pretrain-finetune paradigm.
Instead of directly probing the language model or exhaustively enumerating potential confounding factors, we analyze this issue by perturbing the factual knowledge sources at different scales and comparing the performance of pre-trained language models before and after the perturbation. 
Surprisingly, throughout our experiments, we find that although the knowledge seems to be successfully injected, the correctness of injected knowledge only has a very limited effect on the models' downstream performance.
This finding strongly challenges previous assumptions that the injected factual knowledge is the key for language models to achieve performance improvements on downstream tasks in pretrain-finetune paradigm.

Does the Correctness of Factual Knowledge Matter for Factual Knowledge-Enhanee Pre-trained Language Models?
Boxi Ca Qiaoyu Tang 13; Hongyu Lin') Xianpei Han 1:21
Le Sun
Chiness Information Processing Laboratory SStte Key Laboratory of Computer Science Institute of Software, Chinese Academy of Sciences, Beijing, China 3 University of Chinese Academy of Sciences, Beijing, China [boxi2020, tanggiaoyu2020 020, hongyu, x xianpei, sunle] @ @iscas...... ac cn
Abstract
Knowledge
Knowledge Injection
Downstrcca
Venifla
Vanila PLM
In recent years, the injection of factual knowl- edge has been observed to have a significant positive correlation to the downstream task performance of pre-trained language models. However, existing work neither demonstrates that pre-trained models successfully learn the injected factual knowledge nor proves that there is a causal relation between injected factual knowledge and downstream performance im- provements. In this paper, we introduce counterfactull-bast based analysis framework to ex- plore the causal effects of factual knowledge injection on the performance of language mod- els within pretrain- finttine-fuuueeeee paradigm. Instead of directly probing the language model or ex- haustively enumerating potential confounding factors, we analyze this issue by perturbing the factual knowledge sources at different scales and comparing the performance of pre-trained language models before and after the perturba- tion. Surprisingly, throughout our experiments, we find that although the knowledge seems to be successfully injected, the correctness of in- jected knowledge only has very limited ef- fect on the models' downstream performance. This finding strongly challenges previous as- sumptions that the injected factual knowledge is the key for language models to achieve per- formance improvements on downstream tasks pretrain-finetune finetune paradigm.
Tim Cook is the CEO of Apple.
Foctual Substitattin
Factual perturied
BMI Gates is the CEO of Apple.
Oetelagfree saberifurtio
Ontological parturbed
Microsoft is the CBO Apple.
Langgaag- Madii
This paper explores the impact of factual knowledge by comparing the downstream task perfor- mance before and after knowledge perturbation.
Figure I:
transferability improvements across tasks com- pared with previous architectures (Devlin al., 2019; Radford et al., 2019; Liu al., 2019; Raffel et al., 2020; Scao et al., 2022; Touvron et al., 2023). Some previous investigations contribute the periors of pre-trained language models to their en- tailed various kinds of knowledge learned from the pre-training stage (Petroni et al., 2019; Lewis et al., 2020b; Yin et al., 2022; Cao et al., 2023). Among them, factual knowledge, which reveals the relationships between real-worid entities (c.g.. Tim Cook is the CEO of Apple) and plays critical role in human cognition (Unger, 1968), is regarded critical factor for pre-trained language model to approach trusted intelligent agent (Lewis al., 2020b; Yin al., 2022). Consequently, how improve the acquisition, modeling, and application of factual knowledge of pre-training language mod- els has become a hot research topic. To this end, many studies have been devoted to further injecting factual knowledge to enhance the acquisition and modeling of factual knowledge in pre- pre-trained lan- guage models (Zhang et al., 2019; Liu et al., 2020; Sun et al., 2020; Wang et al., 2021a,b), and have already reported successful performance improve- ments on specific downstream tasks.
1 Introduction
In recent years, pre-trained language models (PLMs) have emerged as the dominant approach natural language processing. Through self- supervised learning on large-scale text corpus, PLMs can acquire different kinds of knowledge automatically without additional manual guidance, which demonstrates significant generalizability and
On the contrary, recent studies have found that pre-trained language models struggle with retaining factual knowledge, and the retained factual knowl- edge can exhibit inconsistencies with the original knowledge sources (Poerner et al., 2020; Elazar
"Equal contribations. 'Corresponding authors.

---

al., 2021; Cao al., 2021). Furthermore, the indirect evaluations through downstream tasks only reflect that injecting factual knowledge corre- lated to performance improvement, but can not establish the causality between them due to the existence of many additional potential confound- ing factors (training data domain, model parameter size, etc.). Consequently, to identify the impact of factual knowledge injection on the downstream per- formance of pre-trained language models, the fol- lowing two critical questions should be answered:
is the CEO of Apple"), as well ontological sub- stitution that thoroughly perturb entities with their counterparts of another type (e.g., substitute "Tim Cook the CEO of Apple" with "Microsoft is the CEO of Apple"). In addition, to analyze the impact of perturbation, we investigate three knowledge in- jection approaches that acquire knowledge from two types of factual knowledge sources, includ- ing plain texts containing factual knowledge and structured factual knowledge bases.
Throughout empirical evaluations on a wide range of representative downstream NLP tasks, our findings surprisingly deviate from previous hy- potheses. Although the knowledge injection ap- proaches seem to inject factual knowledge into PLMs successfully, we find that factual knowl- edge perturbation has only a very limited effect on PLMs' downstream task performance, i.c., the correctness of factual knowledge shows a very limited impact on all evaluated downstream tasks. Furthermore, although the influence of ontologi- cal perturbation is slightly stronger than factual perturbation, also does not cause statistically significant performance divergence most down- stream tasks. Specifically, our experiments show that in most downstream tasks, the performance fluctuation caused by the above-two perturbation is not greater than the fluctuation caused by random seeds, and the results of t-test further demonstrate that there are no statistically significant differences between the performance before and after pertur- bation. Through this counterfactual-basse analy- sis, our findings demonstrate that injected factual knowledge is not the core reason for the perfor- mance improvements of previous knowledge injec- tion approaches on pre-trained language models in the pretrain- finnttn--ii paradigm.
Through the existing knowledge injection methods, is the factual knowledge really in- jected into pre-trained language models?
If so, is it indeed the injected knowledge, rather than other confounding factors, that is responsible for the observed performance im- provements in downstream tasks?
Unfortunately, it is infeasible to directly answer the above-two questions due to the lack of highly effective language model knowledge probing and confounding factor identification measurements. To this end, as shown Figure 1, this paper in- troduces a counterfactual--aase analysis frame- work (Veitch et al., 2021; Guidotti, 2022) to ex- plore the causal effects of injecting factual knowl- edge in "what-if" manner. Moreover, Figure 2 illustrates the applied framework. Instead of di- rectly probing the language model or exhaustively enumerating potential confounding factors, we an- alyze the above-two questions by perturbing the factual knowledge sources at different scales, then comparing the downstream performance of pre- trained language models before and after the per- turbation. The key motivation of our work is that: 1) If the knowledge injection approaches are inef- fective, the performance of the model injected with the correct knowledge and perturbed knowledge should not perform very differently in the knowl- edge probing evaluation; 2) If the correctness of injected factual knowledge is indeed essential for downstream tasks, then injecting perturbed, wrong knowledge should cause significant performance decline. Specifically, in order to observe the as- pect from which factual knowledge affects PLMs, we conduct two kinds of perturbation factual knowledge, factuu substitution, which replaces an entity in factual knowledge with an- other entity with the same type (c.g., substitute "Tim Cook is the CEO of Apple" with "Bill Gates
The following part of this paper is organized as follows. In Section 2, we briefly review the related work about factual knowledge probing and injec- tion of PLMs. Section 3 presents our proposed counterfactual-basee analysis framework. The ex- perimental results and the process leading to our conclusions are presented in Section 4. In Sec- tion 5, we provide a brief discussion and conclude our findings!.
'We openly released our source code at https:// github.

---

Carpus
Knowledge Perturbation
Appk:
Holiban: CHOP
Steve Jobs California
Tii CBO of
Factsal Subatitution
BERT
BERT
Apple bocted in Califienia
caliarni 9 Howal
Class
BASSK
TeeCook
The Deck hover Orti
App
Knowledge Base
Ontolagical
SANIE/K-Adoptee
Tim
CED
Apple
Cook Moreih Calloeni eaarr Paiter
FRNIE/K-AAapter
Steve
Califrrais
OUIZO
[HAAK
Tiat Cock Iwsoees oI
O Ou
RCP
Marzed you fraddd
Npple
Figure 2: The illustration of the counterfactual-bet -based knowledge analysis framework. Given factual knowledge from text corpus or knowledge bases, we first conduct two perturbations to obtain incorrect knowledge instances. Then pre-train LMs the perturbed datasets. Finally, compare the downstream performance of PLMs before and after perturbation to explore the causal effects of factual knowledge injection on the performance of PLMs.
2 Related Work
modeling (Devlin et al., 2019; Liu et al., 2019), denoising auto-encoder (Lewis et al., 2020a), etc. And such pre-trained LMs have been shown to po- tentially entail large scale of factual knowledge in the parameters (Bouraoui et al., 2020; Petroni al., 2019). In addition, many studies intend to explic- itly infuse the factual knowledge from structured knowledge base into LMs (Yang et al., 2021). Pop- ular strategies include: 1) Embedding combined methods (Zhang et al., 2019; Peters et al., 2019; He et al., 2020b), which encode structured knowledge via knowledge embedding algorithms and then en- hance PLMs' text representation with knowledge graph embedding; 2) Knowledge supervised meth- ods (Wang al., 2021a,b; Yamada et al., 2020), which utilize elements from structured knowledge as supervision signals and leverage base PLMS to learn their semantics.
Factual knowledge reveals the relationships be- tween real-world entities and plays crucial role in human cognitive (Unger, 1968). Therefore, lots of studies devoted to probing the factual knowl- edge entailed in PLMs on the one hand (Petroni et al., 2019; Kassner et al., 2021; Sung et al., 2021) 1), and propose to enhance the factual knowledge in large-scale LMs to benefit their performance on downstream tasks on the other (Zhang et al., 2019; Liu et al., 2020; Wang al., 2021b).
Factual Knowledge Probing aims to evalu- ate how well current PLMs are entailing factual knowledge in parameters. Currently, the most popular strategy prompt-based probing (Petroni et al., 2019; Ettinger, 2020; Srivastava et al., 2022), e.g., query BERT with "Tim Cook is the CEO of [MASK]" to determine whether it contains cor- responding knowledge. And recent studies have found that prompt-based probing could be inac- curate (Poerner al., 2020; Zhong al., 2021), inconsistent (Elazar al., 2021; Kassner and Schütze, 2020; Cao et al., 2022; Jang al., 2022), and unreliable (Li et al., 2022; Cao et al., 2021).
3 Counterfactual-base Knowledge Analysis Framework
As mentioned above, this paper intends to investi- gate whether current factual knowledge injection approaches can inject factual knowledge into pre- trained language models and whether the injected factual knowledge can benefit downstream NLP tasks in pretrain-finctune paradigm. However, it is currently not feasible to directly explore these ques- tions due to the lack of effective knowledge probing and confounding factor discovery approaches for pre-trained language models.
Factual Knowledge Injection aims to inject factual knowledge into PLMs. Currently, there are two main sources for injecting factual knowl- edge into language models, including plain text and structured knowledge base. For plain text, LMs typically acquire factual knowledge through self-supervised learning on large-scale text corpus without any knowledge guided supervision. The popular objectives include casual language model- ing (Radford et al., 2019; Brown et al., 2020; Scao et al., 2022; Touvron et al., 2023), masked language
To this end, propose leverage counterfactual-based knowledge analysis frame- work answer these two questions in a "what-if" manner. Specifically, we transform the problem of investigating the characteristics of model with

---

Source Model
Wikipedia BERT
Wikidata
statement" "Bill Gates is the CEO of Apple". On the other hand, ontological substitution thoroughly per- turbs entities with counterparts of different types, which is used to evaluate the importance of factual knowledge ontology on downstream tasks. For ex- ample, ontological substitution replaces "Tim Cook is the CEO of Apple" with "Microsoft is the CEO of Apple".
ERNI::
K-Adapter
Total
14,545,579
6,105,524
5,565.478
Perturbed
13,538,337
5,541,297
4,692.683
Pert. Rate
93.08%
90.76%
84.32%
Table The perturbation details of two kinds of knowl- edge sources. For text corpus, we perturb 93.19 % para- graphs from Wikipedia. As for structured knowledge base, we perturb 90.76% training instances for ERNIE and 84.32% for K-Adapter from Wikidata.
This paper mainly focuses on two kinds of the most widely-used factual knowledge sources, in- cluding learning from plain text corpus and struc- tural factual knowledge bases. For learning from plain text corpus, we use paragraphs in Wikipeida" that contain an anchor linking to Wikidata (Vran- dečić and Krötzsch, 2014) as knowledge sources. For learning from knowledge bases, we direct use entities and their relations in Wikidata knowl- edge sources. The type of an entity is determined by its corresponding "instance of" and "subclass of" properties in Wikidata. In this paper, we mainly focus on perturbing factual knowledge about three kinds of representative entity types, including Per- son, Location, and Organization. Table demon- strates the perturbation details for both kinds of knowledge sources, revealing that the perturba- tion would affect most of their training instances. For learning from text, we utilize 14,545,579 para- graphs from Wikipedia and perturb 13,538,337 of them, resulting in a 93.1 % perturbation rate. For knowledge learning from structured data such as ERNIE, we utilize a total of 6, 105.542 pre-training instances, of which we perturb 5,531,297, leading to perturbation rate of 90.76%.
injected knowledge into comparing the behaviors between models injected with correct and incor- rect knowledge, respectively. Consequently, if the knowledge injection approaches work, models in- jected with incorrect knowledge should exhibit sig- nificantly inferior performance on the knowledge probing evaluation than models injected with cor- rect knowledge. Furthermore, if injecting factual knowledge is indeed helpful to downstream tasks, models injected with correct factual knowledge should perform significantly better than models injected with incorrect factual knowledge down- stream tasks.
The overall counterfactual based analysis frame- work is illustrated in Figure 2. Specifically, given factual knowledge from text corpus or knowledge bases, we first conduct perturbation on the in- stances in them to obtain incorrect knowledge in- stances. Then pre-train language models with several representative factual knowledge injection approaches on both vanilla and perturbed knowl- edge sources. Finally, we compare the performance of knowledge-injected models to reach conclusions about the above-two questions. In the following sections, we will first demonstrate how we con- duct knowledge perturbation on different knowl- edge sources and then briefly introduce the inves- tigated representative factual knowledge injection approaches in our experiments.
3.2 Factual Knowledge Injection Approaches
In recent years, the injection of factual knowledge into large-scale language models has emerged a prominent research area. Various methods have been developed for injecting knowledge, depending on the specific sources of knowledge. In this paper. we explore three representative approaches for in- jecting knowledge, corresponding to the following two types of knowledge sources:
3.1 Factual Knowledge Perturbation
Knowledge perturbation aims to generate counter- factual factual knowledge instances for analysis. In this paper, we mainly employ two kinds of per- substitution
Learning From Plain Text. In this study, we se- lect BERT (Devlin al., 2019) as our experiment architecture for learning from text, as is one of the most representative pre-trained language mod- els. To investigate the impact of factual knowledge on BERT, we conduct pre-trate rie-training of the BERT- base model from scratch with masked language
turbation strategies, including and ontological substitution. Factual substitution studies the influence of the factual correctness of factual knowledge in models by replacing entities with other entities of the same type. For example, factual substitution perturbs the factual knowledge "Tim Cook is the CEO of Apple" with an incorrect
Phttps: //ww///wwwww/...////o////.. wi ki pedieaa..o..

---

modeling as the objective, separately on both the vanilla and perturbed versions of Wikipedia text corpus respectively. The model pre-trained using a batch size of 1024 sequences for 500,000 steps. For optimization, we use Adam optimizer with a learning rate of le - 4, 31 = 0.9, B2 = 0.999, learning rate warmup over the first 10,000 steps. The training process was conducted on 2 Nvidia A100 GPUs with 80G RAM for about 10 days.
tasks, knowledge guided tasks, knowledge apply- ing tasks, and language understanding tasks.
Knowledge Probing Tasks are primarily used to investigate the knowledge entailed in PLMs. We use LAMA (Petroni et al., 2019), the most widely used factual knowledge probing benchmark, as our testbed to determine whether the factual knowledge is successfully injected into PLMs. LAMA evalu- ates the factual knowledge in PLMs by employing cloze-style questions, such as "Tīm Cook is the CEO of [MASK]".
Learning From Structured Knowledge. We se- lect one representative approach for each direc- tion of learming factual knowledge from structured knowledge mentioned in Section 2, including:
Knowledge Guided Tasks aim to evaluate the ability of PLMs to recognize the factual infor- mation within texts, such as entities and entity relations. Specifically,, we evaluate BERT on two widely used named entity recognition (NER) datasets including CONLL2003 (Tjong Kim Sang, 2002) and OntoNotes 5.0 (Pradhan et al., 2013), as well as two representative relation extraction (RE) datasets including ACE2004 (Mitchell et al., 2005) and ACE2005 (Walker al., 2006). For the eval- uation of ERNIE and K-Adapter, to obtain more reliable experimental conclusions, we conduct ex- periments on the same tasks as the original paper, including entity typing (e.g.. Open Entity (Choi et al., 2018) 8) and FIGER (Ling et al., 2015)) and re- lation classification (e.g.. FewRel (Han et al., 2018) and TACRED (Zhang et al., 2017)).
ERNIE (Zhang al., 2019) a typical model that injects knowledge into PLMs through the embedding combined method. first iden- tifies the named entity mentions in the text and aligns them to Wikidata. Then ERNIE aggregates the entity representation with its corresponding token embedding, where entity representation is trained on KG via knowl- edge embedding algorithms like TransE (Bor- des et al., 2013). We perturb the acquired knowledge for ERNIE by substituting the en- tity representation in the input.
K-Adapter (Wang et al., 2021a) is a repre- sentative method that utilizes elements in the explicit knowledge base as the supervision signal. It designs adapter to inject fac- tual knowledge via relation classification task with keeping the original parameters of PLM fixed. We perturb the acquired knowledge for K- K-Adapter by directly substituting the entities in the explicit knowledge base.
Knowledge Applying Tasks focus on evaluat- ing the model's ability to apply factual knowledge to reasoning and moolem--solving tasks (Petroni et al., 2021), e.g., open QA and fact checking. In this paper, we select the open QA datasets Nat- ural Questions (Kwiatkowski al., 2019), Cos- mosQA (Huang et al., 2019), and fact checking dataset FEVER (Thorne et al., 2018).
To ensure fair comparison, we strictly adhere to the pre-training process outlined in the original papers.
Language Understanding Tasks includes var- ious tasks such as text classification, natural lan- guage inference, sentiment analysis, etc. We select GLUE (Wang et al., 2019) as evaluation bench- mark, which is a collection of NLU tasks and widely used in various PLMs' evaluations, and pre- vious knowledge injection studies have reported performance improvements on them (Sun et al., 2019; Liu al., 2020: Sun et al., 2021).
3.3 Downstream Evaluation
To make a comprehensive and thorough evalua- tion, we conduct experiments on a wide range of downstream tasks, most of which have been pre- viously shown to achieve performance improve- ment through knowledge injection (Zhang et al., 2019; Wang al., 202 la; He et al., 2020a; Yamada et al., 2020; Qin et al., 2021: Sun al., 2021). In- spired by Yu et al. (2023), we divide these tasks into four categories based on the stratification and con- nection to factual knowledge: knowledge probing
In the downstream task fine-tuning process, we follow the same dataset split and hyper-parameter settings of the original papers, please refer to Ap- pendix for details due to page limitation. In addi- tion, to avoid the impact of randomness on investi- gation conclusions, all experiments were conducted

---

Perturbation
Factual Query
Output
Cicero
Lorde
vanilla
Cicero was born in [MASK]-
Rome
Factual
Rome
Disney
perturb
Lorde was born in [MASK]-
Disney
Mahatma Gandhi
Luzon
vanilla
Mahatma Gandhi was born in [MASK].
India
Ontological
India
Nevada
perturb
Luzon was born [MASK].
Nevada
Table 2: Example of PLMs' outputs before and after perturbation. The left part shows the perturbation mapping and the right shows the input and output of the models; "factual" and "ontological"" denotes the corresponding substitution strategy; "vanilla" and "perturb" denote the vanilla LM and the perturbed L.M respectively.
Model
Vanilla
Factual
Ontological
8057 JACENe5
3557-40
BERT
28.18
11.62
10.34
ERNIE
29.21
25.31
25.67
Table 3: The factual knowledge probing results on LAMA for both vanilla and perturbed pre-trained lan- guage models.
EAiede: 32/m
viits
under 5 random seed settings, and their means and standard deviations are reported in the later results.
4 Experiments and Findings
Figure Distributions of LMs' performance down- stream tasks. We assume the variation in performance caused by random seed follows a normal distribution, and estimate the parameters based the results of the vanilla language model on various random seeds, and the remaining points on the curve demonstrate the per- formance of LMs before and after perturbation.
Based on the analysis framework presented in Sec- tion 3, we conduct extensive experiments and in- triguingly find that the correctness of factual knowl- edge shows a very limited impact on almost all downstream tasks. In this section, we will elab- orate on our experimental procedures, provide a detailed account of our findings, and illustrate how we arrive at our conclusions.
the performance on LAMA benchmark for PLMs with and without different types of perturbation. is evident that BERT's performance on LAMA significantly decreases from 28.18 to 1 11.62 after factual substitution and further drops to 10.34 af- ter ontological substitution. ERNIE's performance drops from 29.21 to 25.31 and 25.67 after per- turbation. The significant decline in performance demonstrates the effectiveness of injecting incor- rect knowledge. In conclusion, through factual knowledge probing, we can demonstrate the effec- tiveness of knowledge injection on both learning factual knowledge from plain text and structural knowledge bases. This implies that current rep- resentative knowledge injection approaches suc- cessfully influence factual knowledge in language models.
4.1 Does Knowledge Injection Works?
Conclusion 1. Knowledge injection approaches successfully affect factual knowledge in pre-trained language models.
To assess the effectiveness of knowledge in- jection approaches, we compare the perfor- mance of PLMs before and after perturbation on the well-known knowledge probing benchmark LAMA (Petroni et al., 2019). Table 2 demonstrates several illustrative examples of the predictions from both vanilla and perturbed models, showcasing the influence of the knowledge injection process on the model's 's output. For example, in the perturbed corpus, we substitute the factual knowledge <Ci- cero, birthplace, Rome> with <Lorde, birthplace, Disney>, The vanilla BERT predicts "Cicero was born in Rome". while the perturbed BERT predicts "Lorde was born in Disney", indicating that fac- tual knowledge injected during pre-training indeed influences the model's prediction.
4.2 Does Factual Substitutions Affect Downstream Performance?
Conclusion 2. Regardless of the approaches of knowledge injection, factual substitution shows very limited influence on all downstream tasks, i.e.,
To further quantify such effects, Table shows

---

Model
Perturbation
CONLL2003
OntoNotes 5.0
ACE2004
ACE2005
NQ
FEVER
GLUE
No Substirution
91.37 + 0.31 91.22 + 0.05
86,92 0,10
12.18 = 0.52
72.93 = 1.01
50.36
+ 0.21
88 88
88.5
0.21
80.26 = 0.33 80.07 + 0.53
BERT
Factual Substitution
88.87 0.11
71.79 + 0.39
73. 0.63
50.35 + 0.14
88.40 0.188
P-value
0.385
0.542
0.265
0.755
0.849
0.273
0.560
Model
Perturbation
Open Entity
FIGER
FewRel
TACRED
GLUE
No Substitution
73.85 + 0.41
71.43 + 1.25
88.41 + 0.33
66.86 + 0.85
81.74 + 0.59 81.57 + 0.34
ERNIE
Factual Substitution
74.13 + 0.69
71.16 0.96
86.88 0.34
66.31 + 0.45
P- P-value
0.509
0.740
0,000
0.291
0.638
Model
Perturbation
Open Entity
FIGER
TACRED
CosmosQA
GLUE
No Substitution
76.05 0.36 76.05 + 0.43
77.67 0.67
70.81 0.49
80.04 1 0.73 79.52 + 1.05
87.31 1 0.75
K-Adapter
Factual Substitution
77.08
0.74
70.47 + 0.54
87.90 + 0.56
P-value
0.995
0.276
0.376
0.437
0.240
Table 4: Model performance and t-test results on downstream tasks before and after factual substitution. The last row presents p-values from the t-test. Most p-values significamtly exceed threshold 0.05, indicating insufficient statistical evidence to assert a noteworthy difference in model performance before and after perturbation.
the correctness of injected factual knowledge is not the key factor for factual knowledge-enhunced language models to achieve better performance.
For learning from structured knowledge, we con- duct experiments on both embedding combined (ERNIE) and knowledge supervised (K-Adapter) methods. In order to ensure the reliability of our conclusions, we select the same tasks as the origi- nal papers, where knowledge injection was shown to provide benefits. Overall, reach similar find- ings with learning from plain texts, where the ran- dom seeds cause larger performance fluctuation than factual substitution on most benchmarks. The only exception comes from FewRel, where the fac- tual substitution leads to relatively significant per- formance degeneration. However, it is worth noting that FewRel and ERNIE share the same knowledge source, which could lead to significant information leakage and make the model more dependent the correctness of knowledge in the explicit knowl- edge base. We also conduct detailed experiments prove and quantify the information leakage, which is beyond the scope of this paper, and therefore we present the results in the appendix.
To investigate the effect of knowledge pertur- bation, we calculate the mean and standard devi- ation of model performance with 5 different ran- dom seeds for each task and compare the perfor- mance fluctuation caused by perturbation and ran- dom seeds. Morcover, we leverage the t-test to further verify whether a statistically significant dif- ference exists between the model performance be- fore and after perturbation.
Table 4 shows the models' performance on down- stream tasks before and after factual substitution. For lcarning from plain texts, we observe that fac- tual substitution has limited impact on all evalu- ated dataset: On language understanding tasks using GLUE benchmark, the average performance fluctuation caused by factual substitution is 0. 0.19%, which is lower than the performance fluctuation 0.33% caused by random seeds. And the perfor- mance on each task is demonstrated in the appendix due to page limitations: 2) On knowledge applying tasks such as open domain QA and fact checking, we obtain similar findings GLUE benchmark, indicating that the knowledge acquired from the pre-training phase has a limited impact even when the downstream tasks require specific knowledge for answer inference or task solving; 3) Even for the factual knowledge guided tasks such as NER and RE, we surprisingly find that the random seed still causes larger performance fluctuation than fac- tual substitution. Moreover, on relation extraction tasks such as ACE 2005, the average performance of the perturbed models across different random seeds is even higher than the vanilla model without perturbation.
To further quantify the performance divergence, we employ a t-test (Boneau, 1960) to examine the significance of the performance differences be- tween the models before and after factual substi- tution. The null hypothesis posited no alteration in the mean performance, while the alternative hy- pothesis argued for a discernible variation in perfor- mance levels. Following standard conventions, we set the threshold for statistical significance at 0.05 That is to say, p-value greater than 0.05 indicates that there is sufficient statistical evidence to ject the null hypothesis. The p-values of the models on each downstream datasets are presented in the last row of Table 4. In all of these datasets (except FewRel Rel as we mentioned above), the p-values were notably larger than our pre-specified level statis-

---

Model
Perturbation
CONLL2200
OntoNotes 5.0
ACE2004
ACE2005
NQ
FEVER
CLUE
No Substitution
91.37 + 0.31
88.92 + 0.10
72.18 + 0.52
72.93 + 1.01
50:36 + 0.21
88.58 + 0.24
80.26 0.33
BERT
Ontological Substitution
91 18 = 0.09
88. 72 + 0.15
72.56 + 0.33
73.80 = 0.00
50.34 = 0.10
88 23 0.28
80. 18 0.36
P-value
0.290
0.061
0.250
0.154
0.002
0.089
07758
Model
Perturbation
Open Entity
FIGER
FewRel
TACRED
GLUE
No Substitution
73.85 + 0.41
71.43 + 1,25
88.41 + 0.33
66.86 = 0.85
81.74 1 0.59
ERNIE
Ontological Substitution
74.26 + 0.53
70.00 + 1.60
85.15 0.46
66.57 + 0.35
81.67 0.16
P-value
0.261
0.197
0.000
0.541
0.829
Model
Perturbation
Open Entity
FIGER
TACRED
CosmosQA
GLUE
No Substitution
76.05 + 0.36
77.67 = 0.67
70.81 + 0.49
80.04 + 0.73
87
0.75
K-Adapter
Ontological Suhstitution
76.12 0.26
77.66 0.16
70.44 + 0.35
79.99
1.40
88.14 0.16
P-value
0.754
0.978
0.254
0.951
0.060
Table 5: Model performance and P-values of t-test on each downstream tasks before and after ontological substitution.
tical significance (0.05). This outcome suggests an absence of substantial evidence to reject the null hypothesis. Therefore, we could not substantiate the existence of a significant difference in model performance before and after factual substitution. Overall, for language models acquiring factual knowledge from either plain texts or structured knowledge bases, the correctness of factual knowl- edge during pre-training has a very limited im- pact on downstream tasks, which significantly chal- lenges the previous assumptions of factual knowl- edge leading to performance improvements on downstream tasks.
straightforward manner. Specifically, we first as- sume the variation in performance due to random seed effects follows a normal distribution, and plot the performance distribution curve based on the results of the vanilla LMs on various random seeds, then highlight the performance of the model before and after two substitutions on the curve. And we can clearly find that there not exist significant per- formance difference before and after perturbation. We also notice that, overall, the impact of ontolog- ical substitutions is slightly stronger than factual substitutions, which is also understandable because large-scale ontological substitutions not only affect the correctness of factual knowledge in the model but also interfere with the model's understanding of linguistics and semantics, thus undermines the language understanding ability of PLMs.
4.3 Does Ontological Substitution Affect Downstream Performance?
Conclusion 3. Overall, the influence of ontological substitution is slightly stronger than factual substi- tution, but still shows very limited impact most downstream tasks.
Conclusions and Discussions
This paper proposes counterfactual knowl- edge analysis framework to investigate the influ- ence of injecting factual knowledge into LMs. Throughout our experiments, we find that even though current knowledge injection approaches do inject factual knowledge into models, there exist very limited causal relations between the correct- ness of factual knowledge and performance im- provements. Our findings strongly challenge pre- vious assumptions that the injected factual knowl- edge is the core reason of previous factual injec- tion approaches to achieve improvements on down- stream tasks, but very likely due to other confound- ing factors.
The performance comparison and t-test results about ontological substitution are demonstrated in Table 5. Surprisingly, we find that even type-level ontological substitution still has a limited impact on most downstream tasks, even for tasks that sig- nificantly rely entity type information, such as named entity recognition, relation extraction, and entity typing. The results of the t-test further sup- port this finding. In most downstream tasks for all three models, the p-values exceed the thresh- old 0.05, indicating a lack of sufficient evidence to confirm significant difference in model per- formance before and after ontological substitution. The only exception also comes from FewRel in ERNIE, mainly due to information leakage as men- tioned above. Figure illustrates the performance distribution before and after perturbation on sev- cral tasks, which illustrates the limited effect of both factual and ontological substitution in a more
Our conclusions research directions. For example, current extremely large LMs such as GPT-3 (Brown et al., 2020) and LLaMA (Touvron et al., 2023) can perform well on various downstream tasks but would still generate large amounts of incture-incorrect responses, and
also shed light on future

---

our conclusions indicate that we need to improve the evaluation paradigm for comprehensive PLM evaluation on factual knowledge.
Zied Bouraoui, José Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelli- gence Conference, IAAI 2020, The Tenth AAAI Sym- posium on Educational Advances in Artificial Intel- ligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 1466-7463, AAAI Press.
Limitations
This paper focuses on factual knowledge injection, in the future, we can further investigate the im- pact of knowledge injections on other knowledge types such as linguistic knowledge (Ke et al., 2020; Lauscher et al., 2020; Levine et al., 2020; Zhou et al., 2019), syntax knowledge (Zhou et al., 2019; Sachan et al., 2021; Bai et al., 2021) and common- sense knowledge (Bosselut et al., 2019; Guan et al., 2020; Shwartz et al., 2020).
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alce Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtuul.
Due to the huge cost and the limitations of com- puting resources, we have not yet conducted exper- iments on extremely large language models such as GPT-3 (Brown et al., 2020).
Acknowledgement
We sincerely thank the reviewers for their insightful comments and valuable suggestions. This work is supported by the Natural Science Foundation of China (No.62122077 and 62106251). Hongyu Lin is sponsored by CCF-Baidu OpenFund.
Boxi Cao, Hongyu Lin, Xianpei Ham, Fangchao Liu, and Le Sun. 2022. Can prompt probe pretrained language models? understanding the invisible risks from a causal view. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume Long Papers). pages 5796-588666 7796-5888, Dublin, Ireland. Association for Computational Linguistics.
References
Boxi Cao, Hongyu Lin, Xianpei Han, and Le Sun. 2023, The life cycle of knowledge in big language models: A survey. arXiv preprint 2xxiv:3333.076
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, and Yunhai Tong. 2021. Syntax- BERT: Improving pre-trained transformers with syn- tax trees. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 3011-3022 1-3020, Online. Association for Computational Linguisties.
Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingy- ong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable
educated guess? revisiting lan- guage models as knowledge bases. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1tth International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 1860- 1874, Online. Association for Computational Linguistics.
C Alan Boneau. 1960. The effects of violations of assumptions underlying the t test. Psychological bulletin, 57(1):49.
Antoine Bordes, Nicolas Usunier, Alberto García- Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Pro- ceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2787-2795.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettle- moyer. 2018 Ultra-fine entity typing. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 87-96, Melbourne, Australia. Association for Computational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Antoine Bosselut, Hannah Rashkin, Maarten Sap. Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 4762-4779, Flo-
rence, Italy. Association for Computational Linguis- tics.

---

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhi- lasha Ravichander, Eduard Hovy, Hinrich Schützze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transac- lions of the Association for Computational Linguis- tics, 9:1012-1031.
58th Annual Meeting of the Association for Compu- tational Linguistics, pages 7811-7818, Online. Asso- ciation for Computational Linguistics.
Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Min- lie Huang. 2020. SentiLARE: Sentiment-aware lan- guage representation learning with linguistic knowl- edge. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6975-6988, Online. Association for Computational Linguistics.
Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguisties, 8:34-48.
Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- lational Linguistics, 7:452-466.
Minlie Huang. 2020. A Knowledge-enhancet pre- training model for commonsense story generation. Transactions of the Association for Computational Linguistics, 8:98-108.
Riccardo Guidotti. 2022. Counterfactual explanations and how to find them: literature review and bench- marking. Data Mining and Knowledge Discovery, pages 1-55.
Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti. Anna Korhonen, and Goran Glavaš. 2020. Specializing unsupervised pretraining models for word-level se- mantic similarity. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 1371-1383 Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang. Yuan Yao. Zhiyuan Liu, and Maosong Sun. 2018 FewRel: large-scale supervised few-shot relation classification dataset with saate-o--the-aee evaluation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809, Brussels. Belgium. Association for Computational Linguistics.
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020. SenseBERT: Driv- ing some sense into BERT In Proceedings of the 58th Annual Meeting of the Association for Compu- rational Linguistics, pages 4656 6666-667, Online. Asso- ciation for Computational Linguistics.
Bin He, Xin Jiang, Jinghui Xiao, and Qun Liu. 2020a. Kgpim: Knowledge-guide language model pre- training via generative and discriminative learning. arXiv preprint 20X0v52012.03551.
Bin He, Di Zhou, Jinghui Xiao, Xin Jiang, Qun Liu, Nicholas Jing Yuan, and Tong Xu. 2020b. BERT- MK: Integrating graph contextualized knowledge into pre-trained language models. In Findings of the Association for Computational Linguistics; EMNLP 2020, pages 2281-2290, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7871-7880, Online. Association for Computa- tional Linguistics.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qu: Machine reading com- prehension with contextual commonsense reasoning. arXiv preprint arXiv: 1909.00277.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis. Wen-tau Yih, Tim Rock- täschel, et al. 2020b. for knowledge-intensive nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9474.
Joel Jang, Seonghyeon Ye, and Minjoon Seo. 2022. Can large language models truly understand prompts? a case study with negated prompts. ArXiv preprint. abs/2209.1274 12
-uggeeete generation
Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 3250-3258, Online. Association for Computational Linguistics.
Shaobo Li, Xiaoguang Li. Lifeng Shang. Zhenhua Dong, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang. and Qun Liu. 2022. How pre-trained language mod- els capture factual knowledge? causal-inspired analysis. In Findings of the Association for Com- putational Linguistics: ACI. 2022, pages 1720-1732, Dublin, Ireland. Association for Computational Lin- guistics.
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the
Xiao Ling, Sameer Singh, and Daniel S. Weld. 2015. Design challenges for entity linking. Transactions of

---

the Association Computational Linguistics, 3:315- 328.
Yuchen Zhang, and Zhi Zhong, 2013. Towards ro- bust linguistic analysis using OntoNotes. In Proceed- ings of the Seventeenth Conference on Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria. Association for Computational Linguistics.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang. Qi Ju, Haotang Deng, and Ping Wang. 2020. K-BERT: en- abling language representation with knowledge graph. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innova- live Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educa- tional Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 2901-2908. AAAT Press.
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang. Maosong Sun, and Jie Zhou. 2021. ERICA: Improving entity and rela- tion understanding for pre- -raiindd language models via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3350-3363, Online, Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeutlemoee, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv preprint, ab99
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019, Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Alexis Mitchell, Stephanie Strassel, Shudong Huang, and Ramez Zakhary. 2005. Ace 2004 multilin- gual training corpus. Linguistic Data Consortium, Philadelphia, 1:1-1.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li. Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP- -JCNLLP, pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Devendra Sachan, Yuhao Zhang, Peng Qi, and William L. Hamilton. 2021. Do syntax trees help pre-trained transformers extract information? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guisties: Main Volume, pages 2647-2661, Online. Association for Computational Linguistics.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compitational Linguistics: Human Language Technologies. pages 2523-2544, Online. Association for Computational Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint (1XXx::22111.000000
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4615-4629, Online. Association for Computa- tional Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China, Association for Computational Linguistics.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro. Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ArXiv preprint, abs/2206.0066
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. E-BERT: Efffcient-yct-effecttv entity embeddings for BERT. In Findings of the Association for Compu- lational Linguistics: EMNLP 2020, pages 803-818, Online. Association for Computational Linguistics.
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020. CoLAKE: Contextualized language and knowledge embedding. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3660-3670, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng. Anders Björkelund, Olga Uryupina,

---

Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang. Junyuan Shang. Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. 2021. Frnie 3.0: Large-scale knowledge enhanced pre-training for lan- guage understanding and generation. ArXiv preprint, abs/2107.021:1
GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In 7th In- ternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Ruize Wang, Duyu Tang. Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao. Daxin Jiang, and Ming Zhou. 2021a. K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters, In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1405-1418, Online. Association for Computational Linguistics.
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang. Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: Enhanced Rep- resentation through Knowledge Integration. ArXiv preprint, abs/1904.09223.
Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sung- dong Kim, and Jaewoo Kang. 2021. Can language madels he biomedical knowledge bases? In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang. Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021b. KEPLER: A unified model for knowledge embed- ding and pre-trained language representation. Trans- actions of the Association for Computational Linguis- tics, 9: 194.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKEE Deep contextualized entity representations with entity- aware self- attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 6442-6454, On- line. Association for Computational Linguistics.
James Thoree, Andreas Vlachos. Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Jian Yang, Gang Xiao, Yulong Shen, Wei Jiang, Xinyu Hu, Ying Zhang, and Jinghui Peng. 2021. A survey of knowledge enhanced pre-trained models. ArXiv preprint, abs/21 10.00269.
Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-inleeendent named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (C0NL1-2002).
Da Yin, Li Dong, Hao Cheng, Xiaodong Liu, Kei-Wei Chang, Furu Wei, and Jianfeng Gao. 2022. A survey of snowledge-intensive nip with pre-trained language models. ArXiv preprint, 01s/22022.0872..
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 130.
Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng. Zijun Yao, Xiao- han Zhang. Hanming Li, et al. 2023. Kola: Carefully benchmarking world knowledge large language models. arXiv preprint arXiv:2306.09
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 35 45, Copenhagen. Denmark. Association for Com- putational Linguistics.
Peter Unger. 1968. An analysis of factual knowledge. The Journal of Philosophy, pages 177-
Victor Veitch, Alexander D' Amour, Steve Yadlowsky, and Jacob Eisenstein. 2021. Counterfactual invari- ance spurious correlations in text classification. Advances in neural information processing systems, 1111666-66200
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative tities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1451, Florence, Italy. Association for Compu- tational Linguistics.
Denny Vrandežić and Markus Krötzsch. 2014. Wiki- data: a free collaborative knowledgebase. Communi- cations of the ACM, 57(10):78 -85..
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilin- gual training corpus. Linguistic Data Consortium, Philadelphia, 57:45.
Zexuan Zhong and Dangi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.

---

pages 50-61, Online. Association for Computational Linguistics.
A Fine-tuning Details
In the fine-tuning stage, most of the hyper- parameters and model architectures are kept con- sistent with the original papers. Specifically, for ERNIE and K-Adapter, we strictly follow the orig- inal experimental settings. As for BERT, will introduce fine-tuning procedure in detail. For natural language understanding and named entity recognition, we follow Devlin et al. (2019)'s task- specific architecture, which simply incorporates PLMs with an additional output layer. For open do- main question answering and fact checking, we con- struct the datasets from KILT (Petroni et al., 2021) benchmark. Since focus on factual knowledge learned by PLMs, ignore the retrieval stage and instead provide the model with the gold document. For relation extraction, we consider PURE (Zhong and Chen, 2021) as our base architecture, which is an approach for utilizing PLMs relation ex- traction tasks. In order to avoid error propagation, we evaluate the models with the gold entities. In addition, we use the same data split with Zhong and Chen (2021). The detailed metrics and hyper- parameters are shown in Table 7.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]; Learning learning recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017-5033, Online. Association for Computational Linguistics.
Junru Zhou, Zhuosheng Zhang, Hai Zhao, and Shuail- iang Zhang. 2019. Limit-bert: Linguistic informed multi-task bert. ArXiv preprint, abs/1910. 14296..
B Information Leakage Analysis
As we mentioned in Section 4.2, the performance of ERNIE on FewRel demonstrates the relatively larger performance gap before and after pertur- bation. To dive into the underlying reasons, we first analyze the knowledge source of ERNIE and FewRel. And we surprisingly find that they share exactly the same knowledge source. Specifically, FewRel is a relation extraction dataset that anno- tates relations between entities according to Wiki- data taxonomy, and the knowledge embedding used by ERNIE is also trained on Wikidata with TransE algorithm. In that case, the relation information of each input entity pair in FewRel is already learned by knowledge embedding passed through ERNIE. This could lead to severe answer leakage and make the model's ''s outputs more rely on the information leaked from the external knowledge source, further leading to the performance gap.
To further verify and quantify the impact of such information leakage, design simple non- parametric classification model. Specifically, for each input entity pair in FewRel, we acquire the same corresponding entity embedding with ERNIE, and simply use a k-nearest neighbors algorithm (KNN) for the classification regardless of the input

---

CoLA
SST-2
MRPC
STS-B
QQP
MNLI
QNLI
RTE
Avg.
BERT
Vanilla
54.29
91.58
88.70
85.74
88.48
83.84
90.16
59.28
80.26 = 0.33
Factual
52.64
91.74
87.87
85.72
88.37
83.78
90.06
60.36
80.07 = 0.53
Ontological
54.67
91.97
88.70
85.67
88.29
83.75
89.56
58.84
80.18 + 0.36
ERNIE
Vanilla
57.46
91.42
86.17
88.86
89.14
84.24
90.28
66.34
81.74 + 0.59
Factual
55.50
91.82
87.73
88.30
89.08
84.04
90.18
65.94
81.57 = 0.34
Ontological
55.56
91.92
87.02
88.44
89.20
84.04
90.20
66.98
81.67 + 0.16
K-Adapter
Vanilla
62.87
95.99
90.34
92.01
90.67
89.91
93.97
82.82
87.31 = 0.75
Factual
64.58
95.96
91.12
91.81
90.68
89.88
94.31
84.84
87.90 = 0.56
Ontological
65.72
96.24
90.69
91.61
90.71
89.91
94.57
85.70
88.14 = 0.16
Table Results on GLUE dev set.
Hyperparameter
Model ERNIE
Vanilla
Factual
Ontological
Dataset
Task
Metric
Batch
Epochs
88.41
86.88
85.15
GLUE
NLU
2e-5
32
3
NQ
QA
exact match
2o-5
32
5
KNN
69.67
42.10
17.71
FEVER
FC
accuracy
5e-5
32
5
CONI.12000
NER Fl score
2e:5
5
Table 8: The performance of KNN and ERNIE on FewRel before and after perturbation. Vanilla indicates no perturbation, factual indicates factual substitution, and ontological indicates ontological substitution.
OntoNotes 5.0
NER Fl score
5e-5
32
10
ACE2004
RE
Fl score
2c-5
32
10
ACE2005
RE
Fl score
20-5
32
10
Table 7: The metric and hyperparameters of each dataset. *: We follow the metrics mentioned in GLUE (Wang al., 2019) for each tasks. I.R, Batch and Epochs indicate learning rate, batch size and num of epochs, respectively.
C GLUE Performance
Table 6 shows the detailed results GLUE bench- mark. We can find that both factual substitution and ontological substitution show limited influence on most tasks, which consistent with the prior conclusions.
text. Table compares the performance between KNN and ERNIE before and after perturbation. We can see that: 1) The Fl score of KNN without knowledge perturbation achieves 69.67 on FewRel. This model is simple non-parametric classifica- tion model using the same knowledge embedding with ERNIE as input. Such results indicate that the performance of ERNIE may highly rely on the rela- tion information leaked by Wikidata, instead of knowledge injected in the model. 2) The FI score of KNN significantly drops from 69.67 to 40.10 af- ter factual perturbation, and further drops to 17.71 after ontological perturbation, which reaches much larger performance gap caused by knowledge perturbation than ERNIE. Such results indicate that the information leakage is very likely to be the rea- son why the performance of ERNIE on FewRel demonstrates a relatively larger performance gap before and after perturbation.


Reviews:
[{"review_id": "ncuBcGnYGB", "reviewer": "Reviewer_3qik", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and thoughtful suggestions, and we would like to clarify the issues you raised.\n- Regarding the title of our paper: in our initial idea, we think injecting incorrect knowledge into LMs also transforms the original factual knowledge within these models into non-existence, hence, we chose the title \"Does factual knowledge...\" However, your feedback has prompted us to recognize the potential ambiguity associated with this title.  Therefore, we plan to modify the title in the revision to make it more precise and to avoid potential ambiguity.\n- Regarding the scope of our experiments:  \n  - Due the huge computational expenses of our experiments (requiring pre-training each LM from scratch twice), we currently opt to select one representative language model for each of the 3 widely used knowledge injection methods, to ensure the generalizability of our conclusions (please refer to section 3.2 for detail).  \n  - Given the **shared Transformer architecture, type of knowledge source and the fundamental pre-training task (masked language modeling) among BERT, RoBERTa, and ALBERTA**, we chose BERT as the representative model for learning knowledge from plain text, and we think that our conclusions should remain consistent across these 3 models. \n\nTaking into account your valuable feedback, we will include additional experiments involving other pre-trained language models in our revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper targets an interesting problem and gives insightful results.\nThe analyzing procedure is well-designed and convictive.", "reasons_to_reject": "The title is a little misleading since the real problem this paper proposed is \"Does *correct* factual knowledge matter...\" \nIf we want to knowledge \"Does factual Knowledge matter...,\" we need to remove the knowledge from models and measure the differences.\n\nThe limited scale of the experiments may not support the general conclusions proposed in this paper. For example, do these conclusions hold for other pre-trained language models, such as RoBERTa, and ALBERT?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims to check whether the correctness of the knowledge injected by pre-training affects the models' performance on downstream tasks. The authors mainly conducted two types of pre-training, one injects correct knowledge into the models, and the other injects perturbed knowledge. The evaluation results on some downstream tasks show no significant difference between correct knowledge and wrong perturbed knowledge. Based on this observation, the author concludes that the correctness of injected knowledge only has a very limited effect on the models’ downstream performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "huyyTHubAa", "reviewer": "Reviewer_tpiB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and for pointing out the areas where our paper could be further enhanced. \n\nRegarding the selections of representative knowledge augmented pre-trained models, we would likely to kindly clarify our principle for selecting target models. As we mentioned in section 3.2, currently there are two primary knowledge sources for knowledge acquisition by LMs, including plain text and structural KBs. Therefore, we select 3 representative PLMs including BERT, ERNIE and K-ADAPTER, covering both 2 knowledge sources and 3 distinct knowledge acquisition strategies, and the consistent findings among them demonstrate the generalizability of our conclusions.\n\nMeanwhile, the scope of knowledge-enhanced language models is continuously evolving. And we agree with your suggestion that further incorporating the latest knowledge-enhanced LMs could certainly help solidify our points and broaden the research's impact. \n\nIn the future, we will seek more novel knowledge injection strategies and incorporate them into our revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Throughout the proposed counterfactual-based knowledge analysis framework, the authors find that eventhough current knowledge injection approaches do inject factual knowledge into models, there exist very limited causal relations between the correctness of factual knowledge and performance improvements. This is an interesting discovery, which strongly challenge previous assumptions.", "reasons_to_reject": "The authors should add some representative work of current knowledge augmented pre-trained models to increase the convincing power of this paper's arguments.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper mainly demonstrates whether the so-called knowledge injection in the existing pre-training model really has a real impact on related downstream tasks. The authors introduce a counterfactual-based analysis framework to explore the causal effects of factual knowledge injection on the performance of language models within pretrain-finetune paradigm and find the correctness of injected knowledge only has a very limited effect on the models’ downstream performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Oa7Uielqca", "reviewer": "Reviewer_jAz2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we would like to extend our gratitude for your in-depth review and constructive criticism.  \n\n- Regarding the selection of downstream tasks: we have opted for these specific tasks due to prior research, such as ERNIE and K-Adapter, reporting performance enhancements on them through knowledge injection. Therefore, we would like to investigate whether there exist causal relations or just spurious correlations, between knowledge injection and the observed improvements in downstream performance.\n- Regarding the framing of results: firstly, we totally agree with you about \"existing evaluation downstream tasks aren't really meaningful for evaluating whether the model benefits from injecting factual knowledge\", as well as the suggestion that our writing should focus more on evaluation. Meanwhile, in our opinion, the experimental outcomes in our paper concerning both factual perturbation and ontological perturbation can be attributed to two aspects of reasons.\n    - From a task perspective, as you mentioned, part of the downstream tasks can be accomplished without extensive world knowledge, which are not suitable for exiting knowledge-enhanced LMs evaluation. However, this explanation might not entirely account for the outcomes observed in tasks like RE, as well as the outcomes stemming from ontological perturbation.\n    - From a model perspective, our experiments suggest that current knowledge-enhanced LMs might not be able to effectively leverage knowledge acquired from pre-training to downstream tasks via fine-tuning.\n    - Furthermore, we will take these valuable suggestion into account to revise both our title and paper.\n\nWe sincerely thank you for the suggestions about both paper reframing and presentation improvements, which are very insightful and valuable. We will follow the suggestions and carefully revise the paper to make it more precise and comprehensible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is a nice sanity check on knowledge-augmented language models. It's always great to see work that makes us rethink existig methods and evaluation protocols.\n- This paper is technically sound, and I think the experimental setup is appropriate for the claims", "reasons_to_reject": "- I'm not totally sure I agree with this framing of the results (and the framing is somewhat unclear). To me, the core takeaway from these results is that usual downstream evaluations don't really assess knowledge at all---you can be injected with factually incorrect knowledge and still do more-or-less fine.\n  - similarly, I disagree with the premises in the paper that tasks like NER and open-domain QA really require factual knowledge. These are things that could _benefit_ from factual knowledge (e.g., a gazetteer for NER), but you can usually do just fine on NER with minimal world knowledge as long as you're reasonably good at reading comprehension (which these factually-perturbed models seem to be, since they're still pre-trained on a bunch of english).\n- If there was some way of reframing this paper so that it's more centered on the evaluation, I'd like it a lot more. The current evaluations don't really answer the titular question of \"Does Factual Knowledge Matter for Factual Knowledge-Enhanced Pre-trained Language Models?\"---it ultimately depends on what the downstream task is, and this paper, to me, shows that existing evaluation downstream tasks aren't really meaningful for evaluating whether the model benefits from injecting factual knowledge.\n  - Furthermore, the title / much of the paper focuses on _models_, whereas I think this result is a lot more about evaluation / data.\n\n- Anyway, in general, I like this paper and the experiments, so I'd be happy to see it accepted. However, I think the current framing / narrative of the paper somewhat misses the point. It'd be great if the paper could narrow down its claims and make them a bit more forcefully. I know that I can't necessarily force the authors to make these changes, but I really like this paper, and I think it'd be a shame if it were hamstrung by wishy-washy writing. Ultimately just my opinion, though.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores the effect of factuality on methods for factual knowledge injection. In particular, the authors assess:\n\n1. Do knowledge injection methods actually inject knowledge? (yes)\n2. Does the factuality of the injected knowledge affect downstream performance on a variety of tasks? (not really)\n\nExperiments are performed with a variety of models that either learn knowledge through the pre-training process (e.g., BERT MLM), or through adding explicit structured data (e.g., ERNIE).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 2 could be clearer. Maybe arrows between Cicero -> Rome, etc so it's clear what the relations are.\n- 435 4.1 Does Knowledge Injection Works? -> 435 4.1 Does Knowledge Injection Work?\n- The results tables are terribly confusing (e.g., Table 4). \"Vanilla\" seems to mean before knowledge perturbation, and \"Factual\" is after knowledge perturbation? In that case, it's very confusing that the row labeled as \"Factual\" actually refers to the setting where the injected knowledge is _not_ factual. I get that the \"factual\" here refers to \"factual\" vs \"ontological\" perturbtions, but even something like replacing vanilla with \"No Perturbation\" and \"factual\" with \"Factual Perturbation\" would be a lot more clearer."}}]

Author rebuttal:

Meta-review: This paper introduces a novel counterfactual-based analysis framework to investigate the causal effects of factual knowledge injection on pre-trained language model performance within the pretrain-finetune paradigm. The authors challenge existing assumptions by demonstrating that the correctness of injected knowledge has only a limited impact on downstream performance. The findings provide valuable insights and contribute significantly to the understanding of the role of factual knowledge in language models. The reviewers are generally exciting about the insights of this paper. I would therefore recommend to accept this paper to the main conference.
Decision: Accept-Main