Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models
Authors: Paul Youssef
Osman Alperen Koraş
Meijie Li
Jörg Schlötterer
Christin Seifert
Venue: EMNLP 2023 Findings (2023)
Keywords: factual knowledge probing
TL;DR: We survey publications that probe language models for factual knowledge.

Abstract: Pre-trained Language Models (PLMs) are trained on vast unlabeled data, rich in world knowledge. This fact has sparked the interest of the community in quantifying the amount of factual knowledge present in PLMs, as this explains their performance on downstream tasks, and potentially justifies their use as knowledge bases. In this work, we survey methods and datasets that are used to probe PLMs for factual knowledge. Our contributions are: (1) We propose a categorization scheme for factual probing methods that is based on how their inputs, outputs and the probed PLMs are adapted; (2) We provide an overview of the datasets used for factual probing; (3) We synthesize insights about knowledge retention and prompt optimization in PLMs, analyze obstacles to adopting PLMs as knowledge bases and outline directions for future work.

Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models
Paul Youssef Osman Alperen Koras' Meijie Li' Jörg Schlötterer Christin Seifert
"Institute for AI in Medicine (IKIM), University Hospital Essen, University of Duisburg-Essed UUniversity of Mannheim UUniversity of Marburg joerg schloetterer, fesman.koras meijie. Lij@uni-duue de
seistiinnseierree jeeiieetttnniiaaaaaeee Jeuni-mambbrr
Abstract
Non-optimized
Optimized
Sec.
Impur
Cloze prompts Questions
Diversification & mining Direct optimization
Pre-trained Language Models (PLMs) are trained on vast unlabeled data, rich in world knowledge. This fact has sparked the interest of the community in quantifying the amount of factual knowledge present PLMs, as this ex- plains their performance downstream tasks, and potentially justifies their use as knowledge bases. In this work, we survey methods and datasets that are used to prohe PLMS for factual knowledge. Our contributions are: (1) We pro- pose categorization scheme for factual prob- ing methods that is based on how their inputs, outputs and the probed PLMs are adapted; (2) We provide overview of the datasets used for factual probing: (3) We synthesize insights about knowledge retention and prompt opti- mization in PLMs, analyze obstacles to adopt- ing PLMs as knowledge bases and outline di- rections for future work.
Entities
Generation with PLMs
Vanilla
Sec. 2.2
Sec. 2.3
Pretrained
PLM
Adapted
TY
Typed querying
DE
Debiasing
/ Self-supervised
LM
Multi-token entities
Supervised
Figure I: An overview of our categorization scheme of factual knowledge probing methods.
make PLMs more knowledge-rich and obtain gains on various downstream tasks.
Besides the interest in quantifying knowledge for better downstream tasks performance, there is special interest factual knowledge present in PLMs, because they are envisioned to become soft knowledge bases, from which one can easily extract relational knowledge that had been captured during pre- tre-training (Petroni et al., 2019; Sung et al., 2021). Querying PLMs for knowledge would eliminate the complex NLP pipelines used for knowledge extraction, the need for labeled data to train models for relational knowledge extraction, and schema de- signing (Petroni et al., 2019). Furthermore, PLMs would allow users formulate queries knowl- edge bases (KBs) in natural language, which makes them accessible to a wider user base (Heinzerling and Inui, 2021). Despite recent advances enabling smooth conversational interactions, e.g., with Chat- GPT!, factuality is still an open issue (Ray, 2023). Many methods and datasets have been proposed to probe PLMs for factual knowledge. Probing in- volves PLM and dataset. The dataset contains truthful facts. These facts are used to estimate the amount of knowledge PLMs. More specifically, the dataset contains inputs that identify the fact we are looking for, in order to extract from the PLM (c.g., "Dante was born in [MASK]"), and ground
1 Introduction
Pre-trained language models have been a game changer in NLP. Their reliance on large unlabeled corpora for pre-training and the availability of com- putational resources have enabled a speedy scaling of these models. This scaling has been reflected on the performance of numerous downstream tasks in NLP (Devlin et al., 2019; Chowdhery et al., 2022; Touvron al., 2023), and led to the wide adapta- tion of the pre-train then finetune framework.
The success of PLMs is attributed to the rich rep- resentations and the knowledge captured from the pre-training corpora (De Cao et al., 2021; I; Han et al., 2021; Ye et al., 2022). There has, therefore, been huge interest in investigating and quantifying the type and amount of knowledge present in PLMs, e.g., (Davison al., 2019; Jawahar al., 2019; Petroni et al., 2019; Tenney et al., 2019; Roberts al., 2020), in order to have a better understand- ing about which kinds of knowledge are internal- ized during pre- me-traininn, and to develop methods to
hhtp:://opemaiccom//llagagggg

---

answers are indeed correct (e.g.. Florence). The data is often described in terms of relations (e.g.. "place-of-birth" between subjects (e.g.. "Dante") and objects (e.g., "Florence"). To produce prompts, a template is created for each relation (c.g.. '[X] was born in [MASK]"), that is then filled with sub- ject entities. The inputs can also have other forms such questions (e.g., "Where was Dante born?"). truth answers that help evaluate if the retrieved
inputs are not altered in any way.
Cloze prompts are widely used across several methods. Petroni et al. (2019) probe PLMs for factual knowledge by manually constructing cloze- style templates for several relations. Onoe et al. (2022) automatically construct cloze prompts from Wikipedia and Wikidata by masking out spans near entities of interest, in order to evaluate PLMs' knowledge about (unseen) entities. Abaho et al. (2022) construct cloze prompts from annotated PubMed abstracts to use PLMs as health outcome predictors. Chen et al. (2022) finetune PLMs us- ing cloze prompts that consist of task descriptions alongside a few examples to elicit more facts.
In this work, we review recent work about factual knowledge probing. For the survey, we considered papers that cite the seminal work by Petroni et al. (2019) which first introduced the concept of PLMs as KBs.² We make the following contributions: (I) We provide a categorization of factual knowl- edge probing methods that is based on how inputs, PLMs and their outputs are adapted (see Figure 1 and Section 2); (2) We provide an overview of the datasets used for factual knowledge probing and categorize these under three classes based on their goal (Section 3); (3) We synthesize insights about knowledge retention and prompt optimization in PLMs (Section 4), analyze obstacles to adopting PLMs as knowledge bases (Section 5), and outline directions for future work (Section 7). We make our corpus of relevant papers publicly available.
Questions are the second input category. Several Question Answering datasets are used to finetune '5 models (Raffel et al., 2020), and evaluate the amount of knowledge implicitly present in their pa- rameters in (Roberts et al., 2020). Multiple choice questions are used in (Hardalov et al., 2020) by pro- viding PLMs with the questions followed by each option individually. The options are masked, and the final answer is selected based on the normalized log probabilities of the predicted tokens for each option. Kalo and Fichtel (2022) present a dataset based on Wikipedia, where inputs consist of sev- eral questions and answers, i.e., a few examples implicitly indicate the task, and a similar question without an answer for evaluation.
2 Methods for Factual Probing
We categorize factual probing methods based on adaptations to i) input, ii) model, and iii) output. Categories are not mutually exclusive, i.e., one method could adapt input and model simultane- ously. Figure 1 and Table I provide an overview of the probing methods. We only consider prompt- ing methods that have been explicitly used for fac- tual knowledge probing. For a general review of prompting methods, we refer to (Liu al., 2023).
Entities are used in methods that infer relational information or generate descriptions based on these entities. Some methods depend on a simple clas- sifier or cosine similarity between the subject and object representations to determine the presence or absence of a relation. For example, to probe for geographical knowledge, Liétard et al. (2021) use fixed inputs that contain locations (e.g., countries or cities). These inputs are then used to extract representations for the respective locations from PLMs. Using these representations, the authors evaluate based on the ability of a simple classifier to solve certain tasks (e.g., predicting if two coun- tries share border). Dufter et al. (2021) evaluate the amount of knowledge present in static word embed- dings by matching subject entity (the query) to an object entity from a pre-defined set of possible objects based on the cosine similarity between the representations of the subject and object entities. Shi et al. (2021) train generative PLMs to generate entities' descriptions while providing only the en-
2.1 Probing Inputs
We distinguish between non-optimized or fixed in- puts, and optimized inputs that adapted in vari- ous ways to elicit more facts from PLMs.
2.1.1 Non-optimized Inputs
Extracting factual knowledge from PLMs depends on providing them with short inputs that indirectly describe the sought-after information. These meth- ods can take various forms (cloze prompts (Taylor, 1953), questions, or entities). Non-optimized in- puts represent the simplest case, where the probing
Forr more details refer to Appendix A.I

---

tities as inputs, and compare them ground truth descriptions.
ilar to AUTOPROMPT, but allows for the trigger tokens be replaced with vectors from a contin- uous embedding space. In a similar fashion, Qin and Eisner (2021) propose learning an ensemble of continuous prompts per relation. Additionally, they perturb the representations of the prompts in each layer in the probed PLMs using small learn- able vectors. The intuition is to have activation patterns that similar the ones encountered during pre-training, which would make it easier to elicit knowledge from PLMs. Newman et al. (2022) utilize adapters (Houlsby et al., 2019) to map the embedding vectors to continuous prompts in order to make the probed PLMs less sensitive to different phrasings of the same prompts. Saeed and Papotti (2022) augment the masked tokens with a special type of embeddings, called Type Embed- dings. These embeddings are derived from several entities that share the same type, and are shown to help tie the probed PLM's predictions to the ex- pected type of the masked entity. PERO (Kumar and Talukdar, 2021) depends on querying PLMs with prompts containing few training examples (or shots), which demonstrate the task to the queried PLMs. Since PLMs are quite sensitive to the order and the quality of the provided training examples in the prompt, PERO leverages genetic algorithm to find an optimized prompt and a separator token to concatenate the examples in the prompts. (Li et al., 2022c) exploit the symmetry of the task, and optimize prompts in continuous space so that the probability of predicting both the subject and the object is maximized using the resulting prompts.
2.1.2 Optimized Inputs
Probing inputs contribute substantially to the prob- ing procedure. PLMs are sensitive to the in- puts (Petroni et al., 2019; Jiang et al., 2020b; Elazar et al. 2021), and even syntactical variations or dis- tractors, that do not alter the meaning, cause the PLM's predictions to change (Heinzerling and Inui, 2021: Longpre et al., 2021; Pandia and Ettinger, 2021; Podkorytov et al., 2021; Li et al., 2022a). Therefore, depending on the probing inputs, the estimate on factual knowledge we obtain may vary significantly. Optimized inputs represent variations of the inputs, where the inputs are changed to ac- count for the sensitivity of the probed PLMs.
Diversification and mining methods aim to di- versify and optimize prompts by mining Wikipedia or other resources, and selecting the best perform- ing prompts or combination of them. For exam- ple, Jiang et al. (2020b) propose mining- based and paraphrasing- -aaeed approach to create alter- native prompts that outperform manual ones. The final prompts are selected based on their perfor- mance on training set, and can also be combined in an ensemble. Bouraoui et al. (2020) mine for prompts that contain the entities of interest, and filter these based on the ability of the probed PLMs to predict the masked objects. After the filtering step, the remaining prompts are utilized to create dataset that consists of positive inputs, i.e., contain- ing true subject-object pairs, and negative inputs, which contain false pairs. This dataset is then used for the final evaluation.
Generation with PLM methods re-write prompts with the help of secondary PLM. Haviv et al. (2021) re-write manual prompts using another version of the probed model. The re-writing model is trained to produce prompts that help extract more knowledge from the probed one, which is kept unchanged. Zhang et al. (2022) leverage a generative PLM to produce optimized prompts.
Direct optimization methods aim to directly op- timize existing prompts. This optimization happens either in discrete space, to keep the prompts in natural language, or in continuous space where the prompts do not have to correspond to specific tokens from the vocabulary. Optimization could also target only the masked token or the order the examples in the prompt, in case few examples are provided in the prompt to better indicate the task. Shin et al. (2020)'s AUTOPROMPT extends manu- ally created prompts by prompts with pre-defined number of trigger tokens, and employs gradient- based search to sequentially replace the trigger to- kens with concrete tokens. These tokens are chosen to increase the probability of predicting the correct object. OPTIPROMPT (Zhong et al., 2021)(z is sim-
2.2 Probed PLMs
PLMs are probed for knowledge using either their original pre-trained parameters (Petroni et al., 2019; Jiang al., 2020b), after adapting these parame- ters (Roberts al., 2020; Meng et al., 2022b).
2.2.1 Vanilla PLMs
Methods in this category do not induce any changes to the probed PLMs, and depend on pre-training ob-

---

jectives to probe PLMs for factual knowledge. Us- ing the pre-trained parameters is the most straight- forward approach and claimed to preserve the facts learned during pre-training (Elazar et al., 2021; ; Newman et al., 2022).
and are provided with unseen task descriptions and labeled examples to adapt the task. Elazar et al. (2021) further train BERT with a consistency loss to increase its robustness to paraphrases that de- scrihe the same relation. Shi et al. (2021) finetune generative PLMs to generate entity descriptions de- pending only on their knoweldge from pre-training. Qin and Eisner (2021) do not directly change any parameters in PLMs, but rather introduce additional trainable parameters in each layer that change the hidden representations of the prompts help make them more suitable for knowledge extraction.
Most methods leverage the language modeling objectives from pre-training to probe for factual knowledge (Petroni al., 2019; Jiang et al., 2020b; Shin al., 2020; Haviv et al., 2021; Kumar and Talukdar, 2021; Zhong ct al., 2021; Kalo and Fich- tel, 2022; Newman et al., 2022; Onoe et al., 2022; Saeed and Papotti, 2022). Other methods rely on representations that come from the model's body, discarding task-specific parameters altogether (e.g.. the Masked Language Modeling head in BERT-like models) (Liétard al., 2021) or use representations of the subject and object entities in the case of static word embeddings (Dufter et al., 2021).
Self-supervised adaptation. Adaptations in self-supervised manner can introduce changes the model without explicitly finetuning the model to the probing task. For example, Meng et al. (2022h) propose to re-wire the probed PLM in a self-supervised manner. Their method depends on using data from the pre-training phase, splitting each sentence into a head part and tail part, and using contrastive learning objective to push the representations of the matching head and tail pairs (positives) closer to one another, and that of the non-mann ma-maathingg pairs (negatives) to be further apart. The evaluation is based on the similarity between the representations of the prompt and a predefined set of entities that represent potential answers.
2.2.2 Adapted PLMs
Some works adapt the PLMs under evaluation to enable evaluation tasks, that do not correspond to any pre-training objective. The adaptation, how- ever, is also coupled with risks such as train-test overlap (Lewis et al., 2021; Wang et al., 2021a).
Supervised adaptation. Most methods finetune the probed PLMs in supervised manner to adapt them to the probing task. Roberts et al. (2020) fine- tune T5 models for closed book question answer- ing, where models have only questions inputs, while leaving out any context or external knowl- edge sources that might contain the answer. Simi- larly, Wang et al. (2021a) finetune BART to output a related passage, and then the answer. Bouraoui et al. (2020) finetune BERT to classify prompts based on whether the relation between the subject and object entities truly holds or not. Fichtel et al. (2021) finetune BERT model with its masked lan- guage modeling head to predict the masked tokens in the provided prompts. Abaho al. (2022) pro- pose an additional position-attentinn layer on top of transformer models, where the position of the masked token is kept constant, and the remaining tokens are given positions relative to the masked to- ken. This approach is considered to put more focus on the masked tokens and its interaction with the remaining tokens in the prompt. Chen et al. (2022) leverage a task description that depends on the re- lation between the subject and object entity, along- side few labeled examples to train the probed PLMs. At inference time, the PLMs are kept frozen
2.3 Outputs
Methods focusing on the outputs of PLMs address restricting the output space of PLMs, debiasing their outputs, and handling multi-token entities.
Typed querying. Kassner al. (2021) propose to restrict the space of possible values for replacing the masked token (ohject) from the whole vocabu- lary to a specific set of tokens whose type matches the type of the ground truth object. For example, if the PLM is queried with the prompt: "'The smallest country in the world is [MASK]", only entities of type country are considered to replace the [MASK] token. This method has two advantages: it reduces the number of objects under consideration and al- lows for a better comparison across PLMs with different vocabularies (Kassner et al., 2021).
Debiasing. Zhao et al. (2021) identify biases in the predictions of PLMs towards common and re- cent tokens, and propose a method that adapts the output probabilities by first estimating these bi- ases using neutral examples and then correcting them. This debiasing method is shown to reduce

---

the variance across prompts and has a positive ef- fect on fact retrieval. Malkin et al. (2022) propose a method to increase the effect of distant tokens on the predictions of PLMs. The method depends on combining two output distributions over the vocab- ulary. One distribution is based on the full-Iength input, whereas the other is based on a shortened ver- sion of the same input. Wang et al. (2023) identify the problem of object bias in optimized prompts and propose to make all potential objects equally probable when no subject is provided, and increas- ing the probability of the correct object, when the subject is available. Yoshikawa and Okazaki (2023) output predictions only above a sufficient confi- dence threshold. This results in less biased evalu- ation, and reflects the ability of PLMs in excluding uncertain predictions. To address the problems of multiple valid answers and frequency bias, i.e., the co-occurence of some subject and object entities despite not being in a factual relation to one an- other, Dong et al. (2022) use two templates, one contains the correct relation while the other con- tains an erroneous relation between the two entities, and compare the probability for the correct object under both relations.
facts. While 16 datasets are solely English, there are three multilingual datasets (mLAMA (Kass- ner et al., 2021). X-FACTR (Jiang et al., 2020a) and DLAMA (Keleg and Magdy, 2023)). In- dicGLUE (Kakwani et al., 2020) contains 11 In- dic languages. Most datasets consist of cloze prompts, while QA datasets (WebQuestions (Be- rant et al., 2013), TriviaQA (Joshi et al., 2017), NQ (Kwiatkowski al., 2019)), PopQA and Enti- ty tyQuestions (Mallen al., 2023) are also used to quantify factual knowledge (Roberts et al., 2020). Wang et al. (2021a) adapt SQuAD (Rajpurkar et al., 2018) for closed- question answering.
6 out of 8 datasets used for probing domain- specific knowledge target the biomedical do- main (c.g., MedQA (Jin et al., 2021), Bio- LAMA (Sung et al., 2021) and MedLAMA (Meng et al., 2022b)). The multilingual dataset EX- AMS (Hardalov et al., 2020) focuses on scientific QA, whereas LEFT (Ciosici et al.. 2021) contains questions from humanities and social sciences.
The community has constructed further datasets to investigate other aspects of using PLMs as knowledge bases. PARAREL (Elazar et al.. 2021) and its multilingual counterpart mPARA- REL (Fierro and Søgaard, 2022) target the sensitiv- ity of PLMs to paraphrases. Negated/ Negated/Misprimeet Misprimed LAMA (Kassner and Schütze, 2020) focuses on how negaation/misprrmingg affects fact retrieval from PLMs, whereas Pandia and Ettinger (2021) tar- get the effeet of distractors. Updating knowledge in PLMs is considered by Jang et al. (2022a,b); Lee et al. (2022); Meng et al. (2022a); Hase et al. (2023); Hoelscher-Obermalle et al. (2023); Mar- gatina et al. (2023). TEMPLAMA (Dhingra et al., 2022) is concerned with time-dependent facts re- trieval, whereas SituatedQA (Zhang and Choi, 2021) considers both, temporal and geographical contexts. Heinzerling and Inui (2021) use a large dataset to evaluate the knowledge storing and re- trieval capabilities of PLMs, and hence their use as KBs Singhamia et al. (2022) challenge the com- munity to build a KB from PLMs, and provide a dataset to facilitate fact retrieval.
Multi-token entities. To handle multi-token en- tities, Jiang et al. (2020a) propose using a pre- defined number of masked tokens and filling these using different strategies: 1) independent from each other, 2) sequentially (left-to-right for English), 3) starting with the most confident predictions. (Kalin- sky et al., 2023) leverage the masked token repre- sentation to generate multiple tokens using a small generative model.
3 Datasets for Factual Probing
We found a variety of datasets (44 in our corpus) that have been proposed used for probing fac- tual knowledge in PLMs: 18 datasets for probing general knowledge, for domain-specifft knowl- edge and 18 datasets that target other aspects, e.g. consistency of PLMs (cf. Table 2).
Datasets for general knowledge probing are used to quantify generic factual knowl- edge in PLMs with the most prominent being LAMA (Petroni et al., 2019). WIKI-UNI (Cao al., 2021) is similar to LAMA, but with uniform distribution of object entities. LAMA- UHN (Poemer et al., 2020) is a subset of LAMA without casy-to-guess examples. DLAMA (Ke- leg and Magdy, 2023) targets culturally diverse
4 Insights about Knowledge Retention and Prompt Optimization
Two further aspects emerged from the surveyed pupers: i) factors affecting knowledge retention, and ii) whether prompts should be optimized.

---

Paper
Inpot
Ope
Adapt.
Esample
Tested PLMs
Eval.
et (2019)
[MASK].
fairseq feom.
p@k
Transformer XL. BERT
mining trigger prosupts
BERT
[X]
15 the capital of [V].
(2020)
Q
< <A1, A2,
pe
Jiang et (2020s)
MT
Barack obana is [HASK] [MASK] [MASK] by profession.
mBERT. XLM. XLM
Jiang et al. (2020b)
C
DM
prompt mining and paraptrasing
BERT, ERNIE, Know Bert
p01
birectx develaped [MASK] [MASK] released DirectX. DirectX is created by [NASK].
Roberts et (2020)
Q
Who lives in the inpertal palace in Tokyo?
TS BERT, RoBERTa
RM
Shin al. (2020)
DO
prompes optimization in discrete space
p pā
DU renory
branding
MRR
Dulter et al. (2021)
11
sin (scapiiiaaal entity,, <country entitys)
BERT. MBEKT, fantText
p@1
Elazar et al. (2021)
rains PI M with consistency loss Italy is [MAS]] Italy's capital, [MASK].
BERT
por cons,
The
cace
Hichel al. (2021)
Dante was born [MASK].
BERT
per
Haviv et al. (2021)
TM
re-writing with PLM
BERT
will 8 is originally aired [MASK]
Kassner al. (2021)
TY
Berlin is the capital of [MASK]ountry
BERT, MBERT
p@1
Kumar and Talukdar (2021)
DO
examples reardering
BERT
p@
ex2, ex3, lives in <location entity?.
lecated [HASK].
Liéand al. (2021)
BERT. RoBERTa, GPT-2
PER
Qin and Fisner (2021)
prompts optimization continucus space, perturba- tions of representations all layers
BERT, RoRERTa
p
MRR
DJ [V1] [v5] [HASK] [v6]
Shi al. (2021)
B
generatinng descriptions for entities [Carll Nerger]
BAKT, T3
R-L
Wang al. (2021a)
<q> <answer related passagez AAP estimates and corrects biuses was
BART
EM, F1, HE
Zhao al. (2021)
DEB
GPF3
p01
[MASK]
Zhong (2021)
prompes optimization contimuon space
BERT
pet
DX [V1] ... [v5] [MASK]
Ababo et (2022)
Two CMZ patients and norphine patient showed complete [NASK].
BERT, BioBERT, Blomed RoBERTa
EM. PM
SciBERT, UmisBERT
Chen et al. (2022)
<task description> <example>> Dante was born in [MASK].
BERT. DeBERTa
p
Dong et (2022)
C
uses protsbilitiiss for comvet/incoreet relations P(faeasii Obana worked)
TS
False rate
in) P[Hewail
(2022)
Q
<QsAze,
languages
GPT-2,
speak?
et (2022c)
DO
optimiced prompts to predict subject and object
BERT. RoBERTa
P@L. MRR
([HASK]) [V1] [V5] (LMASK))
Malkin et al. (3022)
DFB
combines nan ourpur distriburions
GPT-2, GPT-3
p@
Dante born in [MASK], born [NASK]
Meng (2022b)
tEgvitegrvvir may prevent [HASK], entity)
BERT, BlucBERE BioBERT, TS, BART. PubMedBERR, SeiFive
p@1
Newman et al. (2022)
C
DO
udspter mapping prompts to continous prompts after embedking layer
BERT
p@t cons
[V1] ... [vS] from [NASK] is Canada's capital
Onoe et al. (2022)
C
InRNA veccines] do not affect [MASK].
TS. BART, GPT:Nco
pplk
Snced and Papotri (2022)
no
masked takens with type emhoddings
BERT
p@ pakk
of Obara ([MASK] [TE]).
Zhang (2022)
LM
generating prompts PLM (BART)
BERT
p@l
Marco Benevento and violin yeah much
like trafficking UNESCO partly [MASK].
Kalinsky (2023)
masked token rept. to generate multi
p@
predictions
love [HASK] city.
Wang al. (2023)
DER
reduces object bias
BERT. RoRERTa
po I, MRR. entroyy 1, RC-
The native language
[MASK]. outputs prediction sufficient confidence DX was born [MASK].
Yodhikawa and Okazaki (2023)
BERT. RoRERTa
AUC
Table Overview of probing methods. Input type: cloze prompts questions entities E
Prompt optimizaiini
diversification and mining pm. direct optimization po, generation with PLMs Other methods: debiasing mutli-token entities MT, or typed querying TY. PLM adaptation: supervised (v), or self-supervised (v+). Evaluation: consistency (cons), consistent-accuran (cace), exact match (EM), human evaluation (HE), mean reciprocal rank (MRR), partial match (PM), perplexity (pplx), probe error reduction (PER), ROUGE-L (R-L), and AUC of the risk-coverage curve (RC-AUC).
"adapted and non-adapted PLM

---

Dataset
Cat.
Lang.
Example
Hlust.
Access
LAMA (Petroni al., 2019)
Donte born in [MASK]
Google Analogy(semantic)
CLS
located [X], capitel
(Reuraoui ot all, 2020)
WebQuestions (Roberts et 2020)
What degrees dhana get?
BATS (ency.) (Bouraoui al. 2020)
CLS
on
[X] is the capital of [Y]
0.5k
TriviaQA (Rohens et al., 2020)
en
who won the Nobel Peace Prize in 2009?
96k
NQ (Roberts et al., 2020)
GX
en
Who lives in the imperial palace in Tokyo?
322k
GENERAL KNOWLEDGE
IndicGLUE (Kakwani al., 2020)
indic"
Shanbhupara <HASK> Aarel1 Tehsil, Dujarat State.
importent villege
239k
X-FACTR (liang al., 2020a)
multi
The mother tongue of Obana [MASK]
398k
LAMA-UHI (Poemne al., 2020)
en
naintains diplonatic relations with [M6SK]
LPAQA (Jiang al., 2020h)
en
Directx developed/reeated by [MASK]
mLAMA (Kassner el al., 2021)
muliii
Paris is the capitel of [MASK]
855k
DESCGEN (Shi et ol., 2021)
GX
NLG
en
[Carl Menger] was an Austrian econonist...
37k
WIKI-UNI (Cao et al., 2021)
Gk
en
Turing was barn in [MASK]
70k
SQuAD (Wang et al., 20211)
en
<answer" related passage> <A>
92k
KAMEL (Kalo and Fichtel, 2022)
<ORA>A, What languzges does Confurius speak?
(Keleg Magdy. 2023)
Egypt located in [MASK]
PopQA (Mallen et al., 2023)
en
What is the capital of Loutsiana?
14K
EntityQuestions (Mallen et al., 2023)
en
Who is the author of The Target?
177k
EXAMS (Hardalow al., 2020)
multi
<Q> <AT A2,,,2, A3, + <Ay?
McdQA (Ji et al., 202.1)
DK
en.zh"
<tase> <q> A21,A2,A3, A3,44> - <hp>
61k
DOMAIN-SPECIESS
DisKnE (Alghammi al., 2021)
DX
CLS
ea
The putient has high BP <SEP> Hyype tension
7k
(Yuan al.. 2021)
DK
apraclonidine prevent [MASK]
144K
LEFT (Ciosici al., 2021)
DK CLS
<statements <True/False>
BioLAMA (Sung
Hepatitis has symptons
[MASK]
EBM-NLP (Abaho al., 2022)
en
... patient complete [HASK]
MedLAMA (Meng el al., 20226)
DK
co
Elvitegravrr ney prevent [MASK]
19k
Negated LAMA
en
capital Italy is not [MASK]
(Kassner and Schütoc, 2020)
Misprimed LAMA
Dinosaurs? Munich is located [MASK]
11k
(Kassner and Schiitze, 2020)
ParaRel (Flazar al., 2021)
CD
en
Turing was barn in/is notive to [MASK]
n.a."
(Pandia and Etinger, 2021)
CO
ea
Sebastian lives in France. The capital of
40k
Sebastian's country
SituatedQA (context/ /answer)
CK
Who made the rost point shots In the NBA?
18k
(Zhang Choi, 2021)
(Meinzerling and Inui, 2021)
Turing born in [MASK] Tenatoes are [MASK].
15M
(Podkorytov al., 2021)
en
OTHEX
mParaRel (Piero and Sagaord, 2022)
multi
Turing fron/was born in [HASK]
n.a."
TEMPLAMA (Dhingra et al., 2022)
CK
en
[2012] Cristiano Ronaldo plays for [MASK]
50k
(Singhania et al., 2022)
Kn
en
France shares a land border with [MASK]
(Jang et al., 2022b)
XU
en
DMsKK] is the prime minister of England
30k
TemporalWiii (Jang al., 2022a)
PPL
On December, the Onieron variant... is Tur Turing's en
2sRE
paid player EPL?
168k
CounterFaact (Meng 2022a)
tongue nold,new>
ECBD (Onac al, 2022)
en
[NRNA vaccines] de not affect [MAS]]
(Hase al., 2023)
Kt
en
Mary Lowe Good has relation "wimerr of to [MASK]
17kk
CounterFact-+
KU
ca
The nother tongue of Danielle Darrieux is English.
(Hoelscher-Olee Obbemarrr- 2023)
native language of Montesquteu is [MASK]
DynapicTempLL.ie AMA
KU en
The surnane of the Prina Ministar of the UK is
(Margatina al., 2023)
[MAS]]
Table 2: Datasets for factual knowledge probing. Probed knowledge: general knowledge ox. domain-specific knowledge nk. context-dependent knowledge ex. PL.Ms sensitivity to paraphrases, negation or mispriming io.
related to PLMs as KBs kв knowledge updating kL. misconceptions mc and unseen entities jul NLP task: cloze
prompts question answering Ø classification c... natural language generation NLC. and perplexity PPL. Showing languages, example, and number of instances in the dataset (rounded). Data access: accessible without effort (+), accessible with some effort (o), not accessible (-). @ refers to dynamic datasets, whose number of instances is changeable over time. We only include references to papers, in which the datasets used for factual knowledge probing. References to papers introducing the datasets are added in Table 4 in the appendix.
'11 different languages "including zh-simplified
'number of relations 328, prompts per relation 38
"number of relations 343, prompts per relation 37.13 (avg, over languages)

---

4.1 Factors Affecting Knowledge Retention
Jiang et al., 2020b). These paraphrases might be better fact retrievers because of their similarity the pre-training corpus (Cao et al., 2022). Other prompt optimization methods find better prompts in discrete or continuous spaces (Shin et al., 2020; Zhong et al., 2021). These prompts are largely uninterpretable, and can even retrieve facts from randomly initialized PLMs (Zhong et al., 2021; Ishibashi al., 2023).
PLMs are diverse with respect to their architectures, pre- pre-training objectives and their pre-training data. A compelling question is: how do all these factors affect knowledge retention in PLMs?
Large language models are known to perform generally better and hold more knowledge (Brown et al., 2020; Roberts et al., 2020). However, the model's architecture and pre-training objectives are more decisive for knowledge retention than its size (Li et al., 2022a). For example, pre-training with the Salient Span Masking objective (Guu et al., 2020) helps PLMs to absorb more facts (Roberts et al., 2020; Cole et al., 2023). Similarly, Xiong et al. (2020) demonstrate that training the model to predict if the original entities in the text have been replaced with other entities is beneficial for fact retrieval. More generally, Ye et al. (2021) conclude that masking strategy matching the downstream task, positively affects the performance on that task. A larger pre-training corpus with an encoder- only model (Liu et al., 2020) leads to higher knowl- edge retention (Zhang et al., 2021), but with an cncoder-decoder model (Lewis et al., 2020), larger corpus negatively affects knowledge reten- tion Wang et al. (2021a). Recency (Chiang al.. 2020) and frequency (Kandpal al., 2023), i.e., when and how often the data observed at training, are also essential for knowledge retention.
Performance improvements for optimized prompts can be attributed either to prompts becoming more similar to the pre-training data or overfitting the facts distribution. Evaluation should take the pre-training corpora and the facts distribution in the probing dataset into account (Cao et al., 2021, 2022). Future work should consider adapting prompt optimization methods produce more interpretable prompts, This would keep the performance gains, and increase the trustworthiness of optimized prompts.
Obstacles to Adopting PLMs as KBs
Consistency. A challenge to relying PLMs as knowledge bases is their sensitivity to the in- put queries (Fierro and Søgaard, 2022). PLMs rely on shallow surface features and lexical corre- lations (Kassner and Schütze, 2020; Misra et al., 2020; Poerner et al., 2020; Rogers et al., 2020; Li et al., 2022b), which explains their high sensitivity to the way queries are formulated. Current solu- tions (Elazar al., 2021; Newman al., 2022) train PLMs be robust to variations in inputs, but further improvements are needed to make PLMs reliable knowledge bases. PLMs are known to be highly sensitive to prompts, especially in lan- guages other than English (Fierro and Søgaard, 2022), where less resources are available. Mak- ing PLMs more robust to prompts in non-English languages is promising future work direction.
Larger models and more pre-training data can improve knowledge retention if combined with the right choices for architecture and pre-training ob- jective(s). However, scaling might not be suffi- cient (Kandpal al., 2023). Even though many works propose new architectures and pre- teeettrinin objectives to increase factual knowledge retention in PLMs and their robustness to prompts (Févry et al., 2020; Hosseini et al., 2021; Sadeq et al., 2022; Whitehouse et al., 2022; Min et al., 2023; Zhong al., 2023), this is promising future work direction, as there is more room for improvement.
Interpretability. Identifying where facts are stored and how they are retrieved essen- tial to adopt PLMs as trustworthy knowledge sources. Several approaches locate knowledge in PLMs (Wallat et al., 2020; Podkorytov et al., 2021; Alkhaldi et al., 2022; Dai et al., 2022; Meng et al., 2022a), with different conclusions depending the architecture (e.g., knowledge is located in the middle layers of GPT-like models (Meng et al., 2022a), or in the upper layers in BERT-like mod- cls (Dai et al., 2022)). Another line of work fo- cuses on the data aspect, showing the dependence
4.2 Should Prompts be Optimized?
Prompt Optimizing leads to better probing perfor- mance (Jiang et al., 2020b; Shin et al., 2020; Kumar and Talukdar, 2021; Newman et al., 2022; Zhang et al., 2022) However, it remains unclear whether this improvement is due to optimized prompts leak- ing new knowledge into the probed PLMs.
Optimized prompts can be mere paraphrases of manually created prompts (Bouraoui et al., 2020;

---

of PLMS on word co-oceurrences and positionally close words (Li et al., 2022b), or tracing back pre- dictions to training data (Akyurek et al., 2022: Park et al., 2023). Knowing how PLMs retrieve facts remains challenging, but necessary to make PLMs transparent fact retrievers. The introduction of a fact tracing benchmark (Akyurek et al., 2022) opens the door for works in this direction.
strategies in PLMs based on different levels of KBs supervision. We provide a novel categorization scheme and conduct a systematic analysis of meth- ods for factual knowledge probing that goes beyond
all existing surveys. We additionally provide a cate- gorization of factual probing datasets. Furthermore, we discuss recent findings on knowledge retention, the use of optimized prompts, and challenges with corresponding recent solutions to adopting PL.Ms as KBs, shedding light on several future work di- rections. In contrast to other work, we employed systematic approach to curate and analyze rel- evant literature to a comprehensive and unbiased representation of existing work.
Updating Knowledge. PLMs come with a fixed set of pre-trained parameters that encode knowl- edge about the world. As time passes, this knowl- edge becomes partially outdated. Hence, editing existing knowledge in PLMs and augmenting them with new knowledge is crucial for their use as knowledge bases (Zini and Awad, 2022).
7 Discussion and Future Work
One line of research locates the modules respon- sible for factual predictions and modifies these to update the corresponding facts (Dai et al., 2022; De Cao et al., 2021; Meng et al., 2022a). Other lines of research keep the original PLM unchanged, but augment it with additional parameters to induce the desired changes (Wang et al., 2021b; Lee et al.. 2022), or encode facts with time stamps in PLMs make them "time-aware" (Dhingra al., 2022). When updating facts in PLMs, it crucial that only the targeted facts are affected and that these facts are retrievable using different para- phrases (De Cao et al., 2021; Hase et al., 2023). However, current methods for facts editing (Meng et al., 2022a, 2023) still do not fulfill these require- ments (Hoelscher-Obermiier al., 2023). Meth- ods that introduce additional parameters should be made more scalable (Jang al., 2022b).
Factual probing methods are developed extract as many facts as possible from the new smart pools of knowledge, namely PLMs. This gives us an es- timate about how much PLMs have learned from pre-training, and help us to assess their suitability for use cases such as L.MMs-ss-KBss Improving probing methods should go hand-in-hame with ad- vances in PLMs themselves, to help better assess and make use of PLMs. Our analysis (cf. Section 2) shows that current probing methods focus mostly on one the the three dimensions we use in our cat- egorization (inputs, PLMs, outputs). Introducing adaptations across two or more of these dimen- sions (c.g., optimizing inputs while also debiasing outputs) might lead to further improvements with respect to factual knowledge retrieval.
Besides improving probing methods, it is also essential to pay attention the benchmark datasets, Some probing datasets are shown to be biased towards certain entities (Cao et al., 2021). Con- structing unbiased probing datasets is crucial to have unbiased estimates of factual knowledge in PLMs. At the same time, developing comprehen- sive datasets which correspond to the capacity of the recently published large PLMs, e.g., (OpenAI, 2023; Penedo al., 2023; Touvron al., 2023), is an important future work direction.
6 Related Work
AlKhamissi et al. (2022) elaborate requirements for PLMs as knowledge bases and review recent literature w.r.t. those requirements. These require- ments are widely known (c.g., consistency (Petroni al., 2019) and updating knowledge (De Cao et al., 2021)). Our analysis leads to similar general ob- servations (cf. Section 5), and additionally reviews more recent solutions to these obstacles. Cao et al. (2023) cover probing PLMs as part of the knowl- edge cycle in PLMs, but do not address factual knowledge probing at the same level of detail as we do. Liu et al. (2023) survey prompting meth- ods in detail. However, they cover only a part of factual knowledge probing methods. Safavi and Koutra (2021) survey how PLMs acquire relational knowledge, organizing knowledge representations
We also believe that is necessary for current evaluation schemes to not be limited to counting how often PLMs answer correctly. Instead, we call for a comprehensive evaluation that includes further important factors such as the number and frequency of the answers in the pre- pre-training corpus, creation period of the pre-training corpus, model size, and the number of training epochs.

---

8 Limitations
Badr AIKhamissi, Millicent Li, Asli Celikyilmaz. Mona Diab, and Marjan Ghazvininejad. 2022. A review on language models as knowledge bases. arXiv preprint arXiv:220.00603/
For our corpus construction we relied on all the pub- lications that cited (Petroni ct al., 2019). Although this represents the first work that sparked the com- munity's ss interest in the factual knowledge present in PLMS and their use as KBs, there might be par- allel works or works that go into the same direction but do not directly cite Petroni et al. (2019)'s 's work, which are not included in our corpus. Addition- ally, we relied on the venue information provided by Semantic Scholar's API to filter out irrelevant publications. These information are not always accurate and might have affected our initial corpus. In this work, we focused on works that revolve around factual knowledge, and excluded works that focus on other types of knowledge (e.g., linguistic knowledge and commonsense knowledge). How- ever, there are methods that are used for other types of knowledge that could also be applied to factual knowledge and vice versa. We consciously ex- cluded works that focused on other types of knowl- edge, but this does not mean that such methods are not applicable to factual knowledge probing.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1533-1544 Seattle, Wash- ington. USA. Association for Computational Linguis- tics.
Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7456- 7463.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-191....
Boxi Cao, Hongyu Lin, Xianpei Han, Fangchao Liu, and Le Sun. 2022. Can prompt probe pretrained language models? understanding the invisible risks from a causal view. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 5796-5809 Dublin, Ireland, Association for Computational Linguisties.
Acknowledgements
We thank Jan Trienes, and the three anonymous re- viewers for their insightful comments this work.
Boxi Cao. Hongyu Lin, Xianpei Han, and Le Sun. 2023. The life cycle of knowledge in big language models: A survey. arXiv e-prints, pages arXiv-2303.
References
Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingy- ong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable
Micheal Abaho, Danushka Bollegala, Paula Williamson, and Susanna Dodd. 2022. Position-base based prompting for health outcome generation. In Proce Proceedings of the 21st Workshop Biomedical Language Process- ing, pages 26-36, Dublin, Ireland. Association for Computational Linguistics.
educated guess? revisiting lan- guage models as knowledge bases. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), ), pages 1860-1874, Online. Association for Computational Linguisties.
Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Bin- bin Xiong, lan Tenney, Jacob Andreas, and Kelvin Guu. 2022. Towards tracing knowledge in language models back to the training data. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1429-2446, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yanda Chen. Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2022. Meta-learning via language model in-context tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 719-730, Dublin, Ireland. Association for Computational Lin- guistics.
Israa Alghanmi, Luis Espinosa Anke, and Steven Schockaert. 2021. Probing pre-trained language mod- els for disease knowledge. In Findings of the Associ- ation,for Computational Linguistics: ACL-IJCNLP 2021, pages 3023-3033, Online. Association for Computational Linguistics.
Cheng-Han Chiang, Sung-Feng Huang, and Hung-yi Lee. 2020. Pretrained language model embryology: The birth of ALBERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 681 68133 13-6828, On- line. Association for Computational Linguistics.
Tareq Alkhaldi, Chenhui Chu, and Sadao Kurohashi. 2022. A peek into the memory of T5: Investigat- ing the factual knowledge memory in a closed-book qa setting and finding responsible parts, Journal of Natural Language Processing, 29(3):762-784
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts,

---

Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
knowledge in pretrained language models. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 5937-5947, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Manuel Ciosici, Joe Cecil, Dong-Ho Lee, Alex Hedges, Marjorie Freedman, and Ralph Weischedel. 2021. Perhaps PTLMs should go school task to as- sess open book and closed book QA. In Proceedings of the 2021 Conference on Empirical Methods in Nat- Online
Philipp Dufter, Nora Kassner, and Hinrich Schüitze. 2021. Static embeddings efficient knowledge bases? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2353-2363, Online. Association for Computational Linguistics.
ural Language Processing, pages and Punta Cana, Dominican Republie, Association for Computational Linguistics.
Jeremy R. Cole, Aditi Chaudhary, Bhuwan Dhingra, and Partha Talukdar. 2023, Salient span masking for temporal understanding. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 3052- 3060, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhi- lasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transac- tions of the Association for Computational Linguis- rics, 9:1012-1031.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Voolme 1: Long Papers), pages 8493- 8502, Dublin, Ireland, Association for Computational Linguistics.
Thibault Févry, Livio Baldini Soares, Nicholas FitzGer- ald, Eunsol Choi, and Tom Kwiatkowski. 2020. En- tities as experts: Sparse memory access with entity supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 4937-4951, Online. Association for Computational Linguistics.
Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pre- trained models. In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNI.PIICCNLP), pages 1173-1178, Hong Kong, China, Association for Computational Linguistics.
Leandra Fichtel, Jan- Christoph Kalo, and Wolf-Tilo Balke. 2021, Prompt tuning or fine-tuning - investi- gating relational knowledge in pre-trained language models. In 3rd Conference on Automated Knowledge Base Construction.
Constanza Fierro and Anders Sogaard. 2022. Factual consistency of multilingual pretrained language mod- els. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3046-3051 Dublin, Ireland. Association for Computational Linguisties.
Nicola De Cao, Wilker Aziz, and Ivan Titov, 2021. Edit- ing factual knowledge in language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6491 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Anna Gladkova, Aleksandr Drozd, and Satoshi Mat- suoka. 2016. Analogy- detection morpholog- ical and semantic relations with word embeddings: what works and what doesn't. 't In Proceedings of the NAACL Student Research Workshop. pages 8-15, San Diego, California. Association for Computational Linguisties.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minncapolis, Minnesota. Association for Computational Linguistics.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wa Wei Chang. 2020. REALM: Retrieval- augmented language model pre-training. In Proceed- ings of the 37th International Conference Machine Learning, (CML'20. JMLR.org.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos. Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language mod- els as temporal knowledge bases. Transactions of the Association for Computational Linguisties, 10:257 273.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yugi Huo, Jiczhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre-trained models: Past, present and future. Al Open, 2:225-250.
Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu,
Zhifang Sui, and Lei Li. 2022. Calibrating factual

---

Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslay Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilin- gual question answering. In Proceedings of the 2020 Conference on Empiricul Methods in Natural Lan- guage Processing (EMNLP)) pages 5427-5444, On- line. Association for Computational Linguistics.
Joel Jang, Seonghyeon Ye, Changho Lee. Sohee Yang. Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022a. TemporalWiki: A lifelong benchmark for training and evaluating ever-evolving language models. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6237-62500, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zor- nitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan lyer. 2023. Methods for measuring, up- dating, and visualizing factual beliefs in language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Compu- lational Linguistics, pages 2714-2731, Dubrovnik, Croatia, Association for Computational Linguisties.
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun KIM, Stanley Jungkyu Choi, and Minjoon Seo. 2022b. Towards continual knowledge learning of language models. In Interna- tional Conference on Learning Representations.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3651 3-31-7,,7 Florence, Italy. Association for Computational Linguistics.
Adi Haviy, Jonathan Berant, and Amir Globerson. 2021. BERTese: Learning to speak to BERT. In Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguistics: Main Volume, pages 3618-3623, Online. Association for Computational Linguistics.
Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-FACTR: Multilingual facttal knowledge retrieval from pre- trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 5943-5955 On- line. Association for Computational Linguisties.
Benjamin Heinzerling and Kentaro Inui. 2021, Lan- guage models as knowledge bases: On entity repre- sentations, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 1772-1791 Online. Association for Computational Linguisties.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020b. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Jason Hoelscher-Ohermaier, Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. 2023. Detecting edit failures in large language models: An improved specificity benchmark. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pages 11548-11559, Toronto, Canada. Association for Computational L Linguistics.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng. Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Ap- plied Sciences, 11(14):6421.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computationall Linguistics ( (Vol- ume 1: Long Papers), pages 1601-161 I, Vancouver, Canada. Association for Computational Linguistics.
Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R De- von Hjelm. Alessandro Sordoni, and Aaron Courville. 2021. Understanding by understanding not: Model- ing negation in language models. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguisties:
Human Language Technologies, pages 1301-1312, Online. Association for Computational Linguistics.
Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSuite Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4948- 4961, Online. Association for Computational Lin- guistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In In- ternational Conference on Machine Learning, pages 2790- 2799. PMLR.
Yoichi Ishibashi, Danushka Bollegala, Katsuhito Su- doh, and Satoshi Nakamura. 2023. Evaluating the robustness of discrete prompts. Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 2373- 2384, Dubrovnik, Croatia, Association for Computa- tional Linguistics.
Oren Kalinsky, Guy Kushilevitz, Alexander Libov, and Yoav Goldberg. 2023. Simple and effective multi- token completion from masked language models. In Findings of the Association for Computational Lin- guistics: EACL 2023, pages 2356-2369, Dubrovnik, Croatia. Association for Computational Linguistics.

---

Jan- Christoph Kalo and Leandra Fichtel. 2022. KAMEL: Knowledge analysis with multitoken enti- ties in language models. In Proceedings of the Con- ference on Automated Knowledge Base Construction.
Omer Levy, Minjoon Sco. Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333-342, Vancouver, Canada. Association for Computational Linguisties.
Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2021. AMMUS A survey of transformer-basee pretrained models in natural language processing.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising requence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023, Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 15696-15707 PMLR.
Patrick Lewis. Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open- domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 1000-1008, Online, Association for Computational Linguistics.
Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating knowledge in multiiinuual pretrained language models. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 3250-3258, Online. Association for Computational Linguistics.
Junyi Li, Tianyi Tang, Zheng Gong, Lixin Yang. Zhuo- hao Yu, Zhipeng Chen. Jingyuan Wang, Xin Zhao, and Ji- Ji-Rong Wen. 2022a. ElitePLM: An empirical study on general language ability evaluation of pre- trained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3519-3539, Seatlle, United States. Association for Computational Lin- guistics.
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models; Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational L. Linguistics, pages 781 17, Online. Asso- ciation for Computational Linguistics.
Amr Keleg and Walid Magdy. 2023. DLAMA: A frame- work for curating culturally diverse facts for prob- ing the knowledge of pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6245-6266, Toronto, Canada, Association for Computational Linguistics.
Shaobo Li, Xiaoguang Li, Lifeng Shang, Zhenhua Dong, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang. and Qun Liu. 2022h. How pre-trained language mod- els capture factual knowledge? analysis. In Findings of the Association for Com- putational Linguistics: ACL 2022. pages 1720-1732, Dublin, Ireland. Association for Computational Lin- guistics.
cousal-inspired
Sawan Kumar and Partha Talukdar. 2021. Reorder- ing examples helps during priming-baset few-shot learning. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 4507-4518, Online. Association for Computational Linguistics.
Yiyuan Li, Tong Che, Yezhen Wang. Zhengbao Jiang, Caiming Xiong, and Snigdha Chaturvedi. 2022c. SPE: Symmetrical prompt enhancement for fact prob- ing. In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, pages 1169--11699 Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin. Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wer Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452 466.
Bastien Liétard, Mostafa Abdou, and Anders Søgaard. 2021. Do language models know the way Rome? In Proceedings of the Fourth BlackboxNLPP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 510-517, Punta Cana, Dominican Repub- lic. Association for Computational Linguistics.
Kyungjac Lee, Wookje Han, Scung-won Hwang, Hwaran Lee, Joonsuk Park, and Bang-Woo Lee. 2022. Plug aong-and-payy adaptation for continuously- updated QA. In Findings of the Association for Com- putational Linguistics: ACL 2022, pages 438-447, Dublin, Ireland. Association for Computational Lin- guistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv, 55(9).

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. RoBERTa: A robustly optimized BERT pretraining approach.
Sewon Min, Weijia Shi, Mike Lewis. Xilun Chen, Wen- tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2023. Nonparametric masked language modeling. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2097-2118, Toronto, Canada. Association for Computational Linguisties.
Shayne Longpre. Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7052-7063, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Kanishka Misra, Allyson Ettinger, and Julia Rayz. 2020. Exploring BERT's sensitivity to lexical cues using tests from semantic priming. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pages 4625-4635, Online, Association for Computa- tional Linguisties.
Benjamin Newman, Prafulla Kumar Choubey, and Nazneen Rajani. 2022. P-Adapters: Robustly ex- tracting factual information from language models with diverse prompts. International Conference on Learning Representations.
Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. 2022. Coherence boosting: When your pretrained language model is not paying enough attention. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 8214-8236, Dublin, Ireland. Association for Computational Linguistics.
Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang. lain Marshall, Ani Nenkova, and Byron Wallace. 2018. A corpus with multi-level annotations of pa- tients, interventions and outcomes to support lan- guage processing for medical literature. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1; Long Papers), pages 197-207, Mclbourne, Australia. Association for Computational Linguisties.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non- con-parmettiic mem- ories. In Proceedings of the ō1st Anmual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 9802-9822, Toronto, Canada. Association for Computational Linguistics.
Yasumasa Onoe, Michael Zhang, Eunsol Choi, and Greg Durrett. 2022. Entity cloze by date: What LMs know about unseen entities. In Findings of the Associa- tion for Computational Linguistics: NAACL 2022, pages 693-702, Seattle, United States. Association for Computational Linguistics.
Katerina Margatina, Shuai Wang, Yogarshi Vyas, Neha Anna John, Yassine Benajiba, and Miguel Ballesteros. 2023. Dynamic benchmarking of masked language models on temporal concept drift with multiple views. In Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics. pages 2881-2898, Dubrovnik, Croatia. Association for Computational Linguistics.
OpenAL. 2023, GPT-4 technical report.
Lalchand Pandia and Allyson Ettinger. 2021, Sorting through the noise: Testing robustness of information processing in pre-trained language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 1583- 1596, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and editing factual asso- ciations in GPT. In Advances in Neural Information Processing Systems.
Kevin Meng, Arnah Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023. Mass- editing memory in a transformer. In The Eleventh International Conference on Learning Representa- tions.
Sung Min Park, Kristian Georgiev, Andrew Ilyas. Guil- laume Leclere, and Aleksander Madry. 2023. Trak: Attributing model hehavior at scale. In International Conference on Machine Learning (ICML).
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for Falcon I.M.: Outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116
Zaiqiao Meng, Fangyu Liu, Ehsan Shareghi, Yixuan Su, Charlotte Collins. and Nigel Collier. 2022b. Rewire- then-probe: A contrastive recipe for probing biomed- ical knowledge of pre-trained language models. In Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics ( Volume 1: Long Papers), pages 4798-4810, 10, Dublin, Ireland. As- sociation for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference
Tomas Mikolow, Kai Chen, Greg Corrado, and Jef- frey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint a0Xiv:1301.3781.

---

on Natural Language Processing (EMNLP-IJCNLP). pages 2463-2473, Hong Kong. China. Association for Computational I Linguistics.
Mohammed Saced and Paolo Papotti. 2022. You are my type! type embeddings for pre-trained language models. In Findings of the Association for Computa- rional Linguistics: EMNLP 2022, pages 4583-4598, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Maksim Podkorytov, Daniel Biš, and Xiuwen Liu. 2021. How can the [MASK] know? the sources and limi- tations of knowledge in bert. In 202/ International Joint Conference on Neural Networks (IJCN), pages 1-8.
Tara Safavi and Danai Koutra. 2021. Relational World Knowledge Representation in Contextual Language Models: A Review. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing. pages 1053-1067, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. E-BERT: Seficient--et-effeette for BERT. In Findings of the Association for Compu- tational Linguistics: EMNLP 2020. pages 803-818, Online, Association for Computational Linguistics.
entity embeddings
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee. and Danqi Chen. 2021. Simple entity-centric ques- tions challenge dense retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6138-6148, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5203-5212, Online. Association for Computa- tional Linguistics.
Weijia Shi, Mandar Joshi, and Luke 7ettlemoyer. 2021. DESCGEN: A distantly supervised datasetfor gener- ating entity descriptions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1:
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified tox-xt transformer. Journal of Machine Learning Research. 21(140):
Long Papers), pages 4 415-427, Online, Association for Computational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for SQuAD, In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, Association for Computational Linguistics.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 4222-4233 Online. Association for Computational Linguistics.
Partha Pratim Ray. 2023. ChatGPT: A comprehensive review on background, applications, key challenges. bias, ethics, limitations and future scope. Internet of Things and
Sneha Singhania, Tuan-Phong Nguyen, and Simon Razniewski. 2022. Knowledge base construction from pre-trained language models 2022. In Seman- tic Weh Challenge on Knowledge Base Construction from Pre-trained Language Models. CEUR-WS.
Systems, 3:121-154.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1-5422, Online. Association for Computational Linguistics.
Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sung- dong Kim, and Jaewoo Kang. 2021. Can language models be biomedical knowledge bases? In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, . pages 4723-4734, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Wilson L. Taylor. 1953. "Cloze Procedure": A new tool for measuring readability. Journalism Quarterly, 30(4):415-433.
Nafis Sadeq, Canwen Xu, and Julian McAuley. 2022. InforMask: Unsupervised informative masking for language model pretraining. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 5866-5878, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Ben- jamin Van Durme, Sam Bowman, Dipanjan Das. and Ellie Pavlick. 2019. What do you learn from con- text? probing for sentence structure in contextualized word representations. International Conference on Learning Representations.

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint 2302::::39.1977
in intermediate pre- vre-training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7190-7202, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hiyori Yoshikawa and Naoaki Okazaki. 2023, Selective- LAMA: Selective prediction for confidence-aware
Jonas Wallat, Jaspreet Singh, and Avishek Anand. 2020. BERTnesia: Investigating the capture and forgetting of knowledge in BERT. In Proceedings the Third BlackboxNLP Workshop on Analyzing and Interpret- ing Neural Networks for NLP, pages 174-183, On- line. Association for Computational Linguistics.
evaluation of language models. In Findings of the As- sociation for Computational Linguisties: EACL 2023, pages 2017-2028. Dubrovnik, Croatia. Association for Computational Linguistics.
Zheng Yuan, Yijia Liu, Chuanqi Tan, Songfang Huang. and Fei Huang. 2021. Improving biomedical pre- trained language models with knowledge. Pro- ceedings of the 20th Workshop on Biomedical Lan- guage Processing, pages 180-190, Online. Associa- tion for Computational Linguistics.
Cunxiang Wang, Pai Liu, and Yue Zhang. 2021a. Can generative vre-trained language models serve as knowledge bases for closed-book QA? In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3241-3251 Online, Association for Computational Linguistics.
Michael Zhang and Eunsol Choi. 2021. SituatedQA: In- corporating extra-linguist contexts into QA. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 7371- 7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao. Daxin Jiang, and Ming Zhou. 2021b, K-Adapter: Infusing Knowledge into Pre- Trained Models with Adapters. In Findings of the Association for Computational Linguistics: ACI-IJCNLP 2021, pages 1405-1418, Online. Association for Computational Linguistics.
Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021. When do you need bil- lions of words of pretraining data? In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Popers), pages 1112-1125, Online. Association for Computational Linguistics.
Yuhang Wang, Dongyuan Lu, Chao Kong, and Jitao Sang. 2023. Towards alleviating the object bias in prompt tuning-based based factual knowledge extraction. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4420 4432, Toronto, Canada. Association for Computational Linguistics.
Yue Zhang, Hongliang Fei, Dingcheng Li, and Ping Li. 2022. PromptGen: Automatically generate prompts using generative models. In Findings of the Associ- ationfor Computational Linguistics: NAACL 2022. pages 30-37, Seattle. United States. Association for Computational Linguistics.
Cheaxi Whitehouse, Fenia Christopoulou, and Igna- cio lacobacci. 2022. EntityCS: Improving zero-shot cross-lingual transfer with entity-centric code switch- ing. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 6698-67 14, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing sho-shot performance of language models. In Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020. Pretrained encyclopedia: Weakly supervised knowledge-pretranng language model. In International Conference Learning Representations.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Self-evolution learning for dis- criminative language model pretraining. In Find- ings of the Association for Computational Linguis- tics: ACL 2023, pages 4130-4145, Toronto, Canada. Association for Computational Linguistics.
Jaacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu. and Lingpeng Kong. 2022. ZeroGen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1153-1 1669, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]; Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017-5033, Online. Association for Computational Linguistics.
Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, and Madian Khabsa, 2021. On the influence of masking policies

---

Julia EI Zini and Mariette Awad. 2022. On the explain- ability of natural language processing deep models. ACM Comput, Surv, 55(5).
Which probing methods are used?
2 Which probing methods proposed?
3 Which PLMs probed?
A Corpus Creation & Annotation Methodology
4 Are the probed PLMs fine-tuned?
5 Which knowledge probing datasets are used?
6 What are the sources for the datasets?
A.1 Paper Selection
7 Does the paper present any methods or anal- ysis with respect to factual knowledge prob- ing (e.g.. updability, interpretability, consis- tency)?
We created our corpus of papers about factual knowledge probing by querying the Semantic Scholar API on 31. August, 2023 2:51 pm for all works citing (Petromi al.. 2019), resulting in 1,416 papers. According our knowledge Petroni et al. (2019)'s work was the first to quantify factual knowledge in PLMs, and envisioned using PLMs as KBs. We then separated the papers based on their venue information: the venue papers (1,006 instances), the arXiv papers (375 instances) and the no-venue papers (35 instances) with missing venue information returned by the Semantic Scholar API, We did a web search for the no-venue papers and manually assigned them to their respective group, yiclding a total of 1,008 venue papers. We then proceeded to annotate in two steps, after which our final corpus of papers were reduced to 94 highly relevant papers.
8 Does the paper propose method to inject knowledge into PLM?
9 Is the paper a survey?
Table 3: The information extracted for each paper.
other non-peer-reviewed publications. After dedu- plication, the first annotation step yielded a total of 173 relevant papers.
A.3 Second Annotation Step
We revisited all relevant papers and annotated them based on the questions listed in table 3. We only included publications that perform intrinsic eval- uations (Kalyan et al., 2021) that directly target factual knowledge, and we exclude extrinsic evalu- ations on knowledge-intee intensive tasks. Additionally, we excluded papers that did not train on free-text corpora. For each probing method, we store its name, description and an example. For datasets, we store name, domain (of knowledge), original source (from which the dataset was compiled), number of instances and language of the dataset. We excluded 88 of 173 papers for being irrelevant upon second inspection and added papers, which were not our initial corpus of research papers. These pa- pers introduced relevant datasets. Our final corpus of surveyed papers thus counts 94 highly relevant papers.
A.2 First Annotation Step
Three authors single annotated the peer-reviewed papers as either relevant or not-relevant based on our inclusion and exclusion criteria, discussing certainties with all annotators. When in doubt, the paper was marked relevant for high recall. A paper was marked as relevant based on its title, abstract or body, if it contained one of the following:
Knowledge probing methods, metrics or datasets for quantifying the relational or factual knowl- edge stored in a PLM.
Methods shown increase the amount of rela- tional factual knowledge stored in PLM. Methods update, localize or increase the con- sistency of factual knowledge in PLMs.
We further explicitly included short and long pa- pers, workshop papers, posters, shared tasks, com- petitions and challenge papers with at least 4 pages and explicitly excluded master theses, PhD disser- tations, workshop proposals, workshop reports and
semanticscholar. / citatiiomsFiields-iintess url, title, abstract, yenue,year, referenccctt wtw.renceeccunnccitttiooooo.oooo citationCount, influentiialiiaationcoutt fieldsofStudy, publicationDateellitt-t1e

---

Example
Access
LAMA (Petroii et al., 2019)
CK
en
Danto has born in [MASK]
50k
(Bouranui 2020l) Google
CK CLS
It located in DX. the capital of EV]
(semantic) (Mikolow et al. 2013)
(Roberts at. 2020) WesQuestions illeratt al.
GX
What deprees Obana
GENERAL ANOWLEDGE
(Bouranui et 2020) BATS (ancyclopedic)
CK C1.5
en
DD is the capital of [Y]
0.5%
(Gllddoowa 2016)
(Roberts et al. 2020y TrivisAA (Joohi 2017)
CK
on
Who von the Nobel Peace Prize in 2005?
96k
(Roberts al. 2020) NQ (Knintoossi 2019)
Who lives
imporial. palace Tokye?
523k
IndicCLUE
2020)
Shanchupara ewLSt> is Inportant village
23%
Arreli Tehssl, Gugarat State.
X-FACTR
20250)
CK
muhii
nother congue Chara DUSK]
106K
LAMA-UHN (Pocmer 20200
en
meintains diplossiic relations with DMASK]
LPAQA (Finng et al., 2020k)
GK
en
DirectX is developed/ceested by [MASK]
36
MLAMMA (Kosener nl., 2021)
GK
multi
Paris is the capital of [MASK]
855k
DESCGEN
CK
NL.
[Car] Menger]
Austr ian
WIKI-UNI (Cao al.
cn
firing was born in D4SC].
(Wing al. 202tat (Rajparkar al., 2018)
Gk
Q
cn
<tp -> cansver related passagez dt>
93k
KAMEL (Kalo and Fichtel, 2022)
GK
<08A0 What Tanguagea does Confuzius speak?
47%
DLAMA (Keleg Maply, 2023)
Feynt Toosted in
PopQA (Mallen et 2023)
en
What is the capstal Loutsiana?
(Mallen et 2022)EmdiiyQuusieee (Sciovalino 2t al. 2023)
GK
en
Who is the author of The Target?
177k
2020)
A2, <6,2
DOMAIN-SII SRCINC
MedQA dia et 2012))
DK 0
en.ah
<Casop Op <al A2, <hi>
61k
DiiKnE (Alghanmi et 2021)
DK C1.6
The pationt has high BP <SEP> Hypertonsion
7k
(You 2021)
DK
spraclenzcine prevent (MWSK
144k
(Ciosici all. 2021)
en
<statorents <True/False>
BIOLAMA (Sung at ol., 2021)
DK
en
Hopatitis has sumptons such as [MAS]]
40k
(Absho et al., 2022) EBM P Dyye et al, 2018)
DK
Elvitegravir en
potient showed complete [HASK]
3k
McdLAMA
20226)
DK
(M63K
Nagated LAMA (Kossner
capital of
[MASK]
Misprimed (Kassner and Schilnee, 2021))
en
Dinosaurs? Munich is located [MASK]
ParaRel (Blzzar et 2021)
en
Turring is fromiss born in (HASK]
n.a"
(Pandia and Ettinger, 2021)
CO
en
Sebastian lsves in France. The capital of
40k
Sebastian's country [HISK]
SitustedQA (context) (answer) (Zhang and Choi, 2021)
CK
en
Who made the rost point shots in the NEA?
18k
Oriin
(Huinorllng and Imni, 2021)
KB
en
Turing was bern in DWASK]
15M
(Padinoryto et al., 2021)
MC
en
Tomatoes are [MSS].
0.1k
maaarRell and Sogsand, 2022)
muki
Turing froe/wss
[HASK]
TEMPLAN AMA (Thingra et al., 2027)
en
[211]] Crisciam Bonaldo playa DSSC]
(Singhania et al., 2022)
KB
en
France shares land berder with [MASK]
2k
(lang al., 20226)
KU
[MASK] is the prime ninister of England
30k
TemporalWiki (Jang (las et oll, 2022) mR (lavy et al,, 2017)
2022a)
K1
On Deceeber, Onieron variant...
K11
c
in the most paid player 7P17
CountrFacct (Meug et 2022a)
KU
en
Turdingss nothor tengue is sold, now>
23k
ECBD (Onoe et al. 2002)
UP
[eRNA vaccines] de not affect DUSK].
35k
2003)
K11
Mary food relation 'wineer off [HAS
170k
CounterFact (Hocischer-Obeen Obernaior 2023) DynomicTempLAMM (Margatina al., 2023)
KU
en
nothor congue Cantelle Darrieux English.
The native longuage of Hencesquieu is (HSSK of DSAK]]
Minuster of
Table 4: Datasets for factual knowledge probing. Probed knowledge: general knowledge cx Nomain-speciin knowledge nic, context-dependent knowledge cx, PL.Ms sensitivity to paraphrases, negation or mispriming in. related to PLMs as KBs kв knowledge updating kL. misconceptions mc and unseen entities ug. NLP task: cloze prompts question answering q classification
CLS natural language generation NLG, and perplexity
P9L
refers to dynamic datasets, whose number of instances is changeable over time.
"Il different languages "including zh-simplified
'number of relations 328, prompts per relation
"number of relations 343, prompts per relation 37.13 (avg. over languages)


Reviews:
[{"review_id": "uwmU0UkHFZ", "reviewer": "Reviewer_avt8", "summary": "", "questions": "* Can you describe in detail the main differences and novelty of this review and the other reviews mentioned above?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer avt8 for their feedback and are encouraged that our survey was found easy to follow, with a helpful categorization scheme to organize the comprehensive set of papers. \n\nIn the following we address the two concerns. The first one is closely related to the question for the authors, thus we combine them. \n\n### How our work differs from other surveys:\n \n> many discussions can be found in previous reviews with similar arguments, especially in the most intriguing parts, Section 4 and Section 5, where most of the viewpoints (prompt optimization , consistency, knowledge updating, Interpretability, etc.) have already been summarized in other surveys [1][2][3][4]...\n\n> Can you describe in detail the main differences and novelty of this review and the other reviews mentioned above?\n\nThe reviewer mentioned that our work is similar to other related works. We addressed the relation to other surveys in the paper pdf (L82-86 in the Introduction), limiting ourselves to two works (including [4]). We provide an extension to this part of our work (relation to other surveys) below, as we would include it in a CR version. Additionally, we detail the differences between our work and the 4 works mentioned by the reviewer below. We would also gladly discuss these references in detail in the CR version if the reviewer deems it beneficial. \n\n**Relation to the Surveys (paper version):**\n\nAlKhamissi et al. [2] elaborate requirements for PLMs as knowledge bases and review recent literature w.r.t. those requirements. These requirements are widely known (e.g., consistency [6] and updating knowledge [7]). Our analysis leads to similar general observations (cf. Section 5), and additionally provides more recent solutions to these obstacles (cf. Section 5 and  “Details on Relation to [2]”). Cao et al. [3] cover probing PLMs as part of the knowledge cycle in PLMs, but do not address **factual** knowledge probing at the same level of detail as we do. Liu et al. [4] survey prompting methods in detail. However, this covers only a part of factual knowledge probing methods. Safavi and Koutra [5] survey how PLMs acquire relational knowledge, organizing knowledge representations strategies in PLMs based on different levels of KBs supervision. In contrast to all existing surveys, we conduct a systematic analysis of methods for factual knowledge probing, providing a categorization scheme (that is admittedly novel and well-organized). We additionally provide a categorization of factual probing datasets. Furthermore, we discuss recent findings on knowledge retention, the use of optimized prompts, and challenges with corresponding recent solutions to adopting PLMs as KBs, shedding light on several future work directions. In contrast to other works [2,3,4,5], we employed a systematic approach to curate and analyze relevant literature. This ensured a comprehensive and unbiased representation of existing works.\n\n\n**Details on Relation to [1]:**\n\n[1] is a position paper arguing that PLMs cannot replace KBs. In our work, the potential use of PLMs as KBs is part of the motivation for the survey. In contrast to [1], we do not argue for or against PLMs as KBs, but rather focus on the tool (probing) that would help the community decide in which direction to go. We do cover obstacles to adopting PLMs as KBs in Section 5, in order to review recent solutions that address these obstacles, and to provide directions for future work. In contrast to us, [1] does not cover probing methods and datasets, knowledge retention, and prompts optimization. Even with respect to PLMs-as-KBs, recent solutions for the main challenges (Section 5) are not covered by [1]. To illustrate this on the topic of “Provenance”, [1] states that “LMs have no ability to trace their outputs back to specific source documents (and passages) in the training data”. However, recent work (Towards Tracing Knowledge in Language Models Back to the Training Data (Akyurek et al., 2022)) has introduced a fact tracing benchmark analyzing this issue, laying the groundwork for future research in this direction, and showing that it is possible, albeit challenging, to attribute generated knowledge to training data. \n\n**Details on Relation to [2]:**\n\n[2] focuses on PLMs as KBs, and which requirements PLMs should fulfill to become alternatives for KBs. It does not cover probing methods at the same level of detail and depth as we do, and does not provide any categorization for these as we do (see Table 1). Probing datasets (see Table 2) are also not included in [2], and it does not discuss knowledge retention and prompts optimization as we do. [2] intersects with Section 5 in our work, where we review challenges and solutions to PLMs-as-KBs. Compared to [2] we cover more recent solutions to the identified problems in a dynamic and rapidly changing area of research (e.g., Towards Tracing Knowledge in Language Models Back to the Training Data (Akyurek et al., 2022), Knowledge Neurons in Pretrained Transformers (Dai et al., 2022), Plug-and-Play Adaptation for continuously-updated QA (Lee et al., 2022)).\n\n**Details on Relation to [3]:**\n\n[3] reviews the life cycle of knowledge in PLMs, in which knowledge probing is a part. While [3] is more general and considers knowledge probing in **general**, we focus on **factual** knowledge probing. As a result, our work is more comprehensive and in-depth with respect to factual knowledge probing. To illustrate this, [3] mentions only 4 factual probing datasets (see Table 1 in [3]), whereas our work provides a categorization of more than 35 datasets (see Table 2). Furthermore, some recent important works about knowledge probing are missing  in [3] (e.g., Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models (Meng et al.,2022), PromptGen: Automatically Generate Prompts using Generative Models (Zhang et al., 2022), Entity Cloze By Date: What LMs Know About Unseen Entities (Onoe et al., 2022)).\n\n\n**Details on Relation to [4]:**\n\nWe briefly mentioned the relation to [4] in the Introduction L84-85, and provide more details here: [4] reviews prompting methods in general without a structured literature review. Our work systematically selects and analyzes methods for factual knowledge probing, which include but are not limited to prompting methods. For example, probing datasets are not included in [4], and it does not discuss knowledge retention and recent solutions to LMs-as-KBs. Besides, [4] does not cover works that adapt PLMs (e.g., How Much Knowledge Can You Pack Into the Parameters of a Language Model? (Roberts et al., 2020), Meta-learning via Language Model In-context Tuning (Chen et al., 2022)).\n\n\n[1] Language Models As or For Knowledge Bases - Workshop on Deep Learning for Knowledge Graphs 2022  nonarxiv version https://ceur-ws.org/Vol-3034/paper2.pdf \n\n[2] A Review on Language Models as Knowledge Bases, arXiv version April 2022 - according \nto the first authors web-page ((https://bkhmsi.github.io)  this is under review at JAIR\n\n[3] The Life Cycle of Knowledge in Big Language Models: A Survey, arXiv version March 2023 - this might be under submission or accepted and not yet published, the first author's homepage (https://c-box.github.io) mentions \"Machine Intelligence Research\" for this paper \n\n[4] (reference [Liu et al., 2023] in the paper pdf)  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing - ACM CSUR Jan 2023 https://dl.acm.org/doi/full/10.1145/3560815\n\n[5] (reference [Safavi and Koutra, 2021] in the paper pdf) Relational World Knowledge Representation in Contextual Language Models: A Review - EMNLP 2021\n\n[6] (reference  [Petroni et al., 2019] in the paper pdf) Language Models as Knowledge Bases? - EMNLP 2019\n\n[7] (reference [De Cao et al., 2021] in the paper pdf) Editing Factual Knowledge in Language Models - EMNLP 2021\n\n\n### Extending Future Work: \n> For a comprehensive review, the discussion on future directions and conclusions appears to be too brief, limiting the ability of future researchers to gain deeper insights.\n\nOur short future work section is attributed to the space constraints (the - in our view - very relevant tables 1 and 2 require 2 pages in total already). While we covered the most relevant and broadly acknowledged directions, there are more intriguing avenues for future work. With the additional space allowed upon acceptance, we will include discussions on the following further directions:\n1. Probing Methods: \n    1. Current probing methods focus mostly on one of the three dimensions (inputs, PLMs, outputs). Simultaneously adapting two or more of these dimensions might lead to further improvements (e.g., optimizing inputs while also debiasing outputs). \n2. Probing Datasets: \n    1. Some probing datasets are biased towards certain entities [A]. Constructing unbiased probing datasets is essential to have unbiased estimates of factual knowledge in PLMs. \n3. Knowledge Retention:\n    1. Some architectures and pre-training objectives contribute positively to producing factual outputs [B]. Further developing architectures and training objectives is essential for more progress. \n4. Prompts Optimization:\n    1. Current methods for discrete prompts optimization lead to better performance, but produce highly un-interpretable prompts [C]. Adapting these methods to produce interpretable prompts would keep the performance gains, and increase the trustworthiness of optimized prompts. \n5. Obstacles to PLMs as KBs\n    1. Consistency - Multilingual PLMs are sensitive to paraphrased prompts especially in languages other than English [D]. Making PLMs more robust in languages other than English and low-resource languages is crucial for their usability across languages. \n    2. Interpretability - Tracing the factual knowledge of PLMs back to the data it was learnt from is crucial for interpreting the extracted knowledge. This can easily be accomplished for KBs, but remains an open challenge for PLMs. However, the introduction of a fact tracing benchmark [E] opens the door for works in this direction. \n    3. Updating Knowledge - When updating facts in PLMs, it is essential that only the targeted facts are affected and that these facts are retrievable using different paraphrases [F]. Current methods still do not fulfill these requirements. \n6. Evaluation: \n    1. Currently, factual knowledge in PLMs is mostly evaluated based on how often can PLMs provide correct predictions to a set of inputs. This kind of evaluation neglects many other crucial factors (the number and frequency of the answers in the pre-training corpus, creation period of the pre-training corpus, size of the model, number of training epochs, etc). We call for a more comprehensive evaluation that takes all of these factors into account.\n   \n[A]: Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases (Cao et al., 2021)\n\n[B]: REALM: Retrieval-Augmented Language Model Pre-training (Guu et al., 2020)\n\n[C]: Factual Probing Is [MASK]: Learning vs. Learning to Recall (Zhong et al., 2021)\n\n[D]: Factual Consistency of Multilingual Pretrained Language Models (Fierro & Søgaard, 2022)\n\n[E]: Towards Tracing Knowledge in Language Models Back to the Training Data (Akyurek et al., 2022)\n\n[F]: Editing Factual Knowledge in Language Models (De Cao et al., 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper offers a comprehensive review of factual knowledge probing, the paper is well-written and easy to follow.\n*  The categorization schema is novel and well-structured, providing a clear approach to organizing relevant studies.\n* The article provides a comprehensive summary of the existing probing methods and datasets (as presented in Table 1 and Table 2).", "reasons_to_reject": "* Regarding the novelty of the study: many discussions can be found in previous reviews with similar arguments, especially in the most intriguing parts, Section 4 and Section 5, where most of the viewpoints (prompt optimization , consistency, knowledge updating, Interpretability, etc.) have already been summarized in other surveys [1][2][3][4]...\n* For a comprehensive review, the discussion on future directions and conclusions appears to be too brief, limiting the ability of future researchers to gain deeper insights.\n\n[1] Language Models As or For Knowledge Bases\n\n[2] A Review on Language Models as Knowledge Bases\n\n[3] The Life Cycle of Knowledge in Big Language Models: A Survey\n\n[4] Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing", "questions_to_authors": "* Can you describe in detail the main differences and novelty of this review and the other reviews mentioned above?", "paper_topic_and_main_contributions": "*  This paper provide a comprehensive review on factual knowledge probing in Pre-trained language models. They categorize relevant studies based on probing inputs, outputs and target LMs.\n*  Moreover, this paper provides interesting discussion about knowledge retention, prompt optimization and obstacles of PLMs-as-KBs.\n\n\n---\n\nAfter rebuttal: thanks for providing such a comprehensive response. \nOverall, I appreciate the author's meticulous review and summarization of the entire field.\n\nRegarding the response about differences between other surveys:\n> In contrast to all existing surveys, we conduct a systematic analysis of methods for factual knowledge probing, providing a categorization scheme (that is admittedly novel and well-organized). We additionally provide a categorization of factual probing datasets. Furthermore, we discuss recent findings on knowledge retention, the use of optimized prompts, and challenges with corresponding recent solutions to adopting PLMs as KBs, shedding light on several future work directions.\n\nI appreciate the categorization of relevant studies and probing datasets.\nHowever, I still think that this review lacks novel insights: knowledge retention (the conclusion seems to be obvious), the use of optimized prompts can be found in [3], the discussions about LMs-as-KBs can be found in both [1][2][3].\n\nOverall, I'm willing to increase the score on \"Excitement,\" **but I strongly recommend that the author incorporate more novel discussions in the revision**.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FF75D5rJpW", "reviewer": "Reviewer_nj8g", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer nj8g for their feedback and are encouraged that our survey was found well-organized, comprehensive, and beneficial for researchers in this area. \n\nIn the following we address the reviewer’s concerns. \n\n### Survey Insights and Future Work Expansion: \n> The insight part of the survey is not very exciting. The future work is too short and hasty.\n\nWe are not quite sure which \"insights part\" the reviewer is referring to. Section 4 \"Insights about Knowledge Retention and Prompt Optimization\" discusses insights from our systematically collected papers - and addresses what we know and don't yet know about i) factors affecting knowledge retention and ii) pros and cons of prompt optimization. Section 5 reviews obstacles that currently prevent PLMs from being adopted as KBs. Our future work section was held very brief due to space constraints (the key tables 1 and 2 require 2 pages already). However, with one more page available upon acceptance, we will adapt future work to include the following more detailed and intriguing directions (these are mentioned for Reviewer avt8 as well):\n\n1. Probing Methods: \n    1. Current probing methods focus mostly on one of the three dimensions (inputs, PLMs, outputs). Simultaneously adapting two or more of these dimensions might lead to further improvements (e.g., optimizing inputs while also debiasing outputs). \n2. Probing Datasets: \n    1. Some probing datasets are biased towards certain entities [A]. Constructing unbiased probing datasets is essential to have unbiased estimates of factual knowledge in PLMs. \n3. Knowledge Retention:\n    1. Some architectures and pre-training objectives contribute positively to producing factual outputs [B]. Further developing architectures and training objectives is essential for more progress. \n4. Prompts Optimization:\n    1. Current methods for discrete prompts optimization lead to better performance, but produce highly un-interpretable prompts [C]. Adapting these methods to produce interpretable prompts would keep the performance gains, and increase the trustworthiness of optimized prompts. \n5. Obstacles to PLMs as KBs\n    1. Consistency - Multilingual PLMs are sensitive to paraphrased prompts especially in languages other than English [D]. Making PLMs more robust in languages other than English and low-resource languages is crucial for their usability across languages. \n    2. Interpretability - Tracing the factual knowledge of PLMs back to the data it was learnt from is crucial for interpreting the extracted knowledge. This can easily be accomplished for KBs, but remains an open challenge for PLMs. However, the introduction of a fact tracing benchmark [E] opens the door for works in this direction. \n    3. Updating Knowledge - When updating facts in PLMs, it is essential that only the targeted facts are affected and that these facts are retrievable using different paraphrases [F]. Current methods still do not fulfill these requirements. \n6. Evaluation: \n    1. Currently, factual knowledge in PLMs is mostly evaluated based on how often can PLMs provide correct predictions to a set of inputs. This kind of evaluation neglects many other crucial factors (the number and frequency of the answers in the pre-training corpus, creation period of the pre-training corpus, size of the model, number of training epochs, etc). We call for a more comprehensive evaluation that takes all of these factors into account.\n   \n[A]: Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases (Cao et al., 2021)\n\n[B]: REALM: Retrieval-Augmented Language Model Pre-training (Guu et al., 2020)\n\n[C]: Factual Probing Is [MASK]: Learning vs. Learning to Recall (Zhong et al., 2021)\n\n[D]: Factual Consistency of Multilingual Pretrained Language Models (Fierro & Søgaard, 2022)\n\n[E]: Towards Tracing Knowledge in Language Models Back to the Training Data (Akyurek et al., 2022)\n\n[F]: Editing Factual Knowledge in Language Models (De Cao et al., 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This survey is well-organized  and incorporates numerous relevant studies, reflecting considerable effort and dedication in its compilation. This survey can help people systemically access the recent studies about factual knowledge probing in PLMs.", "reasons_to_reject": "The insight part of the survey is not very exciting. The future work is too short and hasty.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This survey examines the works that deals with factual knowledge in pre-trained language models, encompassing various methods and datasets designed for factual knowledge probing.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mczqpKOSgN", "reviewer": "Reviewer_ASH9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ASH9 for their feedback acknowledging that our survey is the first to address the topic of factual knowledge probing in PLMs. We address the reviewer's concerns (missing representative work, and existing surveys) below.\n\nIn the following we address the reviewer’s concerns.\n\n### The recency of the works covered in our survey:\n> **Some representative works of this year are missing in this survey,** and there are too many references to existing related surveys.\n\nThe reviewer mentioned that our survey does not cover representative works from 2023. Indeed, we did not cover recent works from 2023. We will integrate novel work in the final version of the paper. To assess whether novel papers would impact the main message or structure of our paper, we systematically analyzed the papers of this year's passed main conferences EACL 2023 and ACL 2023. We found 9 relevant papers which we can easily integrate in our current structure. Our findings were the following: \n-  3 probing methods that target the outputs of PLMs. These fall in the **Outputs** category in our taxonomy of probing methods. \n- 3 datasets that target different aspects (culturally diverse facts, temporal drifts, editing facts in PLMs). These can also be added to the **General Knowledge** and **Other** datasets categories we have. \n- 2 papers that propose pre-training objectives to increase knowledge retention. These can be integrated in Section 4.1 **Factors Affecting Knowledge Retention**. \n- 1 paper that targets evaluating editing facts in PLMs. This fits into Section 5 under **Updating Knowledge**.\n\n### How our work differs from other surveys:\n> Some representative works of this year are missing in this survey, **and there are too many references to existing related surveys.**\n\nThe reviewer also remarked that “there are too many references to existing related surveys.” Since we only cite two related surveys in our work (see Introduction L82:85), we believe the reviewer was alluding to other related surveys that we did not cite. Here, we clarify how our survey differs from other surveys, extending the corresponding part in the paper pdf (L82:86 in Introduction) (for more details please check our response to Reviewer avt8): \n\n**Relation to the Surveys (paper version):**\nAlKhamissi et al. [2] elaborate requirements for PLMs as knowledge bases and review recent literature w.r.t. those requirements. These requirements are widely known (e.g., consistency [6] and updating knowledge [7]). Our analysis leads to similar general observations (cf. Section 5), and additionally provides more recent solutions to these obstacles (cf. Section 5 and  “Details on Relation to [2]”). Cao et al. [3] cover probing PLMs as part of the knowledge cycle in PLMs, but do not address factual knowledge probing at the same level of detail as we do. Liu et al. [4] survey prompting methods in detail. However, this covers only a part of factual knowledge probing methods. Safavi and Koutra [5] survey how PLMs acquire relational knowledge, organizing knowledge representations strategies in PLMs based on different levels of KBs supervision. In contrast to all existing surveys, we conduct a systematic analysis of methods for factual knowledge probing, providing a categorization scheme (that is admittedly novel and well-organized). We additionally provide a categorization of factual probing datasets. Furthermore, we discuss recent findings on knowledge retention, the use of optimized prompts, and challenges with corresponding recent solutions to adopting PLMs as KBs, shedding light on several future work directions. In contrast to other works [2,3,4,5], we employed a systematic approach to curate and analyze relevant literature. This ensured a comprehensive and unbiased representation of existing works.\n\n\n[1] Language Models As or For Knowledge Bases - Workshop on Deep Learning for Knowledge Graphs 2022  nonarxiv version https://ceur-ws.org/Vol-3034/paper2.pdf \n\n[2] A Review on Language Models as Knowledge Bases, arXiv version April 2022 - according to the first authors web-page ((https://bkhmsi.github.io)  this is under review at JAIR\n\n[3] The Life Cycle of Knowledge in Big Language Models: A Survey, arXiv version March 2023 - this might be under submission or accepted and not yet published, the first author's homepage (https://c-box.github.io) mentions \"Machine Intelligence Research\" for this paper \n\n[4] (reference [Liu et al., 2023] in the paper pdf)  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing - ACM CSUR Jan 2023 https://dl.acm.org/doi/full/10.1145/3560815\n\n[5] (reference [Safavi and Koutra, 2021] in the paper pdf) Relational World Knowledge Representation in Contextual Language Models: A Review - EMNLP 2021\n\n[6] (reference  [Petroni et al., 2019] in the paper pdf) Language Models as Knowledge Bases? - EMNLP 2019\n\n[7] (reference [De Cao et al., 2021] in the paper pdf) Editing Factual Knowledge in Language Models - EMNLP 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors propose a categorization scheme for factual probing methods and provide an overview of the datasets used for factual probing, then synthesize insights about knowledge retention and prompt optimization in PLMs, analyze obstacles to adopting PLMs as knowledge bases and outline directions.", "reasons_to_reject": "Some representative works of this year are missing in this survey, and there are too many references to existing related surveys.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper survey methods and datasets that are used to probe PLMs for factual knowledge, which is the first to address the topic of factual knowledge probing in PLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a survey paper on factual knowledge probing for pretrained language models. It categorizes and reviews probing methods and datasets. It also discusses knowledge retention and prompt optimization in further detail.

The paper is fairly comprehensive in reviewing past work in an important and well-studied area. However, reviewers have pointed out that it is quite similar to other recent surveys. There are indeed differences between this paper and each of the individual previous papers especially in terms of scope and depth. However, this paper provides incremental insights over recent surveys taken together.
Decision: Accept-Findings