Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models
Authors: Amirhossein Kazemnejad
Mehdi Rezagholizadeh
Prasanna Parthasarathi
Sarath Chandar
Venue: EMNLP 2023 Findings (2023)
Keywords: Knowledge Acquisition
Knowledge Utilization
Pretrained Language Models

Abstract: While pre-trained language models (PLMs) have shown evidence of acquiring vast amounts of knowledge, it remains unclear how much of this parametric knowledge is actually usable in performing downstream tasks. We propose a systematic framework to measure parametric knowledge utilization in PLMs. Our framework first extracts knowledge from a PLM's parameters and subsequently constructs a downstream task around this extracted knowledge. Performance on this task thus depends exclusively on utilizing the model's possessed knowledge, avoiding confounding factors like insufficient signal. As an instantiation, we study factual knowledge of PLMs and measure utilization across 125M to 13B parameter PLMs. We observe that: (1) PLMs exhibit two gaps - in acquired vs. utilized knowledge, (2) they show limited robustness in utilizing knowledge under distribution shifts, and (3) larger models close the acquired knowledge gap but the utilized knowledge gap remains. Overall, our study provides insights into PLMs' capabilities beyond their acquired knowledge.

Measuring the Knowledge Acquisiiio-UUtlizztoon Gap in Pre-trained Language Models
Amirhossein Kazemnejad 1.2 Mehdi Rezagholizadeh"
Prasanna Parthasarathi Sarath Chandar
McGill University; Mila Quebee AI; Huawei Noah's Ark Lab;
4 École Polytechnique de Montréal; *Canada CIFAR AI Chair;
amirhossein. jwa.rrossiinkkaeeneeeeeeeee jademaile@ammmlm... mcgi 11.111..6
Abstract
OPT
LLaMa
Fraction of Encycloodic Ficts
While pre-trained language models (PLMs) have shown evidence of acquiring vast amounts of knowledge, it remains unclear how much of this parametric knowledge is actually usable in performing downstream tasks. We propose systematic framework to measure parametric knowledge utilization PLMs. Our framework first extracts knowledge from PLM's parame- ters and subsequently constructs a downstream task around this extracted knowledge. Perfor- mance on this task thus depends exclusively on utilizing the model's possessed knowledge, avoiding confounding factors like insufficient signal. Employing this framework, we study factual knowledge of PLMs and measure uti- lization across 125M to 13B parameter PLMs. We observe that: (1) PLMS exhibit two gaps in acquired vs. utilized knowledge, (2) they show limited robustness in utilizing knowledge under distribution shifts, and (3) larger mod- els close the acquired knowledge gap but the utilized knowledge gap remains.
0.4
0.2
125M
350M
1.38
2.78
76
136
7R
139
Number Parameters (log scale)
Usable
Figure 1: Parametric knowledge of PLMs Gap
represents the missing facts the model's parametric knowledge (what the model knows). Gap exists in how much of this knowledge can actually be utilized downstream tasks (the usable knowledge). We find that although the first gap mostly shrinks, the second remains as we increase the model's size.
given the prompt "The capital of France is", PLM can be queried to complete the input and extract the fact "Paris".
1 Introduction
A common assumption about parametric knowl- edge is that if the model poses a certain type of knowledge, it utilizes it when performing down- stream tasks related to that knowledge. For exam- ple, if a model knows about X and Y (such that X and are similar), and taught to perform a task on X, the convention that the model generalizes the application of the task on Y and all other similar knowledge Such is the foundation for the recent interest in instruction tuning (Wei et al., 2022a; Chung et al.. 2022), and the SFT-RLHF pipeline (Ouyang et al., 2022). In this paradigm, LLMs are finetuned learn how to follow instructions on a few tasks the model is capable of and are subse- quently expected to generalize and follow instruc- tions for novel tasks by utilizing their pre-training knowledge (residing in their parameters).
Recent research has demonstrated that language models pre-trained on vast amounts of internet data acquire broad range of knowledge about linguis- tic structures (Tenney et al., 2019b; Blevins et al., 2022), eneyclopedic relations (Petroni et al., 2019; Hao et al., 2022), levels of commonsense (Zhou al., 2020; Liu al., 2022a), and even coding and reasoning rules (Chen al., 2021: Wei al., 2022b). Recent studies on behavioral parametric probing and prompting (Jiang et al., 2020; Qin and Eisner, 2021; Brown et al., 2020) has demonstrated that such knowledge, collectively referred to as "parametric knowledge," resides reliably within subset of trained parameters in pre-trained models (PLMs). Importantly, this knowledge can be iden- tified without additional finetuning. For instance,
Equal advising.
However, it is not clear to what extent this as-

---

Downstream Task Consistent with Model's Knowledge (A.²)
Where did Barack Obumaa gradunte from?
Model's Parametric Knowledge (D*)
Ohama graduated from Harvard.
went Stanford.
K train
DO
Bill Gates studied at Harvrrd
Obama left Harvard.
(Barack Obama, gradusted_ fron, Haraad)
PLM Mo
Crose
Down:
streat
(Alen Turing, born.in in, Londan)
Tisk
Where is the Alan Turing's birthplac?
birthplace Turing
Alan Turing born in London.
Kg Kest
Steve Jobs originated from London
Mahatma Gandhi died London.
Figure 2: XTRAEVAL Framework: (1) From a pretrained LM, Me. the model's parametric knowledge are extracted as D (2) Following which downstream task training and test split, Klrain and Ki are created from D (3) The evaluation the application of acquired knowledge estimated through the performance the test split, after finetuning Ma on the downstream task.
sumption holds in practice, giving rise to central question: how much of parametric knowledge will get applied in downstream tasks? If the causal link between "identifiable knowledge" and its practical application in downstream tasks is not established (Kulmizev and Nivre, 2021), the mere presence of knowledge within a model's parameters does not necessarily guarantee its utilization in such tasks. This raises questions about the assertion of pre- trained language models (PLMs) as differentiable knowledge bases (Hao et al., 2022) and their overall capabilities. For instance, as demonstrated by Qin et al. (2023), ChatGPT's performance lags behind its foundational model, GPT-3.5, in areas including commonsense and logical reasoning tasks.
ing et al. (2021) demonstrate that the quantity of "evidence" presented in the finctuning dataset in- fluences the features that PLMs rely on during the finetuning process. Specifically, the model may re- sort to lexical heuristics when the finetuning signal toward linguistic features is insufficient.
In this work, we are interested in more general sense of knowledge and propose XTRAEVAL EXTRACT, TRAIN, AND EVALUATE atically measure how much of parametric knowl- edge is utilized in downstream tasks. XTRAEVAL sidesteps potential confounders (such as shortcuts or insufficient signal) that arise from the nature of arbitrary crowd-sourced tasks used in prior work by carefully creating the downstream task from the model's knowledge. Specifically, given pre-trained language model, our framework first identifies and extracts knowledge residing in its pa- rameters. Subsequently, using the extracted knowl- edge, we construct a downstream task on which we finetune the model. Finally, we measure knowl- edge utilization based on its performance on the downstream task. By constructing the task based the model's pre-existing knowledge, ensure that (1) the model is evaluated solely its pos- sessed knowledge, avoiding penalties for lacking information and (2) successful task completion re- lies explicitly on utilizing the model's parametric knowledge, eliminating the insufficient training sig-
system-
Previous studies have investigated this question within linguistic domains and have demonstrated that although PLMs have the capacity encode linguistic knowledge, they may not effectively em- ploy it in downstream tasks. For example, McCoy et al. (2019) illustrates that PLMs employ syntactic heuristics to solve NLI even though they are able to represent proper linguistic hierarchies (Tenney et al., 2019a), even after finetuning (Merchant et al., 2020; Zhou and Srikumar, 2022). Warstadt al. (2020) provide evidence that RoBERTa requires data inoculation or pre-training with extensive data in order to effectively utilize its hicrarchical lin- guistic knowledge. In a more recent study, Lover-

---

nal issue and dataset shorteuts.
retrieval or following instructions in coding) such that the model can only solve the task by utilizing the knowledge identified in the first step. More formally, we create K and Kfesa as the non- overlapping train and test sets of downstream task K. where the model learns the task from K Step 3. Finally, the performance on the test set Klyss is used measure of the model's ability utilize its knowledge.
In this paper, we provide the first instantiation this paradigm based on encyclopedic knowledge facts and conduct an extensive study to measure knowledge utilization of PLMs across a wide range of parametric scales (ranging from 125M to 13B). We observe the following:
PLMs show two different but equally impor- tant gaps: (1) The gap in the acquired knowl- edge and (2) and the gap in parametric knowl- edge that can be actively applied to down- stream tasks (Section 3).
Constructing the downstream task based on the model's knowledge ensures that the model not evaluated on the knowledge did not acquire dur- ing pre-training. Also, the I.ID. nature of this paradigm (i.e. the model is only exposed to imputs it is already familiar with) allows us to measure whether the model can utilize its knowledge at all.
PLMs not robust to finetuning distribution shifts, and failure utilize knowledge wors- ens with such shifts, further questioning their generalization capabilities (Section 4).
2.2 Encyclopedic Knowledge
Factual parametric knowledge in encyclopedic facts is well-studied in PLMs (Petroni et al., 2019; Jiang et al., 2020) and allows for an objective and systematic evaluation of our framework (Figure 2), Therefore, in this paper, we instantiate XTRAEVAL to measure the utilization of parametric knowledge concerning encyclopedic facts. In this case, the diagnostic dataset 'D is set of encyclopedic facts D (h, r, t) acquired from an off-the-shelf knowledge base (e.g. Wikipedia). Each fact m; E D is a tuple the form (head, relation, tail), such as (Barack Obama, GraduatedFrom Harvard). In the extraction phase, pre-trained model Mo has to zero-shot predict the tail entity t given the head entity h and the relation We use soft- prompting (Qin and Eisner, 2021) obtain the model's predictions, it enhances prediction con- sistency compared to discrete prompts, particularly for moderate-sizer models. The extracted knowl- edge D C D is the subset of tuples the model can predict correctly.
Although scaling the number of parameters helps to close the first gap, the second still remains in larger sizes (Section 5).
In the next sections, first describe our frame- work and its instantiation in detail (Section 2), and finally present our experimental results in Sec- tions 3 5.
2 Framework
2.1 EXTRACT, TRAIN, AND EVALUATE
Principles The primary objective of our evalu- ation framework is measure how much of the knowledge present in the model's parameters is actually usable downstream tasks. Ideally, down- stream tasks must be designed in a way that solely attributes any success to the model's knowledge being used, while ensuring that failure in perform- ing the task is not due to lack of pre-training knowledge.
The Paradigm To this end, we propose EX- TRACT, TRAIN, AND EVALUATE, which consists of three main steps:
Our downstream task K. is standard document retrieval task (Karpukhin al., 2020). Given a query q, the model retrieves the relevant document from set of candidates. We construct KC from the extracted knowledge D by converting each fact x E D into a retrieval instance k E K This conditions the downstream task on the model's knowledge. The conversion generates a query g by removing the tail entity from æ. It then gen- erates relevant and irrelevant documents using a stochastic generator
Step I. Given a pre-trained model Ma with parameters 0 and a diagnostic dataset 'D (e.g. a set of encyclopedic facts or coding problems), we first extract and identify parametric knowledge as a set of data instances r E 'D the model can solve without further training (zero-shot). We denote such set as 2 realization of My's parametric knowledge w.r.t D.
Step 2. We construct a downstream task IC around the model's own knowledge D (e.g. fact
d~ P(d H h, R T-T=t))
(1)

---

Encyclopedic Fact:
last token for decoder-only models) and outputs a scalar as the similarity measure (Figure A.1). Fi- nally, we evaluate the model on Keestt by measuring its accuracy in retrieving the relevant document d among ,..,...., for a given query q-
r.t) (Barack Obama, FraduutedFrom, Harvard)
Input
Sampled Document
Barack Obama graduated from Harvard.
(.,).))
Gold document (d*)
Barack Obama eamed degree from Stanford.
(h,r,-)
The task design ensures that the association be- tween knowledge query qi and gold fact document df' relies solely on the parametric knowledge repre- sented by z; E De This is because other variables, like text overlap, are randomly sampled from the same distribution for both query and documents.
Randomly replacing the tail entiry.
Bill Gates received his degree from Harvard.
(..,,) L)
Rendonll replacing the head entity.
Barack Ohama was horn in Harvard.
(h,,,))
Randomly replacing the relation.
Steve Jobs died in Harvard.
Keeping the tail entity and samplinn others entities.
Thus, the model can only solve the task by uti- lizing its internal knowledge. F'ieetuiin on ACB train should only trigger the utilization of the parametric knowledge.
McGill is the alma mater of Justin Trudeau.
(.
Keeping the relation and sampling others entities.
(h Barack Ohama is located in London.
Keeping the head entity and sampling others entities.
Michacl jordan football player by profession. Unconditional sampling.
Training The document generator P(d |) can generate various types of documents for each fact DD Please refer to Table for list of all the types. For training, we use three types for negative documents
Table 1: All possible inputs to the document generator P(d H. R, T) per each fact x and examples of the corresponding sampled documents. The dot means that the corresponding entity or relation is not given, and the document generator will randomly choose it from D The gray text provides an explanation of the sampled document. Note that we do not force the document generator to generate a factual document and the model itself has to predict the relevancy of each document.
's with uniform weights: (h, r, :), ( r, t). and (h,,t) t) as they are the hardest ones since they only differ in one entity from the query. To keep the GPU memory usage under control, we sample four documents per each type (refer to Sec- tion 3.1 for the effect of the number of negatives on the results), which results in total of m 12 negatives. We resample the documents on each epoch to avoid overfitting and use validation set to choose the best checkpoint. Also, we keep the learning rate low and use no weight decay to pre- vent any forgetting. We use three seeds for the extraction phase, three seeds for splitting DO into train and test, and three seeds for finetuning the downstream task, which results in 27 different runs per each model.
where d depends on the head entity h. relation and tail entity t. The document generator, P(d +), selects template at random and fills in the blanks with the input entities. If H, R, or T are missing, the generator chooses a random entity from D to complete the input. Specifically, we generate relevant document d' by sampling from P(d] | .) with gold entities in x as input, and create irrelevant documents d by omitting one or more entities. Each & comprises tuple (g. {dt ........ . }) where is the number of irrelevant documents.
Inference During inference, the model must iden- tify the gold document d' amidst distractor docu- ments d' 's. To ascertain that the model genuinely recognizes the correct answer, we employ varied assortment of distractors. Initially, we use doc- ument type (h,r,-), ensuring all non-gold tails are included. Subsequently, we utilize the remain- ing non-gold document types listed in Table as distractors, sampling 50 documents for each type. Lastly, we also sample 50 irrelevant but factually correct documents from the test set to assess the model's 's sensitivity to factual accuracy.
We partition 2 randomly (60%-40%) gen- erate Kitrain and K which serve as the training and test sets for the downstream task, respectively. We finetune the model on IC0 KGGaan in cross-encoder setup (Nogueira and Cho, 2020) with the InfoNCE objective (van den Oord et al., 2019):
exp(sim(q,d") ))
C(k) = log Ede .t .... exp(sim(q,d)) d))
The similarity score sim(.,) is computed sim(g, d) (MMe( d)),
We evaluate pre-trained models across various fam- ilies: OPT (Zhang et al., 2022), GPT-Nco (Black et al., 2021), RoBERTa (Liu et al., 2019), and
where h is a randomly initialized value head that takes the representation of the [CLS] token (or the

---

Accurcy on Dounstram Task (AQ.))
Auuuccy on Diegnnstie (D)
0.8
0:20
0.01
0.82
0.4
0.35
0.34
0.4
BERT
RoBERTs
GPT-N Neo
OPT
BERT
RoBERTs
GPT-Neo
OPT
(a) Encyclopedic Knowledge (Zero-shot)
(b) Knowledge Utilization in Downstream (Finetuned)
Figure 3: (a) The fraction of encyclopedic facts the pre-trained LM can predict correctly without any training. Reported over seeds (standard deviation 0.004 for models). (b) The model performance downstream task (created based correctly predicted facts) measured top-1 retrieval accuracy. Averaged over 27 runs per each model (ar & 0.011 for all models). Refer to Appendix B for detailed results.
BERT (Devlin et al., 2019). Unless otherwise stated, use the base size (125M) of these mod- els. We investigate the scaling behavior of OPT and LLaMa (Touvron et al., 2023) Section We initialize the diagnostic dataset from LAMA (Petroni et al., 2019), which has 34K facts over 40 relations. Our results are reported over 1134 finetuning runs (Refer to Appendix A for detailed hyperparameters.)
0.8
Accuracy
0.6
0.4
0.2
0.2
0.4
12
18
24
30
(a) Fraction of D
(b) Number Negatives
BERT
RoBERTa
GPT-Neo
O
OPT
3 Evaluating the Knowledge Utilization
Figure 4: (a) Knowledge utilization when using dif- ferent fractions parametric knowledge to create the downstream task. (b) The effect of number of negative training documents (df 's) used for creating the down- stream task.
We separately report the fraction of facts (D) that can be extracted and the downstream performance of models in Figure 3.
First, find that, par with previous work (Qin and Eisner, 2021), there is significant gap in the encyclopedic facts the models can correctly predict and the entire facts present in the diagnostic dataset D (Figure 3a). Note that one can arbitrarily increase the number of correctly predicted by con- sidering a prediction as correct if the gold entity is among the model's top-A predictions. However, we only consider k 1 to only focus on the facts that the model can confidently predict. Nonetheless, find that BERT and RoBERTa extract slightly more encyclopedic facts than GPT-Neo and OPT.
BERT and RoBERTa by small margin.
This finding suggests that models struggle to utilize their entire parametric knowledge in down- stream tasks. In the next sections, we investigate the potential causes of this gap.
3.1 Role of Downstream Training Data
The effect of initial knowledge D As we utilize Do to create the downstream task, examining the impact of its size (DD)) on knowledge utilization is crucial. If consistent behavior observed for different sizes, it implies that the utilization gap does not stem from the amount of initial knowl- edge and must be rooted in inductive biases (e.g., the model or finetuning process), allowing measure and compare utilization with different ini- tial knowledge.
Critically, all models demonstrate pronounced gap in downstream task performance, or knowledge utilization, (Figure 3b). This unexpected outcome occurs despite the downstream task being seem- ingly simple since (1) models are trained and eval- uated examples based on their accurate encyclo- pedic knowledge predictions, and (2) both K and Klest are sampled from the same distributions (L.ID), so the models only encounter seen entities. Notably, OPT and GPT-Neo manage to outperform
To measure such effect, for each model, we first compute D and then instead of directly using it for KC we sub-sample smaller sets of at various

---

fractions and construct the downstream task using each sub-sampled D In Figure 4.a, we observe the knowledge utilization fairly consistent (at least for fractions > 0.4) across different sizes of D for all models. Larger fractions seem to have less variance as well. This suggests that the uti- lization performance is intrinsic to the downstream knowledge transfer rather than the initial knowl- edge residing in the model.
Fraction of Encyelopdicc Facts
0.6
0.2
0:28
contiifable
Usable
The effect of the number of negatives The model learns to apply its parametric knowledge by optimizing the retrieval objective. To ensure the training signal, produced by contrastive loss on XCD train strong, we vary the number of nega- tive documents for creating X& Krraain If the training signal is weak, we expect knowledge utilization to improve with more negatives.
Figure 5: Gaps in parametric knowledge represents the missing facts in parametric knowledge 'D' (what the model knows). Gap exists in how many the known facts the model can actually utilize in downstream tasks (the usable knowledge).
Gap
we see that although RoBERTa manages to extract more facts than GPT-Neo, due to Gap 2, it performs the same GPT-Nco in downstream tasks.
To answer this question, we follow the same setup deseribed in Section 2 and increase the number of negative documents sampled per type from 4 to 10. We also consider reducing it to neg- atives per type to better understand its effectiveness. We keep the initial knowledge D fixed.
4 Robustness of Knowledge Utilization
We intentionally design the downstream task K. to straightforward and free of any distributional shift as we want to measure the maximum knowl- edge utilization of the model. However, in real- world applications, it is likely that the model en- counter samples that are different from the training distribution. In this section, investigate the bustness of knowledge application in the presence of such distributional shifts.
Figure 4.b summarizes our findings. Knowledge utilization remains the same for all models as increase the number of negatives. This pattern is observed with few two negatives per type. This suggests that the training signal strong enough across the board and the gap in knowledge utilization not rooted in the training objective.
3.2
Gap 1 vs. Gap
4.1 Non-L.LD. KC train and KCB
Findings in Section 3.1 shows that the gap in knowl- edge utilization (i.e. accuracy on Kebest does not depend on the size of D and fairly consistent across different number of negatives. Moreover, we find that the variation across the random splitting of D to create train and test sets of the downstream task is negligible.
Recall that we randomly divide D into two sets as the data source for the creation of K-O and CC In this experiment, however, we split D such that the relation types (r) in KC and KCl are disjoint. Specifically, of the relations and their corresponding facts for KCB train and the rest for K We repeat this pro- cess over three seeds to create three different splits. We still follow the same procedure for converting knowledge triples to document retrieval examples as explained in Section 2. In this way, we ensure we don't 't change the task's 's nature, i.e. the model still needs to apply its parametric knowledge to solve the task, but the distributional shift between KC Krrain and KC9 can represent real-world scenarios. If the model learns systematically apply its knowl- edge, we expect its downstream performance be similar to or close the I.I.D. setting (Section 3). We observe downstream task performance drops
randomly select 60%
The robustness such design choices allows us define Usable Knowledge, which basically indicates the portion of facts from 'D that the model can actually utilize in the downstream task. We compute this metric by multiplying the accuracy on K by the fraction of correctly predicted facts in D. We report the results in Figure 5.
These results clearly demonstrate that there exist two gaps in the models' knowledge. Gap in how many facts the model knows after pre-training. Gap in how many of facts the model knows can be truly utilized in downstream tasks. Indeed,

---

Acaincy on Dovnstream Task (Kio))
Froction of Encyclopedic Ficts
0.75
0.6
0.50
0.25
0.2
125M
350M
1.38
2.78
78
138
(log scale)
UID
OOD Relation (x)
Model Family
Gap Type
LLaMa
Gap
Cap
Figure 6: Robustness to distributional shift In the OOD setting, we produce a distributional shift (over the relation types) between the examples the train and test set of the downstream task K., All models fail to generalize to unseen relations. The IID setting is the the described in Section and repeated from Figure 3b for comparison.
Figure 7: Gaps in parametric knowledge Knowl- edge gaps directly compute
different model sizes. Specifically, we use 1 (Accuracy on D²) for Gap and (Accuracy on D x (1 downstream accuracy) for Gap 2).
suggesting they acquire more knowledge from pre- training data. Secondly, find that the gap be- tween identifiable and usable knowledge persists larger models, and their ability to apply knowledge in downstream tasks does not improve with scaling. Figure 7 illustrates these gaps directly, demonstrat- ing that while Gap 1 decreases in larger models, Gap 2 remains relatively unchanged.
significantly for all models when evaluated OOD (Figure 6). This indicates the models cannot use their knowledge on examples with unseen relation types, though all relations and facts originate in D Thus, knowledge usage downstream tasks sensitive to distribution shifts, suggesting failure to apply pre-training knowledge may be more severe in worldworld applications.
The results suggest that while PLMs, even small scales, pose considerable knowledge, ex- tracting an equivalent amount of usable knowl- edge necessitates much larger models. For in- stance, OPT-125M accurately predicts 34% of en- cyclopedic facts, but only OPT-13B (approximately 100x larger) can reliably apply the same volume in downstream tasks. Enhanced pre-training tines, including the use of more data higher qual- ity data, can bolster knowledge acquisition, as is clearly demonstrated by LLaMa models. Notably, LLaMa-78 significantly outperforms OPT-13B. While LLaMa models possess a greater amount of usable knowledge due to superior initial knowledge, gap in knowledge utilization remains discernible (Figure 7).
5 Effect of Scaling law On The Gaps
Recent NLP success has come from scaling up pre- training model parameters (Brown et al., 2020). With larger models and increased compute, capa- bilities such as in-context learning and chain-of- thought reasoning emerge (Wei et al., 2022b). The expanded capacity allows these models to absorb more knowledge from pre-training data, improving their usefulness knowledge sources. However, it remains uncertain scaling boosts the propor- tion pre-training knowledge applicable to down- stream tasks. Ideally, we like to see a narrowing gap in pre-training knowledge alongside superior knowledge utilization.
Discussion
To investigate this, we evaluate XTRAEVAL on increasing sizes of OPT and LLaMa models. Specifically, at each scale, we first extract the model's parametric knowledge and then create the downstream task based on it using the same proce- dure as described in Section 2. Figure I reports the results of this experiment.
Lately, pre-trained language models with chat- bot interfaces have increasingly been served as knowledge bases (Ouyang et al., 2022). These chatbots typically employ the model's parametric
tWe conducted the experiments for OPT-6.7B multiple times, and observed a dip in performance in all runs. We suspect that this consistent decline may he attributed to issues that arose during the pre-training phase.
First, we confirm that greater fraction of knowl- edge triples in can be identified in larger models,

---

knowledge to respond queries and offer infor- mation. Our study examines the dependability of this knowledge and its impact on downstream task performance. We discover that, regardless of in- ductive biases, PLMs face difficulty utilizing their full knowledge in downstream tasks (Section 3). This unreliability of parametric knowledge could constrain the concept of "PLMs as differentiable knowledge bases."
Liu et al., 2022a). Recently, the emergent abilities of LLMs have shown that they acquire skills like coding (Chen et al., 2021), reasoning (Chowdh- ery et al., 2022), and in-context learning (Brown et al., 2020), in addition to the previously men- tioned knowledge.
Using the Parametric Knowledge Roberts al. (2020) finetune a pre-trained T5 model for question answering in a closed-book setting and showed that can perform on par or better than models that use explicit knowledge bases. Wang et al. (2021) made similar observation for the BART model. More recently, PLMs are being used to generate facts and documents for noowledge-intensive tasks (Li et al., 2022; Liu et al., 2022b; Yu et al., 2023). In this paradigm, in order answer factual questions, instead of retrieving relevant documents, the model has to first generate the facts and then answer the question with those facts as context. This paradigm shows that the models may not be able to use their parametric knowledge on their own and need ex- plicit grounding to be able use it. Furthermore, there is plethora of work that investigates whether the model employs its linguistic knowledge when solving downstream language understanding tasks. McCoy et al. (2019) shows that RoBERTa does not use its linguistic knowledge for solving NLI. In- stead, it relies on shallow heuristics, Lovering et al. (2021)'s observation aligns with this finding and shows the training data used for the downstream task needs to have enough evidence to trigger the model's linguistic knowledge. In work, we use more general notation of parametric knowledge and investigate utilization in cases where sufficient evidence is present in the finetuning data.
Additionally, our findings show that the utiliza- tion gap persists even with scaling (Section 5). No- tably, while models at each scale capture more knowledge from pre-training data, obtaining the same amount of usable knowledge requires sig- nificantly larger models. This exposes a potential constraint in the recent trend of adopting mid-sized PLMs (Li et al., 2023).
Lastly, we discover that knowledge utilization depends on the peculiarities of finetuning data for downstream tasks. Specifically, as seen in Sec- tion 4, PLMs struggle to apply their knowledge relation types not encountered during finetun- ing, even if they accurately predicted such facts in step 1. This generalization gap could highlight challenges within the recent SFT-RLHF paradigm (Ouyang et al., 2022). For instance, the model may only adhere to instructions and excel at tasks resembling the finetuning data. Consequently, it might be necessary to meticulously craft finetuning data activate and utilize all aspects of parametric knowledge in downstream tasks. However, it re- quires elaborate studies to establish the systematic issues in knowledge application beyond encyclope- dic knowledge like procedural and task knowledge.
7 Related Work
Parametric Knowledge Petroni et al. (2019) constructed probing dataset to measure the fac- tual knowledge present in PLMs. They showed that many encyclopedic facts can be extracted without further training of the model and proposed PLMs as a new type of knowledge base, which can be trained on the unstructured text and queried using natural language. Follow-up up work improves the methods for probing and extracting world knowledge from PLMs (Jiang et al., 2020; Shin et al., 2020; Qin and Eisner, 2021; Newman al., 2022). Apart from encyclopedic facts, studies have explored PLMs parametric knowledge in other areas, such as lin- guistic structures (Tenney et al., 2019 9b; Blevins et al., 2022), and commonsense (Zhou et al., 2020;
Conclusion
In this study, we presented EXTRACT, TRAIN, AND EVALUATE (XTRAEVAL ), framework de- signed to assess the parametric knowledge of pre- trained language models. Employing XTRAEVAL we identified a previously unnoticed gap in what models know and how much of it they can actually use. Our findings reveal that this gap exists not only in smaller models but also persists in larger ones. Additionally, we demonstrate that distri- butional shift finetuning data can result even larger gaps between the model's knowledge and its practical application in downstream tasks.

---

Limitations
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- tre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex: Humun Lunguage Tech- nologies, Volume (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Although XTRAEVAL is agnostic to the specific type of parametric knowledge, our work primarily focuses on encyclopedic facts as one type of world knowledge that PLMs can acquire. It is plausible that similar results would hold for other knowledge types, however, further work is needed for a precise investigation.
Shibo Hao, Bowen Tan, Kaiwen Tang, Bin Ni, Hengzhe Zhang, Eric P Xing, and Zhiting Hu. 2022. Bert- net: Harvesting knowledge graphs from pretrained language models. ArXiv preprint, abs/2206. 14268.
While there are various downstream tasks that could be evaluated, we primarily focus on doc- ument retrieval as it allows us to systematically demonstrate the key issue of knowledge applica- tion that we aim to highlight. We also acknowledge that our study was limited to few model families and parameter scales due compute constraints. However, our evaluation protocol is model agnostic. enabling future work to explore this phenomenon on other tasks and with different models.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru. Todor Mihaylov, Daniel Simig. Ping Yu, Kurt Shuster, Tianlu Wang. Qing Liu, Punit Singh Koura, Xian Li, Brian O' O'Horo, Gabriel Pereyra, Jeff Wang. Christo- pher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. 2023. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. ArXiv preprint.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
References
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow. Zenodo.
Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih, 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022.
Prompting language models for linguistic structure. ArXiv preprint, abs/2211 0/s/2211..7735 1.07830.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual.
Artur Kulmizev and Joakim Nivre. 2021. Schrödinger's tree - on syntax and neural language models. ArXiv preprint, 10.02117.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff. Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii. Terry Yue Zhuo, Thomas Wang. Olivier Dehaene. Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, et al. 2023. Starcoder: may the source be with you! ArXiv preprint, abs/2305.06161.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto. Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, et al. 2021. Evalu- ating large language models trained on code. ArXiv preprimt, abs/2107.03372
Yanyang Li, Jianqiao Zhao, Michael Lyu, and Li- wei Wang. 2022. Eliciting knowledge from large pre-trained models for unsupervised knowledge- grounded conversation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 10551-10564, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. ArXiv preprint, abs/2204.02311.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, et al. 2022. Scal- ing instruction- instructionffieetueed language models. ArXiy preprint, abs/2210. 1416.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Pe- ter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022a. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages

---

3154-3169, Dublin, Ireland. Association for Compu- tational Linguistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? ArXiv preprint, abs/2302.06417
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta:
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts, In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 5203- Online. Association for Computa- tional Linguisties.
robustly optimized bert pretraining ap- proach. ArXiv preprint, abs/1907. 19692
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping. Mohammad Shoeybi, and Bryan Catanzaro. 2022b. Multi-stage prompting for knowledgeable dialogue generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1317-1337, Dublin, Ireland. Association for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, Online, Association for Computational Linguistics.
Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. 2021. Predicting inductive biases of pre- trained models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222 4235, Online. Association for Computational Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- Proceedings of
tics natural language inference. the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 3428 -344344, Florence. Italy, Association for Computational Linguisties.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 20. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computational Linguistics.
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. What happens to BERT embed- dings during fine-tuning? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and In- terpreting Neural Networks for NLP, pages 44, Online. Association for Computational Linguisties.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipan- jan Das, and Ellie Pavlick. 2019b. What do you learn from context? probing for sentence structure in contextualized word representations. In 7th Inter- national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Benjamin Newman, Prafulla Kumar Choubey, and Nazneen Rajani. 2022. P-adapters: Robustly extract- ing factual information from language models with diverse prompts. In The Tenth International Con- ference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Rodrigo Nogueira and Kyunghyun Cho. 2020. Pas- sage re-ranking with hert. ArXiv preprint, 104000055.00......
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv preprint, abs/2203.00155.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. Representation learning with contrastive predictive coding. ArXiv preprint, abs/1187.00374.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pages 2463-2473, Hong Kong. China, Association for Computational Linguistics.
Cunxiang Wang, Pai Liu, and Yue Zhang. 2021. Can generative pre-trained language models serve as knowledge bases for closed-bowd ings the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 3241-3251. Online. Association for Computational Linguistics.
QA? In Proceed-

---

Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu. and Samuel R. Bowman. 2020. Learning which fea- tures matter: RoBERTa acquires preference for linguistic generalizations (eventually). Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 217-235, Online. Association for Computational Lin- guistics.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022b. Emer- gent abilities of large language models. Transactions on Machine Learning Research. Survey Certifica- tion.
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models strong context generators. In The Eleventh Inter- national Conference on Learning Representations.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Victoria Lin, Todor Mi- haylov, Myle Ott. Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models. ArXiv preprint, Abs/2205.01066
Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. Evaluating commonsense in pre- trained language models. In The Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Shirry--Sccnel Innovative Applications of Artificial Intelligence Conference, IAAl 2020, The Tenth AAAI Symposiun Educational Advances in Artificial In- telligence, FAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9733-9740. AAAI Press.
Yichu Zhou and Vivek Srikumar. 2022. A closer look how fine-tuning changes BERT. In Proceedings of the 60th Annual Meeting of Association for Computational Linguistics (Volume I: Long Papers). pages 1046-10000 1061, Dublin, Ireland. Association for Computational Linguistics.

---

Simillaiit sim(g. (g.
Parameter
Value
Optimizer
Adam W x 10
Learning rate
Weight Decay
Pre-trained Model Mo
Batch size
64
Learning Rate Scheduler
Polynomial 6% of training steps
Warm Up
# Train Epochs
20
"[ct.5] Where did Obema gradunte from! Obumna gradusted from Harvou"
Table A.I: Summary hyperparameters used in knowl- edge extraction stage (stage I).
Figure A.1: Cross-encoder document retrieval setup (Nogueira and Cho, 2020). For decoder-only models, the value head takes the representation of the last input token.
Parameter
Value
Optimizer
AdamW x 10
A Training Details
Learning rate
Weight Docay
0
Batch size
32
A.1 Knowledge Extraction
Learning Rate Scheduler
Polynomial 6% of training steps
Warm Up
We adopt the same procedure as outlined by Qin and Eisner (2021) for extracting knowledge facts from frozen PLM. Specifically, we utilize soft- prompts instead of discrete prompts. We insert three soft prompts before and after the head entity and allocate distinct soft-prompts for each rela- tion type. We then train them using the training set provided by Qin and Eisner (2021) and em- ploy a validation set to select the best checkpoint. The hyperparameters used in this stage, borrowed from Qin and Eisner (2021), are summarized in Table A.I.
# Train Epochs
20
Table A.2: Summary hyperparameters used fine- tuning downstream task (stage 2).
extraction accuracy obtained by the model.
A.4 Scaling Experiment Details
We adhere to the same procedure as other models in the scaling experiments of OPT and LLaMa. For larger models, we only increase the batch size following lyer et al. (2023) to ensure better fine- tuning stability. Due to the extensive computational and cost requirements of fully fine-tuning the 7B and 13B models, we limit the number of seeds for these variants. Specifically, we trained four seeds for LLaMa-7B, two for OPT-13B, and one for LLaMa-13B.
A.2 Finetuning
For fine-tuning the models, follow a straight- forward procedure, training the models in cross-encoder setup (Figure A.1). The hyper- parameters used for fine-tuning are listed in Ta- ble A.2. In initial experiments, we tried Ir e 1 x 10-5, 3 10-5.5 x 10-5, but we did not find any significant difference between them for all models. Thus, opted to the same learning rate for all models.
A.5 Computational Resources
For the experiments in this study, we exclusively use NVIDIA V100-32GB GPUs. Models with < 350M parameters were trained on a single GPU, For knowledge extraction, we utilized parallel train- ing implemented by DeepSpeed for larger parame- ter sizes. Table A.4 displays the number of GPUs used and the approximate duration of single train- ing run.
A.3 Dataset Details
We employ the LAMA dataset (Petroni et al., 2019) as our diagnostic set, consisting of 34,039 facts. The training and validation sets for poft-prompt training, provided by Qin and Eisner (2021), con- tain 29,029 and 7,255 triples, respectively. The size of the downstream dataset varies from one model instance another, as we construct the downstream task based on its extracted knowledge. The size such datasets can be easily calculated by multi- plying the number of facts in D by the knowledge
A.6 Reproducibility
For the experiments in this study, we exclusively use NVIDIA V100-32GB GPUs. Models with s 350M parameters were trained on a single GPU. For knowledge extraction, we utilized parallel train-

---

Model
Knowledge Extraction
Downstream Finetuning
OPT
125M
64
32
350M
32
1.3B
2.7B
32
6.7B
128
64
13B
128
64
LLaMa
7B
128
64
13B
128
64
Table A.3; Batch size used in the scaling experiments.
Model
Knowledge Extraction
Downstreum Finetuning
OPT
125M
1 (Oh 30m)
(Oh 30m)
350M
(Oh 30m)
0m)
(1h 40m)
2 (1h 40m)
2.7B
4 (3h 30m)
4 (3h 50m)
6.7B
8 (1h 20m)
8 (Sh 40m)
13B
8 (5h 20m)
8 (1d 4h 40m)
L.aMMa
7B
8 (8h 30m)
8 (17h 40m) (1d 15h 15m)
13B
(17h 30m)
Table A.4: Number of GPU and approximate training time for each model size.
ing implemented by DeepSpeed for larger parame- ter sizes. Table A.4 displays the number of GPUs used and the approximate duration of single train- ing run.
B Full Results
B.1 Detailed IID Results
We present the comprehensive results of our exper- iments from Section in Tables B.5 to B.8.
B.2 Detailed OOD Results
Table B.9 provides a detailed account of the experi- ment results Section
B.3
Detailed Scaling Results
The detailed results of Section 5 are presented in Tables 1.10 and B.11.

---

Model
Knowledge Extraction Accuracy
Downstream Accuracy
BERT RoBERTa
0.4722 1 0.0006
0.7760 0.0112
0.3681 - 0.0007
0.7852 0.0092
GPT-Neo
0.3531 1 0.0005
0,8081 0.0099
OPT
0.3444 + 0.0038
0.8177 0.0083
Table B.5: -lean--standad deviation of results presented in Figure and Figure 3b Each number computed over 27 runs and models are in 125M parameter regime.
Downstream Accuracy (Per Number of Negatives)
Model
12
18
27
30
BERT
0.7604 0.0190
0.7760
0.0112
0.7908
0.0075
0.7965 1 0.0081
0.7970 + 0.0055
RoBERTa
0.7660 0.0192
0.7852
0.0092
0.7889
0.0086
0.7961 1 0.0100
0.7927 1 0.0117
GPT-Neo
0,8000 0.0115
0.8081
0.0099
0.8129 0.0128
0.7577 + 0.2186
0.7586 + 0.2189
OPT
0.7966 0.0109
0.8177 0.0083
0,8179 0.0113
0.8294 + 0.0054
0.8269 + 0.0095
Table B.6: standadddd deviation of results presented in Figure 4. Each number is computed over 27 runs and models are 125M parameter regime.
Downstream Accuracy (Per Fraction DD)
Model
0.2
0.6
0.8
BERT
0.5371 +: 0.0707
0.7015 + 0.0272
0.7376 + 0.0173
0.7678 +:: 0.0138
0.7760 + 0.0112
RoBERTa
0.7329 + 0.0554
0.7763
0.0153
0.7824 + 0.0122
0.7769 + 0.0145
0.7852 + 0.0092
GPT-Neo
0.8139 + 0.0633
0.8313 0.0268
0.8149 + 0.0318
0.7948 + 0.0164
0.8081 = 0.0099
OPT
0.7542 0.0710
0.8312 0.0135
0.8100 + 0.01588
0.8065 + 0.01
0.8177 = 0.0083
Table B.7: standard deviation of results presented in Figure 4. Each number is computed over 27 runs and models are in 125M parameter regime.
Fraction of Encyclopedic Facts
Knowledge Gaps
Model
Identifiable
Usable
Gap
Gap 2
BERT RoBERTa
0.4722 + 0.0006
0.3665 + 0.0052
0.5277 0.0006
0.1058 0.0053
0.3681 t 0.0007
0.2890 = 0.0035
0.6319 0.0008
0.0791 0.0034
GPT-Neo
0.3531 1 0.0005
0.2854 1 0.0035
0.6468 0006
0.0678 0.0035
OPT
0.3444 t 0.0038
0.2816 + 0.0036
0.6556 0.0038
0.0628 0.0030
Table B.8: Full results including stanard deviation of experiments presented in Figure 5.
Downstream Accuracy
BERT
RoBERTa
GPT-Neo
OPT
IID
0.7760 = 0.0112
0.7852 +: 0.0092
0.8081 + 0.0099
0.8177 + 0.0083
OOD Relation (r) All
0.2467 1 0.0396
0.3559 0.0415
0.1312 0.0365
0.0976 1 0.0458
OOD Relation (r) Seen entities
0.2451 = 0.0337
0.3778 + 0.0474
0.1330 + 0.0346
0.1001 + 0.0489
0OD Relation (r) Unseen entities
0.2361 = 0.0608
0.3176 + 0.0616
0.1317 + 0.0503
0.0971 + 0.0499
Table B.9: Full results including mtanddstandard deviation of experiments presented in Figure 6, Each number is
computed over 27 runs.

---

Model
Knowledge Extraction Accuracy
Downstream Accuracy
OPT
125M
0.3677 + 0.0045
0.6836 1 0.0049
350M
0.3839 1 0.0017
0,7451 i 0.0053
1.3B
0.4403 = 0.0052
0.7485 + 0.0026
2.7B
0.4761 0.0019
0.7458 + 0.0149
6.7B
0.5090 0.0016
0.3819 + 0.0220
13B
0.5407 = 0.0008
0.7155 + 0.0013
LLaMa
0.6952 = 0.0078
0.7548 0.0099 0.7508
13B
0.7014 - 0.0025
Table B.10: Results of stage 1 and stage for various parameter sizes, which is used to present plots in Figures
and 7
Fraction Encyclopedic Facts
Knowledge Gaps
Model
Identifiable
Usable
Gap 1
Gap 2
OPT
125M
0.3677 1 0.0045
0.2513 1: 0.0013
0.6323 0.0033
0.0941
0.0234
350M
0.3839 1 0.0017
0.2860 1 0.0019
0.6161 0.0015
0.0811
0.0174
1.3B
0.4403 + 0.0052
0.3289 + 0.0037
0,5606 + 0.0037
0.0945
0.0166
2.7B
0.4761 +: 0.0019
0.3551 1 0.0077
0,5239 0.0016
0.1029 0.0196
6.7B
0.5090 + 0.0016
0.1944 + 0.0112
0,4910 + 0.0014
0.2712 + 0.0467
13B
0.5407 + 0.0008
0.3874 + 0.0007
0.4586 + 0.0000
0.1325 0.0249
I.LaMa
0.6952 + 0.0078 0.7014 + 0.0025
0.5282 + 0.0062
0.3002 0.0010
0382 0.0358
13B
0.5288
0.2957 0.0000
0.1503 0.0356
Table B.I Mean- deviation of results presented Figures and 7.


Reviews:
[{"review_id": "HTtFfXlSa7", "reviewer": "Reviewer_STEM", "summary": "", "questions": "A) A comment about line 043: I would argue that it is commonly assumed that as-is after pre-training an LLM it is often not the case that the acquired knowledge is readily applied to downstream tasks and that different fine-tuning approaches aim to exactly improve that are great examples that demonstrate how more parametric knowledge can be harvest. So, I would not think that it is actually a common assumption that 'if the model poses a certain type of knowledge, it utilises it when performing down-stream tasks.'. Nonetheless, I find it important to quantify and understand when it is used and when it is not.\nB) A comment about line 147: You mean ideally for your purpose because I would say generally it is not necessarily the case that we want to ensure that 'failure in performing the task is not due to a lack of pre-training knowledge.\"\nC) Are you ensuring that each test instance has also at least one candidate that has exactly the same head entity combined with exact same relation phrasing as the gold candidate but only differs in terms of the tail entity? \nD) I wonder how reliable the identification of knowledge is. One could have also used the retrieval task to identify knowledge and then then tail prediction as the downstream task. I would guess the findings are similar, no? So, I would guess that the zero-shot tail prediction also does not identify all knowledge the model captures and suffers form the same knowledge utilaisation problem.\nC) I would love to see a qualitative error analysis.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe greatly appreciate your thorough review and valuable feedback. It is heartening to observe that you found our framework **\"novel in testing knowledge utilization,\"** and that you consider our **analysis to be \"interesting.\"** We're also encouraged by your recognition of the **impact of our work,** notably in charting an \"interesting research direction\" and in \"tracking progress for finetuning methods.\"\n\nSimilarly, we're thankful for Reviewer 3 (GyXC), who commended our **\"systematic\" and \"careful\" design** of the framework. This enhances our confidence in the rigor of our approach, and is in alignment with Reviewer 2 (twpn)'s positive remarks about the **\"comprehensiveness of experiments.\"** Furthermore, we're honored by the kind affirmation from Reviewers 2 (twpn) and 3 (GyXC) regarding our work's **contribution**. They highlight its role in fostering a \"better understanding of knowledge utilization\" and in illuminating \"the extent to which pretrained knowledge can be leveraged.\"\n\nWe now address the key questions and concerns in detail below:\n\n> Novelty of Findings\n\nThank you for bringing attention to the perceived overstatement concerning the knowledge gaps in our evaluation. We recognize the importance of clearly defining the scope and context in which our findings apply. Our intention was to examine the utilization of pretraining knowledge in downstream tasks, which in our work, particularly refers to tasks *a PLM is finetuned on* and present the novelty of our  findings particularly on **Gap 2** (Please refer to the discussion below for more details). We have attempted to articulate this in lines 108-111 and provide an example in lines 54-59. However, acknowledging your observation, we realize that further clarification may be necessary to prevent any confusion and plan to revise the introduction and abstract to address this.\n\nIn lights of the fruitful discourse started by the reviewer, we would like to clarify some of our points in more details to better ground our discussion:\n\n**Gap 1**, as we mention in the paper, exists in the acquired knowledge from the pretraining stage: Some facts might not have been seen by the model or captured appropriately, suggesting that this gap might be rooted in the model's capacity or the quality of pretraining data. For instance, as depicted in Fig. 1, larger models within the same family generally exhibit more identifiable facts (despite the same pretraining data); LLaMa, with a more expansive training corpus [1], showcases greater identifiable knowledge in comparison to OPT in the same parameter regime. As correctly highlighted by the reviewer, previous research has addressed this gap across various contexts[2,3,4], which we cite in lines 515-524.\n\n**Robustness in Knowledge Identification:** We argue that the robustness in knowledge identification is inherently a property of the knowledge probing method: A PLM might possess knowledge on a subject, but the probe might not detect it (e.g. there are better or worse prompts for extracting some facts). Indeed, Petroni et al [2] acknowledge “...we are measuring a lower bound for what language models know...”. This is why improving the accuracy and reliability of such probes is an active area of research [5,6]. We believe that while the question of this type of robustness is important, it is more concerned with the probing method rather than the model itself and is not the primary focus of our study. Nevertheless, we utilise a standard probing method from existing literature [3], which is known for improved robustness. Should a better probing method emerge, our framework can readily harness it to produce more accurate measurements.\n\n**Gap 2:** We, as discussed in the paper, refer to this gap as potential inefficiency of a PLM in utilizing its knowledge during the finetuning phase for a downstream task. This definition is irrespective of the probing method employed and the probe's capacity to identify the model's knowledge. Our framework employs a specific probing method without making assumptions about its accuracy. In doing so, it measures the utilization gap solely on the *knowledge set correctly identified by the probing method*, circumventing the imperfections of the probe. We believe findings of this framework points to an aspect of model behavior that remains relatively unexplored in current literature. For example, Gudibande et al [7] recently attributed the underwhelming performance of open-source LLMs (in comparison to OpenAI models) to a better base pretraining than to the finetuning or the instruction data. Our framework's meticulous design, recognized by reviewer GyCX, allows us, to the best of our knowledge, to demonstrate and accurately quantify this gap for the first time. Nonetheless, we are more than happy to explore any references the reviewer might be aware of that address and quantify this specific gap.\n\nWe understand that these details may not have been sufficiently discussed in the original draft but we make sure to have a more comprehensive discussion in the updated manuscript.\n\n> Reliability of Knowledge Identification & Question D\n\nThank you for raising the concern regarding the reliability and robustness of knowledge identification probing. As mentioned in the previous section (*Robustness in Know....*), we argue that the robustness of knowledge identification is predominantly determined by the probing technique rather than the intrinsic properties of the model itself. We differentiate this from the knowledge utilization issue happening in the model (after finetuing) that we are exploring in gap 2. Additionally, we would like to note that our framework takes such innate unreliability into consideration by only quantifying the gap 2 on *the set of knowledge the probe can correctly and reliably identify*. Thus, we expect a consistent conclusion with a different probing method as our framework only makes observations about the identified knowledge. \n\nWe appreciate the reviewer’s comment about using “retrieval” in knowledge probing. Indeed, this paradigm is agnostic to the type of tasks employed at various stages. Employing \"retrieval\" as knowledge identification, however, might be challenging since this task is vastly different from what a PLM is usually trained on (e.g. language modeling). The model could potentially underperform in a zero-shot retrieval setting, leading to a suboptimal probing result.\n\n> The Prevalence of Knowledge Utilization Assumption in the Literature (Question A)\n\nWe acknowledge the reviewer's sentiment about the prevalence of such assumptions in the literature. To address this, we’d like to highlight some quotes from the literature that explicitly make this assumption: Zhou et al. 2020 [8] states that “syntactic, semantic and word sense knowledge are contained in such [PLM] representations, which explains why they benefit such tasks.” Furthermore, a growing body of recent work proposes to use minimal instruction-tuning datasets [9,10,11], (some as limited as 1000 examples), motivated by the hypothesis that instruction-tuning only teaches the model to interact with the user and utilizes the pretraining knowledge. Particularly, Zhou et al, 2023 [9] states that “alignment can be a simple process where the model learns ...[the] format for interacting with users, to expose the knowledge [...] already acquired during pretraining”. We agree with the reviewer that such assumptions might have become less dominant, but we hope our work sheds light on the hidden consequences of them, especially given the emergence of such recent works.\n\n> Question B\n\nThank you for your comment. Yes, we indeed meant “ideally” with respect to the goal of our evaluation. We’ll revise the wording to avoid any misinterpretation.\n\n> Question C \n\nYes, candidates with the same head entity and relation phrasing as the gold standard, but different tail entities, are included. This type of candidate is the one we care the most, and we’ve ensured it’s well represented in the task construction. We tried to convey this in lines 264-275. We will improve the clarity in this section.\n\n> Additional Qualitative Error Analysis\n\nWe appreciate the reviewer's interest in a qualitative error analysis of models' performance. In our preliminary analysis, we focused on evaluating the distribution of similarity scores $sim(q, .)$ (line 227) between the query and different candidate types. Our findings indicate that the model faces more difficulties in distinguishing among candidate types $(\\mathbf{h}, \\mathbf{r}, \\cdot)$, $(\\mathbf{h},, \\cdot, \\mathbf{t})$, and $(\\cdot,, \\mathbf{r}, \\mathbf{t})$. Specifically, these candidate types generate similarity scores that exhibit overlapping distributions with those of the gold documents.\n\nThis issue could potentially be attributed to the nature of these candidate types, where only one entity element—be it the head, relation, or tail—is distinct from the query, thereby confounding the model's decision-making process. Additionally, we discovered an intriguing behavior in our models: they struggle to differentiate between unfactual and factual candidate types both denoted as $(\\cdot, \\cdot, \\cdot)$. We hypothesize that this lack of distinction could be a consequence of these candidates sharing no entities with the query, leading the model to deprioritize or even discard them.\n\nWe find these patterns to be consistent across the models (BERT, RoBERTa, GPT-Neo, and OPT) and will include this analysis in the appendix for further scrutiny.\n\n\n> Application to RLHF domain\n\nThe reviewer makes an astute observation regarding the potential applicability of the framework to the RLHF domain. This connection, as also noted by reviewer GyxC, is indeed intriguing and noteworthy. While our proposed paradigm is compatible with instruction-tuning, there are several critical aspects that must be meticulously designed to accurately isolate the phenomenon of knowledge utilization. Specifically, the development of a method for 'task' extraction from scratch, for the knowledge identification phase, is non-trivial as there is no well-established technique available in the literature. Moreover, the process of converting the extracted knowledge into a downstream task remains ambiguous and needs exploration. Given these complexities, we believe that a comprehensive analysis of both knowledge types surpasses the scope of a single paper. Nevertheless, we recognize the intriguing nature of this question and believe that the findings in our work could motivate such in-depth future studies, opening up an important avenue for research.\n\nWe hope the clarifications provided have resolved any remaining questions. Would you kindly consider increasing your ratings based on the key points clarified?\n\n\n- [1] Touvron et al. https://arxiv.org/abs/2302.13971, 2023\n- [2] Petroni et al, https://aclanthology.org/D19-1250/, EMNLP 2019\n- [3] Qin et al, https://aclanthology.org/2021.naacl-main.410/, NAACL 2021\n- [4] Tenney et al, https://openreview.net/forum?id=SJzSgnRcKX, ICLR 2019\n- [5] Shin et al, https://aclanthology.org/2020.emnlp-main.346/, EMNLP 2020\n- [6] Newman et al, https://openreview.net/forum?id=DhzIU48OcZh, ICLR 2022\n- [7] Gudibande et al, https://arxiv.org/abs/2305.15717, 2023\n- [8] Zhou et al, https://arxiv.org/abs/1911.11931, AAAI 2020\n- [9] Zhou et al, https://arxiv.org/abs/2305.11206, 2023\n- [10] Touvron et al, https://arxiv.org/abs/2307.09288, 2023\n- [11] Taori et al, https://github.com/tatsu-lab/stanford_alpaca, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The framework to test knowledge utilisation is novel and highlights an interesting research direction. Their framework is useful to track progress across different fine-tuning methods in terms of knowledge utilisation.\n- The authors carry out interesting additional analysis. Specifically, their findings around scaling and how it is not reducing the knowledge utilisation gap.", "reasons_to_reject": "- Their identification of two types of knowledge gaps itself is not novel. We know that large amounts of factual knowledge are not captured  by the PLM (prior papers have shown that performance on zero-shot Q&A or tail entity prediction is not perfect). Already the fact that predictions across paraphrases, different language queries or styles of querying for knowledge (Q&A vs. multiple-choice vs. tail prediction) leads to different predictions, shows that there is a knowledge utilisation gap. Therefore, some of their findings are over-claimed, e.g., \"We discover that, [...] PLMs face difficulty utilizing their full knowledge in downstream tasks.\"\n- The method of knowledge identification potentially suffers the knowledge utilisation problem itself (see question D).\n- They could have evaluated different fine-tuning paradims. The authors speculate how their finding could translate to RLHF but have not carried out experiments around that.", "questions_to_authors": "A) A comment about line 043: I would argue that it is commonly assumed that as-is after pre-training an LLM it is often not the case that the acquired knowledge is readily applied to downstream tasks and that different fine-tuning approaches aim to exactly improve that are great examples that demonstrate how more parametric knowledge can be harvest. So, I would not think that it is actually a common assumption that 'if the model poses a certain type of knowledge, it utilises it when performing down-stream tasks.'. Nonetheless, I find it important to quantify and understand when it is used and when it is not.\nB) A comment about line 147: You mean ideally for your purpose because I would say generally it is not necessarily the case that we want to ensure that 'failure in performing the task is not due to a lack of pre-training knowledge.\"\nC) Are you ensuring that each test instance has also at least one candidate that has exactly the same head entity combined with exact same relation phrasing as the gold candidate but only differs in terms of the tail entity? \nD) I wonder how reliable the identification of knowledge is. One could have also used the retrieval task to identify knowledge and then then tail prediction as the downstream task. I would guess the findings are similar, no? So, I would guess that the zero-shot tail prediction also does not identify all knowledge the model captures and suffers form the same knowledge utilaisation problem.\nC) I would love to see a qualitative error analysis.", "paper_topic_and_main_contributions": "The authors introduce a framework to measure parametric knowledge utilisation of PLMs in downstream tasks. Instead of crowd-sourced tasks, they create the tasks from the model’s own knowledge. They first extract knowledge from a PLM (zero-shot tail prediction), subsequently design a task around this knowledge (document retrieval) and fine-tune the model. They target encyclopaedic knowledge. The authors show that i) PLMs did not acquire all factual knowledge par of Wikipedia, ii) not all acquired knowledge is utilised in downstream tasks, iii) model scaling does not close the knowledge utilisation gap and iv) fine-tuning distribution shifts negatively impact knowledge utilisation.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- The related work section should also cover work highlighting robustness problems of knowledge extraction.", "Typos_Grammar_Style_and_Presentation_Improvements": "- Caption in Figure 5: 'Gap 2 exists in how many of the known facts the model can actually utilize in downstream tasks (the usable knowledge).' There is something wrong with this sentence."}, "extra_text": {}}, {"review_id": "UJ8129SGo5", "reviewer": "Reviewer_twpn", "summary": "", "questions": "1. Have you considered using more comprehensive probe methods in combination?\n2. Why did you choose to predict relationships as the probe method for knowledge?\n3. Why did you choose document retrieval as the automatically generated downstream task?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nThank you for investing time in reviewing our paper and offering valuable feedback. We are encouraged to find that you commend our work for its **\"extensive and comprehensive experiments and analyses,\"** and acknowledge its contribution towards a **\"better understanding of knowledge utilization.\"**\n\nWe are heartened by the unanimous recognition from several reviewers regarding the quality and significance of our work. Reviewer 1 (STEM) kindly acknowledged our framework's **\"novelty in testing knowledge utilization\"** and found our finding and analysis \"interesting\". We are grateful to reviewer 3 (GyXC) acknowledging \"systematic\" and \"careful\" design of our framework, further increasing our confidence in the **rigor** of our work.Regarding the **impact**, Reviewer 1 (STEM) praised our work for outlining an \"interesting research direction\" and for **\"tracking progress in finetuning methods.\"** These sentiments were also echoed by Reviewer 3 (GyXC), who highlighted the work's significance in illuminating \"the extent to which pretrained knowledge can be leveraged.\"\n\nWe now address the key questions and concerns in detail below:\n\n> Exploration of Other Tasks and Knowledge Types & Question 2\n\nThank you for your insightful comments on the need for additional exploration of other tasks. We agree with your sentiment that exploring other types of knowledge or tasks would strengthen our work. However, it's worth mentioning that our choices in designing this XTraEVAL were deliberate for several reasons, which lead to the creation of robust yet novel paradigm as highlighted by reviewers 1 (STEM) and 3 (GyXC).\n\nFirst, one of our primary goals was to create meticulous evaluation protocol that exclusively measures *knowledge utilization* within pretrained language models (PLMs). A necessary definition in such a protocol is a clear and tangible formulation of knowledge itself. As also mentioned in lines 183-186, encyclopedic facts serve a basic, yet fundamental formulation. More importantly, they are a well-established convention in the field and has rich literature around it [1,2,3,4].\n\nFurthermore, the ground truth for these facts is sourced from Wikipedia, a reliable and well-curated repository. This minimizes ambiguity in these knowledge facts (e.g. both `Obama was born in Hawaii` and `Obama was born in <some-other-place>` can’t be true at the same time), thereby allowing us to produce a robust evaluation. \n\nThirdly, we’d like to note that in our paradigm, encyclopedic facts consider all entities—head and tail—together, forming a cohesive piece of knowledge. While structured as triples (head, relation, tail), these facts are presented to the model in a natural language format, which is quite generic and versatile. This is particularly advantageous for zero-shot knowledge probing, aligning closely with the objectives these PLMs were trained on. Such an approach significantly minimizes any distributional shift between the pretraining and knowledge extraction phases. In leveraging this property, we were indeed able to evaluate a range of models, spanning from encoder-only to decoder-only (autoregressive) architectures, resulting in the comprehensive analysis you kindly acknowledged.\n\nWhile our investigation is focused, we have taken care to conduct a thorough study. We believe our work could serve as a foundational effort in establishing a framework for measuring knowledge utilization and sets a precedent for future work in other domains like RLHF, as acknowledged by reviewers 1 (STEM) and 3 (GyXC). \n\n>  Use of Other Probing Methods (Question 1)\n\nWe appreciate your insightful suggestion about incorporating other probing methods. We are indeed fortunate that the type of knowledge we consider enjoys a rich literature of probing methods.\nIn the preliminary stage of the work, we considered various methods and chose soft-prompts introduced by Qin et al [2]. This method was selected for its strong performance, especially when applied to smaller models, which tend to exhibit more inconsistencies when interacting with different discrete prompts. Moreover, the simplicity and robustness of this approach have made it well-established in the field, as evidenced by prior work [5,6].\n\nIt is important to note that our framework treats the knowledge identification probing mechanism as a black-box component, without making any assumptions about its internal workings. This is because the evaluation protocol makes an observation *only* on a *set of knowledge the probe reliably identifies*. While comparing different probing methods is not the focus of our work, the design of our paradigm allows for easy adaptability; if a more advanced or effective probing method becomes available, it can be seamlessly integrated into the existing framework.\n\n>  Motivation Behind Document Retrieval (Question 3)\n\nThank you for posing this insightful question.. As outlined in lines 144-151 of our paper, it is crucial that the selected downstream task directly reflects the model's ability to utilize its parametric knowledge. The failure in successfully performing the task should not be attributed to other factors. In document retrieval task, the model is responsible for matching a query to the correct document solely based on its parametric knowledge ($\\mathcal{D}^\\theta$). If the downstream task included an additional component like reasoning, it would complicate the interpretation of the model's performance, leaving us uncertain whether any failure was due to inadequate reasoning skills or insufficient utilization of pre-trained knowledge. We believe document retrieval as a task not only meets the framework's criteria and enables us to perform systematic study avoid of confounders but also makes our findings even more interesting, as it shows models fail even in this seemingly simple case. \n\nWe hope that the clarifications provided have resolved any lingering questions. Would you kindly consider increasing your scores given the main clarifying points above?\n\n\n- [1] Petroni et al, https://aclanthology.org/D19-1250/, EMNLP 2019\n- [2] Qin et al, https://aclanthology.org/2021.naacl-main.410/, NAACL 2021\n- [3] Shin et al, https://aclanthology.org/2020.emnlp-main.346/, EMNLP 2020\n- [4] Newman et al, https://openreview.net/forum?id=DhzIU48OcZh, ICLR 2022\n- [5] Lester et al, https://arxiv.org/abs/2104.08691, 2021,\n- [6] Schucher et al, https://aclanthology.org/2022.acl-short.17.pdf, ACL 2022\n- [7] Hao et al, https://arxiv.org/abs/2206.14268, ACL 2023 Findings", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces XTRAEVAL (EXTRACT, TRAIN, AND EVALUATE) to measure how much parametric knowledge is used in downstream tasks. \n2. The paper presents extensive and comprehensive experiments and analyses, yielding valuable insights through comparisons of multiple pre-trained models. These insights contribute to a better understanding of knowledge utilization in natural language processing.", "reasons_to_reject": "The conclusions drawn in the paper would benefit from additional supporting evidence. While detecting knowledge presence in PLMs through relation prediction tasks might be more favorable for BERT-like models, it is essential to explore multiple methods. By employing various approaches and tasks, the study can provide a stronger foundation for the conclusions, ensuring robustness and generalizability of the findings regarding knowledge utilization in pre-trained language models.\n", "questions_to_authors": "1. Have you considered using more comprehensive probe methods in combination?\n2. Why did you choose to predict relationships as the probe method for knowledge?\n3. Why did you choose document retrieval as the automatically generated downstream task?", "paper_topic_and_main_contributions": "This paper introduces XTRAEVAL (EXTRACT, TRAIN, AND EVALUATE) to measure how much parametric knowledge is used in downstream tasks. It provides valuable insights by comparing multiple pre-trained models, enhancing our understanding of knowledge utilization in natural language processing.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "51TxbgpujG", "reviewer": "Reviewer_GyXC", "summary": "", "questions": "- Are the prompts generated by the document generator diverse enough to avoid overfitting? The data might lack diversity and hurt the model generalization capabilities (for example, it seems that many examples start with entities). Could the authors provide some thoughts on the relation between the data format used in the finetuning phase and the catastrophic forgetting of pretrained knowledge?\n- Is there any specific motivation for why only the relations were used for creating the distributional shift? Could the same be done with the entities to investigate whether the model can maintain the knowledge facts for specific entities?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe are grateful for your meticulous review and insightful feedback on our work. It is especially encouraging to hear your positive remarks, which highlight the **\"careful and systematic design of our framework.\"** We are also pleased that you find the **usefulness** of our investigation compelling, particularly in its connection to the emerging paradigm of **\"instruction tuning,\"** and its potential to illuminate “the extent the pretrained knowledge can be leveraged.”\n\nWe're also encouraged by the supportive feedback from other reviewers. Specifically, Reviewer 1 (STEM) highlighted the **novelty** of our framework for measuring knowledge utilization. Reviewer 2 (twpn) appreciated our “**extensive and comprehensive** experiments and analyses”, a view also shared by Reviewer 1 (STEM) who also found our analysis to be of particular interest. Both reviewers further acknowledged the **impact of our work** in \"highlighting an interesting research direction,\" providing a way to \"track progress across different approaches,\" and offering a \"better understanding of knowledge utilization.\"\n\nWe now address the key questions and concerns in detail below:\n\n> Use of Other datasets\n\nWe are glad our thought processes match. At the preliminary stage of our research, we did indeed consider using commonsense datasets (e.g. ConceptNet[1]) in the framework (for populating $\\mathcal{D}$). However, with further investigation, we encountered challenges with the ambiguity inherent in this type of knowledge. Specifically, entities and relations are often not uniquely mapped.  For example, if the gold triple is (fish, AtLocation, aquarium), another equally correct triple could be (fish, AtLocation, sea). When we use such data, we often find that the model's predictions are actually correct, but do not match the dataset’s answer:\nQuery: “Where is fish found?” Gold: “Something you find at the aquarium is a fish”, Model’s Retrieval Result: “Something you find under the Sea is fish.”\n\nOther examples like this include:\n- Query: “What do people have?” Gold: “The people have eyes”, Model’s Retrieval Result: “People have babies.”\n- Query: “What are the vultures?” Gold: “Vultures are scavengers”, Model’s Retrieval Result: “A vulture is a animal”\n\nIn light of these observations, we made the deliberate choice to forgo this type of knowledge to avoid introducing a potential confounder into our study.\n\nFor encyclopedic facts, we also considered other knowledge bases than DBPedia​​ (Wikipedia’s relational knowledge base), including Freebase, an older but separately crafted knowledge base. But, we found similar patterns (to those of DBPedia). Based on these initial observations, we opted to use a well-established DBPedia-sourced dataset in our study, rather than introducing a new one, and instead we made sure to perform a thorough study with various model architectures, families, and sizes, with many seeds. Nevertheless, we recognize the importance of exploring the findings of this framework in other domains and fully endorse introduction of new dataset and knowledge types as an important avenue for future research.\n\n> Robustness Experiments in Section 4 & Question 2\n\nWe appreciate the reviewer’s thoughtful feedback on the experiments in Sec. 4 regarding the potential risk of forgetting. To address this, we like to kindly note that our experiments in Section 4 were designed to *closely* mimic the setup of the experiments in Section 3 (of course except for the distribution of relations). This was done deliberately to make everything the same across both setups (please refer to lines 397-411 for more details), including the same number of negatives, identical hyperparameters, same document generator and so on; the number of examples for training and evaluation also remained approximately equivalent. Therefore, we believe that given these considerations, if significant forgetting were occurring during fine-tuning, we wouldn't expect to see the marked performance gap between the OOD and IID scenarios that our results show. While these methodological choices were intended to carefully isolate the effect of distributional shift, we acknowledge that the fine-tuning process could indeed introduce an element of forgetting. But, we wish to note that our fine-tuning approach adheres to standard practices commonly used in the field. This implies that even if some level of forgetting is present, it actually helps to reinforce the core message of our work, which is to highlight noteworthy issues in methodologies that are commonly accepted within our field.\n\nRegarding your second question, the reviewer makes an astute observation in exploring the distributional shift with entities in addition to relations. Although the idea of segmenting $\\mathcal{D}^\\theta$ based on specific entities, like the \"head\" of a triple, is noteworthy, this approach comes with its own set of challenges, primarily due to the sparse availability of knowledge triples for each entity. For instance, the number of triples involving \"Barack Obama\" as the head entity would be substantially fewer compared to triples with the relation \"born_in.\" This scarcity of data for individual entities makes it difficult to draw robust conclusions about whether the model retains or forgets its pre-trained knowledge concerning specific entities. Thus, we opted to employ the relation for this particular experiment to have reliable evaluation.\n\n> Diverse Document Generation (Question 1)\n\nThank you for raising the insightful question regarding the diversity of generated prompts and its impact on the forgetting of pretrained knowledge. In our study, we utilized the prompts from the work of Qin et al. [2] and augmented them as necessary. We indeed acknowledge that employing a more diverse set of prompts could be beneficial as such diversity would offer a more faithful representation of real-world scenarios in which users interact with the model. However, we like to highlight that the design of our framework can inherently mitigate the issues that may stem from a lack of prompt diversity. We employ the same document generator for both the training and testing phases, thereby ensuring that the model is exposed to and evaluated on the same set of prompts. This design choice aims to provide an unbiased evaluation environment. Essentially, the model is never tested on prompts it hasn't seen during training. So, even if the model overfits to specific sentence structures, it won't be tested to utilize its knowledge in unfamiliar sentences. This setup allows us to focus on the core question of how well the model applies its pretrained knowledge to the task at hand.\n\n\n- [1] Speer et al, https://arxiv.org/abs/1612.03975, AAAI 2017\n- [2] Qin et al, https://aclanthology.org/2021.naacl-main.410/, NAACL 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes a framework to analyse acquisition and utilization of knowledge during pretraining/fine tuning phases in pretraining language models, which might help to shed light on the extent the pretrained knowledge can be leveraged during subsequent finetuning phases.\n- The systematic design enables the evaluation of the model solely based on its current learned knowledge, avoiding cases where the model does not have knowledge about the information.\n", "reasons_to_reject": "- The paper only evaluates encyclopedic knowledge using one dataset based on wikipedia which might be restrictive to derive conclusive findings. It would be interesting to investigate other datasets (e.g., WebNLG [1]) and types of knowledge (e.g., commonsense [2])\n- It seems not not very clear that the robustness experiment (Section 4) would be the accurate way to capture that the model can't use its pretrained knowledge since the fine-tuning procedure might be making the model to overfit learning to catastrophic forgetting. A further investigation could be done with more diverse data and prompts (similar to what instruction tuning does) in order to avoid overfit and to preserve the pretrained knowledge (see questions for the authors).\n\n[1] The WebNLG Challenge: Generating Text from RDF Data (Gardent et al., INLG 2017).\n[2] UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark  (Lourie et al. 2021).\n", "questions_to_authors": "- Are the prompts generated by the document generator diverse enough to avoid overfitting? The data might lack diversity and hurt the model generalization capabilities (for example, it seems that many examples start with entities). Could the authors provide some thoughts on the relation between the data format used in the finetuning phase and the catastrophic forgetting of pretrained knowledge?\n- Is there any specific motivation for why only the relations were used for creating the distributional shift? Could the same be done with the entities to investigate whether the model can maintain the knowledge facts for specific entities?\n", "paper_topic_and_main_contributions": "This paper investigates the usage of parametric knowledge from pretrained large language (LLM) models in downstream tasks. It proposes an approach to quantify and extract knowledge from the pretrained LLM, using KB/dataset facts and implement a downstream task based on the extract knowledge. The setup is careful design to assess the impact of only the parametric knowledge on downstream tasks. This is an useful investigation since recently supervised finetuning / instruction tuning / RLHF are used to enable the model to follow instructions based on its parametric knowledge.\n\nSpecifically, given a pretrained LLM, the proposed approach:\nExtract knowledge present in the model parameters\nConstruct a downstream task based on the extracted knowledge\nMeasure knowledge \"retrieval\" based on the performance in the downstream task.\n\nExperiments indicate that there is a gap in knowledge utilization (downstream performance) suggesting that it can be hard for the models to make use of their parametric knowledge during downstream fine tuned tasks. \n\nThe paper conducts an experiment where the model needs to use its parametric knowledge to solve the downstream task with a distributional shift where the model is tested with relations which are not used during fine tuning. It is observed that the performance drops when evaluating in such out-of-domain scenarios and the model cannot employ its knowledge on examples with unseen relations even though they were present in its parametric knowledge before fine tuning.\n\nFinally, the paper investigates whether the scaling (# of model parameters) improves the amount of pretrained knowledge employed during the downstream task and shows that the gap between identifiable and used knowledge remains for larger models.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree this work conducts a comprehensive investigation into an important topic of analysis, and yields insights into the changing nature of the knowledge acquisition and utilization gaps with model scale. While reviewers note challenges in deriving robust and generalizable findings without exploring many datasets, finetuning and probing methods, they do acknowledge the authors have thought deeply about the problem and experimental setup to make their findings as useful and possible.
Decision: Accept-Findings