P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts
Authors: Benjamin Newman
Prafulla Kumar Choubey
Nazneen Rajani
Venue: ICLR 2022 Poster (2022)
Keywords: NLP
Prompting
Commonsense
information extraction
factual extraction
Large Language Models

Abstract: Recent work (e.g. LAMA (Petroni et al., 2019)) has found that the quality of the factual information extracted from Large Language Models (LLMs) depends on the prompts used to query them. This inconsistency is problematic because different users will query LLMs for the same information using different wording, but should receive the same, accurate responses regardless. In this work we aim to address this shortcoming by introducing P-Adapters: lightweight models that sit between the embedding layer and first attention layer of LLMs. They take LLM embeddings as input and output continuous prompts that are used to query the LLM. Additionally, we investigate Mixture of Experts (MoE) models that learn a set of continuous prompts (the "experts") and select one to query the LLM. These require a separate classifier trained on human-annotated data to map natural language prompts to the continuous ones. P-Adapters perform comparably to the more complex MoE models in extracting factual information from BERT and RoBERTa while eliminating the need for additional annotations. P-Adapters show between 12-26% absolute improvement in precision and 36-50% absolute improvement in consistency over a baseline of just using natural language queries alone. Finally, we investigate what makes P-Adapters successful and conclude that a significant factor is access to the LLM's embeddings of the original natural language prompt, particularly the subject of the entity pair being queried.

P-ADAPTERS: ROBUSTLY EXTRACTING FACTUAL IN-
FORMATION FROM LANGUAGE MODELS WITH DI-
VERSE PROMPTS
Benjamin Newman" Stanford University
Prafulla Kumar Choubey Salesforce Research
Nazneen Rajani Salesforce Research
ABSTRACT
Recent work (e.g. LAMA (Petroni et al. 2019h) has found that the quality of the factual information extracted from Large Language Models (LLMs) depends on the prompts used to query them. This inconsistency is problematic because differ- ent users will query LLMs for the same information using different wording, but should receive the same, accurate responses regardless. In this work we aim to ad- dress this shortcoming by introducing P P-dapters: lightweight models that sit be- tween the embedding layer and first attention layer of LLMs. They take LLM em- beddings input and output continuous prompts that used query the LLM. Additionally, we investigate Mixture of Experts (MoE) models that learn a set of continuous prompts ("experts") and select one query the LLM. They require a separate classifier trained on human-annotated data to map natural language prompts to the continuous ones. P-Adapters perform comparably complex MoE models in extracting factual information from BERT and RoBERTa while eliminating the need for additional annotations. P-Adapters show between 12-26% absolute improvement in precision and 36-50% absolute improvement in consistency over baseline of only using natural language queries. Finally, we investigate what makes P-Adapters successful and conclude that a significant fac- tor is access to the LLM's embeddings of the original natural language prompt, particularly the subject of the entity pair being queried.
the
INTRODUCTION
Recently, there has been an interest in meeting users' factual information needs using large language models (LLMs) as knowledge bases in place traditional, index-based IR methods (Petroni et al.) 2019 Metzler et al. 202I). In this conception of LLM knowledge bases, LLMs accumulate factual know wewgeedee during pretraining, have their parameters frozen to maintain this knowledge, and then queried by users using handcrafted, natural language prompts. For example, if one wants to know what the capital of Canada is, they might query a frozen model with the prompt, "The capital of Canada [MASK]. ." while another might different wording: "Canada, which has the capital city [MASK]." "" In order for LLMSs to be effective knowledge bases, they have to be robust to different queries users could provide. Unfortunately, prior work has shown that LLMs are not robust: queries that are semantically equivalent may lead to inconsistent predictions (Jiang et al. 2020 Elazar et al.) 202T]. For example, taking the prompts above, the first prompt does extract the correct answer "Ottawa" from BERT Base, but the second extracts the incorrect "Winnipeg". This observation has led many works to try to find the optimal prompt or set of prompts for a given relation: the one(s) that will allow models to extract factual information the best (e.g.. for the relation capital.o These works have proposed ensembling sets of prompts (Jiang et al. 2020), optimizing which tokens included in the prompt (Haviv et al. 2021 Shin al. 2020). or forgoing the difficult discrete optimization problem entirely and instead optimizing the continuous embeddings that are input to
the L.MM (Li & Liang. 2021 Qin & Eisner 2021 Zhong al. 2021).
In our work, rather than focus on finding a single optimal prompt, we aim to help LLMs overcome this variability by adapting natural language prompts into continuous representations that allow
bb newman@cs stanford.eedeeoo...o.ooo Work conducted during internship Salesforce Research.

---

LLMs to accurately predict factual information. Because are motivated by user-focused setting, we want our adaptation method to require only a natural language prompt (e.g. "The capital of Canada is [MASK] ") at inference time. No additional annotation for the relation (capital of) between the subject ("Canada") and object ("Ottawa") of the entity pair in the prompt should be required. Nor should we need the identity of the subject separate from the prompt. At training time, our ideal method would only rely on (prompt, object) pairs (e.g. ("The capital of Canada is [MASK]", ", "Ottawa")), so collecting new data would not require additional annotation. Our research question is then as follows: in a setting where we are trying to prompt a frozen LLM using natural language alone, how can we encourage models to produce more consistent and accurate responses to potentially varied prompts.
We introduce class of models called P- Adapters (short for "prompt adapters") to per- form this adaptation, which satisfy our infer- ence and training desiderata. P-Adapters sit between the embedding layer and the first at- tention layer of the frozen LLM, and modify the LLM's embeddings so factual information can be more effectively predicted (Figure [ They are optimized end-to-end, only requiring (prompt, object) pairs at training time, and im- plicitly encourage consistency by learning to map variable training prompts to the same ob- ject.
Ottawa
Pretrained LLM
0 Ó 0 0
P-Adapter
0 I I
LLM Embedding
[MASK] Canada's capital
Captial of Canada si [MASK]
Capital Canada is [MASK]
The P- P-Adapter Framework: P-Adapters sit between the embedding layer and the first at- tention layer of the LLM. They take input the LLM embeddings and output continuous prompts that are fed into the LLM. The LLM is frozen (embeddings included), while the parameters of the P-Adapter are trained. The adapter helps miti- gate variability among different phrasings and ty- pographic errors in the prompts (as shown in the example inputs).
Figure 1:
We also investigate other models that could increase the consistency of natural language prompts: Mixture of Experts (MoE) models. Rather than adapt a natural language prompt, a MoE model maps it to an "expert" continu- ous prompt based on the relation between the prompt's 's subject and object. The MoE model then queries the frozen LLM with that expert. This method reduces variability by mapping all natural language prompts concerned with the same relation to a single prompt optimized for that relation. These methods require additional annotations during training and inference (namely the relation between the entities and the identity of the entity subject), and therefore do not conform to our desiderata. However, we include them in our discussion because they are useful for comparing to previous work and for understanding why P-Adapters are successful. We find that P-Adapter and MoE methods both strongly outperform a baseline of using the natural language prompts alone by between 12-28% on precision and 36-86% on consistency. While MoE models usually perform better, the small increase comes at the cost of requiring relation annotations at training time and spec- ifying the subject of the entity at inference time. To simulate the variable kinds of natural language prompts expect methods to handle, tings: on a different set of natural language prompts, on a different distribution of entities (uniform rather than the distribution of Wikidata), and on natural language prompts with typographic errors. These OOD evaluations are especially important in light of recent work has found that models that appear to have knowledge of factual information often rely on surface-level feature associations. For example, models ignore negation, utilize stereotypical associations when making predictions (Kass- ner & Schütze 2020 Poerner et al., 2020). and pick up on the distribution of objects from their training corpus rather than learn how to access arbitrary facts (Zhong et al.] [021]] CCo et al. 2021) We find that the OOD distribution of entities is the most challenging setting, and while typographic
evaluate them in three -ittof--dittiuutiinn (OOD) set-
errors reduce model performance, they do so by less than might be expected.
Finally, we investigate what makes P-Adapters effective, finding that it is important to keep some of the original natural language prompt embeddings available to the LLM. In particular, the availability of the subject of the prompt is the most important factor, in contrast with some other work suggesting the subject matters less (Cao et al. 2021
2

---

Overall, we conclude that P-Adapters are helpful in reducing the impact of variable prompts, and are able to do so without requiring additional annotations. To encourage the use of P-Adapters to effectively extract factual information, release the code used to train them.
2 RELATED WORK
Prompting I Language Models for Factual Knowledge. Prompting offers low-parameter alter- native finetuning I.LMs. In prompting, the LLMSs are usually frozen and their pretraining task is used to fill in the desired information, which is usually single token (Trinh & Le 2018 Davison ct al. 2019: Petroni al., 2019). Cao et al. (202I) outlines three main approaches for prompting LLMs for factual information. The one we focus on here involves querying them zero-shot with a prompt that contains the single entity of interest (like in Figure !
(Petroni et al. 2019).
Much previous work tries to lower-bound how much factual information LLMs store. Petroni et al. (2099) use one natural language prompt for each relation, while subsequent work argues that this approach underestimates LLM''s abilities given how sensitive they are to input prompts. To support his argument, [iing al. (2020) and |Elazar et al. (202T) generate paraphrases of prompts in the LAMA dataset and ensemble them to get score for each relation. Others argue for optimizing discrete (Shin et al. 2020, Haviv et al. 202T) or continuous (Qin & Eisner 2021] Zhong et al.,
2021 Liu al. 2021b) prompt for each relation instead of working with language ones. |Eazar et al. (021) thoroughly investigate prompt sensitivity in this setting, finding models often predict different entities for semantically equivalent prompts. Further, they propose continuing to train LLMs with a consistency loss function to improve their robustness. In this work, however, we focus on a user-inspired and lightweight approach that does not require updating LLM parameters.
Robustness and Extracting Facts. Training robust models ones that are invariant to semantics- preserving input perturbations is challenge in many settings (Szegedy al.. 2013 Belinkov & Bisk 2018) and has to led to theoretical and empirical innovations such as adversarial training
SSinha et al. 2017), provably robust models (Dvijotham et al.) 2018), or more robust encoding schemes (Jones et al. 2020 Our works falls into this last category. In NLP, prior work investigates
invariances to typos, word substitutions, irrelevant sentences, and syntactic perturbations (Jones
ct al. 2020 Alzantot al. 2018 Jia & Liang. 2017: Lyyer et al. 2018).
For robustly extracting factual information, Poerner et al. (2020) and Kassner & Schütze (2020)
argue that LLMs exploit surface form regularities when making predictions. Zhong al. 2221) and Dufter et al. (2021) make a related observation, concluding that simpler models like randomly initialized LLMS static embeddings. and even a Naive Bayes model can achieve a precision better than majority baseline. |Cao et al. (2021) argue that prompts that were found to do well in previous work overfit the distribution of objects in the training data rather than enabling knowledge extraction. They show that prompting with different sets of entities leads to very similar predictions. Therefore, we adopt an OOD evaluation setting address these concerns. Low parameter finetuning methods (Houlsby et al. 2019 Ben Zaken et al., [222 Liu et al.) (201a) have also been shown to have better OOD performance for generation and classification tasks (Li Liang 2021 which we hope to leverage for factual extraction here.
Lester et al.] 2021)
3 TERMINOLOGY
Taking inspiration from Liu et al. 102 we use the following terminology throughout this work. Each fact want to extract consists of two parts: entiry pair-a a subject (æ) and an object (v) and the relation (r) between them. We use the 41 relations in the LAMA dataset (Petroni et al, 2019). Each relation is expressed through a number of templates (c.g., one is "The capital of [X] is [Y]"). To query a language model for factual information, we substitute the subject in for the "[X]" in the template, and a [MASK] token into the object's place (the "[Y]") to form prompt (p). We refer to this as a natural language prompt to emphasize that it consists of natural language tokens. Our prediction problem then consists of mapping p to y.
https:///iithub.comm///
3

---

LLMs perform this mapping by using their embedding layer, e, embed p into sequence of continuous vectors, and then inputting this sequence to their first attention layer. We refer to this sequence of continuous vectors that are imput to the LLM as a continuous prompt (Peon). We make this distinction because P-Adapters sit between the embedding layer and the first attention layer of the I.L,, they take input e(p) and output continuous prompt Pront that differs from e(p). After passing Poont through the attention layers of the LLM, we take the LLM's prediction (j) to be the token assigned the highest probability in the location of the [MASK] token in p. and compare it to to assess the model knowledge of the fact. Formally, a model is consistent if for any two prompts Pr. P2 for the same (z. y, r) triple, its top-1 predictions ÿ1 and f/2 are equal.
4 METHODS
To test extracting factual knowledge (rather than overfitting to training templates or entity pair dis- tributions), we evaluate our P-Adapters in four settings:
1. ID templates and objects for testing in-domain generalization.
2. 0OD Prompts for testing generalization to novel natural language prompts.
3. OOD Objects for testing whether our P-Adapter models learn to match the distribution of objects in the training set rather than predict arbitrary object entities.
4. OOD Keyboard Errors for testing robustness to typos in the natural language prompts.
4.1 DATA
Entity Pairs. We use the entity pairs and relations from the T-Rex split of the LAMA work (EI- sahar et al.] 2018 Petroni et al. 2019) in our experiments. This data includes 41 relations and ap- proximately 1000 entity pairs for each relation sampled from Wikidata, including citations where the relation manifests in Wikipedia. This data is used for evaluation. For training and validation, we use separate sets of entity pairs for each relation collected by [Shin al. (2020), which they use to optimize their discrete prompts. The entities pairs in the training, validation, and evaluation sets are all disjoint, though the distribution of object entities is similar. For the OOD Objects setting, we use the entity pairs in the uniform-wikide constructed so that each object appears the same number of times for each relation, in contrast to the ID evaluation set which contains a less uniform distribution of objects from Wikidata.
dataset from Cao et al. (202I). This dataset was
Templates. The templates we use are pooled from prior work: LAMA, LPAQA, and ParaRel datasets (Jiang et al. 2020; Elazar et al. 202T). LPAQA includes templates created automatically with a paraphrase model, mined from Wikipedia, and written by annotators, and ParaRel contains high-quality human- human-written written templates. Additionally, we augment the ParaRel templates using the BERT lexical substitution system from Lee et al.] (2021) by generating five paraphrases for each template, and then manually filtering out generations that do not preservens semantics. This gives us on average 81.4 prompt templates per relation (For more statistics, see Figure 6). We split the templates into two cqual-sized groups: one for training and one for OOD Prompt evaluation. For the OOD Keyboard Errors setting, we use the training templates, except we introduce least one typographic error in each template using the nipaug package (Mal 2019)
4.2 MODELS
First we introduce our P-Adapter models, and then we introduce our MoE models, and we finish with our oracle and baseline models. Recall that P-Adapters intercede between the LLM's embedding layer, e, and the first attention layer (Figure [). We extract information from BERT Base Cased, BERT Large Cased, and RoBERTa Large (Devlin et al.) 2018 Liu et al., 2019). Because these models' pretraining corpera contain Wikipedia, they should have access to the facts test.
P-Adapter. P-Adapter models take input the embeddings of the natural language prompt, e(p), and output a new sequence of continuous embeddings, Peoms that are used as input to the LLM in place of e(p). Formally, P-Adapter is function fP.Adupter e(p) - Peont trained maximize
4

---

Pous
Poot
Point
Rewrite
Prefix
P-Tuning
MLP
MLP
nax pool
nax pool
BiLSTM
BiLSTM
BiLSTM
e(p)
=(p)
(a) P-Adapter Proposals
MoE
100000
100000
Oracle
BERT (Finetuned)
Gold Relation
<(p)
<(p)
(b) Mixture of Experts and Oracle
Figure 2: P-Adapters lie in between the LLM Embeddings and the rest of the model 1 We propose end-to-end P-Adapters (2a) as well as a Mixture of Experts model (2b). Figures 2b and 2a are sub- stituted into the "P-Adapter" block in [ The subject embedding in blue; the [MASK] embedding is red; embeddings generated by the P-Adapter are yellow; and other unmodified embeddings are gray. Dotted arrows represent inputs and outputs model components and solid arrows represent copying from the input of the P-Adapter to its output.
PLM(y Peout) The LLM's prediction is then:
arg max PLM (v (ee))))er(e(p))). VEV
where the LLM vocabulary. There are many different possible parameterizations of JP-Adapiers and we describe three here. Two (Rewrite P-Adapter and Prefix P-Adapter) require no additional annotations, fitting the training and inference data criteria put forth in the introduction. The third (P-Tuning P-Adapter) does not fit our criteria, but we include it because it provides insight into what makes a P-Adapter effective (See Figure 2a for an illustration of each.)
The simplest P P-Adapter we investigate is the Rewrite P-Adapter. rameterized by d/2-dimensional bidirectional LSTM followed by single MLP applied to each hidden state independently (where d is the LLM's hidden dimension). The output of the MLP, Peants input to the LLM. We use LSTMs to better compare our results with those of Liu et al. (202 who also use them, and because want use only small number of parameters. Additionally, the prompts tend to be short and we do not do any pre-training, so LSTMs are effective.
The Rewrite P-Adapter is pa-
Next look at the Prefix P-Adapter. The Prefix P-Adapter is parameterized similarly to the Rewrite P-Adapter; however, its output is a prefix embeddings that are prepended to e(p). Like other works that learn prompt prefixes, the prefix has fixed length (we use nine compare with later models) (Li & Liang 2021 Lester et al. (021I). Formally, using the Prefix P-Adapter, the input the LLM is [(((p)); e ep)]] where g the application of Bi-LSTM, a max pool, and an MLP to project the max pool's output into the size of the prefix. Unlike the Rewrite P-Adapter, the Prefix P-Adapter keeps all of the embeddings ep)) accessible the LLM's first attention layer.
Finally, we investigate the P- P-Tuning P-Adapter. This P-Adapter is based on work by Liu et al. (202Tb) which presents method called p-Tuning that learns continuous prompt for each relation in the LAMA dataset. The method learns function - Rdx9 The Peom consists of the output of g. the LLM's unmodified embedding of the subject, and the LLM's unmodified embedding the mask token. It has the following form:
poa)) e(x); 9()) e( [MASK] ))
where bracket notation represents python-style list indexing.
The P-Tuning P-Adapter is implemented in a similar fashion, except g takes as input e(p) rather than and is parameterized similarly to the Prefix P-Adapter. Like the Prefix P-Adapter, the method out-
5

---

puts fixed number of embeddings: however, unlike the Prefix P-Adapter, only the embeddings for the subject and [MASK] token are kept unchanged, rather than all e(p). Note that this P-Adapter additionally requires annotations for the identity of the subject during training and inference. This means that it does not fit the desiderata laid out in the introduction, but is included because allows us investigate what makes P-Adapter successful.
Mixture of Experts and Oracle. Enforcing consistency does not require modifying a natural lan- guage prompt like the P-Adapters do. One can also map each natural language prompt to a canonical continuous prompt, and use this prompt to query the frozen LLM. We explore this option next with our MoE and Oracle methods. In these, the canonical prompt is a continuous prompt optimized for the relation between the entities in the natural language prompt Liu et al. (2021b). These con- tinuous prompts are learned using P-Tuning, and have the same format as described previously in the P- P-Tuning P-Adapters section; they ignore e(p), depending only the z, MASK, and (Liu ctt al. 20. We train these continuous prompts with P-Tuning using the same method and hyper- paramters reported in Liu et al. (2021b), with one prompt for each of the 41 relations. In contrast to the P-Adapter methods, these require additional annotations with the relation of the prompt (See Figure 2bp.
The Mixture of Experts (MoE) model consists of two components. The first is a classifier that predicts the relation between the entities of natural language prompt. This classifier is BERT Base Cased model finetuned on a 41-way classification task. The second component is look-up table to map the predicted relations to the canonical continuous prompts.
Note that this is similar to a traditional mixture of experts model approach except that we do not use a weighted combination of prompts from different relations and instead just use the single prompt from the predicted relation.
The Oracle method is similar to the MoE approach, except rather than using a classifier to predict the relation, the gold relation is used at inference time. This method is an oracle because it achieves perfect consistency-
makes the same prediction for all prompts with the same entity pair. The oracle also serves as direct comparison to the prior work of P-Tuning because P-Tuning assumes access to the gold relation.
Baseline. Finally, the baseline is model that takes input the natural language prompt without any prefixes or optimization. Any effective and useful P-Adapter network would have to perform better than just using the natural language prompt itself.
4.3 METRICS
We report two metrics: precision@) @1 (P@ 1) and consistency. P@1 is a stringent measure of whether prompt can extract a fact. It is the fraction of prompts where the correct object the LLM's top prediction. Consistency measures whether a model's 's predictions for a given entity pair match across the different prompts p's that contain the entities. We calculated consistency using the method from Elazar et al. (2021): the consistency of knowledge of fact is the proportion of pairs of prompts for the fact where the model makes the same prediction. Formally, given set of un- ordered pairs of prompts with the same entity pair, P, where there are unique prompts, and letting Tp) = arg max,ey PLM (v Sprompal (e(p)) return the model's top prediction given a natural language prompt x', the consistency is defined as:
consistency(Top-117 E(ppDer 1 (Top-1(p)) = Top-
1 nn 1)
Note: a model can be consistent and inaccurate by always predicting the same incorrect object.
5
RESULTS
We present our main results for BERT Base in Table I The results for BERT Large and RoBERTa Large show similar trends and are available in the Appendix (Tables Figures and and Qualitative Results are in Appendix[D] Across all evaluation settings, we find optimized prompts lead to higher precision than natural language ones, the low performance of the baseline indicates.

---

Metric
P-Adapter
ID
OOD Prompes
0OD Objects
OOD KE
Baseline
0.157
0.157
0.069
0.092
Rewrite
0.258 1 0.00
0.247 = 0.00
0.078 1 0.00
0.167 1: 0.00
Prefix
0.425
0.415 = 0.01
0.193 1 0.00
0.326 0.01
P@
P-Tuning
0.442 = 0.00
0.422 = 0.00
0.203 + 0.00
0.325 + 0.00
MoE
0.188 = 0.01
0.418 0.00
0.237 + 0.02
0.331 0.00
Orucle
0.496 1 0.01
0.496 = 0.01
0.238 1: 0.02
0.496 1: 0.01
Baseline
0.126
0.133
0.097
0.068
Rewrite
0.476 0.01
0.448
0.02
0.456 + 0.01
0.223 0.01
Consistency.
Pretix
0.656 0.02
0.613 0.02
0.588 = 0.03
0.452 + 0.03
P. P-Tuning
0.730 + 0.00
0.646
0.01
0.656 + 0.01
0.476 +: 0.01
MoF.
0.947 +: 0.03
0.658 = 0.03
0.916 1: 0.05
0.439 1: 0.01
Oracle
1.000 0.00
1.000 0.00
1.000 + 0.00
1.000 0.00
Table I: P@ and Consistency for BERT Base across all our settings. P-Adapters are separated based on whether they require additional training/inference data. Note that the consistency of the Oracle is 1.0 and that the Baseline does not have any standard deviation because there no op- timization across runs. Results are microaveraged over all relations. (OOD KE: OOD Keyboard Errors)
Precision. Comparing the different evaluation settings, observe the following. First, the OOD Objects was the most challenging setting, on average 20.41 % lower precision than the ID setting, even for the oracle. However, the models performed similarly the OOD Prompts they did on the ID ones. At first this might seem to conffict with previous work that finds that the prompt has a large impact on performance (Jiang et al.. 2020). but these results make claims about average rather than individual prompt performance. That said, precision is still higher ID than OOD, particularly for the MoE model. Looking the MoE's relation classifier predictions, we can see it achieves an F1 of 99% for the ID and OOD Objects settings, but only 81% for OOD Prompts (See Appendix F). While this suggests some overfitting, our methods still outperform the baseline, so what they learn is still useful for the OOD natural language prompts. Finally, when evaluated on the OOD Keyboard Errors, we find that models still outperform the baseline. However, precision on the corrupted prompts is lower than the uncorrupted ones, and drops by larger absolute percentage than the baseline does.
Consistency. Additionally, across all variation types we can see that optimized prompts lead to more consistency among the models" predictions. The consistencies dramatically increase from less than 0.2 for the baseline over 0.4 (Table [ Models were the least consistent on the OOD Keyboard Errors, and interestingly, were similarly consistent between the OOD Prompts and OOD Objects evaluations despite having a much higher precision for the OOD Prompts. The oracle has perfect consistency of 1.0, because it uses the same continuous prompt for all facts with the same relation, so the predictions for that fact are the same. The MoE model has a high consistency for a similar reason: as long as the classifier predicts two prompts for the same entity pair come from the same relation, the predictions will be the same, and therefore consistent. The P-Tuning P-Adapter often has a high consistency as well, sometimes even higher than the MoE. This suggests that in P-Tuning P-Adapter, the main factor driving the model's prediction the subject of the entity pair, because the unmodified embedding of the subject is shared among all of the prompts for the same object.
In the P-Adapters, this consistency is implicitly encouraged by the current training procedure by training P- Adaatrrss to map from various natural language prompts to the same object. We investigate adding an additional loss term to explicitly encourage consistency and find that doing so increases consistency by 5% on average. This suggests that implicitly encouraging consistency can be helpful but is not necessary. The procedure and results are detailed in Appendix B
Unmodified Embeddings. We observe that giving the LLM access its unmodified embeddings of the natural language prompt was helpful. The Prefix P-Adapter and the P-Tuning P-Adapter, which have access to some or all of these embeddings, achieve higher precisions than the Rewrite P-Adapter, which alters all of them. Because of this, we perform additional experiments to see what part of natural language prompt important keep.
7

---

Unmodified Embeddings
Precision vs a
0.4
Rewrite
0.4
P-Tuning
P@1
0.2
0.2
0.0
0.0
None
[MASK]
[MASK]
Subject
Subject+ [MASK]
Subject+ [MASK]
Q = 0.00
0 0.10
0 0.25
0 0.50
0.75
0 = 0.90
0 = 1.00
Rewrite
P-Tuning
Figure 3: The above results show which of the LLM's 's embeddings important keep unmodified. On the left, the x-axis has the embeddings that are not modified by the P-Adapter (e.g. "None" is the Rewrite Addaperer, where all embeddings Base and the ID data. In the plot on the right, we compare different a's to the P-Tuming P- P-Adapter and the version of the "Rewrite" P-Adapter with the unmodified subject and [MASK] embeddings, finding the a 0.5 performs as well as the other adapters that require additional annotation.
modified). These results computed with BERT
One hypothesis is that the [MASK] token embedding should remain unmodified, and the Rewrite P- Adapter performs poorly because it corrupts this embedding. We investigate this in two ways. First, we train a version the Rewrite P-Adapter that replaces the P-Adapter's 's output in the [MASK] to- ken position with the LLM's [MASK] token embedding. Second, we train version of the P-Tuning P-Adapter whose output only has the LLM's [MASK] embedding, not the subject embedding. How- ever, we find these do not increase performance much compared the Rewrite P-Adapter.
Another hypothesis is that the subject embeddings must remain unmodified, and that the Rewrite P-Adapter corrupts the subject, leading to lower precision. To address this, we train version of the Rewrite P-Adapter model that replaces the embeddings for the subject in the P-Adapter output with the LLM's embeddings for the subject. We find this much more effective, increasing precision by 13%. For comparison with the P-Tuning P-Adapter, which keeps the subject and [MASK] em- beddings the same, we also train a version of the Rewrite P-Adapter that replaces the embeddings at both the subject and [MASK] token positions with the LLM'' embeddings and find that this per- forms only 4% worse than the P-Tuning P-Adapter. The results are visible in Figure 3 From these experiments, we conclude that the most important aspect of the prompt is the subject token. This also helps explain why the Prefix P-Adapter does well: it has access to the LLM's embeddings for the entire natural language prompt, which includes the subject.
Unfortunately, while these changes to the Rewrite P-Adapter models show promising results, they require knowing the index of the subject tokens training and inference time, which contradicts our requirement to not use extra annotations. Fortunately, there is another way to incorporate the unmodified LLM's embeddings of the natural language prompt into the output of the P-Adapter: we can interpolate between the P-Adapter output and the unmodified embeddings. To do this, we train a third type of Rewrite P-Adapter. If frewtit--alapper is the original Rewrite P-Adapter, the equation for the new Rewrite P-Adapter is:
fP-Adapter (p) = we(p)) + (1 a- (eeeriteeaddpeee(((()))))
where a is a hyperparameter. When a = 0, this P-Adapter is equivalent to the original Rewrite P- Adapter, and when is the baseline. We test {0.1, 0.25, 0.5, 0.75, 0.9). and find 0.5 to perform the best. outperforms the Rewrite P-Adapter when the subject and [MASK] tokens are substituted in, though barely underperforms compared the P-Tuning P-Adapter (See Figure 3)
8

---

6 DISCUSSION
Our goal was to create models to adapt arbitrary user queries predictions from frozen LLMs were accurate and consistent. One desiderata was to train with only (prompt, object) pairs, so that user's experience would match current IR system experiences. Despite this requirement, some of the methods we introduce do use additional information-for comparison to other methods and understanding our own. For example, the P-Tuning P-Adapter and the MoE models both need the identity of the subject of the user's prompt, and requiring a user to provide this feels cumbersome and redundant. One could obviate this need using an off-the the shelf NER system as well, and we leave exploring this to future work.
Additionally, the MoE models train their classifiers using annotations for the relation between the entities in the prompts. We use these annotations to compare to previous work: previous work as- sumes that relation is the most useful intermediate variable for factual extraction, but this does not have to be the case. For example, good prompts across different relations like "Ottawa is the capital of [MASK]" and "The second largest country by land area is [MASK] might share components even though they do not share relation. Our P- P-Adapters allow for greater flexibility by condition- ing on the natural language prompt itself rather than the relation. Importantly, while using extra information leads better results in some cases, the Prefix P-Adapter usually achieves compara- ble precision. And while the Rewrite P-Adapter flounders, ensembling its outputs with the LLM embeddings leads greater success.
We speculate that there are two likely reasons for the P-Adapters' success. First, not updating the LLM parameters likely maintains factual knowledge learned during pretraining- -lazarr et al. (202I) show that continuing to finetune the models on extracting factual information can lead to lower accuracy. Second, the natural language prompts are likely close to being effective prompts, so small changes from P-Adapters are sufficient to elicit the correct objects.
is also worth noting that while the Prefix P-Adapter approaches the performance of the techniques with more information, this performance is still quite modest. The oracle was able to achieve only a precision of 50% in the cases with ID objects, missing half of the facts. For the OOD Objects evaluation set, it performed much worse, only correctly predicting about 25% of the facts. While the P@ metric is quite strict, these numbers do not bode well for this approach. Cao et al. (202T) blame this poor performance and they observe that when the subjects are changed, the LLM's distributions do not change all that much. They even observe that the LLMS's distributions when prompts have subjects are similar to those when the subjects are completely omitted. Our findings point to a more nuanced result. While it might be true that the LLM's distributions in these cases are correlated, the subject does matter when considering the top prediction. We find that without the LLM's embeddings of the subject accessible, the P-Adapters performs even more poorly.
the optimized prompts overfitting to the training object distribution,
7 CONCLUSION
LLMs implicitly learn factual information during pretraining. but accurately and consistently ex- tracting this information is challenging. Our work complements previous work on extracting factual information from LLMs by introducing more user-oriented setting: we want our models only take in varied natural language prompts, and return the objects that meet information needs. To do this, we propose P-Adapter models, which sit between the embedding layer and the first attention layer of LLMs. These are effective compared to other methods that require additional annotations (e.g., the relation between the entities in the prompt), and we perform ablations determining that their success is due to keeping the LLM's embeddings of the subject available to the LLM. While we focus on the task of extracting factual information from LLMs, P-Adapters potentially provide general framework for adapting to variable inputs, for example in reading comprehension di- alogue. While there is still room for the further improvement in using LLMs as knowledge bases, and we P-Adapters as important part of the future of this endeavor.
9

---

8 ETHICS STATEMENT
While our work is inspired by how users interact with current IR systems, we do not perform any experiments with human subjects. Additionally, our method improves ability to extract factual information from pretraining corpora, this means it may be possible to exploit our method to elicit the biased or harmful predictions from these corpora as well, though we do not study this. Finally, improving the ability to accurately and consistently extract information from text sources has privacy concerns LLMs are pretrained on private or otherwise sensitive data. For the LLMs study, this is not concern as they trained Wikipedia and Books Corpus data rather than web data.
REPRODUCIBILITY STATEMENT
The descriptions of our models and training procedure in Section 4 and the code available at the link below is sufficient for reproducing our results. The data we use is available in the Github repositories of the work we cite that creates these resources. For running experiments, see the README. file here: https: / t.hub. com/ sa les Force/Facct..
ACKNOWLEDGMENTS
The authors would like to thank Wenhao Liu, John Hewitt, and Alex Tamkin for their valuable discussions and feedback. We would also like to thank our reviewers for suggesting additional experiments and analyses to strengthen the claims made the work.
REFERENCES
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai- Wei Chang. Generating natural language adversarial examples. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2890-2896, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/ vI/D18-1316. URL (ttps:::: aollanhhhooyyy... org/D1 1316
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine trans- lation. In International Conference on Learning Representations, 2018.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-effiiiet fine-tuning for transformer-based masked language-models. arXiv e-prints, pp. arXiv-2106, 2021.
Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. Knowledgeable educated guess? revisiting language models knowledge bases. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1870-1874, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ vl/2021.acl-long.. 146. URL//ttpes//eelaattooloo.......
attp:/2222..c111omom/.. 146
Joe Davison, Joshua Feldman, and Alexander M Rush. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCNLP) pp. 1 1173-1178, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04880 2018.
Ming-Wei
Philipp Dufter, Nora Kassner, and Hinrich Schütze. Static embeddings as efficient knowledge bases? In Proceedings of the 2021 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies. pp. 2353-2363 Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/ 1/202/ naacl- main. 186. URL https: //acl antho ogy. www.orgg/2222 naacl -ma in. 186
10

---

Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O'Donoghue, Jonathan Uesato, and Pushmeet Kohli. Training verified learners with learned ver- ifiers. arxiv preprint arXiv:1805.1022 10265, 2018.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod- els. arXiv preprint arXiv:2102.01017 2021.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Associa- tion (ELRA).
Adi Haviv, Jonathan Berant, and Amir Globerson. BERTese: Learning speak to BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 3618 3623, Online, April 2021. Association for Computational
Linguisties. URL
org/ 2021 eacl mani.. 316
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings Machine Learning Research, pp. 2790-2799, PMLR, 09-15 Jun 2019. URL https:: http:::://poocednnggs...
pxess/v97/houlsbyLis html
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example genera- tion with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1885, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/11/N18-1177 URL https:/7eee antho log..ooog....................... Scgy.org/N18-117
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In EMNLP, 2017.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, 8:423-434 07 2020. ISSN 2307-387X. doi: 1a116///aclaa000220 URL http:::://cc.ooo//////tt Eacl a 00324
Erik Jones, Robin Jia, Aditi Raghunathan, and Percy Liang. Robust encodings: framework for combating adversarial typos. Transactions of the Association for Computational Linguistics, 2020.
Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7811-7818, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020.aao actp://222202cc222//ea www.mmmmi........ URL. htt.s:://cclattholl.
org/
2020. acl main. 698
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR (Poster), 2015. WRRLhttp://aaxit org. /as//411 6980
Mina Lee, Chris Donahue, Robin Jia, Alexander Iyabor, and Percy Liang. Swords: A benchmark for lexical substitution with improved data coverage and quality. Proceedings of the 2021
Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4362-4379, Online, June 2021. Association for Computa- tional Linguistics. doi: 10.1 10.18853/v1/2000 1at.001..aaclmaaaaa. ma...-m URL
org/2021, naacl maa i...ann.11 345
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. ArXiy, abs/2104.082 2021.
11

---

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Natural Language Processing (Volume 1: Long Papers),
11th International Joint Conference pp. 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 18653/ URL https: http::::///cclaathoolggg...///// Grgg acl-logg 353
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.13586, 2021a.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. arXiv preprint arXiv:2103.10385 2021b.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
Edward Ma. NIp augmentation. http://gghhbbooom/nkkccccaaaaau 2019.
Donald Metzler, Yi Tay, Dara Bahri, and Marc Najork. Rethinking search: making domain experts out of dilettantes. In ACM SIGIR Forum, volume 55, pp- 1-27. ACM New York, NY, USA, 2021.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMMN.P-JCCNLP),, pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/1/DD19-1222 URL
Nina Poerner, Ulli Waltinger. and Hinrich Schütze. E-BERT: Efficient-yet-effeee entity em- beddings for BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 803-818, Online, November 2020. Association for Computational Linguistics. doi: 10. 1118653/11/2000 122020 /tt.2/00.iningeeeenp.....
URL. htthh::////e ptttps://aclanthcllooo
anthelogy..o org/2020.
Findings ema.p...-mmmp..... 71
Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings the 2021 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5203-5212, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2222 11t.65///12222...ammmmm....l URL
https: www.aaaananntto....c.oooaooooole org/2021 www.naaaaaaaaaaam.aa 410
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto- Prompt:
Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222-4235, Online, November 2020. Association for Computational Linguis- TRLhttps: /tt///tcpp////cllaaccccoooo.o org/200000.0
tics. doi: 1ttp:///1122200.mmmmmmaaama aunnppmain.. 346
Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10577 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Trieu H Trinh and Quoc Le. A simple method for commonsense reasoning. arXiv: 1et.1806602477 2018
arXiv preprint
12

---

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing- In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstratinns, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. URL https: //ww: aclweb. www./otttggggggotooooooo 2020. Seannp-ddmeee
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]: Learning vs. learn- ing to recall. Proceedings of the 2021 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5017-5033, On- line, June 2021. Association for Computational Linguistics. doi: 10. 18653/v1/222) .naccl main. 398. URL UtlL:/tttss//aa..tttoooo.oooooo/
mwacl-maalee 398
A
ALL RESULTS
See Figures
and 5
as well as Tables 2 and
LLM
P-Adapter
ID
OOD Prompts
OOD Objects
OOD KE
Baseline
0.157
0.157
0.069
0.092
Rewrite
0.258 = 0.00
0.247 + 0.00
0.078 = 0.00
0.167 0.00
BERT Base
Prefix
0.425 + 0.01
0.415 0.01
0.193 + 0.00
0.326 0.01
P-Tuning
0.442 + 0.00
0.422
0.203 + 0.00
0.325 0.00
MoE
0.488 + 0.01
0.418 0.00
0.237 = 0.02
0.331 - 0.00
Oracle
0.496 + 0.01
0.496 1 0.01
0.238 = 0.02
0.496 0.01
Baseline
0.163
0.160
0.077
0.099
Rewrite
0.117 - 0.06
0.115 0.06
0.042 + 0.02
0.072 + 0.03
BERT Large
Prefix
0.431 + 0.03
0.424
0.207 + 0.01
0.337 0.02
P-Tuning
0.470 0.00
0.443 0.00
0.222 = 0.00
0.346 0.01
MoE
0.498 + 0.01
0.431 + 0.01
0.250 - 0.02
0.339 0.01
Oracle
0.510 + 0.02
0.509 + 0.02
0.258 + 0.03
0.510 + 0.02
Baseline
0.122
0.121
0.054
0.071
Rewrite
0.222 + 0.04
0.214 0.04
0.068 + 0.01
0.148 0.03
RoBERTa
Prefix
0.385 + 0.03
0.385 + 0.03
0.179 + 0.02
0.311 + 0.02
Large
P-Tuning
0.360 = 0.07
0.359 + 0.06
0.167 0.03
0.306 + 0.03
MoE
0.335 + 0.11
0.290 + 0.09
0.156 - 0.05
0.231 0.06
Oracle
0.340 + 0.11 0.340 0.11
0.157 = 0.06
0.340 0.11
Table 2: Tabular format showing precision from Figure 4
The Baseline does not have any standard
deviation because there randomness across runs. Results microaveraged over all relations. (OOD KE: OOD Keyboard Errors)
B
TEMPLATES
See Figure and Table
13

---

LLM
P-Adapter
ID
OOD Prompts
OOD Objects
OOD KE
Baseline
0.126
0.133
0.097
0.068
Rewrite
0.476 + 0.01
0.448 + 0.02
0.456 + 0.01
0.223 + 0.01
BERT Base
Prefix
0.656 0.02
0.613 0.02
0.588 = 0.03
0.452 0.03
P-Tuning
0.730 + 0.00
0.646 0.01
0.656 = 0.01
0.476 0.01
MoE
0.947 = 0.03
0.658 + 0.03
0.916 = 0.05
0.439 0.01
Oracle
1.000 = 0.00
1.000 = 0.00
1.000 = 0.00
1.000 0.00
Baseline
0.122
0.123
0.094
0.069
Rewrite
0.550 = 0.22
0.538 0.23
0.571 = 0.22
0.399 0.29
BERT Large
Prefix
0.619 = 0.07
0.583 + 0.04
0.566 = 0.06
0.446 = 0.02
P-Tuning
0.759 + 0.01
0.656 + 0.01
0.694 + 0.02
0.484 + 0.02
MoF
0.955 0.03
0.680 0.04
0.933 + 0.04
0.456 0.03
Oracle
1.000 = 0.00
1.000 = 0.00
1.000 = 0.00
1.000 = 0.00
Baseline
0.093
0.096
0.076
0.050
Rewrite
0.438 + 0.03
0.403 + 0.02
0.421 = 0.02
0.217 0.01
RoBERTa
Prefix
0.540 + 0.07
0.531 0.06
0.491 = 0.08
0.425 0.04
Large
P-Tuming
0.731 0.13
0.707 0.15
0.702 = 0.18
0.600 0.21
MoE
0.974 = 0.01
0.661 + 0.03
0.970 = 0.02
0.436 = 0.03
Oracle
1.000 0.00
1.000 0.00
1.000 + 0.00
1.000 0.00
Table 3: Tabular format for the consistency from Figure The Baseline does not have any standard
deviation because there is no randomness across runs. Note that the Oracle models achieves consistency of 1. Results are microaveraged over all relations. (OOD KE: OOD Keyboard Errors)
Precision across all Datasets
BERT Base
0.5
0.4
0.3
0.1
0.0
ID
OOD Prompts
OOD Objects
COD Keyboard Errors
BERT Large
0.5
0.4
@
0.3
0.2
0.1
0.0
ID
OOD Prompts
COD Objects
COD Keyboard Errors
RoBERTa Large
0.5
0.4
0.3
0.2
0.0
ID
OOD Prompts
OOD Objects Prefix
COD Keyboard Error MoE
Baseline
Rewrite
P. Tuning
Oracle
three LLMss. Each row is different model, the x-axis is the dataset and error bars show standard deviation across either runs. We can see that the oracle method outperformed all others, while the baseline and rewriter did the worst. The trends are similar across all models-thh the only difference come from some RoBERTa MoE models having higher variance than others which is explained by some runs not finding the best P- P-Tuning templates.
Figure 4: Results across our ID and OOD datasets
14

---

Consistency across all Datasets
BERT Base
1.0
0.8
0.6
0.0
OOD Prompts
OOD Objects
OOD Keyboard Errors
BERT Large
K0
OOD Prompts
OOD Objects
Keyboard Errors
RoBERTa Large
0.6
0.4
0.2
0.0
ID
OOD Prompts PTTnning
OOD Objects
OOD Keyboard Errors Oracle
Baseline
Rewrite
Oracle
Figure 5: Consistency across all LLMs and evaluation sets.
Number of Templates per Relation
40
60
80
100
120
Figure 6: Distribution of number of templates across the 41 relations we tested. Minimum is 32, maximum 123 and mean is 81.44.
subject
Canada
object
Ottawa
relation
IS CE..APITTA
template
The capital of [X] is [Y]. The capital of Canada is [MASK). Toronto
natural language prompt prediction
Table Here the terminology is applied to the example from the introduction. Note that the model is incorrect in this example.
15

---

C TRAINING DETAILS
All of our P-Adapters were trained using the hyperparameters from Liu et al. (2021b): Adam opti- mizer with learning rate of le weight decay of 5e batch size of 128, and exponential learning rate decay schedule with decay rate of 0.98 (Kingma & Ba 2015). Fach result is the mean over five random seeds.
Our MoE classifiers were trained using an Adam optimizer with learning rate of 0.001 and linear learning rate decay (Loshchilov & Hutter 2018). We use HuggingFace Transformers train the model for 3 epochs on the same training data used to train the P-Adapter models (Wolf et al.) 2020). Each result is the mean over three random seeds rather than five because finetuning the classifier was more costly than training the the P-Adapters.
D QUALITATIVE ANALYSIS
We also performed qualitative analysis to determine how P-Adapters' predictions compare to the Baselines' and MoE models', We performed this analysis using the BERT base models, and focused on the Prefix P-Adptter.
In general, we find that the Baseline tends to include more nonsensical or untopical vocabulary items. For example, for P36, the capital_or relation, the baseline predicts the capital of "Iraq" to be '!" 'She', and 'It' for some prompts. There are also situations where despite this variability,, the baseline fails to predict the correct object for any of the prompts. For example, for the relation there 60 natural language prompts. The correct object of the sub- ject "Raahe" is "Finnish", and it does not appear in the baseline's 's 28 distinct predictions. However, half of the P-Adapter's predictions are the correct "Finnish".
While the P-Adapter's predictions are more consistent than the baseline's, they are less consis- tent than the MoE's. While this is undesirable from the perspective of consistency, it can lead to greater accuracy in some cases. For example, for the relation 1 wagguge..spoen the MoE model predicted the object "French" for the subject "Guy Maddin" for 100% of the prompts, while the variability in P- P-Adapters allowed them to predict the only correct answer, "English" in 89.3% of the prompts (and "French" in 9.3% of them).
There are downsides to this flexibility, however. We also observe examples where P-Adapters are tricked by the surface form of the prompt. For example, for the relation place of death, "Aki- hiko Saito" is predicted be Tokyo by 95% of phrases, even though the correct answer is "Iraq". There are also examples where P-Adapters' (particularly, the Prefix P-Adapters') predictions appear to be copied from the input where the MoE models' are not. For example, for prompts like "Windows Server 2003 was manufactured by [MASK). P-Adapters correctly predict "Microsoft" for 87% of paraphrases, but they also predict "Windows" 13% of the time, which we do not observe the MoE models doing.
E EXPLICITLY OPTIMIZING FOR CONSISTENCY
Currently, P-Adapters encourage consistency in LLMs' predictions implicitly, the loss encour- ages LLMs to predict the same, correct object for variable natural language inputs. However, one could imagine that explicitly encouraging consistency could be helpful. One way explicitly train for consistency is by encouraging not just the top-1 prediction to be the same, but the distribution over objects to the be the same for different natural language prompts for the same fact (Elazar al.) 2021). To do this, we train with pairs of natural language prompts for the same fact, and encourage the distributions over the predicted objects be similar using the symmetric KL-Divergence. For- mally, given two prompts for the same entity pair and relation Pr, P2, and an LM PLM we define our loss as:
Econsistency = DDK DDuL(PLss (PL))(P)) (1 PLM(P2)) + DD (PLM (LM)(1))
16

---

We then add this to the loss, e that previously calculating the cross-entropy loss between the distribution over the object and the correct object, giving a total loss of:
lotal l Reomsistency
where we use À 0.5.
This approach is inspired by the approach taken by Hlazar et al. (2021), with the differences being that they finetune all of the parameters of the I.L,, calculate this loss over all prompts with the same fact, and also find that in their setting they need to additionally continue with masked-language modeling training.
We train new Prefix-P- P-Aappeers on the BERT Base with this new loss, and find that the consistency increases slightly, and the P@] stays the same or slightly decreases. This boost in consistency is beneficial, but the small increase (compared to the baseline) does not suggest that explicitly encour- aging consistency is necessary (See Table
ID
OOD Prompts
OOD Objects
OOD KE
P@
Baseline
0.157
0.157
0.069
0.092
Prefix P-Adapter
0.425 + 0.011
0.415 = 0.009
0.193 + 0.004
0.326 + 0.012
+ consistency
0.415 0.002
0.408 = 0.001
0.190 0.002
0.326 + 0.001
MoE
0.488 + 0.007
0.418 = 0.002
0.237 + 0.020
0.331 + 0.003
Consistency
Baseline
0.126
0.133
0.097
0.068
Prefix P-Adapter
0.656 = 0.023
0.613 = 0.020
0.588 + 0.026
0.452 + 0.031
+ consistency
0.680 + 0.008
0.645 + 0.007
0.628 0.005
0.507 + 0.010
MoF
0.947 0.034
0.658 + 0.032
0.916 + 0.046
0.439 + 0.013
Table 5: P@ I and Consistency of Prefix P-Adapters on BERT Base. +consistency indicates training with the consistency loss (averaged over three runs). Consistency increases small amount, while accuracy stays the same. The Baseline and MoE results are included for reference.
F RELATION CLASSIFIER
We finetune models. The F1 scores are microaveraged across three random seeds are visible in Table 6
to predict the relation from the natural language prompt for the MoE
ID
OOD Prompts OOD Objects
OOD KE
0.989 + 0.004
0.812 + 0.011 0.990 + 0.003 0.607 + 0.015
Table 6: Microaveraged Fl score for the relation classifier component of the MOE models in each of the evaluation settings we investigate.
17


Reviews:
[{"review_id": "yhaZzqxYq1T", "reviewer": "Reviewer_jddf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The problem setting--mapping natural language prompts to more optimal continuous prompts--is interesting. Most prior work on prompting attempts to find optimal prompt templates for each task (or in this setting, KB relation type), with the assumption that the task label is known at test time. Methods like P-Adapters for learning a transformation of natural language prompts could be an interesting alternative to standard prompting (with a fixed template per task) and fine-tuning, with possible applications to meta-learning/transfer learning.\n* The authors compare methods with different assumptions about the availability of annotations, and under different kinds of distribution shifts.\n* The paper is clearly written and the proposed methods are sensible and easy to understand.\n\nMajor comments:\n* The paper does not draw a connection to prior work on robust optimization (for example, see references in [1]), which offers a more principled framework for formulating the objective of invariance to a class of input perturbations. At the very least, it would be good to give a formal definition of robustness/consistency and cite this line of prior work.\n* The paper does not provide a clear justification for why P-Adapters would be expected to improve consistency, and nothing in the training objective encourages the model to be consistent. The argument might be more convincing if you could compare it to a training objective that does explicitly promote consistency--for example, by encouraging the hidden representations of different prompt paraphrases to be similar.\n* As this paper notes, the factual probing benchmark has class imbalances and represents a very particular use case, so while P-Adapters appear to improve consistency here it’s unclear if these results will generalize to other settings. Would it be possible to evaluate P-Adapters on a wider variety of tasks, following prior work on prompt optimization [2, 3]? For example, [4] provide 100 prompt templates for a wider variety of NLP tasks. The paper would be more compelling if you could show that P-Adapters improve consistency in other settings too.\n* Much of the motivation of this paper is based on assertions about user preferences, but there is no human evaluation to validate these claims. For example, would a typical user prefer to give a natural language input but get an inconsistent response? Or would they rather pick a pre-defined relation type with better promise of consistency? In particular, it’s not clear why user preferences would place any restrictions on training.\n* MoE model: How accurate is the relation classifier? There is a big performance drop in the “OOD Prompts” setting, which leads me to wonder if the relation classifier was adequately trained. Either way, relation classification results should be included and discussed in the main paper--a perfect relation classifier would lead to 100% consistency.\n* All of the models suffer a performance drop on the “OOD Objects” setting, which indicates that the models have over-fit to the (imbalanced) distribution of entities in Wikidata. It’s unclear how to interpret consistency in this setting, because the models are consistently producing the incorrect response. It might be more informative to report something like the “Consistent-Acc.” measurement from [5].\n* Table 2 does not compare results to any prior work. For example, can these results be compared to the “Consistency Improved PLM” results from [5]?\n\nMinor comments:\n* The metric called “precision@1” should be called “accuracy@1”\n* Table 2 does not explain how the results are aggregated (prior work takes either the micro- or macro-average over relation types).\n* The description of the methods and the illustration (Figure 2) are somewhat unclear. Is there one MLP applied at each position? Three MLPs? The meaning of the different-colored tokens could also be explained in the caption.\n* The paper is missing some implementation details, such as the size of the BiLSTM.\n* The results section (section 5) might be easier to read if it were divided into subsections or paragraphs.\n* It would be interesting to see the accuracy breakdown by relation type.\n\n[1] Sinha, Aman, Hongseok Namkoong, and John Duchi. \"Certifying Some Distributional Robustness with Principled Adversarial Training.\" International Conference on Learning Representations. 2018.\n\n[2] Shin, Taylor, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. \"Eliciting Knowledge from Language Models Using Automatically Generated Prompts.\" EMNLP. 2020.\n\n[3] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT\nunderstands, too. arXiv preprint arXiv:2103.10385, 2021b.\n\n[4] Gao, Tianyu, Adam Fisch, and Danqi Chen. \"Making pre-trained language models better few-shot learners.\" ACL. 2021.\n\n[5] Elazar, Yanai, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. \"Measuring and improving consistency in pretrained language models.\" arXiv preprint arXiv:2102.01017. 2021.", "summary_of_the_paper": "This paper explores methods for improving the consistency of prompt-based factual probing of pre-trained language models. The main contributions are (1) several methods for mapping natural language prompts to continuous prompts that empirically improve accuracy and consistency on a factual probing benchmark and (2) analysis comparing reasonable alternatives in terms of accuracy and consistency.", "summary_of_the_review": "The paper proposes methods for mapping natural language prompts to continuous prompts, with the goal of improving accuracy and consistency on a factual probing benchmark. The problem is interesting and the method appears to work on this benchmark, but I there are three main changes I would like to see before recommending this paper for acceptance, possibly at a future conference:\n1. Drawing a connection to prior work on robust optimization, and providing a clearer formal justification for why this method will improve consistency.\n2. Providing more detailed empirical results and discussion (in particular relation classification accuracy). The current results are hard to interpret because models can be consistent but inaccurate, and because models can perform well by over-fitting the entity distribution.\n3. Ideally applying the method to a wider range of prompting tasks, to show whether the results will extend beyond the particular setting of factual probing.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sjeckEVrfE6", "reviewer": "Reviewer_AfJQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths of the paper:\n\n- The problem of extracting factual and consistent information from large language models is of high interest to the NLP community. Given how LLMs dominate NLP at the moment, making sure these models are robust and consistent is a timely problem,\nThe paper is overall well written, with only a couple of confusing parts (see below),\n\n- The proposed architecture for intervening between the input embeddings and the first hidden layer of the language model is quite comprehensive. I enjoyed seeing the different options, and in particular, thought the use of the MoE for relation classification to be quite insightful,\n\n- The experimental analysis of the work is well executed, and demonstrated convincingly which interventions were most useful in make predictions more accurate and consistent,\n\n- I liked the analysis in Figure 6, showing the importance of the subject entity on the precision of the fact extraction task,\n\n### Weaknesses of the paper:\n\n- The main weakness in this work is one that relates to the overall goal of fact extraction from language models. The “Oracle'' results from Table 1 are thought provoking: with perfect knowledge regarding the predicate/relation of test examples, and a subsequent 100% consistent response, the LLM is only able to obtain ~50% correct responses from T-Rex, which is an admittedly limited evaluation (41 “head” predicates, mostly of well known entities). While I understand that this work is clearly focused on the consistency issue, not necessarily correctness, it puts into question whether fact extraction from LMs is a worthwhile pursuit.\n\n- I would have liked for the paper to dig a little deeper into this headroom question from the previous point.  Would it be possible to conduct a sampled qualitative evaluation of errors of the Oracle model in the ID cases?  Are the errors due to unseen triples during training time (e.g., not in Wikipedia), or maybe there are issues with model capacity (maybe a 10x version of the LM would be able to recall the prompted fact)?\n\n- In terms of writing, the most confusing section in the paper is Section 4.1. After re-reading it twice, I was still not able to ascertain: (1) what data was used to train the models, and (2) what data was used to evaluate the models. The section makes reference to LAMA’s T-REX, LPAQA, ParaRel, as well as augmentations using BERT lexical replacements, as well as data from “Shin et al, 2020”. The section also talks about examples from these sources as well as templates (presumably filled in with WikiData triples?). I really think this section needs to be rewritten and the training, eval and test datasets should be much more precisely described.  I would also encourage authors to release the exact datasets and splits to allow others to reproduce/improve on this work. But even with a data release, a precise description of how this data was constructed is very important.\n\n- For the MoE and Oracle layers, the description in the paper is insufficient to determine the outputs presented to the first layer of the model. The depiction in Figure 2 hints that the entire sequence is rewritten using the fixed-length learned embeddings, and perhaps the subject or MASK embeddings are preserved? But actually sub-section 4.2 never formally describes how the embeddings are used to create the continuous prompts? Are they prepended/appended to the original inputs? Or do they rewrite the original inputs? Do either the MASK or subject tokens get copied?\n\n- The LAMA benchmarks have one unfortunate characteristic: since it was constructed for BERT-style single token prediction, it has stripped down the original datasets (see the original version of T-Rex, which contains over 600 unique predicates vs. the 41 from LAMA: https://hadyelsahar.github.io/t-rex/  and https://aclanthology.org/L18-1544.pdf ). I wonder if a more comprehensive version of this would be to evaluate on a larger sequence-to-sequence model like BART https://arxiv.org/abs/1910.13461  or T5 https://arxiv.org/abs/1910.10683 (both available as HuggingFace models). Given that this work leverages frozen LLMs, it seems that training and evaluation could be done relatively cheaply even for larger models with proper decoders.\n\n\n### Other comments:\n\n- With respect to the MoE solution, the paper claims that the model does not use a weighted combination and opts to use the top-1 predicted relation. I wonder if authors have tried using a weighted combination instead? If the relation classifier is trained with cross-entropy softmax loss, most of the weights will be close to one-hot (similar to top-1) except when the model is uncertain. Therefore combining prompt embeddings may yield some benefit over top-1. Does this make sense?\n\n- Note sure this is a good idea, but: given that the LLM is frozen, it seems plausible that the continuous prompt embeddings learned in some of the models resemble existing embeddings from the original vocabulary. As such, would it make sense to attempt to “decode” the continuous prompt embeddings into the existing vocabulary? One could use a greedy decoding strategy of extracting the nearest neighbor (via dot product or cosine distance) from each continuous prompt embedding to the vocabulary input embedding table. Have the authors tried  inspecting the continuous prompts in this way? I wonder if the output is informative or whether these prompts are modeling purely latent variables.\n\n- Typo in Figure 1 “Canada si” -> “Canada is”,\n\n- Typo in page 6: “Cannonical”  -> “canonical”\n", "summary_of_the_paper": "This paper addresses the problem of robustness for extracting factual information from large language models. It first describes and motivates the problem of inconsistent predictions of large language models on fact-seeking prompts when these prompts are perturbed or rephrased.  It then proposes a few different methods for addressing this inconsistency that operate on the same portion of the language-model, namely, between the input token embeddings and the first hidden layer of the language model. The work evaluates the performance of the variants using a pooled collection of fact-seeking prompts (e.g., LAMA, LPAQA and ParaSel). The results employ a consistency metric and show that different interventions in the input embeddings cause large differences in inter-prompt consistency.\n", "summary_of_the_review": "The problem of extracting factual and consistent information from large language models is of high interest to the NLP community, and this work in particular should be of interest to the ICLR community. Overall, this work was well-written throughout (easy to follow in most places except for a few rough parts detailed above). The experimentation work was also of high quality, with interesting results. To highlight a few findings: (1) the use of a relation-classification MoE and its consistently high performance on consistency metric seems promising, (2) the analysis demonstrating the importance of the “subject” is correct fact prediction, and (3) analysis demonstrating the negatives effects of uniformizing objects in train/test sets, which is strong indication that LLMs still do not generalize well to unseen objects.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "f4vcjxlAmK", "reviewer": "Reviewer_2hiF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nAuthors also presented a Mixture of Experts models that learn a set of prompts and select one to query the LLM. Experimenta results show that P-Adapters perform comparably to the more complex MoE models in extracting factual information from BERT and RoBERTa while eliminating the need for additional annotations.\n\nThe model and training procedure are well described and results are promising. It is also great that the data used is in the Github repositories. This said, the reader could benefit from better error analysis, as it is not clear how much hallucination and grounding the model is producing.\n", "summary_of_the_paper": "The paper presents an approach for extracting factual information from LLMs, where authors discuss a user oriented setting where they take in varied natural language prompts and return the objects with information needs. Authors proposed an approach, denoted P-Adapters, which is a model that is between the embedding layer and first attention layer of LLMs. It takes LLM embeddings as input and output prompts used to query the LLM. \n\nAuthors also presented a Mixture of Experts models that learn a set of prompts and select one to query the LLM. Experimenta results show that P-Adapters perform comparably to the more complex MoE models in extracting factual information from BERT and RoBERTa while eliminating the need for additional annotations.\n\nThe model and training procedure are well described and results are promising. It is also great that the data used is in the Github repositories. This said, the reader could benefit from better error analysis, as it is not clear how much hallucination and grounding the model is producing.\n", "summary_of_the_review": "The obtained results and model itself will benefit the reader and researchers working in this important research area.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SLqgFY_4gHu", "reviewer": "Reviewer_anXM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ves:\n+ Overall the paper is clear\n+ Use of LLM for extracting factual information is well motivated\n+ The motivation behind the use of natural language prompts to improve the user experience is generally clear. However, it can benefit from further citation of the literature in that domain. Having a few citations of works that compare the user experience with and without the additional information will make your argument in the paper stronger. \n+ Empirical results  suggest that P-Adapters improve the consistency\n\n\n-ves:\n- research questions are dispersed all over the paper. What I mean by this:\n\n    Abstract: you say that you tackle the inconsistency problem and you want to use a lightweight approach. Hence, I can see that there are two research questions: 1) inconsistency of LLM and 2) efficiency - how to tackle the inconsistency problem with a light weight model.\n\n    Introduction: you say that focus is the continuous prompts and user-friendly prompts. These are another two research questions.\n\n    Discussion: you hypothesising that the success of P-Adapter is due to not updating the LLM parameters. Such that it is likely to maintain all of factual knowledge learned during pretraining. Which to me is yet another research question - addressing the inconsistency of LLM with frozen weights vs fine-tuning. In Introduction you talk about the frozen weights of LLM but do not motivate why you decide to keep them frozen until the Discussion section.\n\n   Can you please state and motivate clearly your (full) research question somewhere in Introduction?\n\n- Lack of experiments:\n    1) (Minor) Why didn’t you train BERT fully with the proposed natural language prompts. You emphasise the importance of this experiment yourself in the Discussion section.\n\n    2) (Minor) Why didn’t you train (fine-tune) BERT with P-Adapters? This experiment will also allow you to answer your hypothesis in the Discussion section.\n\n    3) (Major) Did you try to freeze the parameters of LLM (or use an adapter) in the MoE model? You claim that MoE performs better than P-Adapters because of the additional supervision information (in form of relations)  the model gets during the training. With the present experiments it is hard to tell. Maybe MoE perform better because you also tune parameters of LLM?\n\n", "summary_of_the_paper": "The focus of the paper:\n\nThe paper proposes a way to increase the consistency of extracted information from the Large Language Models (LLMs) with respect to the different wordings of the queries that users use to prompt the models.\n\nResearch Directions:\n\nThere are two orthogonal directions that the authors explore. The first direction, is the use of adapters to tackle the aforementioned problem. The second direction is the reduction of supervision data that is needed to train and to test (query) the models. More specifically the authors advocate for the use of natural language prompts only. That is they do not require additional relation(s) that one can incorporate into the prompts to further specify the type of information that one can expect to be retrieved from the model. \n\nMotivation:\n\nThe first direction is motivated by two factors: 1) efficiency - the authors aim for a lightweight approach to increase the consistency and 2) to maintain all of factual knowledge learned during pretraining.  The second direction is motivated by user experience - the authors claim that the natural language prompts are more user friendly.\n\nContributions:\n\nPropose an adapter model - “P-Adapters” that rely only on natural language prompts during the training and testing. It has also been shown that the P-Adapters allows to increase consistency of the LLM to queries that are used to prompt the models.\n", "summary_of_the_review": "I cannot recommend the acceptance of the paper in its present state. There are two main reasons for this. The first reason is the lack of explicitly stated  (unified) research question. Without it, it is hard to tell if the authors have conducted enough experiments to support their research question. I did my best to identify these research questions in the paper and then try to link them to the experiment they conducted. However, I had a hard time doing this. Sometimes motivation given by the authors can be ambiguous. For example, do you use adapters just for efficiency or you are interested in exploring how one can address the inconsistency problem with the adaptors?   The second reason is the lack of experiments with the frozen LLM parameters of MoE. Depending on the outcome of the experiment the conclusions that are made in the paper may change. For example: “While MoE models usually perform better, the small increase comes at the cost of requiring relation annotations at training time and specifying the subject of the entity at inference time.”\n\nHowever, if those two issues are addressed, in my opinion, the paper has a good chance to be accepted. Reduction of  supervision data that is needed to tackle the inconsistency problem has both academic interest and also can be of the great interest to practitioners. Moreover, adapters are presently has high research interest in the NLP community - demonstrating how they can be used to address the inconsistency problem will also be valuable to the community.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3pTVn6qzprI", "reviewer": "Reviewer_v5xA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ This paper tackles one of the most important issues in the large language model: inconsistency results obtained from different prompts which have the same information needs. The problem itself is real and must be resolved.\n\n+ The proposed adaptor, P-adaptor is a simple but effective solution to alleviate the inconsistency.\n\n+ This paper is well-written and easy to follow. Furthermore, this paper provides comprehensive experiments including several qualitative analyses and discussions to show the effectiveness proposed method.\n\nConcerns:\n- Although the proposed method involves experiments in four different settings including OOD keyword error, it might be valuable to investigate an OOD syntax error setting. That is, in a real scenario, users write the natural language prompt which may have a grammatical error. Assessing the robustness of the proposed method in terms of grammatical errors can enhance the quality of the paper.\n\n- In Table 2, the performances of P-tuning in RoBERTa-large are better than the ones of Oracle. Without any explanation, it is not convincing. Instead of listing the result in the tables, it would be clearer if the authors provide more explanations about them although they are in the appendix.", "summary_of_the_paper": "This paper provides an interesting direction to extract factual information from large language models accurately and consistently. Since the quality of the factual information depends on the prompts used to query them, this paper tries to minimize the inconsistency by introducing an intermediate representation. To do this, this paper proposes an adaptor that uses the token embeddings of the natural language prompt as input and outputs continuous prompts that are used to query the large language model to extract factual information accurately and consistently.", "summary_of_the_review": "The proposed method is reasonable and analyses of experiments are well described.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a prompting technique for eliciting factual knowledge from frozen pretained transformer LMs. The key idea is to modify the embeddings produced by the embedding layer before they are passed to the first attention layer and the paper investigates several different design choices. The Reviewers all agree that the paper tackles an important problem with interesting methods, that it is well written and has strong results. The main concerns, raised by Reviewer jddf, were about clarifying the connections to the robust optimization literature and evaluating on OOD relations. The former has been addressed in the revised version. While the latter point remains valid, I find that the paper in its current state has enough useful experiments and analysis to warrant publication. The authors have clarified most of the other points raised by the reviewers in their rebuttal.
Decision: Accept (Poster)