SlowGuess/NAACL_2021_OCR huggingface dataset:
Zexuan Zhong* Dan Friedman* Danqi Chen
{zzhong, dfriedman, danqic}@cs.princeton.edu
Petroni et al. (2019) demonstrated that it is possible to retrieve world facts from a pretrained language model by expressing them as cloze-style prompts and interpret the model's prediction accuracy as a lower bound on the amount of factual information it encodes. Subsequent work has attempted to tighten the estimate by searching for better prompts, using a disjoint set of facts as training data. In this work, we make two complementary contributions to better understand these factual probing techniques. First, we propose OPTIPROMPT, a novel and efficient method which directly optimizes in continuous embedding space. We find this simple method is able to predict an additional  $6.4\%$  of facts in the LAMA benchmark. Second, we raise a more important question: Can we really interpret these probing results as a lower bound? Is it possible that these prompt-search methods learn from the training data too? We find, somewhat surprisingly, that the training data used by these methods contains certain regularities of the underlying fact distribution, and all the existing prompt methods, including ours, are able to exploit them for better fact prediction. We conduct a set of control experiments to disentangle "learning" from "learning to recall", providing a more detailed picture of what different prompts can reveal about pre-trained language models.<sup>1</sup>
Pre-trained language models like BERT are optimized to predict the distribution of words in an Internet corpus (Devlin et al., 2019). Naturally, this distribution encodes information about world facts. Recently, researchers have taken an interest in measuring how much factual information language models acquire from pre-training. Petroni et al. (2019) formally define this project in the LAMA
Linguistic probe:
Training data:
The chef made pizzas
dependency label
Factual probe:
Training data:
(X, place_of_birth, ?)
BERT
#
↓ London
(Dante, place of birth, Florence)
(John Donne, place_of_birth, London)
(Guy Deghy, place_of_birth, Budapest)
Figure 1: A linguistic probe is trained to predict linguistic annotations given the representations returned by a language model, and evaluated on a held-out set of sentences. A factual probe is trained to predict an object for a subject and a relation using a pre-trained language model, and evaluated on a held-out set of subject-object pairs that express the same relation.
benchmark, which consists of (subject, relation, object) triples along with human-written templates that express each relation. They show that BERT can predict objects given cloze-style prompts—for example, “Dante was born in [MASK]”—and they present their result as a lower bound on the amount of factual information BERT encodes. Subsequent work has attempted to tighten this bound by finding better prompts. Jiang et al. (2020) use text mining and paraphrasing to find a set of candidates and select the prompts that lead to the highest accuracy on a training set. Shin et al. (2020) train a model to generate prompts automatically by searching for the sequence of tokens that maximizes expected likelihood of the gold object label. Both of these methods collect additional triples from Wikidata to use for tuning their prompts.
In this paper, we first take a natural next step in the search for better prompts: rather than confining our search space to discrete input tokens, we di
1The code is publicly available at https://github.
com/princeton-nlp/OptiPrompt.
Association for Computational Linguistics: Human Language Technologies, pages 5017-5033
rectly optimize in the input embedding space, finding the real-valued input vectors that are most effective at eliciting facts. We also find that initializing with manual prompts can provide a better starting point for the search process. Our approach, OPTIPROMPT, is simple and compute-efficient, and improves accuracy on the LAMA benchmark from  $42.2\%$  to  $48.6\%$ , compared to previous discrete alternatives. On the more difficult LAMA-UHN split (Poerner et al., 2019), which filters out easy-to-guess entity names, OPTIPROMPT improves accuracy from  $31.3\%$  to  $38.4\%$ .
At the same time, we observe that prompts that are optimized on training data may exploit some regularities in the underlying distribution of facts. How can we make sure our prompts are recovering information solely from the language model? An analogous question has been explored recently in linguistic probing, which aims to explore the linguistic properties encoded in contextualized word representations (Belinkov et al., 2017; Tenney et al., 2019; Lin et al., 2019)—for example, by seeing if a classifier can predict that “chef” is the nominal subject of “made” given the representations returned from a language model (Figure 1). Recent work has attempted to disentangle the information encoded in the representations from the information learned by the probe (Hewitt and Liang, 2019; Pimentel et al., 2020; Voita and Titov, 2020; Zhu and Rudzicz, 2020). However, this question has not been yet explored in factual probing, in part because it is assumed that there is no way to predict a knowledge fact simply from observing a non-overlapping set of facts about other entities. For example, learning that Dante was born in Florence should tell you nothing about the birthplace of John Donne.
We analyze our training data and find that this assumption is not warranted. Even though the training data was collected independently of the LAMA benchmark, there are sufficient regularities in the underlying distribution of Wikidata relations that a naive classifier fit to the training data can achieve surprisingly good performance. Furthermore, our experiments reveal that all the data-driven prompt-search methods, including previous methods and our proposed OPTIPROMPT, are able to exploit this
information to achieve better prediction accuracy. Given some training data, a good search algorithm can find prompts that recover a non-trivial number of "facts" from a neural network with randomly initialized parameters, exploiting both simple class statistics and higher order lexical regularities.
This finding makes it challenging to interpret relative accuracy scores on the knowledge probing task. We show how our control experiments allow us to form a more detailed understanding of the behavior of different probes. For example, by partitioning the test set into "easy" examples, which can be predicted by random controls, and "hard" examples, we can form some conclusions about which facts are less likely to have been learned from training data. OPTIPROMPT outperforms prior methods in both subsets, suggesting it is both better at learning from training data and better at eliciting facts from a language model. We conclude with suggestions for future work that might be less susceptible to the confounding effect of training data.
2 Background: Prompting for Facts
2.1 LAMA
The factual probing setting was introduced by the LAMA benchmark (Petroni et al., 2019), which is designed to measure the amount of factual information encoded in a pre-trained language model (LM). In LAMA, a fact is defined as a triple  $\langle s, r, o \rangle$ , where  $s$  is a subject (e.g., Dante),  $r$  is a relation from a fixed set of relations  $\mathcal{R}$  (e.g., place of birth), and  $o$  is an object (Florence). LAMA facts are drawn from a number of sources, including Wikidata, ConceptNet (Speer and Havasi, 2012), and SQuAD (Rajpurkar et al., 2016). We follow recent factual probing work (Jiang et al., 2020; Shin et al., 2020) in focusing on the T-REx split (Elsahar et al., 2018), which contains up to 1000  $\langle s, r, o \rangle$  triples for each of 41 Wikidata relation types. The relation types are divided into three categories: 1-1 includes relations like capital of; N-1 includes relations like place of birth; and N-M includes relations like shares border with. In the LAMA evaluation, each relation is associated with a human-written prompt that contains a single [MASK] token—for example, “[X] was born in [MASK].” To accommodate masked language models such as BERT, LAMA is restricted to facts for which the object label is a single token in a predefined vocabulary
2In knowledge base completion or link prediction, researchers study how to predict a fact (Barack Obama, nationality, ?) from other triples such as (Barack Obama, place_of_birth, Honolulu) and (Honolulu, city_of, USA). In knowledge probing, the underlying assumption is that one can't predict facts from the other facts of the same relation.
Method	Prompt	Data-driven?
LAMA (Petroni et al., 2019)	[X] is [MASK] citizen	X
LPAQA (Jiang et al., 2020)	[X] is a citizen of [MASK]	✓
AUTOPROMPT (Shin et al., 2020)	[X] m3 badminton pieces internationally representing [MASK]	✓
OPTIPROMPT	[X] [V]1 [V]2 [V]3 [V]4 [V]5 [MASK]	✓
OPTIPROMPT (manual)	[X] [V]1 := is [MASK] [V]2 := citizen	✓
Table 1: Comparison of prompts for the relation country of citizenship. [X] denotes the name of the subject and [MASK] is single-token object label to be predicted. In our OPTIPROMPT approach, we optimize a sequence of learned embeddings  $[\mathrm{V}]_i \in \mathbb{R}^d$  for each relation type.  $[\mathrm{V}]_i := w$  indicates that the vector is learned but initialized by the pre-trained embedding of word  $w$  and OPTIPROMPT (manual) indicates that we use a manual prompt as initialization (see Section 3 for more details).
$\mathcal{V}$ .<sup>3</sup> Given a subject  $s$ , a relation prompt  $t_r$ , and a masked language model, we can identify the word  $\hat{o} \in \mathcal{V}$  to which the LM assigns the highest probability of  $P([\mathrm{MASK}] = \hat{o} \mid t_r(s))$ , where  $t_r(s)$  represents the prompt template with the subject placeholder [X] replaced by  $s$ . If  $\hat{o}$  is the same as the gold object  $o$ , we conclude that the LM encodes information about the fact.
LAMA is an evaluation benchmark, so there is no training data. It is constructed so that a pretrained language model can be evaluated "off-the-shelf" with no additional fine-tuning. Petroni et al. (2019) remark that their benchmark provides only a lower-bound estimate of the amount of factual information stored in an LM, because their manually written prompts might not be optimal for eliciting facts. Accordingly, subsequent work has focused on tightening this bound by using additional training data to find more optimal prompts.
2.2 LPAQA
Jiang et al. (2020) use a range of text-mining and paraphrasing techniques to generate a set of candidate prompts for each relation. They collect a training dataset from Wikidata, ensuring that there is no overlap with subject-object pairs in the LAMA benchmark, and select prompts by measuring accuracy on this training data. They consider a number of rules for selecting prompts, including top- $K$  baselines and an "optimized ensemble", which consists of multiple prompts per relation with weights tuned on the training data. Their prompt dataset, LPAQA, is available online. $^4$
2.3 AUTOPROMPT
Shin et al. (2020) take prompt optimization one step further by training a statistical model, AUTO-PROMPT, to search over the space of input tokens for prompts that elicit correct predictions. They collect  $1000\langle s,r,o\rangle$  triples for each relation type, either from the original T-REx dataset (Elsahar et al., 2018) or from Wikidata, with no triples that appear in the LAMA benchmark. They define a prompt for a given relation  $r$  as the subject followed by a fixed number of "trigger" tokens:
$$
t _ {r} = [ X ] [ T ] _ {1} [ T ] _ {2} \dots [ T ] _ {m} [ M A S K ],
$$
where  $[\mathrm{X}]$  is replaced by the subject,  $[\mathrm{T}]_i$  represents a "trigger" token which can be any token in the vocabulary, and the number of  $[\mathrm{T}]$  tokens is set as a pre-defined number  $m$ . The tokens are initialized as [MASK] tokens and then iteratively updated, at each step using a gradient-based searching algorithm (Wallace et al., 2019) to replace one of the trigger tokens with the token that is estimated to maximize the likelihood of the gold label on the training set.
3 Our Approach: OPTIPROMPT
Our approach is motivated by the view that restricting the search to the space of vocabulary tokens is a suboptimal and artificial constraint. In the case of AUTOPROMPT, optimizing over a discrete subspace is also inefficient: at each step we have to enumerate a set of candidate tokens, replace the selected trigger token, and re-run the model (Shin et al., 2020). The examples in Table 1 also illustrate that optimized textual prompts can be opaque, despite consisting of tokens from the English vocabulary. This undermines one argument in favor of natural language prompts, which is that they are
3Subject names are usually longer, with an average length of 3.7 tokens using the BERT-base-cased vocabulary.
4https://github.com/jzjbjyb/LPAQA
Method	1-1	N-1	N-M	All	UHN
Majority	1.8	23.9	22.0	22.0	23.8
LAMA (manual)	68.0	32.4	24.7	31.1	21.8
LPAQA (manual + paraphrased)	65.0	35.9	27.9	34.1	28.7
AUTOPROMPT (5 [T]s)	58.0	46.5	34.0	42.2	31.3
OPTIPROMPT (5 [V]s)	49.6	53.1	39.4	47.6	37.5
OPTIPROMPT (10 [V]s)	60.7	53.2	39.2	48.1	37.9
OPTIPROMPT (manual)	59.6	54.1	40.1	48.6	38.4
Table 2: Micro-averaged results (top-1) on the LAMA benchmark using the BERT-base-cased model, averaged over relations. UHN stands for UnHelpfulNames (Poerner et al., 2019), which is a subset of LAMA where questions with helpful entity names were deleted. The LAMA results are broken down by relation category. Examples from each category are capital of (1-1), place of birth (N-1), and shares border with (N-M).
human readable so might be easier to interpret.
OPTIPROMPT In this view, we propose OPTIPROMPT, a method for continuous prompt optimization. Rather than limiting the search to the space of discrete tokens, OPTIPROMPT searches for optimal prompts directly, composing prompts using any vector in the embedding space. We first follow AUTOPROMPT and define a prompt in the following form:
$$
t _ {r} = [ \mathrm {X} ] [ \mathrm {V} ] _ {1} [ \mathrm {V} ] _ {2} \dots [ \mathrm {V} ] _ {m} [ \mathrm {M A S K} ],
$$
where each  $[\mathrm{V}]_i\in \mathbb{R}^d$  is a dense vector with the same dimension as the LM's input embedding (e.g., 768 for BERT-base) and the number of [V] vectors is set to a pre-defined number  $m$ .
Treating prompts as dense vectors allows us to search for optimal prompts much more efficiently. Given some initial values for  $[\mathrm{V}]_i$ , we keep all other model parameters fixed and use gradient-descent to minimize the negative log-likelihood of a training set:
$$
\mathcal {L} _ {r} = - \frac {1}{| D _ {r} |} \sum_ {(s, o) \in D _ {r}} \log P \left(\left[ \operatorname {M A S K} \right] = o \mid t _ {r} (s)\right),
$$
where  $D_r$  is the set of (subject, object) pairs with relation  $r$  and  $t_r$  represents the prompt template for relation  $r$  with subject tokens  $s$  substituted for the placeholder [X].
In this basic form, we pick a fixed value for  $m$  (treated as a hyperparameter) and randomly initialize all the [V] tokens. We also consider a more sophisticated form of using manual prompts (we use the prompts provided in the LAMA benchmark) to decide the number as well as the position of the [V] tokens for each relation and initialize each
$[\mathrm{V}]_i$  with the pre-trained input embedding for the corresponding tokens in the manual prompt. As shown in Table 1, we can convert a manual prompt  $[X]$  is [MASK] citizen" into
$$
t _ {r} = [ \mathrm {X} ] [ \mathrm {V} ] _ {1} [ \mathrm {M A S K} ] [ \mathrm {V} ] _ {2},
$$
and use the embeddings of  $is$  and citizen to initialize  $[V]_1$  and  $[V]_2$  respectively. Our motivation is that a good initialization is likely to be important in this challenging non-convex optimization problem.
Setup We train OPTIPROMPT using the data collected by Shin et al. (2020), which contains 800 training examples with 200 held out for development. For our main experiments, we probe the BERT-base-cased model and we compare other pre-trained language models in Appendix C. We report top-1 micro-averaged accuracy:
$$
\frac {1}{| \mathcal {R} |} \sum_ {r \in \mathcal {R}} \frac {1}{| D _ {r} |} \sum_ {(s, o) \in D _ {r}} \mathbb {1} [ \hat {o} = o ],
$$
where  $\mathcal{R}$  is the set of relations,  $D_r$  is the set of (subject, object) pairs with relation  $r$ , and  $\hat{o} = \arg \max_{o} P([\mathrm{MASK}] = o \mid t_r(s))$ . More implementation details can be found in Appendix B.1.
LAMA results Our results are in Table 2. Overall, OPTIPROMPT outperforms the previous reported results in terms of accuracy on the LAMA benchmark. Compared to AUTOPROMPT<sup>5</sup>, our
For AUTOPROMPT, we obtain a slightly different accuracy  $42.2\%$  by evaluating their released prompts, instead of  $42.9\%$  reported in their paper. We suspect that this is due to a discrepancy in the vocabulary used in different papers. We use the vocabulary provided in the LAMA benchmark for all the evaluation: https://github.com/facebookresearch/LAMA#unified-vocabulary.
models perform  $5.4\% - 6.4\%$  higher on LAMA and  $6.2\% - 7.1\%$  on the more-difficult LAMA-UHN benchmark. The improvement is consistent across all categories, with the exception of the "1-1" category, which contains two relations, capital and its inverse, capital of. Interestingly, the prompt that yields the best results in this category is the manual prompt, with LPAQA and AUTOPROMPT prompts performing steadily worse. We speculate that there are very few prompts that elicit this relation with high accuracy and they are difficult to find via stochastic, non-convex optimization.
We also find that initializing the prompt vectors using the manually written prompts improves performance consistently. This confirms our intuition that the manual initialization provides a good prior for finding a good solution in the non-convex optimization problem. The results are broken down by relation in Table 8 in the Appendix.
4 Can We Trust Optimized Prompts?
Our factual probing results confirm that OPTIPROMPT is an effective approach, outperforming the best previous method by  $6.4\%$  on the LAMA benchmark. However, can we conclude that BERT encodes  $6.4\%$  more facts than was previously known? Our prompts, like LPAQA and AUTOPIOMPT, are optimized on in-distribution Wikidata relations, which raises the possibility that they exploit some regularities in the underlying fact distribution. In this section we aim to answer two questions. First, are there patterns in the Wikidata fact distribution that statistical model could theoretically exploit to predict unseen facts? Second, are optimized prompts capable of exploiting these patterns in practice?
4.1 Facts can be predicted from training data
We first examine whether it is possible to predict any facts by just looking at the training data. The simplest pattern is the class prior  $P(o \mid r)$ : if one or two object labels dominate the relation  $r$ , it is easier to guess them regardless of the subject entity. A more sophisticated pattern is to find a correlation between subject tokens and object labels—that is, to estimate  $P(o \mid r, w_1, \dots, w_{|s|})$ , where  $w_1, \dots, w_{|s|} \in \mathcal{V}$  are the tokens of the subject name. To see whether such patterns exist, we fit two simple probabilistic models to the Wikidata training set collected by Shin et al. (2020). The first model always predicts the majority class, with class priors learned from the training data, and the sec
Relation	Class Prior	Naive Bayes
All	17.3	24.6
1-1	0.2	0.3
N-1	23.2	28.6
N-M	11.0	21.8
member of	2.2	59.6
manufacturer	8.9	62.0
Table 3: Results for simple classifiers fit to the Wiki data training data and evaluated on the LAMA test set. We highlight two relations for which object labels are correlated with particular subject tokens: In the member of category, the model appears to learn that any subject with "football" in its name, such as Ghana Football Association, is likely to be a member of FIFA. In the manufacturer category, the model learns to predict that Chevrolet manufactures the Chevrolet Impala, BMW manufactures the BMW M Coupe, and so on.
ond is a Naive Bayes classifier (bag-of-words) with add-one smoothing (see details in Appendix B.2). Table 3 shows the accuracy of these models on the LAMA benchmark, averaged over relations. The majority class model performs well because, on some relations, well over half of the examples are from the majority class. $^{6}$  The Naive Bayes baseline performs even better in all categories by learning correlations between subject tokens and object labels. This analysis complements an observation of Poerner et al. (2019), who point out that BERT can exploit superficial information in a cloze prompt to “guess” the correct answer—for example, predicting that people with stereotypically Italian names were likely born in Rome. Our results show that it is possible to learn these correlations even without prior information about entity names, and there might be other, subtler patterns in the Wikidata distribution.
4.2 Prompts can exploit training data
We have shown that the training data clearly encodes certain regularities and simple statistical models can learn to fit the training data. In the following, we study whether a prompt optimization method built with pre-trained language models, is expressive enough to exploit these regularities in practice. We attempt to answer this question by means of two random controls, inspired by similar proposals from linguistic probing. In our Random Model (RM) baseline, we optimize prompts to elicit
6These include native language (60% French) and continent (72% Antarctica).
Majority Label
Other Labels
Random Embeddings
Pre-trained BERT
Figure 2: Accuracy on LAMA obtained by prompting BERT-base-cased, either the pre-trained model, reinitializing the input embeddings, or reinitializing all parameters. Each bar represents total accuracy micro-averaged over relations and divided into two categories: accuracy obtained by predicting the training set majority class label, and accuracy obtained by predicting other object labels. We also fine-tune BERT, which, in the random control settings, can be thought of as a better lower bound on the entropy of the task distribution.
Random Model
facts from a neural network with the same architecture as the pre-trained LM but with randomly initialized parameters. This is analogous to a control function Pimentel et al. (2020), a function that removes information from a linguistic representation. Any successful predictions in this setting must be the result of optimizing on training data. We also consider a Random Embeddings (RE) baseline, where we reinitialize only the input embeddings. This is analogous to a control task (Hewitt and Liang, 2019), a variant of the probing task in which word types are associated with random labels. Our motivation is that the Random Model setting is more difficult to optimize, so might underestimate the ways a prompt model could exploit information from the training data. Finally, we directly fine-tune a reinitialized BERT model on the training data with the goal of getting a better estimate of the number of LAMA facts that could be predicted from the training data.
The results are shown in Figure 2 (see implementation details and more results in Appendix B.1 and Table 8). In the Random Embeddings setting, both AUTOPROMPT and OPTIPROMPT are capable of finding prompts that elicit some correct predictions. In the Random Model setting, AUTOPROMPT gets
$0\%$  of predictions correct, presumably because it is more difficult to optimize, but OPTIPROMPT is still capable of finding successful prompts. Most successful predictions are obtained by finding a prompt that elicits the majority class label, although OPTIPROMPT also makes a number of correct predictions that cannot be attributed to this strategy. Our qualitative analysis suggests that these prompts exploit both class statistics and correlations between objects and subject tokens (Appendix A.2).