SlowGuess/NAACL_2021_OCR huggingface dataset:
Fine-tuning BERT results in even higher accuracy, indicating that there are patterns that prompts fail to exploit. The random controls represent a challenging setting for prompt optimization, and it is possible that the prompts are better exploiting the training data when they have access to full pretrained BERT model. We find evidence that this is the case by calculating how often each prompt elicits the training class majority label on LAMA, plotting the results in Figure 3. Both AUTOPROMPT and OPTIPROMPT are prone to over-predicting the majority class label. For example, although AUTOPROMPT gets  $0\%$  accuracy in the RM setting, it finds a prompt that elicits the majority label more than  $95\%$  of the time for six relations when optimized on the pre-trained BERT model.[9]
LPAQA prompts predict the majority class less often, possibly because they are less effective at
In the RE setting, the classifier head of the model is also reinitialized, as the output embeddings are tied to the input embeddings.
Hewitt and Liang (2019) consider tasks like part-of-speech tagging, where each word type can be associated with a randomly selected tag. We randomize the inputs rather than the labels, which preserves most of the statistical correlations between subject token types and object labels but removes lexical information from the embeddings.
<sup>9</sup>Shin et al. (2020) attempt to prevent the model from using this strategy by filtering out prompts that contain proper nouns or gold object labels, but this evidently is not enough. For example, the prompt for the position held relation is "[X] explorers voting municipal → consecrated [MASK].", which elicits bishop for 100% of LAMA examples.
Figure 3: The percentage of LAMA examples for which a prompt elicits the training set majority label, compared with the percentage of training and test facts with that label. Optimized prompts show a strong tendency to over-predict the majority class relative to manual prompts and the ground truth. "Train (oracle)" is calculated from the set of Wikidata facts collected by Shin et al. (2020), which is used to train AUTOPROMPT and OPTIPROMPT.
Method	All (34,039)	Easy (10,546)	Hard (23,493)
Manual	31.1	41.5	24.3
LPAQA	34.1	47.0	25.6
AUTOPROMPT	42.2	68.2	26.7
OPTIPROMPT	48.6	75.6	33.0
Table 4: Accuracy on LAMA partitioned into easy examples or hard examples, micro-averaged over relations. Easy facts are the facts that can be predicted by fine-tuning a BERT model with either randomly initialized parameters or randomly initialized token embeddings, or by the Naive Bayes model described in Section 4. Hard examples are everything else. The numbers in parentheses denote the size of each subset.
fitting the training distribution. However, it is still clear that LPAQA prompts also encode distribution of the training data. For instance, the highest ranked occupation prompts discovered by LPAQA include prompts such as “[MASK] and actors [X]” and “[MASK] and player [X]”,[10] which reflect several of the most common occupations in Wikidata. We also discuss examples in Appendix A.2 of cases where LPAQA finds subtle changes to the prompt template that leads the model to predict the majority label more often than the manual prompt and the true test distribution. All the above evidence shows that optimized prompts can learn new facts to some extent.
5 How to Interpret Probing Results?
Our analysis in Section 4.2 shows that optimized prompts can predict new facts from training data. How can we interpret our factual probing results in
this light? In order to get another perspective of the relative improvement, we partition LAMA into an easy subset and a hard subset (examples from each subset can be found in Table 5). The easy subset consists of the facts that can be correctly predicted by any of three models fit to the training data: the Naive Bayes model described in Section 4.2 and a fine-tuned BERT model with either token embeddings reinitialized or all parameters reinitialized. The easy subset serves as an estimate of the set of facts that can be predicted from training data. The hard subset consists of the remain facts. Table 4 shows the results of each prompt on these two subsets of LAMA (the per-relation results are given in Table 9). First, we observe that all the probing methods achieve a much higher accuracy on the easy subset. Using more sophisticated prompt optimization techniques tends to result in big improvements on the easy subset of LAMA and smaller improvements on the hard subset. OPTIPROMPT outperforms AUTOPROMPT by  $7.4\%$  on the easy examples; while on the hard examples, where we filtered out facts that we know can be predicted from the training data, OPTIPROMPT also yields a big improvement  $(+6.3\%)$ . This suggests that OPTIPROMPT is both better at learning from training data and better at eliciting facts from an LM.
For a more qualitative analysis, we randomly sample ten facts from each subset, keeping only facts that are predicted correctly by at least one model and exclude examples that have the majority class label. The examples, shown in Table 5, give a better idea of the types of predictions elicited by different prompts. For example, both AUTO-PROMPT and OPTIPROMPT appear to be exploiting the training data in some cases. In the easy subset, they elicit more accurate predictions on cases when
10https://github.com/jzbjyb/LPAQA/blob/master/prompt/paragrapher/P106.json
Rel.	Fact (manual template)	NB	Manual	LPAQA	Auto	Opti
P103	The native language of Jan van Krimpen is Dutch .	Dutch	Dutch	Dutch	Dutch	Dutch
P279	edible mushroom is a subclass of mushroom .	protein	mushroom	category	mushroom	mushroom
P1001	Governor of Tasmania is a legal term in Tasmania .	Canada	Australia	Australia	Tasmania	Tasmania
P106	Dude Harlino is a actor by profession .	actor	lawyer	wrestler	politician	actor
P27	Jens Evensen is Norway citizen .	Norway	Danish	Sweden	Norway	Norway
P176	Porsche Cayenne is produced by Porsche .	Honda	Porsche	Porsche	Porsche	Porsche
P279	United States H-class submarine is a subclass of submarine .	protein	submarines	submarine	submarine	submarine
P138	Milwaukee Mitchell International Airport is named after Milwaukee .	Peter	Mitchell	Mitchell	Milwaukee	Milwaukee
P176	BMW E9 is produced by BMW .	BMW	BMW	BMW	BMW	BMW
P1412	Tom Mann used to communicate in English .	French	English	English	English	English
P937	Francis Hagerup used to work in Oslo .	London	London	London	Copenhagen	Oslo
P127	Apple Store Online is owned by Apple .	Germany	Apple	Apple	Apple	Apple
P1412	Berengaria of Castile used to communicate in Spanish .	French	Spanish	Latin	Spanish	Spanish
P176	SNES-CD is produced by Sony .	Honda	Sega	Sony	IBM	IBM
P47	Honduras shares border with Guatemala .	Lyon	Guatemala	Guatemala	Guatemala	Guatemala
P937	David Ben-Gurion used to work in Jerusalem .	London	Jerusalem	Jerusalem	Jerusalem	Jerusalem
P19	Peter I of Serbia was born in Belgrade .	Paris	Belgrade	Belgrade	Belgrade	Belgrade
P31	Dally M Medal is a award .	album	prize	prize	award	award
P30	Snowdon is located in Europe .	Antarctica	Wales	Europe	Antarctica	Antarctica
P937	William Lyon Mackenzie King used to work in Ottawa .	London	Canada	London	Montreal	Ottawa
Table 5: Randomly sampling 10 examples each from LAMA-easy (the first block) and LAMA-hard (the second block), only keeping examples that are predicted correctly by at least one model and that do not have the majority label. NB: the Naive Bayes model (Section 4.2), Auto: AUTOPROMPT, Opti: OPTIPROMPT. The correct predictions are underlined.
the answer is a token in the subject name. In the hard subset, they show signs of having over-fit to the training distribution, incorrectly predicting the most common object labels for continent (Antarctica) and manufacturer (IBM). OPTIPROMPT performs better than the other prompts on some facts in both categories. On an easy profession example, while AUTOPROMPT incorrectly predicts the majority label (politician), OPTIPROMPT—along with our Naive Bayes model—apparently encodes a lexical correlation between some aspect of the subject's name and the correct label, actor. On the other hand, OPTIPROMPT out-performs the other prompts on two more difficult examples: "Francis Hagerup used to work in Oslo" and "William Lyon Mackenzie Kingused to work in Ottawa." In both cases, LPAQA predicts the training majority label (London), AUTOPROMPT gets geographically closer (Copenhagen and Montreal), and OPTIPROMPT predicts the correct city.
We note that we cannot conclude that there is no way to predict these "hard" facts from training data. A more general limitation of this analysis is that it does not allow us to say which strategy a model uses to make a particular prediction. Many facts can be predicted either by learning the class prior; by learning a lexical correlation between subject tokens and objects; by exploiting lexical information from the LM; or because the LM genuinely encodes information about a particular entity. Still,
the qualitative examples reveal interesting patterns in the behavior of the different prompt models that could not be observed from the summary accuracy results on the LAMA benchmark, and looking at specific predictions across a number of prompts gives us more evidence for deciding what kind of information the LM encodes about a particular fact.
Our experiments show that OPTIPROMPT is an effective optimization algorithm, outperforming prior work at the task of eliciting facts from a pretrained language model. However, our results are complicated by the fact that any data-driven optimization can find prompts that encode new information from the training data. This leaves open the question of which method we should select if we are interested in factual probing.
Continuous vs. discrete prompts We find that both continuous and discrete optimization are capable of finding prompts that exploit the training data. Even when the prompt is discrete, it is rarely clear why a prompt elicits a particular prediction.[11] Hence, we believe that continuous prompting is more preferable, because it is easier and more efficient to optimize, and makes better predictions (in both easy and hard subsets). On the other hand,
<sup>11</sup>For an illustration, see Appendix A.2 for a list of the AUTOPROMPT templates that elicit the majority class label more than 95% of the time.
one drawback of OPTIPROMPT (which is shared by AUTOPROMPT) is that we need white-box access to the LM to compute the gradients. Discrete prompts will still be necessary in cases where the model parameters are not available, for example in the case of very large language models that are provided over an API.
Learning vs. learning to recall Regardless of how we choose to optimize prompts, it remains difficult to say why a model made a particular prediction—whether it was learned from training data or encoded in the LM. Some avenues for future work might be to consider techniques for attributing predictions to specific training instances, with the goal of developing a causal understanding of how facts are acquired during pre-training or prompt optimization. More generally, our real goal is to understand how pre-trained language models learn and represent information. Prompt-based probing might provide some insight into this question, but we hope that future research will eventually be able to provide more mechanistic explanations for neural network behavior. For example, it would be interesting to understand how information about entities is laid out in neural network parameters and later retrieved in response to an input prompt.
Our work follows from the line of factual probing experiments initiated by Petroni et al. (2019), who introduced the LAMA benchmark for clozestyle factual probing. Subsequent work on LAMA has introduced data-driven methods for optimizing prompts (Jiang et al., 2020; Shin et al., 2020). Poerner et al. (2019) point out that many facts in LAMA can be predicted using lexical clues, and they introduce a new benchmark, LAMA-UHN, that is less susceptible to these heuristics. Our work follows these projects by introducing (a) more effective techniques for optimizing prompts, and (b) a more comprehensive approach for accounting for the role of train/test overlap. Concurrently with this work, other authors explore continuous prompt optimization: Haviv et al. (2021) use an encoder to map a manually written prompt to a sequence of continuous vectors, which are then replaced with the discrete tokens that are nearby in embedding space; Li and Liang (2021) propose Prefix-Tuning, which fine-tunes the left-most hidden representations in auto-regressive language models; Liu et al. (2021) use an LSTM to generate a sequence of prompt
vectors. Prompting has been explored more generally as a method for achieving “few-shot” learning with language models (Brown et al., 2020; Schick and Schütze, 2020; Gao et al., 2020).
Linguistic probing is an extensive area of research that we do not attempt to summarize here (see Rogers et al., 2020 for an overview). Our work is most related to recent proposals about how to measure whether a probe is extracting information from a representation or learning to predict the annotation from probe training data. These include random baselines (Hewitt and Liang, 2019) and information-theoretic measurements (Voita and Titov, 2020). We adopt the notion of control functions from Pimentel et al. (2020). Our study also relates to a larger category of work diagnosing "short-cut learning" (Geirhos et al., 2020) in neural NLP models. McCoy et al. (2019) discover that models like BERT are often "right for the wrong reason", exploiting shallow heuristics rather than underlying linguistic structure, and similar effects have been discovered in many other tasks (Sugawara et al., 2018; Wallace et al., 2019).
We introduce OPTIPROMPT, an effective continuous method for optimizing prompts. Applied to factual probing, OPTIPROMPT outperforms the best previous prompt method by  $6.4\%$  on the LAMA benchmark. We find that the typical training data used for prompt optimization reveals useful information about the underlying task distribution, to the point that search algorithms can find prompts that recover "facts" even from a randomly initialized model. By comparing the predictions of different prompt methods across our different controls we can form a more detailed understanding of how different prompts behave and what they can reveal about pre-trained language models.
Our experiments illustrate that the "facts" recovered from a pre-trained language model should not be considered real facts. Optimizing any kind of statistical model for factual prediction is likely to devolve into stereotype-learning as the model learns lexical correlations between entity names and object labels. This problem is more pronounced if our training distribution comes from a source like Wikidata, which we find to be imbalanced. More generally, language models that are trained on the Internet will model the toxic and harmful language that is found there, a well-documented finding for pre-trained language models like BERT (e.g., Gehman et al., 2020; Nadeem et al., 2020). Using such models for factual prediction is liable to amplify those biases. OPTIPROMPT is intended to be a diagnostic tool and general-purpose optimization method, not a way to use BERT as a knowledge base.
We thank Zhengbao Jiang for answering questions about LPAQA. We thank the members of the Princeton NLP group and the anonymous reviewers for their valuable comments and feedback. This work is supported in part by a Graduate Fellowship at Princeton University.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Association for Computational Linguistics (ACL), pages 861-872.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), pages 4171-4186.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Eleventh International Conference on Language Resources and Evaluation (LREC).
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3356-3369.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. arXiv preprint arXiv:2004.07780.
Adi Haviv, Jonathan Berant, and Amir Globerson. 2021. BERTese: Learning to speak to BERT. In European Chapter of the Association for Computational Linguistics (EACL).
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Empirical Methods in Natural Language Processing (EMNLP), pages 2733-2743.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association of Computational Linguistics (TACL).
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR).
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. Open sesame: Getting inside bert's linguistic knowledge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 241-253.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv preprint arXiv:2103.10385.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. ArXiv, abs/2004.09456.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Empirical Methods in Natural Language Processing (EMNLP), pages 2463-2473.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theoretic probing for linguistic structure. In Association for Computational Linguistics (ACL), pages 4609-4622.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2019. BERT is not a knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa. arXiv preprint arXiv:1911.03681.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 2383-2392.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how bert works. Transactions of the Association of Computational Linguistics (TACL), 8:842-866.
Timo Schick and Hinrich Schütze. 2020. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Automatic prompt construction for masked language models. In Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235.
Robyn Speer and Catherine Havasi. 2012. Representing general relational knowledge in conceptnet 5. In LREC, pages 3679-3686.
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In Empirical Methods in Natural Language Processing (EMNLP), pages 4208-4219.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations (ICLR).
Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In Empirical Methods in Natural Language Processing (EMNLP), pages 183-196.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), page 2153-2162.
Thomas Wolf, Julien Chaumont, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45.
Zining Zhu and Frank Rudzicz. 2020. An information theoretic view on selecting linguistic probes. In Empirical Methods in Natural Language Processing (EMNLP), pages 9251-9262.
A Detailed Results
A.1 Breakdown Accuracy for LAMA
Table 7 shows the per-relation accuracy for each prompting method. In many cases, we can better understand the probing results by examining the specific predictions each method makes.
A.2 Exploiting Training Data
Majority class baseline Figure 3 shows that all optimized prompts have a tendency to overpredict the majority class label. This behavior is most pronounced in the gradient-based methods (AUTOPROMPT and OPTIPROMPT). It is not always clear why a particular prompt elicits these predictions. For example, Shin et al. (2020) attempt to prevent AUTOPROMPT from "cheating" by filtering out prompts that contain proper nouns or gold object labels, but there are still six relations for which AUTOPROMPT elicits the majority label more than  $95\%$  of the time. The AUTOPROMPT prompts for these relations are:
- genre = jazz: “[X] freaking genre orchestra fiction acid [MASK].”
- position played = midfielder: “[X] played colors skier \u2194 defensive [MASK].”
- occupation = politician: "[X] supporters studied politicians musician turned [MASK]."
- employer  $=$  IBM: "[X] 1987adeneNBC computing succeeded [MASK]."
- instrument = piano: "[X] playingdrum concentoative electric [MASK]."
- position held = bishop: “[X] explorers voting municipal \u2192 consecrated [MASK].”
This illustrates that even discrete prompts are capable of finding prompts that elicit a specific label from an LM, and the mechanism by which these prompts elicit the prediction is often obscure.
Perhaps more surprisingly, even LPAQA occasionally finds prompts that are more likely to elicit the majority label compared to the manual prompt. The changes in these cases are often very subtle. For example, the manual prompt for the position of relation is “[X] has the position of [MASK)” and the LPAQA prompt is “[X] has the position of a [MASK]”. Simply inserting the determiner “a” into the prompt leads BERT to predict the majority label, bishop, more than five times as often