SlowGuess/NAACL_2021_OCR huggingface dataset:
Marco Baroni, Georgiana Dinu, and German Kruszewski. 2014. Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In *Association for Computational Linguistics (ACL)*, pages 238-247.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. PIQA: Reasoning about physical commonsense in natural language. In Association for the Advancement of Artificial Intelligence (AAAI).
Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In Association for the Advancement of Artificial Intelligence (AAAI), volume 34, pages 7456-7463.
David Broughton. 1995. The assumptions and theory of public opinion polling. In Public Opinion Polling and Politics in Britain, pages 15-33. Springer.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Association for Computational Linguistics (NAACL).
M. C. Donaldson. 1978. Children's Minds. W. W. Norton.
Max Eichler, Gözde Gül Şahin, and Iryna Gurevych. 2019. LINSPECTOR WEB: A multilingual probing suite for word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 127-132.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Elena Simperl, and Frederique Laforest. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Language Resources and Evaluation Conference (LREC), page 5.
M. A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In International Conference on Computational Linguistics (COLING).
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics (TACL).
D. P. Kingma and J. L. Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), pages 1-15.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In North American Association for Computational Linguistics and Human Language Technology (NAACL-HLT), pages 260-270.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Association for Computational Linguistics (ACL).
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Roektaschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. arXiv preprint arXiv:2005.11401.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv preprint arXiv:2103.10385.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. S. Zettlemoyer, and V. Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach.
Dave Orr. 2013. 50,000 lessons on how to read: A relation extraction corpus. https://github.com/google-research-datasets/ relation-extraction-corpus.
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. S. Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL).
F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel. 2019. Language models as knowledge bases? In Empirical Methods in Natural Language Processing (EMNLP).
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Delip Rao, Paul McNamee, and Mark Dredze. 2013. Entity linking: Finding extracted entities in a knowledge base. In Multi-Source, Multilingual Information Extraction and Summarization, pages 93-115. Springer.
E. Riloff and R. Jones. 1999. Learning dictionaries for information extraction by multi-level bootstrapping. In Association for the Advancement of Artificial Intelligence (AAAI), pages 474-479.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics (TACL).
Kenneth Rose. 1998. Deterministic annealing for clustering, compression, classification, regression, and related optimization problems. Proceedings of the IEEE, 80:2210-2239.
Timo Schick and Hinrich Schütze. 2020a. Exploiting cloze questions for few-shot text classification and natural language inference. arXiv preprint arXiv:2001.07676. Accepted to EACL 2021.
Timo Schick and Hinrich Schütze. 2020b. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP).
Push Singh, Thomas Lin, Erik T. Mueller, Grace Lim, Travell Perkins, and Wan Li Zhu. 2002. Open Mind Common Sense: Knowledge acquisition from the
general public. In On the Move to Meaningful Internet Systems 2002: CoopIS, DOA, and ODBASE, volume 2519, pages 1223-1237. Springer.
Daniil Sorokin and Iryna Gurevych. 2017. Context-aware representations for knowledge base relation extraction. In Empirical Methods in Natural Language Processing (EMNLP), pages 1784-1789.
R. Speer, J. Chin, and C. Havasi. 2017. ConceptNet 5.5: An open multilingual graph of general knowledge. In Association for the Advancement of Artificial Intelligence (AAAI).
Mihai Surdeanu and Heng Ji. 2014. Overview of the English slot filling track at the TAC2014 knowledge base population evaluation. In Proceedings of the TAC-KBP 2014 Workshop.
Alon Talmor, Yanal Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics - On what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743-758.
A Statistics of Relational Databases
The statistics of the various relational databases are shown in Table 8.
B Statistics of the Initial Prompts
Table 7 shows some statistics of the prompts we use to initialize the SoftPrompt model.
C Visualization of Soft Prompts
Figure 1 shows what a mixture of soft prompts looks like when we tune only layer 0. The soft prompts are not too interpretable. The words closest to the tuned tokens (shown in blue) seem to be largely on the music topic. However, the soft templates do not seem to form meaningful phrases, nor is it obvious why they would prime for  $y$  to be an instrument when  $x$  is a musician.
D Entropy of the Mixture Model
For any given relation  $r$ , the entropy of the mixture weights is
$$
H = \sum_ {\mathbf {t} \in \mathcal {T} _ {r}} p (\mathbf {t} \mid r) \cdot \log_ {2} p (\mathbf {t} \mid r) \tag {3}
$$
We then take  $2^{H} \in [1, |\mathcal{T}_{r}|]$  as a measure of the effective number of prompts that were retained. Table 10 shows some statistics of the effective number of prompts. In some cases, tuning the mixture weights essentially selected a single prompt, but on average, it settled on a mixture of several variant prompts (as illustrated by Figure 1).
E Challenging dataset with distinct  $y$ 's
As described in §4.3, we conducted an additional experiment to determine whether the prompts could generalize to novel  $y$  values. We conduct another experiment and ensure that there are no common  $y$  values among the train / dev / test sets. We use T-REx as the base relational database and split the datasets to make the ratio close to 80-10-10. The experiment results are shown in Table 9. We can observe that our method again improves the results, just as in Tables 5 and 6, which shows the generalizability of our method.
Figure 1: Visualization of the LPAQA mining prompts for relation P1303 Instrument (i.e.,  $x$  plays instrument  $y$ ) from T-REx extended. We show the effect of tuning the layer-0 token embeddings (but not higher layers) on BERT-large-cased. The prompts are sorted in decreasing order by mixture weight. Each prompt's weight is shown at left; note that after the first 12 prompts, the remaining ones have negligible contribution. We show each soft prompt in blue, followed by the original (mined) prompt in red. To visualize the tuned vector  $\mathbf{v}$ , we display the blue word  $w$  that maximizes  $p(w \mid \mathbf{v})$ . The brightness of the blue word  $w$  and the original red word  $w_0$  are respectively proportional to  $p(w \mid \mathbf{v})$  and  $p(w_0 \mid \mathbf{v})$ . The red word has size 1, and the blue word has size  $||\mathbf{v}|| / ||\mathbf{v}_0||$ , where  $\mathbf{v}_0$  is the original untuned vector (the embedding of  $w_0$ ). In this example, the blue probabilities  $p(w \mid \mathbf{v})$  range from 6.5e-5 to 9.7e-5 (mean 8.6e-5 ± 8.1e-6), the red probabilities  $p(w_0 \mid \mathbf{v})$  range from 7.7e-5 to 1.1e-4 (mean 9.5e-5 ± 7.8e-6), and the relative magnitudes  $||\mathbf{v}|| / ||\mathbf{v}_0||$  vary from 1.00 to 1.49 (mean 1.12 ± 0.13).
LM	Method	Precision@1	Precision@10	MRR
init → soft → deep	init → soft → deep	init → soft → deep	init → soft → deep
BEb	LAMA	31.1	59.5	40.3
LPAQA	34.1	62.0	43.6
Soft (sin.)	31.1 +14.6? 45.7 + 2.0 47.7	59.5 +16.3? 75.8 + 3.2 79.0	40.3 +15.9? 56.2 + 2.2 58.4
Soft (min.)	34.1 +14.7? 48.8 + 1.9 50.7?	62.0 +15.6? 79.6 + 1.1 80.7?	43.6 +15.8? 59.4 + 1.7 61.1?
Soft (par.)	34.1 +12.8? 46.9 + 1.5 48.4	62.0 +16.8? 78.8 + 0.8 79.6	43.6 +14.2? 57.8 + 1.3 59.1
Soft (ran.)	0.7 +46.6 47.3 + 0.8 48.1	4.6 +74.0 79.1 + 0.0 79.1	2.3 +56.1 58.4 + 0.5 58.9
BEl	LAMA	28.9†	57.7†	38.7†
LPAQA	39.4†	67.4†	49.1†
Soft (sin.)	28.9 +16.9 45.8 + 5.3 51.1	57.7 +19.0 76.7 + 4.4 81.1	38.7 +17.8 56.5 + 5.0 61.5
Soft (min.)	39.4 +11.6 51.0 + 0.6 51.6	67.4 +14.0 81.4 + 0.5 81.9	49.1 +12.5 61.6 + 0.5 62.1
Soft (par.)	39.4 + 9.2 48.6 + 2.5 51.1	67.4 +12.6 80.0 + 1.7 81.7	49.1 +10.5 59.6 + 2.1 61.7
Soft (ran.)	2.3 +47.1 49.4 + 1.9 51.3	8.0 +73.0 81.0 + 0.7 81.7	4.5 +55.9 60.4 + 1.5 61.9
Rob	LPAQA	1.2†	9.1†	4.2†
AutoPrompt	40.0	68.3	49.9
Soft (min.)	1.2 +39.4 40.6 - 7.3 33.2	9.1 +66.3 75.4 - 22.3 53.0	4.2 +48.8 53.0 - 12.1 40.8
BAb	LPAQA	0.8†	5.7†	2.9†
Soft (min.)	0.8 +39.1 39.9	5.7 +69.7 75.4	2.9 +49.2 52.1
BAI	LPAQA	3.5†	5.6†	4.8†
Soft (min.)	3.5 +22.3 25.8	5.6 +62.4 68.0	4.8 +36.2 41.0
Table 5: Experimental results on T-REx original datasets. In the LM column, BEb is BERT-base-cased, BEl is BERT-large-cased, BAb is BART-base-cased, BA1 is BART-large-cased, Rob is RoBERTa-base, and Rol is RoBERTa-large. In the results block, "init" uses the initial untuned prompts; "soft" starts at "init" and tunes the prompts (layer 0) and mixture weights; and "deep" starts at "init" and tunes all the layers. Numbers above the arrows are the relative change in the performance. Within each block, we boldface the best system and all those that are not significantly worse (paired permutation test,  $p < 0.02$ ). We also boldface the relative changes that are significantly different from 0. Other symbols are as in Table 1.
LM	Method	Precision@1	Precision@10	MRR
init → soft → deep	init → soft → deep	init → soft → deep	init → soft → deep
BEb	LAMA	26.4	54.3	35.8
LPAQA	31.2	57.3	39.9
Soft (sin.)	26.4 +22.2? 48.6 +1.0 49.6	54.3 +23.3? 77.6 +0.3 77.9	35.8 +22.9? 58.7 +0.6 59.3
Soft (min.)	31.2 +19.0? 50.2 +0.3 50.5?	57.3 +21.9? 79.2 +0.5 79.7?	39.9 +20.2? 60.1 +0.4 60.5?
Soft (par.)	31.2 +18.5? 49.7 +0.0 49.7	57.3 +21.3? 78.6 +0.6 79.2	39.9 +19.6? 59.5 +0.3 59.8
Soft (ran.)	0.8 +46.3 47.1 +3.5 50.6	4.0 +70.4 74.4 +4.9 79.3	2.2 +54.3 56.5 +3.9 60.4
BEi	LAMA	24.0†	53.7†	34.1†
LPAQA	37.8†	64.4†	44.0†
Soft (sin.)	24.0 +26.2 50.2 +1.2 51.4	53.7 +24.9 78.6 +0.9 79.5	34.1 +25.9 60.0 +1.2 61.2
Soft (min.)	37.8 +13.4 51.2 +1.3 52.5	64.4 +15.1 79.5 +1.6 81.1	44.0 +17.0 61.0 +1.4 62.4
Soft (par.)	37.8 +12.5 50.3 +1.4 51.7	64.4 +14.3 78.7 +2.1 80.8	44.0 +16.1 60.1 +1.6 61.7
Soft (ran.)	1.4 +46.1 47.5 +4.4 51.9	5.4 +68.9 74.3 +6.3 80.6	5.7 +51.2 56.9 +5.0 61.9
Table 6: Experiment results on T-REx extended datasets.
prompts	T-REx-min.	T-REx-par.	Goog-sin.	Goog-min.	Goog-par.	ConceptNet
#relations	41	41	3	3	3	16
avg. prompts	28.4	26.2	1	32.7	28.0	9.3
min #prompts	6	13	1	29	24	1
max #prompts	29	30	1	40	30	38
avg. #tokens	5.1	4.5	4.7	5.3	4.2	7.1
Table 7: Statistics of prompts. The "Goog" stands for "Google-RE." We do not list the statistics of randomized prompts, as they should match the statistics of the mined prompts ("min.") from which they are derived.
database	T-REx original	T-REx extended	Google-RE	ConceptNet
#relations	41	41	3	16
avg. #unique x	1580	834	1837	511
avg. #unique y	217	151	372	507
min #(x,y)	544	310	766	510
max #(x,y)	1982	1000	2937	4000
mean #(x,y)	1715	885	1843	1861
Table 8: Statistics of the relational databases.
Model	P@1	P@10	MRR
LPAQA (BEb)	18.9	40.4	26.6
Soft (BEb)	23.0 (+4.1)	45.2 (+4.8)	30.5 (+3.9)
LPAQA (BEI)	23.8	47.7	32.2
Soft (BEI)	27.0 (+3.2)	51.7 (+4.0)	35.4 (+3.2)
Table 9: Results with distinct  $y$ 's. We use the BERT-base-cased and BERT-large-cased LMs and the LPAQA mining based prompts as initial prompts. The experiments are conducted on the T-REx original dataset.
statistic	mean	std	min	max
T-REx original + min.	12.5	4.0	4.6	21.0
T-REx extended + min.	12.5	4.0	4.6	20.3
T-REx original + par.	5.4	4.0	1.1	17.1
T-REx extended + par.	5.4	3.9	1.2	18.4
Table 10: Statistics of effective number of prompts.