SlowGuess/NAACL_2021_OCR huggingface dataset:
compared to the manual prompt (50.9% vs. 9.5%), and almost twice as often relative to the true distribution in the LAMA benchmark (27.3%). This suggests that even simple data-driven methods can find prompts that encode some regularities in the training data and result in over-estimates of the number of facts in the language model.
Control result details Table 8 shows the accuracy of optimized prompts under our random controls (Section 4.2) and also shows how much accuracy can be attributed to predict the majority class label. AUTOPROMPT cannot predict any facts in the Random Model setting but performs decently on several relations in the Random Embeddings setting by predicting the majority class. For reasons we cannot entirely explain, there is one relation, occupation, for which AUTOPROMPT's performance cannot be attributed to the class prior. The correct predictions in this category are all a result of predicting actor, which AUTOPROMPT predicts  $23.3\%$  of the time. (The most frequent label in the training data is politician.) Other high frequency predictions for this relation include jet, wool, and smart. Notably, even when AUTOPROMPT finds a prompt that can draw out the class prior, it typically does not elicit the class prior  $100\%$  of the time.
OPTIPROMPT is more successful at exploiting the training data. In the Random Model setting, virtually all correct predictions can be attributed to the majority class, which OPTIPROMPT can frequently elicit for all inputs. One noteworthy exception is languages spoken, where OPTIPROMPT is able to successfully classify subjects as speaking either English or French in some cases. It is not immediately clear what decision rule the model learns for these predictions—for example, it could be that the model predicts either English or French at random, in rough proportion to the training distribution; or the model is able to use the correlations between names and spoken languages. In any case, the results illustrate that optimized prompts can learn more sophisticated strategies than simply predicting the majority class, even given a Transformer that contains no prior information at all.
A.3 LAMA-easy and LAMA-hard
Table 9 shows the accuracy of different prompts on the easy and hard subset of LAMA described in Section 5. All of the optimized models tend to perform better on LAMA-easy compared to LAMA-hard, and OPTIPROMPT out-performs AU-
Method	BERT (110M)	BERT (330M)	RoBERTa (330M)	ALBERT (235M)†
Pre-T.	Rand M.	Pre-T.	Rand M.	Pre-T.	Rand M.	Pre-T.	Rand M.
Manual	30.6	-	32.2	-	23.6	-	27.4	-
LPAQA	35.6	-	36.2	-	29.3	-	29.8	-
AUTOPROMPT	44.6	0.0	44.5	0.1	38.6	0.0	33.2	0.0
OPTIPROMPT	50.8	19.9	52.7	19.3	47.8	19.6	44.6	16.9
Fine-tuning	51.9	19.8	54.9	19.6	52.3	21.4	52.8	21.1
Table 6: Comparison of different pre-trained LMs. We downsample the LAMA test set to make sure that in each sample, the object is a single token for all the models. †: there is parameter sharing in ALBERT models so the actual models are much bigger. Pre-T.: pre-trained language models. Rand M.: randomly initialized models.
TOPROMPT in both categories. For example, on the shares border relation, OPTIPROMPT achieves an improvement on both easy questions (Campagnano di Roma, Rome) and hard ones (Chiapas, Veracruz). But note that high accuracy on LAMA-easy does not necessarily mean that a prompt encodes information about the fact distribution. For example, all prompts, including the manually written prompts, perform well on the easy examples in the capital relation. This category includes such facts as "The capital of Sarajevo Canton is Sarajevo," which evidently do not require very much tuning to predict.
B.1 Prompt Optimization
We implement OPTIPROMPT based on the HuggingFace's Transformers (Wolf et al., 2020) library. During trianing, we use an Adam optimizer and a scheduler with a warmup ratio of 0.1. We use an Adam optimizer and a linear scheduler with a warmup ratio of 0.1. We train our OPTIPROMPT model for 10 epochs with a learning rate of 3e-3 and a batch size of 16. For fine-tuning, we use an Adam optimizer and a linear scheduler with a warmup ratio of 0.1. We fine-tune the language models for 10 epochs with a learning rate of 2e-6 and a batch size of 2.
We report AUTOPROMPT's performance based on the prompts released by Shin et al. (2020). When we apply AUTOPROMPT to a control task (e.g., the Random Embeddings model), or compare AUTOPROMPT with different language models on a different dataset (see Appendix C), we run AUTOPROMPT for 1000 iterations for each model to search for the prompt of a relation.
B.2 LAMA Classifiers
In Section 4.2 we fit two simple probabilistic models to the Wikidata training data collected
by (Shin et al., 2020). Given a relation  $r$  and a subject  $s$  consisting of tokens  $w_{1},\ldots ,w_{|s|}\in \mathcal{V}$ , the Class Prior model predicts  $\hat{o} = \arg \max_{o}P(o\mid r)$ , the object label that is most frequently associated with relation  $r$  in the training data. The Naive Bayes model predicts  $\hat{o} = \arg \max_{o}P(o\mid s,r)$ , with
$$
P (o \mid s, r) = P (o \mid r) \prod_ {i = 1} ^ {| s |} P (o \mid w _ {i}).
$$
The probabilities are estimated from the corpus with add-one smoothing:
$$
P (o \mid w _ {i}) = \frac {\operatorname {c o u n t} (o , w _ {i}) + 1}{\sum_ {w \in \mathcal {V}} (\operatorname {c o u n t} (o , w) + 1)}.
$$
C Comparing Pre-trained Language Models
We compare different pre-trained language models (BERT, RoBERTa (Liu et al., 2019), and ALBERT (Lan et al., 2019)) with different probing methods. We collect at most 1000 training samples for each relation from the TRE-x dataset and constrain the object of each sample to be a token for all the models<sup>12</sup>. During testing, we downsample the LAMA test set to make sure that the object in each sample is a single token for all the models. In Table 6 shows the results of different probing methods applied to four pre-trained language models, along with our Random Model baseline. We make the following observations:
- Base vs. Large: The larger version of BERT performs better on LAMA than BERT base
12The original data collected by Shin et al. (2020) is not applicable when we compare different language models, because some object tokens are not in the vocabulary of RoBERTa or ALBERT.
in the OPTIPROMPT probe. We might hypothesize that BERT-large is simply more capable of finding patterns in the training data, but our baseline result does not indicate that this is the case—on the contrary, BERT-large performs marginally worse on the Random Model baseline. This could lead us to believe that BERT-large truly does store information about 1 or  $2\%$  more LAMA facts compared to BERT-base.
- BERT vs. RoBERTa vs. ALBERT: Shin et al. (2020) find that RoBERTa performs significantly worse on LAMA than BERT. We find this is true for our prompts as well (comparing with BERT-large), but the magnitude of the difference decreases in the fine-tuning setting. Our baseline result gives a possible hint as to why: RoBERTa performs better in the RM setting with fine-tuning, indicating that part of the difference between OPTIPROMPT and fine-tuning might be due to better exploitation of training data. This change is even more dramatic in ALBERT. Perhaps these models store less factual information due to pre-training on a wider variety of genres.
We believe that further comparisons along these lines are a promising area of future work—for example, if we could show that probing results are correlated with downstream task performance and use probes to guide model selection.
Relation	Type	Name	#	Manual	LPAQA	Auto	Opti
P1376	1-1	capital of	233	73.8	67.8	56.2	56.7
P36	1-1	capital	702	62.1	62.1	59.7	61.3
P103	N-1	native language	977	72.2	72.2	79.7	86.8
P127	N-1	owned by	687	34.8	32.5	44.3	49.6
P131	N-1	located in the administrative territorial entity	881	23.3	22.8	28.9	41.4
P136	N-1	genre	931	0.8	16.8	55.3	63.6
P138	N-1	named after	642	61.4	59.5	70.7	73.4
P140	N-1	religion	473	0.6	59.8	60.5	76.5
P159	N-1	headquarters location	967	32.4	35.6	35.7	37.4
P17	N-1	country	930	31.3	39.8	51.0	57.8
P176	N-1	manufacturer	973	85.5	81.5	87.5	87.3
P19	N-1	place of birth	944	21.1	21.1	19.5	20.6
P20	N-1	place of death	953	27.9	27.9	29.8	33.8
P264	N-1	record label	429	9.6	6.3	4.2	45.5
P276	N-1	location	958	41.5	41.5	43.0	47.1
P279	N-1	subclass of	964	30.7	14.7	54.9	64.7
P30	N-1	continent	975	25.4	16.9	78.6	86.3
P361	N-1	part of	932	23.6	31.4	37.0	46.4
P364	N-1	original language of film or TV show	856	44.5	43.9	45.0	51.3
P37	N-1	official language	966	54.6	56.8	52.7	58.6
P407	N-1	language of work or name	877	64.2	65.2	68.4	71.0
P413	N-1	position played on team / speciality	952	0.5	23.7	41.7	44.0
P449	N-1	original network	880	20.9	9.1	33.1	36.0
P495	N-1	country of origin	909	28.7	32.2	35.8	40.8
P740	N-1	location of formation	936	8.9	13.7	13.1	15.0
P1001	N-M	applies to jurisdiction	701	70.5	72.8	80.5	85.2
P101	N-M	field of work	696	9.9	5.3	12.1	14.1
P106	N-M	occupation	958	0.6	0.0	13.6	35.7
P108	N-M	employer	383	6.8	5.7	7.8	11.2
P1303	N-M	instrument	949	7.6	18.0	23.1	23.6
P1412	N-M	languages spoken, written or signed	969	65.0	64.7	71.5	76.1
P178	N-M	developer	591	62.9	59.4	64.3	67.9
P190	N-M	twinned administrative body	992	2.2	1.7	2.4	3.1
P27	N-M	country of citizenship	966	0.0	41.5	45.8	47.1
P31	N-M	instance of	922	36.7	36.7	53.6	64.9
P39	N-M	position held	892	8.0	16.1	27.2	42.8
P463	N-M	member of	225	67.1	57.3	64.0	64.0
P47	N-M	shares border with	920	13.7	13.7	19.2	22.2
P527	N-M	has part	976	11.2	10.6	22.1	34.8
P530	N-M	diplomatic relation	996	2.8	3.9	2.8	3.3
P937	N-M	work location	954	29.8	39.1	34.4	43.3
Table 7: The accuracy of different prompts on LAMA for each relation using BERT-base-cased. Manual: the manually written prompts included in LAMA; LPAQA: manually written + paraphrased prompts from Jiang et al. (2020); Auto: the five-token AUTOPROMPT prompts released by Shin et al. (2020). Opti: OPTIPROMPT initialized using the manually written templates.
Relation	Type	Name	Random Embeddings	Random Model
Auto	Opti	FT	Auto	Opti	FT
P1376	1-1	capital of	0.0/0.0	0.0/0.0	0.0/0.0	0.0/0.0	0.0/0.0	0.0/0.0
P36	1-1	capital	0.0/0.1	0.0/11.8	0.0/0.1	0.0/0.0	0.0/0.0	0.0/0.0
P103	N-1	native language	26.5/26.5	60.1/60.1	57.6/63.5	0.0/0.0	60.1/60.1	60.1/60.5
P127	N-1	owned by	0.0/0.0	6.8/7.0	6.7/17.2	0.0/0.0	6.8/6.8	6.4/12.4
P131	N-1	located in...	0.0/0.0	0.0/2.3	0.2/1.5	0.0/0.0	0.2/0.2	0.6/0.6
P136	N-1	genre	27.6/27.6	55.4/55.4	54.7/56.1	0.0/0.0	55.3/55.3	55.4/55.4
P138	N-1	named after	0.0/0.0	1.2/3.9	0.8/50.8	0.0/0.0	1.6/1.6	1.6/1.6
P140	N-1	religion	0.0/0.0	53.7/53.7	53.7/53.7	0.0/0.0	53.7/53.7	53.7/53.7
P159	N-1	headquarters location	0.0/0.0	9.0/9.0	7.7/7.7	0.0/0.0	9.0/9.0	9.0/9.0
P17	N-1	country	0.0/0.1	2.8/2.8	1.6/7.7	0.0/0.0	0.5/1.9	2.6/2.6
P176	N-1	manufacturer	0.2/0.2	8.9/8.9	8.8/80.0	0.0/0.0	8.9/8.9	8.8/70.2
P19	N-1	place of birth	0.0/0.0	2.9/3.2	5.3/5.4	0.0/0.0	0.0/0.3	6.0/6.4
P20	N-1	place of death	0.0/0.0	7.3/11.4	7.8/12.7	0.0/0.0	10.2/10.2	9.7/11.6
P264	N-1	record label	0.0/0.0	37.5/37.5	37.5/37.5	0.0/0.0	37.5/37.5	37.5/37.5
P276	N-1	location	0.2/0.2	2.7/2.7	3.8/5.6	0.0/0.0	0.7/0.9	4.2/4.4
P279	N-1	subclass of	0.0/0.1	15.7/26.2	14.4/36.9	0.0/0.0	15.9/15.9	16.0/16.0
P30	N-1	continent	57.5/57.5	72.3/72.3	72.3/72.3	0.0/0.0	72.3/72.3	72.3/72.3
P361	N-1	part of	4.9/4.9	6.0/6.0	6.0/7.6	0.0/0.0	6.0/6.0	6.0/6.0
P364	N-1	original language...	0.0/0.0	21.1/25.1	23.5/27.8	0.0/0.0	24.1/25.5	22.4/25.5
P37	N-1	official language	0.0/0.0	17.0/17.0	14.7/16.0	0.0/0.0	17.0/17.0	17.0/17.0
P407	N-1	language of work...	0.7/0.9	46.4/46.4	45.8/46.4	0.0/0.0	46.4/46.4	46.4/46.4
P413	N-1	position played...	30.3/30.3	41.6/41.6	41.7/41.7	0.0/0.0	41.7/41.7	41.7/41.7
P449	N-1	original network	3.2/3.2	25.9/31.0	23.9/32.2	0.0/0.0	28.9/31.9	21.5/29.8
P495	N-1	country of origin	1.9/1.9	8.9/10.8	9.6/12.3	0.0/0.0	10.9/10.9	8.6/13.1
P740	N-1	location of formation	0.0/0.0	7.4/7.4	6.8/7.6	0.0/0.0	4.5/5.6	7.4/7.4
P1001	N-M	applies to jurisdiction	0.1/0.1	8.0/42.8	7.4/54.9	0.0/0.0	9.6/9.6	9.6/9.7
P101	N-M	field of work	0.0/0.0	9.6/10.1	10.3/10.8	0.0/0.0	10.5/10.5	10.5/10.8
P106	N-M	occupation	0.0/8.9	6.2/27.5	5.2/30.8	0.0/0.0	14.3/14.3	6.8/26.4
P108	N-M	employer	0.0/0.0	7.6/8.9	3.4/9.1	0.0/0.0	4.2/6.3	5.7/9.4
P1303	N-M	instrument	0.0/0.0	22.8/22.8	21.9/22.7	0.0/0.0	10.6/10.6	22.8/22.8
P1412	N-M	languages spoken...	0.0/0.0	13.9/27.7	10.5/28.0	0.0/0.0	7.3/25.3	12.0/28.2
P178	N-M	developer	0.0/0.0	2.7/11.3	4.2/29.4	0.0/0.0	4.7/5.4	4.6/8.8
P190	N-M	twinned admin...	0.0/0.0	0.0/1.1	1.7/2.1	0.0/0.0	1.4/2.1	0.3/2.4
P27	N-M	country of citizenship	1.3/1.3	9.7/9.9	8.8/13.4	0.0/0.0	10.0/10.0	9.9/10.1
P31	N-M	instance of	0.3/0.3	8.0/11.2	8.4/24.9	0.0/0.0	8.8/8.8	8.9/8.9
P39	N-M	position held	0.0/0.0	20.9/28.7	23.8/32.4	0.0/0.0	27.2/27.2	18.5/30.4
P463	N-M	member of	1.3/1.3	2.2/45.8	2.2/60.4	0.0/0.0	2.2/2.2	2.2/56.4
P47	N-M	shares border with	0.0/0.0	0.0/0.1	0.2/0.9	0.0/0.0	0.0/0.0	0.0/1.1
P527	N-M	has part	0.0/0.0	17.3/17.3	12.0/18.0	0.0/0.0	14.4/14.4	17.4/17.4
P530	N-M	diplomatic relation	0.0/0.0	0.3/0.5	0.0/1.2	0.0/0.0	0.9/0.9	0.0/0.6
P937	N-M	work location	0.0/0.0	14.6/14.6	12.1/16.8	0.0/0.0	13.2/13.2	14.8/14.8
Table 8: Control result details. The value in each cell is  $Maj./Acc$ , where  $Acc$  is the percentage of facts of relation  $r$  that the model predicts correctly and  $Maj$  is the percentage of facts  $\langle s, r, o \rangle$  such that (a) the model predicts  $o$  correctly, and (b)  $o$  is the most frequent object for relation  $r$  in the training data. We probe the BERT-base-cased model, reinitializing either the token embeddings or all of the parameters.
Relation	Type	Name	LAMA-easy	LAMA-hard
#	Man.	LPAQA	Auto	Opti	#	Man.	LPAQA	Auto	Opti
P1376	1-1	capital of	1	100.0	0.0	100.0	0.0	232	73.7	68.1	56.0	56.9
P36	1-1	capital	2	50.0	50.0	50.0	50.0	700	62.1	62.1	59.7	61.3
P103	N-1	native language	668	75.7	75.7	94.0	95.7	309	64.4	64.4	48.9	67.6
P127	N-1	owned by	131	53.4	87.0	89.3	90.8	556	30.4	19.6	33.6	39.9
P131	N-1	located in...	72	16.7	23.6	56.9	63.9	809	23.9	22.7	26.5	39.4
P136	N-1	genre	535	1.1	15.7	96.1	92.9	396	0.3	18.2	0.3	24.0
P138	N-1	named after	341	84.2	85.0	94.7	95.3	301	35.5	30.6	43.5	48.5
P140	N-1	religion	254	0.0	80.7	100.0	96.9	219	1.4	35.6	14.6	53.0
P159	N-1	headquarters location	97	45.4	57.7	82.5	75.3	870	30.9	33.1	30.5	33.2
P17	N-1	country	131	44.3	52.7	71.8	86.3	799	29.2	37.7	47.6	53.2
P176	N-1	manufacturer	791	94.7	90.8	95.6	95.8	182	45.6	41.2	52.2	50.0
P19	N-1	place of birth	101	77.2	77.2	77.2	75.2	843	14.4	14.4	12.6	14.0
P20	N-1	place of death	183	78.7	78.7	78.7	74.9	770	15.8	15.8	18.2	24.0
P264	N-1	record label	195	6.2	6.7	2.6	81.5	234	12.4	6.0	5.6	15.4
P276	N-1	location	90	60.0	60.0	68.9	81.1	868	39.6	39.6	40.3	43.5
P279	N-1	subclass of	374	32.9	28.3	79.1	92.2	590	29.3	6.1	39.5	47.3
P30	N-1	continent	705	34.0	19.1	98.0	99.9	270	3.0	11.1	27.8	50.7
P361	N-1	part of	73	1.4	11.0	16.4	98.6	859	25.5	33.2	38.8	41.9
P364	N-1	original language...	373	69.4	71.8	76.9	82.3	483	25.3	22.4	20.3	27.3
P37	N-1	official language	197	43.7	53.8	88.8	87.3	769	57.3	57.6	43.4	51.2
P407	N-1	language of work...	460	84.6	85.9	92.8	91.3	417	41.7	42.4	41.5	48.7
P413	N-1	position played...	397	1.3	56.7	100.0	99.5	555	0.0	0.2	0.0	4.3
P449	N-1	original network	428	26.4	11.9	54.7	64.5	452	15.7	6.4	12.6	9.1
P495	N-1	country of origin	187	41.2	44.9	51.9	80.2	722	25.5	28.9	31.6	30.6
P740	N-1	location of formation	79	43.0	57.0	74.7	86.1	857	5.7	9.7	7.5	8.4
P1001	N-M	applies to jurisdiction	408	79.9	84.1	90.7	95.1	293	57.3	57.0	66.2	71.3
P101	N-M	field of work	104	9.6	12.5	38.5	36.5	592	10.0	4.1	7.4	10.1
P106	N-M	occupation	386	0.0	0.0	29.8	74.1	572	1.0	0.0	2.6	9.8
P108	N-M	employer	54	27.8	22.2	53.7	72.2	329	3.3	3.0	0.3	1.2
P1303	N-M	instrument	243	7.8	37.9	87.7	88.1	706	7.5	11.2	0.8	1.4
P1412	N-M	languages spoken...	473	86.7	81.8	89.4	90.7	496	44.4	48.4	54.4	62.1
P178	N-M	developer	259	79.5	81.1	90.3	95.4	332	50.0	42.5	44.0	46.4
P190	N-M	twinned admin...	44	2.3	2.3	18.2	11.4	948	2.2	1.7	1.7	2.7
P27	N-M	country of citizenship	217	0.0	74.7	54.8	75.1	749	0.0	31.9	43.1	39.0
P31	N-M	instance of	316	48.1	48.1	79.7	94.0	606	30.7	30.7	39.9	49.7
P39	N-M	position held	421	11.2	28.5	55.3	63.2	471	5.1	5.1	2.1	24.6
P463	N-M	member of	139	87.1	71.9	91.4	95.0	86	34.9	33.7	19.8	14.0
P47	N-M	shares border with	35	5.7	5.7	14.3	22.9	885	14.0	14.0	19.4	22.1
P527	N-M	has part	296	9.1	3.7	29.1	61.8	680	12.1	13.5	19.1	23.1
P530	N-M	diplomatic relation	18	5.6	5.6	5.6	0.0	978	2.8	3.9	2.8	3.4
P937	N-M	work location	258	77.1	86.4	75.6	88.0	696	12.2	21.6	19.1	26.7
Table 9: The accuracy by relation on LAMA-easy and LAMA-hard. LAMA-easy consists of the facts that are predicted correctly by any of three models: the Naive Bayes model described in Section 4; BERT-base-cased with randomly initialized token embeddings; and BERT-base-cased with all parameters reinitialized. LAMA-hard contains all the remaining facts.