SlowGuess/NAACL_2022_OCR huggingface dataset:
International Conference on Web Search and Data Mining (WSDM), pages 105-113, Melbourne, Australia.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962-977.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Feihu Jin, Jinliang Lu, Jiajun Zhang, and Chengqing Zong. 2022. Instance-aware prompt learning for language understanding and generation. arXiv preprint arXiv:2201.07126.
Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 3250-3258, Online.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 7871-7880, Online.
Guiliang Liu, Xu Li, Jiakang Wang, Mingming Sun, and Ping Li. 2020. Extracting knowledge from web text with monte carlo tree search. In Proceedings of the Web Conference (WWW), pages 2585-2591, Taipei.
Fabio Petroni, Tim Roktaschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 5203-5212, Online.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Online.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), pages 5998-6008, Long Beach, CA.
Zhuoyi Wang, Yu Lin, Yifan Li, Feng Mi, Zachary Tian, Latifur Khan, and Bhavani M. Thuraisingham. 2021. Unsupervised perturbation based self-supervised adversarial training. In Proceedings of the 7th IEEE International Conference on Big Data Security on Cloud (BigDataSecurity), pages 20-25, New York City, NY.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, VG Vydiswaran, and Hao Ma. 2022. IDPG: An instance-dependent prompt generation method. arXiv preprint arXiv:2204.04497.
Jinxing Yu, Yunfeng Cai, Mingming Sun, and Ping Li. 2021. Mquade: a unified model for knowledge fact embedding. In Proceedings of the Web Conference (WWW), pages 3442-3452, Virtual Event / Ljubljana, Slovenia.
Jingyuan Zhang, Mingming Sun, Yue Feng, and Ping Li. 2020. Learning interpretable relationships between entities, relations and concepts via bayesian structure learning on open domain facts. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 8045-8056, Online.
Yue Zhang, Hongliang Fei, and Ping Li. 2021. Read-sRE: Retrieval-augmented distantly supervised relation extraction. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 2257-2262, Virtual Event, Canada.
Yue Zhang, Hongliang Fei, and Ping Li. 2022a. End-to-end distantly information extraction with retrieval augmentation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval.
Yue Zhang, Mingming Sun, Jingyuan Zhang, and Ping Li. 2022b. Explainable concept graph completion by
bridging open-domain relations and concepts. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM), Alexandria, VT.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 12697-12706, Virtual Event.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 5017-5033, Online.
Relation	Type	Name	Maunal	LPAQA	AutoPro	OptiPro	Ours
P1376	1-1	capital of	73.8	67.8	56.2	56.7	61.6
P36	1-1	capital	62.1	62.1	59.7	61.3	52.2
P103	N-1	native language	72.2	72.2	79.7	86.8	86.9
P127	N-1	owned by	34.8	32.5	44.3	49.6	54.0
P131	N-1	located in the admin. territorial entity	23.3	22.8	28.9	41.4	40.3
P136	N-1	genre	0.8	16.8	55.3	63.6	68.4
P138	N-1	named after	61.4	59.5	70.7	73.4	76.1
P140	N-1	religion	0.6	59.8	60.5	76.5	80.9
P159	N-1	headquarters location	32.4	35.6	35.7	37.4	37.6
P17	N-1	country	31.3	39.8	51.0	57.8	54.2
P176	N-1	manufacturer	85.5	81.5	87.5	87.3	91.6
P19	N-1	place of birth	21.1	21.1	19.5	20.6	22.8
P20	N-1	place of death	27.9	27.9	29.8	33.8	35.8
P264	N-1	record label	9.6	6.3	4.2	45.5	5.6
P276	N-1	location	41.5	41.5	43.0	47.1	46.5
P279	N-1	subclass of	30.7	14.7	54.9	64.7	65.6
P30	N-1	continent	25.4	16.9	78.6	86.3	89.1
P361	N-1	part of	23.6	31.4	37.0	46.4	41.1
P364	N-1	original language of film or TV show	44.5	43.9	45.0	51.3	54.6
P37	N-1	official language	54.6	56.8	52.7	58.6	62.9
P407	N-1	language of work or name	64.2	65.2	68.4	71.0	68.2
P413	N-1	position played on team / speciality	0.5	23.7	41.7	44.0	51.5
P449	N-1	original network	20.9	9.1	33.1	36.0	39.8
P495	N-1	country of origin	28.7	32.2	35.8	40.8	37.7
P740	N-1	location of formation	8.9	13.7	13.1	15.0	17.3
P1001	N-M	applies to jurisdiction	70.5	72.8	80.5	85.2	87.0
P101	N-M	field of work	9.9	5.3	12.1	14.1	19.4
P106	N-M	occupation	0.6	0.0	13.6	35.7	31.3
P108	N-M	employer	6.8	5.7	7.8	11.2	12.5
P1303	N-M	instrument	7.6	18.0	23.1	23.6	45.8
P1412	N-M	languages spoken, written or signed	65.0	64.7	71.5	76.1	77.1
P178	N-M	developer	62.9	59.4	64.3	67.9	68.6
P190	N-M	twinned administrative body	2.2	1.7	2.4	3.1	3.9
P27	N-M	country of citizenship	0.0	41.5	45.8	47.1	46.5
P31	N-M	instance of	36.7	36.7	53.6	64.9	68.9
P39	N-M	position held	8.0	16.1	27.2	42.8	69.6
P463	N-M	member of	67.1	57.3	64.0	64.0	73.8
P47	N-M	shares border with	13.7	13.7	19.2	22.2	21.2
P527	N-M	has part	11.2	10.6	22.1	34.8	38.7
P530	N-M	diplomatic relation	2.8	3.9	2.8	3.3	2.8
P937	N-M	work location	29.8	39.1	34.4	43.3	48.2
Table 4: The accuracy of different prompts on LAMA for each relation using BERT-base-cased.
We adopt "BART-large" as our generative module and "BERT-base-cased" as our MLM module, both of which are collected from Huggingface website<sup>2</sup>. We use the Adam optimizer with learning rate  $5e - 5$ , set warm-up ratio to 0.1, and weight decay to 1e-3. We repeat our experiments five times and report the average metrics on the test set.
A.2 Detailed Results
Table 4 shows the per-relation accuracy for each prompting method. We see that our method achieves the best performance for most cases.
2https://huggingface.co/models