Open-World Biomedical Knowledge Probing and Verification
Xi TianPeng Wang

Shengyu Mao

Zhejiang University
Hangzhou, Zhejiang, China

Zhejiang University
Hangzhou, Zhejiang, China

Zhejiang University
Hangzhou, Zhejiang, China

ABSTRACT
Recently, there has been a surge of interest in the NLP community in using Pre-trained Language Models (PLMs) as general or
domain-specific Knowledge Bases. However, previous evaluation
settings are under the Closed-World Assumption. In this paper, we
propose open-world knowledge probing and provide an empirical analysis of biomedical PLMs on three datasets. We find thatthe previous evaluation setting may underestimate the knowledge
from the PLMs. We further integrate scientific knowledge into theprompt design and propose SciPrompt, leading to better performance for biomedical knowledge probing. We hope our work canbetter understand the knowledge learned from PLMs and inspire
further research for scientific knowledge discovery.

Pre-trained Corpus: People with COVID-19 have
had a wide range of symptoms reported Feveror chills Cough Shortness of breath or difficulty
breathing Fatigue Muscle or body aches
Headache New loss of taste or smell ...

， ，
，
，
，

COVID-19

：

，

Cough
variant

Sore throat

Fever

has_symptom

Verify and Update

Loss oftaste

Tiredness

1. Cough
2. Fatigue

Pre-trained Language Model

3. Sore throat
4. Fever5. Tiredness...

Covid-19 variant Omicron has symptom
Triple:<Omicron, has_symptom , ?>19.Loss of taste
20.Muscle Ache
Open-world

Closed-world

Assumption

Assumption

CCS CONCEPTS• Computing methodologies → Natural language processing.

Figure 1: Illustration of open-world knowledge probing forbiomedical PLMs.

KEYWORDS
Pre-trained Language Models, Probe, Verification, Open-world
ACM Reference Format:
Xi Tian, Peng Wang, and Shengyu Mao. 2023. Open-World Biomedical
Knowledge Probing and Verification. In Proceedings of The 12th International
Joint Conference on Knowledge Graphs (IJCKG’23). ACM, New York, NY,
USA, 5 pages. https://doi.org/XXXXXXX.XXXXXXX

1INTRODUCTION

Recent studies [1, 7, 14] have demonstrated that Pre-trained Language Models (PLMs) implicitly contain different kinds of knowledge [11] in their parameters without the need of human supervision. This is typically done by formulating knowledge triples
as cloze-style queries with the objects being masked and using
the PLM to fill the single or multiple [Mask] token(s) [3, 8, 20]. In
the biomedical domain, it has been shown that specialized PLMs
(e.g., BioBERT [9], Bio-LM [10]) potentially contain the implicit
Knowledge Graphs (KGs) [2, 13, 18].
Existing research [16] on knowledge probing operates under
the Closed-World Assumption (CWA) [15] in which all entities andrelations already exist in the KG – they are only knowledge which
has been discovered. However, the introduction of PLMs may bringin much-unseen knowledge, which is considered incorrect under
CWA, wrongly lowering the knowledge probing performance. As
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
IJCKG’23, Dec 08–09, 2023, Toykyo, Japan© 2023 Association for Computing Machinery.
ACM ISBN 978-1-4503-XXXX-X/18/06. . . $15.00
https://doi.org/XXXXXXX.XXXXXXX

shown in Figure 1, for a triple query (Omicron, has_symptom, ?),the PLM model gives many correct tail entities, but only Cough,Fatigue and Sore throat are considered correct under CWA since itexists in KGs.
In this work, we borrow the idea of open-world assumption[4, 17] to develop the open-world knowledge probing for PLMs
[12]. With an empirical study by biomedical PLMs on three datasets,
we notice that the previous evaluation setting may underestimate
the performance of knowledge probing, and parts of the probed
knowledge are correct via expert verification. These results indicatethat previous knowledge probing methods miss many essential
correct facts that may contribute to knowledge discovery from selfsupervised pre-training. Moreover, we conduct a comprehensive
analysis to find essential patterns of those “new” triples (not existingin KG under CWA). We observe that similar semantic facts accountfor lots of them, while some other facts are not related to existing
KG, which is interesting for further investigation. We also propose
a new simple probing method, SciPrompt, which utilizes scientificknowledge in prompt design and yields better performance.
The rest of the paper is organized as follows. Section 2 introduces
our new probing method, SciPrompt, along with the knowledge
verification procedure. Section 3 introduces our experimental setting and analyzes the intrinsic patterns of those “new” triples probed
from PLMs under open-world assumption. Section 4 and Section
5 conclude the paper while discussing the existing challenges andfuture directions.

2

OPEN-WORLD KNOWLEDGE PROBING

In this section, we introduce the open and close-world assumptions and detail the probing method, SciPrompt, and knowledge
verification procedure.

IJCKG’23, Dec 08–09, 2023, Toykyo, JapanXi Tian, et al.

Table 1: Main results in CWA setting. We report Acc@1/Acc@5 of each model, including the macro average across three different
knowledge sources. The highest scores are boldfaced.

BioBERT

2.1

Bio-LM

SourceManualOpti.

SciPrompt

ManualOpti.

SciPrompt

Wikidata
UMLS
ProteinKG25

3.67 / 11.2
1.15 / 3.81
0.06 / 0.28

3.21 / 10.75
4.91 / 12.71
8.41 / 18.81

3.50 / 11.17
5.85 / 13.93
8.43 / 24.60

11.97 / 25.92
3.44 / 8.88
0.58 / 2.15

10.09 / 24.76
8.01 / 19.04
6.10 / 20.26

10.92 / 26.50
9.37 / 20.90
8.11 / 21.99

Open vs. Close-World Setting
Candicate
Triples

Closed-world assumption (CWA) believes that the triples that do notappear in a given knowledge graph are wrong, which is essentiallyan approximation. Open-world assumption (OWA) assumes thatthe triples contained in the KG are not complete, which is moreaccurate and closer to the real scenario. The knowledge not in
KGs is not false, but unknown. However, it requires additional
human annotations to verify those triples carefully. In this paper,
we introduce open-world knowledge probing for PLMs. We ask
human experts to verify triples and tag them as correct, incorrect,
and unknown (triples that can not be verified through resourceson the Web). All annotated and verified data will be released forresearch purposes.

2.2

5Classification

Disease-type
Entities
2

Drug-type
Entities
3

Gene-type
Entities
4

Knowledge Sources

Wikidata Wikidata1 is a public Knowledge Base with many factual
knowledge across various domains.
UMLS The UMLS Metathesaurus2 is a biomedical knowledge graphthat contains various vocabularies and concepts in the biomedical
domain.
ProteinKG25 ProteinKG253 is a knowledge graph for protein science that contains descriptions and protein sequences (entity nodes).
We use a sub-set of the ProteinKG25 with the relation is a, is partof, and has part.

2.3

1

Probing Methods

Vanilla Prompt. We use a fill-in-the-blank cloze statement for
probing and adopt two baseline methods: manual prompts [14], and
OptiPrompt [22]. For each relation, we follow Sung et al.[18] to create manual prompts with domain experts. In addition, OptiPrompt
automatically obtains continuous embeddings, which are trainedwith disjoint instances.
SciPrompt. We additionally propose a new prompt-based method,
SciPrompt, for probing. Specifically, we utilize scientific terms toconstruct discrete prompt tokens. We also add some continuous
tokens, which make prompts using any vector in the embedding
space. Formally, our prompt has the following form:
𝑡𝑟 = [Term] 1 . . . [Term]𝑚 [ P] 1 . . . [P]𝑛 [MASK],(1)

where each [Term] is a discrete token from scientific terms related to relation 𝑟 , each [P]𝑖 ∈ R𝑑 is a dense vector with the same
1 https://wikidata.org/
2 https://www.nlm.nih.gov/research/umls/
3 https://zjunlp.github.io/project/ProteinKG25

Figure 2: Procedure of human experts verification.

dimension as the LM’s input embedding. We initialize the continuous embeddings with the embeddings of manual prompts, which
worked consistently better than random initialization. With thosescientific terms and learnable tokens, our SciPrompt can take advantage of domain knowledge as knowledge-informed prompts toelicit scientific factual knowledge. Following Sung et al.[18], we
use gradient descent to minimize the negative log-likelihood forprompt optimization.
From Table 1, we observe that SciPrompt can obtain better
performance, indicating the advantages of scientific knowledge
guidance. Notably, manual prompt also perform well in Wikidata,
possibly because Wikidata belongs to general domain. We leverage the best-performed model to generate candidate triples. For
Top-1 triples that do not exist in the KG under OWA, we leverageknowledge verification addressed in the following section.

2.4

Knowledge Verification

During knowledge verification, we randomly sample 100 Top-1
triples for each relation that do not exist in the KG under CWA,
resulting in a total number of 2.4K triples across three datasets.
Each triple is annotated by 5 human experts with a biomedical
background and proficiency in English as a second language. These
human experts are trained for the well-designed knowledge verification procedure.

Open-World Biomedical Knowledge Probing and VerificationIJCKG’23, Dec 08–09, 2023, Toykyo, Japan

Table 2: Main results in OWA setting. We report CR@1/PR@1 of each relation.

Relation ID

Relation Name

Subject

Object#Triples

CR@1

PR@1

medical condition treateddrug used for treatment
genetic association
therapeutic area
symptoms

chemicaldiseasegene
chemicaldiseasediseasechemicaldiseasediseasesymptom

704
435
830
344
303

20
34
11
14
44

21
40
13
14
49

clinically associated withmay treatcausative agent ofis finding of disease
biological process involves gene productcause ofgene mapped to disease
may be finding of disease
may be molecular abnormality of disease
may be prevented by
may be treated byphysiologic effect of
mechanism of action oftherapeutic class ofprocess involves genedisease has associated genediseasediseasediseasediseasegene
diseasediseasediseasediseasechemicalchemicalchemicalchemicalchemicalgene
genediseasechemicalvertebratebody substancefunction
diseasegene
symptom
genetic aberrant
diseasediseasediseasefunctiontype
diseasedisease

773
531
560
419
736
548
247
614
262
507
864
834
755
738
750
465

14
24
63
12
27
21
10
33
7
23
25
1
5
13
16
4

24
27
66
20
56
26
30
46
58
24
28
37
38
91
77
7

is a
is part ofhas part

gene function
gene function
gene function

gene function
gene function
gene function

19891
3374
301

7
12
31

96
94
99

Wikidata
P2175
P2176
P2293
P4044
P780
UMLS
UR116
UR124
UR173
UR180
UR211
UR214
UR221
UR254
UR256
UR44
UR45
UR48
UR49
UR50
UR588
UR625
ProteinKG25
P0
P1
P3

As shown in Figure 2, we ask these human experts to verify thosecandidate triples under OWA. Human experts are asked to search
the triple knowledge on Wikipedia to verify the correctness. For
those candidate triples not existing in Wikipedia, human experts
will search MalaCards4 (an integrated database of human maladies
and their annotations) for disease-related entities, Drugbank5 (acomprehensive, freely accessible, online database containing information on drugs and drug targets) for drug-related entities, and
GeneCards6 (a searchable, integrative database that provides comprehensive, user-friendly information on all annotated and predicted human genes) for gene-related entities. Human experts will
utilize Google to verify triples when the candidate triple does notexist in the database mentioned above. The triple is tagged withcorrect or wrong when there is a shred of solid evidence in the
text from the existing authoritative biomedical corpus. The triple
is tagged with unknown when there is no substantial evidence on
the Web and these human experts are also unable to confirm.
4 https://www.malacards.org/
5 https://go.drugbank.com/
6 https://www.genecards.org/

For each triple, we follow above annotation process and statistics
votes of 5 human experts, ultimately concluding with the final result
based on the majority consensus. The entire annotation procedure
is an exhaustive endeavor, spanning approximately one month intotal. To further assess the quality and agreement of annotations,
we calculate the average inter-rater agreement between annotators
using Fleiss’ Kappa scores [5], finding that annotations show perfectagreement (𝜅 = 0.9).

3 EXPERIMENTS
3.1 Settings
We follow Sung et al.[18] to reprocess all the datasets. For the openworld knowledge probing setting, we utilize CR@1 and PR@1 asevaluation metrics, which indicates absolute correct and partial
correct (including absolute correct and those unknown triples),respectively. We argue that those unknown triples (triples thatcannot be verified based on the present information on the Web) are
valuable, and possibly part of them can be proved experimentallyin the future.

IJCKG’23, Dec 08–09, 2023, Toykyo, JapanXi Tian, et al.

（Glyxambi，therapeutic area，[mask]）(amphotericin B liposomal ,therapeutic_class_of ,[mask])
Antimycotic
Agentssynonym

antifungal
agentsDiabetes

14%

subtype ofmaturity-onset
diabetes of theyoung type 2

34%

14%
(Cloxacillin sodium ,may_be_prevented_by ,[mask])
Staphylococcus aureus

subtype of(Ascariasis , may_treat ,[mask])

38%

gram-positivebacterial infections

albendazole

independent

ivermectin,
and so onGolden is a subtype of prediction

Golden is incomplete

prediction

Prediction is a subtype of golden

Prediction is synonym of golden

goldenFigure 3: Case study for triples in OWA setting. We analyze those triples tagged correctly by human experts and divide theminto four groups.

3.2

Results

Is this knowledge from PLMs correct? From Table 2, we noticethat part of the knowledge from all datasets is actually correctwhen verified by human experts, which indicates that we mayunderestimate the rich knowledge from the PLMs. Note that the
triples with causative agent of even obtain 63 CR@1 and 66 PR@1
scores, illustrating that PLM has the potential ability to discover“new” scientific knowledge.
What are the characteristics of this knowledge? To further investigate the intrinsic characteristic of this knowledge in the OWA
setting, as shown in Figure 3, we conduct a manual analysis of theseannotated correct triples and categorize them into four groups
as follows :Group 1: The predicted tail entity is independent of thegolden entity. We observe that 38% of the tail entities in predicted
triples have little relevance to the golden entities but are actuallycorrect by human expert verification based on public information.
We argue that this may be because, during pre-training, the PLMs
have seen lots of patterns, leading to knowledge recalling.
Group 2: The predicted tail entity is a subtype of golden
entities. We observe that 14% of the predicted entities are subtypesof golden entities, indicating that PLMs have the powerful ability
to capture useful lexical-type knowledge, which is consistent with
[6].
Group 3: The golden entity is a subtype of predicted tailentities. We observe that 34% of the golden entities are subtypesof predicted entities, demonstrating that PLMs can capture lots ofhierarchy knowledge (e.g., a subtype of).
Group 4: The golden entity is a synonym of the golden
entity. We observe that 14% of the predicted entities are synonymsof golden entity, which illustrates that PLMs have the powerfulability to memorize synonyms.

Overall, we notice that this knowledge with open-world knowledge probing from PLMs has part of intrinsic patterns but also
contains unknown emerging abilities of knowledge discovery[21]. We think it is interesting to study the potential ability of largescale PLMs via open-world knowledge probing, as it promises to
uncover even more hidden knowledge.

4

DISCUSSION

Recently, PLMs have been suggested as a possible complement
to KGs. However, previous studies focus on probing the existingknowledge (CWA Setting) rather than exploring the “new” (OWA
Setting). We think this approach is complementary to the existingevaluation system (e.g., LAMA, BioLAMA) and can serve as a scaffold for investigating open knowledge in PLMs. We hope that byuncovering the potential ability of PLMs with open-world knowledge probing, we can continue to motivate exploring the positivesof pre-training intrinsically and apply the technology for automaticscience knowledge discovery.

5

CONCLUSION AND FUTURE WORKIn this paper, we propose open-world knowledge probing for biomedical PLMs. We provide an empirical analysis of three datasets andfind that the previous evaluation setting may underestimate theknowledge from the PLMs. We further propose SciPrompt, which
integrates scientific knowledge into the prompt, obtaining better
performance for biomedical knowledge probing. We hope that ourwork can contribute to a deeper understanding of the knowledge
learned from PLMs and inspire further research for scientific knowledge discovery. In the future, we plan to study the evaluation strategies for open-world knowledge probing via fact verification [19]and systematically re-evaluate the existing PLMs to reveal theirreal performance.

Open-World Biomedical Knowledge Probing and Verification

REFERENCES[1] Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. A Review on Language Models as Knowledge
Bases. CoRR abs/2204.06031 (2022). https://doi.org/10.48550/arXiv.2204.06031
arXiv:2204.06031[2] Diego Calvanese, Davide Lanti, Tarcisio Mendes De Farias, Alessandro Mosca,
and Guohui Xiao. 2021. Accessing scientific data through knowledge graphswith Ontop. Patterns 2, 10 (2021).
[3] Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong
Xue, and Jin Xu. 2021. Knowledgeable or Educated Guess? Revisiting Language
Models as Knowledge Bases. In Proceedings of the 59th Annual Meeting of the
Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, Chengqing Zong, Fei Xia, Wenjie Li, andRoberto Navigli (Eds.). Association for Computational Linguistics, 1860–1874.
https://doi.org/10.18653/v1/2021.acl-long.146[4] İsmail İlkan Ceylan, Adnan Darwiche, and Guy Van den Broeck. 2016. OpenWorld Probabilistic Databases. In Principles of Knowledge Representation andReasoning: Proceedings of the Fifteenth International Conference, KR 2016, CapeTown, South Africa, April 25-29, 2016, Chitta Baral, James P. Delgrande, and Frank
Wolter (Eds.). AAAI Press, 339–348. http://www.aaai.org/ocs/index.php/KR/
KR16/paper/view/12908[5] Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters.
Psychological bulletin 76, 5 (1971), 378. https://psycnet.apa.org/record/197205083-001[6] Michael Hanna and David Marecek. 2021. Analyzing BERT’s Knowledge ofHypernymy via Prompting. In Proceedings of the Fourth BlackboxNLP Workshop on
Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2021,
Punta Cana, Dominican Republic, November 11, 2021, Jasmijn Bastings, Yonatan
Belinkov, Emmanuel Dupoux, Mario Giulianelli, Dieuwke Hupkes, Yuval Pinter,and Hassan Sajjad (Eds.). Association for Computational Linguistics, 275–282.
https://aclanthology.org/2021.blackboxnlp-1.20[7] Benjamin Heinzerling and Kentaro Inui. 2020. Language Models as Knowledge
Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries.
CoRR abs/2008.09036 (2020). arXiv:2008.09036 https://arxiv.org/abs/2008.09036[8] Benjamin Heinzerling and Kentaro Inui. 2021. Language Models as Knowledge
Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries.
In Proceedings of the 16th Conference of the European Chapter of the Association
for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23,
2021, Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for
Computational Linguistics, 1772–1791. https://doi.org/10.18653/v1/2021.eaclmain.153[9] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim,
Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinform. 36, 4 (2020),
1234–1240. https://doi.org/10.1093/bioinformatics/btz682[10] Patrick Lewis, Myle Ott, Jingfei Du, and Veselin Stoyanov. 2020. Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending
the State-of-the-Art. In Proceedings of the 3rd Clinical Natural Language Processing Workshop. Association for Computational Linguistics, Online, 146–157.
https://doi.org/10.18653/v1/2020.clinicalnlp-1.17[11] Peng Liu, Lemei Zhang, and Jon Atle Gulla. 2023. Pre-train, Prompt and Recommendation: A Comprehensive Survey of Language Modelling Paradigm Adaptations in Recommender Systems. arXiv e-prints, Article arXiv:2302.03735(Feb. 2023), arXiv:2302.03735 pages. https://doi.org/10.48550/arXiv.2302.03735
arXiv:2302.03735 [cs.IR][12] Xin Lv, Yankai Lin, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie
Zhou. 2022. Do Pre-trained Models Benefit Knowledge Graph Completion? A
Reliable Evaluation and a Reasonable Approach. In Findings of the Association for
Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, Smaranda
Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, 3570–3581. https://aclanthology.org/2022.findings-acl.282[13] Zaiqiao Meng, Fangyu Liu, Ehsan Shareghi, Yixuan Su, Charlotte Collins, andNigel Collier. 2022. Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models. In Proceedings of the 60th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
ACL 2022, Dublin, Ireland, May 22-27, 2022, Smaranda Muresan, Preslav Nakov,
and Aline Villavicencio (Eds.). Association for Computational Linguistics, 4798–
4810. https://aclanthology.org/2022.acl-long.329[14] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton
Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language Models as
Knowledge Bases?. In Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on Natural
Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019,
Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for
Computational Linguistics, 2463–2473. https://doi.org/10.18653/v1/D19-1250

IJCKG’23, Dec 08–09, 2023, Toykyo, Japan[15] Raymond Reiter. 1977. On Closed World Data Bases. In Logic and Data Bases,
Symposium on Logic and Data Bases, Centre d’études et de recherches de Toulouse,
France, 1977 (Advances in Data Base Theory), Hervé Gallaire and Jack Minker(Eds.). Plemum Press, New York, 55–76. https://doi.org/10.1007/978-1-4684-33845_3[16] Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How Much Knowledge
Can You Pack Into the Parameters of a Language Model?. In Proceedings of the2020 Conference on Empirical Methods in Natural Language Processing, EMNLP
2020, Online, November 16-20, 2020, Bonnie Webber, Trevor Cohn, Yulan He, andYang Liu (Eds.). Association for Computational Linguistics, 5418–5426. https://doi.org/10.18653/v1/2020.emnlp-main.437[17] Baoxu Shi and Tim Weninger. 2018. Open-World Knowledge Graph Completion.
In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence,(AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18),
and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence(EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, Sheila A. McIlraith
and Kilian Q. Weinberger (Eds.). AAAI Press, 1957–1964. https://www.aaai.org/
ocs/index.php/AAAI/AAAI18/paper/view/16055[18] Mujeen Sung, Jinhyuk Lee, Sean S. Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang. 2021. Can Language Models be Biomedical Knowledge Bases?. In
Proceedings of the 2021 Conference on Empirical Methods in Natural Language
Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11
November, 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, andScott Wen-tau Yih (Eds.). Association for Computational Linguistics, 4723–4734.
https://doi.org/10.18653/v1/2021.emnlp-main.388[19] David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh
Hajishirzi. 2022. MultiVerS: Improving scientific claim verification with weaksupervision and full-document context. In Findings of the Association for Computational Linguistics: NAACL 2022. Association for Computational Linguistics,
Seattle, United States, 61–76. https://doi.org/10.18653/v1/2022.findings-naacl.6[20] Cunxiang Wang, Pai Liu, and Yue Zhang. 2021. Can Generative Pre-trained
Language Models Serve As Knowledge Bases for Closed-book QA?. In Proceedingsof the 59th Annual Meeting of the Association for Computational Linguistics and the11th International Joint Conference on Natural Language Processing, ACL/IJCNLP
2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, Chengqing Zong,
Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational
Linguistics, 3241–3251. https://doi.org/10.18653/v1/2021.acl-long.251[21] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian
Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H.
Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William
Fedus. 2022. Emergent Abilities of Large Language Models. CoRR abs/2206.07682(2022). https://doi.org/10.48550/arXiv.2206.07682 arXiv:2206.07682[22] Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual Probing Is[MASK]: Learning vs. Learning to Recall. In Proceedings of the 2021 Conferenceof the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021,
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür,Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao
Zhou (Eds.). Association for Computational Linguistics, 5017–5033. https://doi.org/10.18653/v1/2021.naacl-main.398