Title: IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models

Benchmark:  IELM
Summary: They introduce a new open information extraction (OIE) benchmark designed to evaluate the relational knowledge stored in pre-trained language models (LMs) such as BERT and GPT (published in 2022). Their method involves transforming these pre-trained LMs into zero-shot OIE systems to assess their performance on both existing and novel factual OIE datasets. Their results show that pre-trained LMs achieve competitive performance, even surpassing state-of-the-art supervised OIE methods on certain datasets without any additional training data.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: open information extraction i.e. answering “fill-in-the-blank” questions when given
a pre-defined relation category
Phenomenon defined by authors: Yes
Phenomenon definition: "In this work, we set up a new open information extraction (OIE) benchmark, called IELM, towards testing the general and open relational information stored in pre-trained LMs."
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Purpose notes: For definition_integrity - the paper looks at both standard OIE and factual OIE.
Task definition: "In this work, we set up a new open information extraction (OIE) benchmark, called IELM, towards testing the general and open relational information stored in pre-trained LMs. We refer to OIE as it is a task that is designed to extract open relations from massive corpora without requiring a pre-defined relation category."
Task item definition: "For open information extraction (OIE), we take an input as a NP-chunked sentence and output a set of triples. Below is an example.
Input DylanNP was born in MinnesotaNP, and was awarded Nobel PrizeNP.
Output (Dylan; born in; Minnesota), (Dylan; awarded; Nobel Prize).
NP denotes the noun phrase."
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Based on knowledge graphs (KG) e.g. Wikidata
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 27,400,440 triples 6,096,709 arguments 5,418 predicates 9,925,937 documents
Dataset size notes: The dataset size above is summed over 4 datasets in Table 2.
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Response format detail: Output is a set of triples
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric statistics: The authors carry out some error analysis: "We argue that we are measuring a lower bound for what LMs know. To further understand the shortcomings of the current method, we conduct an error analysis of the errors in precision on all datasets. We choose BERTLARGE for the study. We sample 100 documents from the Wikidata-OIE dataset, and manually check the reasons for the errors"
Metric subscores: Yes
Metric subscores detail: Metrics are reported for each OIE dataset (CaRB(existing), Re-OIE206 (existing), TAC KBP-OIE (novel), Wikidata-OIE (novel)).
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: No, link is broken
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: They carry out an error analysis:
"We argue that we are measuring a lower bound for what LMs know. To further understand the shortcomings of the current method, we conduct an error analysis of the errors in precision on all datasets. We choose BERTLARGE for the study. We sample 100 documents from the Wikidata-OIE dataset, and manually check the reasons for the errors."

They find error from: incorrect arguments, missing pairs in predicate mapping, correct triples that are not covered by Wikidata, and incorrect predicate phrases.
Taxonomy: NLP / Extraction
New taxonomy: General Application / Extraction