Title: Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction
Benchmark: ROBUST
Summary: The paper introduces ROBUST, a benchmark designed to evaluate open information extraction models by measuring their ability to generalize knowledge extraction across syntactically diverse sentences that share the same semantic content.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: the generalization of open information extraction
Phenomenon defined by authors: Yes
Phenomenon definition: [...] each example is a knowledge-invariant clique that consists of sentences with structured
knowledge of the same meaning but with different syntactic and expressive forms. [...] a model is judged to be robust if its performance is consistently accurate on the overall cliques.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Open Information Extraction (OpenIE) aims to extract n-ary knowledge tuples {(a1,p,a2,...,an)} consisting of n arguments and one predicate from the natural text.
Task item definition: Sentences with arguments and one predicate form a set (clique), where sentences are semantically invariant.
Task definition detail: The base task is OpenIE. Each tuple of sentence+arguments+predicate within a clique is analyzed. The "meta-task" is doing well on the worst tuple within one clique.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 1272 cliques, 4971 sentences
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Response format detail: n-tuples of text are extracted from the resonse.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: For each tuple, the F1 is computed, then across a clique the minimum is computed and aggregated across the dataset as mean.
Metric subscores: No
Metric metascoring: minimum
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Extraction
New taxonomy: General Application / Extraction
Procedural notes: I agree that studying the minimum score achieved by a set of semantically equivalent items captures a notion of robustness. However, the authors often mention "distribtuion shift". Unfortunately, it is not clear what the training distribution is and what the test distribution is in this work, and subsequently it is not clear how the distribution shifts between these two. In my humble opinion, "distributional shift" is a misnomer, they just "enrich the existing data generating process", not change it.