Title: COPEN: Probing Conceptual Knowledge in Pre-trained Language Models
Benchmark: COPEN
Summary: The paper introduces COPEN, a benchmark designed to probe conceptual knowledge in pre-trained language models (PLMs). It includes three tasks evaluating whether PLMs can group entities by concepts, understand concept properties, and identify concepts in context. Results show that PLMs struggle with conceptual reasoning and often rely on spurious correlations.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: conceptual knowledge
Phenomenon defined by authors: Yes
Phenomenon definition: "implicit commonsense behind texts"
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Assessing whether PLMs can judge conceptual similarity, recognize conceptual properties, and conceptualize entities based on context.
Task item definition: A single item represents one probe instance for a specific conceptual task. E.g
In CPJ, an item includes a statement about a property and a concept or concept chain, along with a true/false label.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 11,035
Dataset size notes: Train 10,624, Validation: 2,661
Dataset metadata available: Yes
Dataset metadata detail: Task types: conceptual similarity, recognize conceptual properties, and conceptualize entities based on context.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Task Types
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors explicitly connect each probing task to specific cognitive functions and conceptual structures - grounding their design in existing literature.
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding