Title: CoDA21: Evaluating Language Understanding Capabilities of NLP Models With Context-Definition Alignment
Benchmark: CoDA21
Summary: CoDA21 is a challenging benchmark to assess NLU capabilities of pretrained language models (PLMs). Performance of PLMs is assessed versus humans.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: natural language understanding
Phenomenon defined by authors: No
Phenomenon definition: N/A -- not explicitly defined
Definition scope: Comprehensive
Definition integrity: Authors' description is unclear
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a set of contexts with masked target words and a set of definitions corresponding to these masked words, the task is to find the correct alignment between contexts and definitions. 
Task item definition: Each sample consists of words and associated definitions.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset metadata available: Yes
Dataset metadata detail: statistics for groups of related words
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), cosine similarity, log generation probability
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: difficulty, clean vs. noisy
Metric access: Model access required (e.g. logits)
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding