Title: Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
Benchmark: CURRICULUM
Summary: Current models do not provide insight into how well a language model captures distinct linguistic skills essential for language understanding and reasoning. In this paper, authors introduce CURRICULUM as a new format of NLI benchmark for evaluation of broad-coverage linguistic phenomena. CURRICULUM contains a collection of datasets that covers 36 types of major linguistic phenomena and an evaluation procedure for diagnosing how well a language model captures reasoning skills for distinct types of linguistic phenomena. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Linguistic phenomena in NLU
Phenomenon defined by authors: Yes
Phenomenon definition: For this phenomenon, authors try to measure how well a language model captures distinct linguistic skills essential to language understanding and reasoning.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Natural language inference (NLI). More specifically, authors provide a group of tasks motivated by three benchmarks: GLUE Diagnostic, Rainbow, and DNC. In addition, we include many more subtasks focusing on complex reasoning types such as deductive logic and analytical thinking.
Task item definition: Each single item has a premise, a hypothesis, and a target label.
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 171,252
Dataset size notes: 540,437
Dataset metadata available: Yes
Dataset metadata detail: difficulty level
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), Correlation (Matthew's correlation, Pearson's r)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: sub-phenomenon, difficulty 
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding