Title: MELA: Multilingual Evaluation of Linguistic Acceptability
Benchmark: MELA
Summary: The paper intorduces a multilingual acceptability judgement benchmark covering a diverse set of 10 languages, all annotated by expert linguists.  The acceptability judgment task tests a language model’s ability to distinguish syntactically acceptable sentences from unacceptable ones in a human language. The paper establishes LLM baselines on this benchmark, and investigates cross-lingual transfer in acceptability judgements with XLM-R.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Linguistic Acceptability
Phenomenon defined by authors: Yes
Phenomenon definition: The acceptability judgment task tests a language model’s ability to distinguish syntactically acceptable sentences from unacceptable ones.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: The acceptability judgment task tests a language model’s ability to distinguish syntactically acceptable sentences from unacceptable ones.
Task item definition: a sentence
Task source: hand-written by linguists in respective languages, taken from textbooks, handbooks and journal articles in theoretical syntax + some examples taken from previous benchmarks 
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 46k
Dataset size notes: train set: 33'293, validation:3'970
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall), Matthews Correlation Coefficient (MCC, Matthews), which is a measure of similarity between binary distributions taking values from -1 to 1 and always yielding 0 for any two uncorrelated distributions, regardless of class imbalance.
Metric aggregation: Simple Mean
Metric statistics: simple mean and standard deviation 
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Multilinguality / 
New taxonomy: General Capability / Multilinguality