Title: SLING: Sino Linguistic Evaluation of Large Language Models
Benchmark: SLING
Summary: The SLING benchmark is introduced to evaluate the linguistic knowledge of pretrained Chinese language models, featuring 38,000 minimal sentence pairs in Mandarin Chinese that highlight syntactic and semantic phenomena. These sentences are naturally-occuring and annotated, from the Chinese Treebank 9.0. Evaluating 18 LMs, the study found that their average accuracy is significantly lower than human performance (69.7% vs. 97.1%), with BERT-base-zh achieving the highest accuracy at 84.8%.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon:  Chinese language (Mandarin)
Phenomenon definition: "To understand what kinds of linguistic knowledge are encoded by pretrained Chinese language models (LMs)"
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The tasks consist of short sentence pairs in Mandarin Chinese, classified into nine major linguistic categories. Each pair highlights the difference in acceptability for a particular syntactic or semantic phenomenon (e.g., "The keys are lost" vs. "The keys is lost").
Task item definition: short sentence pairs in Mandarin Chinese
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 38,000
Dataset metadata available: Yes
Dataset metadata detail: Linguistic Phenomena
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Choice of one input sentence
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Linguistic phenomena
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Comprehensiveness: "there are still phenomena that are important but not included in the current work: for example, the ba and bei constructions. For those structures, unacceptability can have different sources (e.g., syntax or pragmatics).19 Simple syntactic structure restrictions are not enough. When deciding which phenomena to include in SLING, we deliberately avoid such cases because the (un)acceptability of these phenomena can be mitigated by contextual or world knowledge. As a result, human judgement can vary significantly"
Taxonomy: Multilinguality / 
New taxonomy: General Capability / Multilinguality