Title: RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs
Benchmark: RuBLiMP
Summary: Minimal pairs are a well-established approach
to evaluating the grammatical knowledge of language
models. This paper introduces
the Russian Benchmark of Linguistic
Minimal Pairs (RuBLiMP), which includes
45k pairs of sentences that differ in grammaticality
and isolate a morphological, syntactic,
or semantic phenomenon. In contrast to existing
benchmarks of linguistic minimal pairs,
RuBLiMP is created by applying linguistic
perturbations to automatically annotated sentences
from open text corpora and decontaminating
test data. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: grammatical knowledge, specifically across morphological, syntactic, and semantic phenomena in the Russian language. 
Phenomenon defined by authors: Yes
Phenomenon definition: This paper introduces the Russian Benchmark of Linguistic Minimal Pairs (RuBLiMP), which includes 45k pairs of sentences that differ in grammaticality and isolate a morphological, syntactic, or semantic phenomenon. Our benchmark covers morphological, syntactic, and semantic phenomena well-represented in Russian theoretical linguistics.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The task is defined as a forced-choice acceptability judgment between two sentences in a minimal pair, where the model must assign a higher probability to the grammatical sentence over the ungrammatical one. 
Task item definition: A pair of sentences with one being grammatically correct and the other one is incorrect, with the respective label
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The task is a constructed benchmark using linguistic minimal pairs to test grammatical knowledge in LMs. This setup is a representative proxy for evaluating capabilities that are critical in applications like machine translation, dialogue systems, and text generation.
Dataset size: 45k
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean
Metric statistics: simple mean,  inter-annotator agreement with WAWA and the Dawid-Skene method for vote aggregation.  delta-scores to measure performance differences between models under different dataset filtering conditions
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: NLP / 
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Natural Language Understanding