Title: RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
Benchmark: RussianGLUE
Summary: In this paper, we introduce an advanced Russian general language understanding evaluation benchmark – RussianGLUE. This benchmark consists of nine tasks, collected and organised analogically to the SuperGLUE methodology (Wang et al., 2019),  it was developed from scratch for the Russian language. We provide baselines, human level evaluation, an open- source framework for evaluating models and an overall leaderboard of transformer models for the Russian language.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: natural language understanding
Phenomenon defined by authors: Yes
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The RussianSuperGLUE benchmark evaluates LM on a set of nine diverse natural language understanding tasks in Russian. These include diagnostics, commonsense reasoning, natural language inference, machine reading comprehension, and world knowledge.
.
Task item definition: A single item in the dataset consists of a natural language input (e.g. a sentence, paragraph, or question) and a corresponding label or output (e.g. classification label, entailment judgment, or text). The exact format varies by task.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: total test 22,119
Dataset size notes: Total (some tasks have none): 97,090 and 14,104
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), exact match, MCC (Matthews Correlation Coefficient)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: it is not in the paper, but available online
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: NLP / Understanding
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Natural Language Understanding