Title: MERA: A Comprehensive LLM Evaluation in Russian
Benchmark: MERA
Summary: The paper proposes a new instruction benchmark, MERA, for measuring LM capabilities in Russian. MERA comprises 21 tasks that have been (i) specifically created for MERA, (ii) translated from English tasks, or (iii) taken from existing Russian resources. The authors tried to culturally adapt MERA to the Russian context, for example by replacing historical concepts in translated English tasks with Russian ones. The paper also provides model baselines (evaluation of 19 LMs) as well as a human baseline for MERA.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications), MERA as a whole tries to measure general "capabilities", but the individual tasks evaluate more specific applications (e.g., question answering).
Target phenomenon: The authors say that MERA covers ten "skills": mathematics, logic, reasoning, common sense, natural language inference, world knowledge, dialogue system, algorithms, computer code, and ethics. Specifically, they assign each of the 21 tasks in MERA to one or several (up to three) of these skills. 
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Task definition: The 21 tasks are split into three categories: "problem-solving tasks" that are defined to constitute "general intelligence evaluation tasks," "exam-based tasks" that "require expertise for solution," and "diagnostic (ethics) tasks," aimed to "identify models' ethical biases, including toxicity harms" (p. 9922). The individual tasks are mapped to one of these three categories and defined in a mostly procedural way (i.e., by describing how the data were put together).
Task item definition: Each task contains items of different types.
Task source: Human exam questions (e.g. GRE questions), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task source detail: Task source varies for the 21 tasks.
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Task ecology varies for the 21 tasks.
Dataset size: Between 164 and 6000 examples per task.
Dataset size detail: All tasks have a test set. Most tasks, except for three in the "Ethics" category, have a train set. Only four out of 21 tasks have a validation set.
Dataset size notes: Train: between 0 and 29,376 examples per task. Validation: between 0 and 900 examples per task.
Dataset metadata available: Yes
Dataset metadata detail: For some tasks.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Response format detail: Response format varies for the 21 tasks.
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Correlation (Matthew's correlation, Pearson's r), (school) grade
Metric definition detail: Metric varies for the 21 tasks.
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: The test sets of the benchmark are private, but the train and validation sets are available, and the leaderboard is being actively updated.
Task face validity: Tricky to say since the paper does not provide a principled definition of the target phenomenon. It just talks of general "capabilities," as well as the ten skills mentioned above. As for the ten skills, face validity varies -- for some (e.g., mathematics) it seems higher than for others (e.g., ethics).
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: Given that the benchmark is trying to measure general capabilities, it is unclear how a more realistic setting would look like.
Human baseline: Yes
Authors discuss validity: No
Taxonomy: NLP / 
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Multilinguality