Title: LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low Resource and Extinct Languages
Benchmark: LINGOLY
Summary: The paper introduces LINGOLY, a new benchmark built on Linguistics Olympiad puzzles in low-resource and extinct languages to test genuine reasoning capabilities in LLMs. The benchmark is crafted covering diverse reasoning complexity, linguistic subject areas, instruction types, and high/low resources. The paper uncovers error pattenrs between high and low resource settings and show the ongoing challenges in multi-step, out-of-domain reasoning.
Contribution: The important contribution is to define the reasoning tasks with necessity and sufficiency: the task cannot be done without reasoning and can be done via reasoning. For the fair evaluation, the paper propose to use low-resource languages to learn the linguistic and grammatical patterns (necessity) that are rare online (sufficiency). The error patterns shows that the LLMs still struggle with the complex (multi-step, ood) reasoning tasks.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multi-step, out-of-domain linguistic reasoning, low-resource languages, 
Phenomenon defined by authors: Yes
Phenomenon definition: We argue that a benchmark task measures reasoning if the task 1) cannot be
done without reasoning (necessity) and 2) can be done via reasoning (sufficiency). However, the combination of these features is difficult to achieve in practice since memorisation and contamination may reduce the necessity of reasoning, and in tasks which draw on background knowledge, as in most ‘commonsense’ benchmarks[7], reasoning itself is insufficient to complete the task.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Purpose notes: No-context baseline -- evaluate if the model performance drops when the context is removed. This concept is to assess the performance if the model has relied on memorization or reasoning from the linguistic clues in the context.
Task definition: The task is to understand genuine reasoning capabilities of LLMs by providing context of low-resource linguistic information and questions to solve based on the given context (or without context to penalize the memorized knowledge). The expected output is a concise textual answer that can be matched up with ground-truth labels.
Task item definition: Below is a problem sheet…
{PREAMBLE}
{CONTEXT}
{QUESTIONS}
{SUBQUESTIONS}
Now respond to the following…
{REPEAT 1 QUESTION}
Format your response as…
{FORMAT TEMPLATE}
Task definition detail: Compare the model performance with and without contextual information to penalize the memorized knowledge and evaluate the genuine reasoning abilities of LLMs using the linguistic cues from the given knowledge.
Task source: Human exam questions (e.g. GRE questions), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: The task from LINGOLY is adapted from official Linguistics Olympiads puzzle sets rather than everyday language usage scenarios or standard benchmarking corpora. 
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: While the benchmark comes from authentic Linguistic Olympiad puzzles, they are still competition-style questions rather than real world usage scenarios. Hence it can be categorized as representative task of a speciflized exam setting.
Dataset size notes:  1,133 questions all for testing.
Dataset metadata available: Yes
Dataset metadata detail: human difficulty, linguistic subjects, task format, language
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Structured response (e.g. valid JSON, API call alone)
Response format detail: Free response is existed but excluded from evaluation (The only case where an instance has a missing answer is when the intended answer was a free response, e.g., “explain your reasoning”. These questions are included in the dataset but removed from the scoring as they are not compatible with being machine-scored.)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: The metric report Exact Match with a standard script for LLMs to output JSON in a single pass. This is different from employing "LLM post-processing" in the sense of an additional LLM-based step to reformat and judge the responses.

They exclude all questions where the answer is “fuzzy” (i.e., accepts synonyms or free
text response) because they cannot automate the evaluation of synonym similarity across languages.
Metric aggregation: Simple Mean
Metric statistics: The authors use a weighted mean in calculating an approximate human performance threshold but not for model performance. They take a weighted average of the annual medal thresholds for ‘Advanced’ problems. 
Metric subscores: Yes
Metric subscores detail: Human difficulty, puzzle format, linguistic subject, language resourcedness
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Across models, performance is consistently higher on problems with easier human difficulty and higher resource languages than those of harder difficulty and lower-resource languages.
(LLMs tested have limited reasoning abilities about low-resource languages and do not achieve the multi-step reasoning required in the harder questions, in addition to errors of following instructions alongside core reasoning tasks.)
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning
Other notes: One critical point is whether language models provide poor performances due to the unfamiliar format or out-of-domain reasoning -- the mismatch between the puzzle's presentation style and the distribution of model instruction templates may cause certain reasoning failures depending on model types.
It would be nice to see how benchmarks have certain patterns with model types.