LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation
Authors: Jude Khouja
Lingyi Yang
Karolina Korgul
Simeon Hellsten
Vlad A. Neacșu
Harry Mayne
Ryan Othniel Kearns
Andrew M. Bean
Adam Mahdi
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Reasoning
Benchmark
Linguistic reasoning
Permutation
Primary area: foundation or frontier models, including LLMs
TL;DR: An inductive reasoning benchmark about natural languages designed to minimise the ability to solve with knowledge or memory

Abstract: Frontier language models demonstrate increasing ability at solving reasoning
problems, but their performance is often inflated by circumventing reasoning and
instead relying on their expanding knowledge and memorisation capacity. We
introduce LINGOLY-TOO, a challenging reasoning benchmark of 1,203 questions
and a total of 6,995 sub-questions that counters these shortcuts by applying expert-
designed obfuscations to Linguistics Olympiad problems. These obfuscations
preserve the underlying solution logic while reducing the likelihood problems
are solvable with via knowledge or memorisation. Our experiments show that
models exploit shortcuts on the original question as performance markedly drop
upon obfuscation. Even the best reasoning models remain highly sensitive, with
scores dropping from around 0.59 on original problems to 0.48 after obfuscation.
LINGOLY-TOO disentangles reasoning from knowledge, offering a clearer measure
of true reasoning capabilities.

Published as a conference paper at ICLR 2026
LINGOLY-TOO: G REASONING FROM KNOWLEDGE WITH TEMPLATISED ORTHOGRAPHIC OB- FUSCATION
Jude Khouja Lingyi Yang-.21, Karolina Korgul-.1, Simi Hellsten 3.411 Vlad Neacşu 5.6
Harry Mayne', Ryan Othniel Kearns', Andrew Bean Adam Mahdi
University of Oxford, Univeesitty of Nottingham, UUniveritty of Glasgow
United Kingdom Linguistics Olympiad, $ "Asia-Pacifie Linguistics Olympiad,
'orgu lordan Al. Rosetti' Institute of Linguistics of the Romanian Academy
oxrml. wtt.e/oooooiggotoooooooo
jkttouja/LingDly-T00
jxhoujaa/lnngoly-oo
ABSTRACT
Frontier language models demonstrate increasing ability at solving reasoning problems, but their performance often inflated by cireumventing reasoning and instead relying on their expanding knowledge and memorisation capacity. We introduce LINGOLY-TOO, a challenging reasoning benchmark of 1,203 questions and a total of 6,995 sub-questions that counters these shorteuts by applying expert- Linguistics Olympiad problems. These obfuscations
designed obfuscations preserve the underlying solution logic while reducing the likelihood problems are solvable with via knowledge or memorisation. Our experiments show that models exploit shortcuts the original question as performance markedly drop upon obfuscation. Even the best reasoning models remain highly sensitive, with scores dropping from around 0.59 original problems to 0.48 after obfuscation. LINGOLY- TOO disentangles reasoning from knowledge, offering a clearer measure of true reasoning capabilities.
1.0
Clouds 3.7 Sonmet (poowin))
Map
0.9
Dasp T-Tmmiuus
Cloude 3 Sonnet the thiitigg
Mai (ours)
0.8
Clsuce Opus 1
0.7
Gemmi 2.5 Pro
GPT5
G aude 5.5 Sormet
preeeeeww
68 enin
Cambei 55 Pro
Lieme 9.5 705- trstrct
GPT 4.5
0.3
GPT40
oS-min
0.2
Phd
0.1
Figure 1: Reasoning performance LINGOLY-TOO. Results without controlling for knowledge and memorisation abilities overestimate reasoning abilities (light blue). Obfuscation mitigates this effect and offers improved estimates (dark blue). There are two scores per model: Mog is based on the original problems and Mody is based on obfuscated problems.
INTRODUCTION
Experts often define reasoning the ability to apply abstract rules derive novel judgments (Koralus
2022
Huang & Chang 2023 Lampinen et al, 2024). This ability includes inductive, spatial, causal,
'qqual first author contribution
"Equal last author contribution
Correspondence khoujaaioooj@oii.oooo ax..ukk..... <adam. maadimmmaaiiee occ.ox.ac.uk>

---

Published as a conference paper at ICLR 2026
and commonsense reasoning skills (Kazemi al. 2025). Benchmarks aim measure reasoning by testing models on unseen tasks. For such measurements to be valid, reasoning must be both a necessary and sufficient condition for success. However, as training sets grow in size and models' memorisation capacity increases, distinctions between train and test sets, and in- and out-of-domain tasks are blurred (Raji et al. 202T), leading to bias in benchmark estimates.
We distinguish two capabilities resulting from training LLMs that help models solve linguistic tasks. We refer knowledge as information stored in model parameters which captures linguistic, factual, and commonsense patterns useful for downstream tasks. Stored knowledge, while generally useful, can confound the measurement of symbolic reasoning capabilities. For example, GPT-5 can already translate from Welsh to English, hence a linguistic reasoning task based on deriving rules from paired Welsh-Fnglish translations would be trivial. We refer to memorisation as when models exploit prior exposure to evaluation datasets and report answers based on previously seen problems in training (Li & Flanigan 2024 Zhou al. 2023).
Consequently, and amidst the rapid saturation in reasoning benchmarks (Kazemi et al. 2025 Phan al. 2025), novel methods for generating challenging, unbiased benchmarks are paramount for measuring true reasoning abilities. To prevent models from bypassing reasoning, test cases should minimise the chance that problems are answerable using knowledge or memorisation. Consider the following scenario: user asks large language model (LLM) to solve language puzzle, but is written in an unfamiliar script or orthography|] In this situation, the model's knowledge of the language is rendered useless. The model must deduce grammatical rules and patterns from context, then apply them to solve the given puzzle, Prior work has explored using synthetic test cases (Saparov & He, 2023) evaluate reasoning. Similar interventions have also been applied to logic puzzles and mathematics, e.g. using symbolic templates to permute variable values (Mirzadeh et al.) 2024). However, these scall-scale perturbations are less effective because questions remain similar to examples the model may have previously encountered. By contrast, permuting the orthography of a language results in test cases with minimal chance of appearing training corpus, while the underlying solution logic remains intact.
Our benchmark LINGŌLY-TOO operationalises this idea. Original problems are taken from the UK Linguistics Olympiad (UKLO (United Kingdom Linguistics Olympiad 2023)), which can be solved by high-school students without specific linguistic or domain knowledge. The problems undergo obfuscation: linguistically ennforre grapheme-level permutations that preserve the underlying solu- tion logic, remove clues a model could match to prior knowledge, and provide thousands distinct, solvable variants. Our benchmark aims to test whether model can symbolic inductive reasoning to infer the abstract rules and patterns needed to answer the questions from context.
Our evaluation results show that models often rely language knowledge rather than reasoning, inflating apparent ability, especially in high-resource languages. Although reasoning models, trained to leverage inference-time compute, outperform general- reneral-purpose LLMs, they remain sensitive to permutations and exhibit failures such as inconsistent reasoning loops. LINGOLY-TOR addresses key issues in benchmarking LLMs: saturation, contamination, and weak construct validity (Reuel Lamparth et al., 2024). Our primary contributions are as follows:
An unsaturated benchmark for frontier reasoning models. The top language model on our benchmark, GPT-5, scores 48% overall and only 31% on the highest difficulty problems (Section 4.1
A method to quantify knowledge effects. The difference between performance on the unperturbed and perturbed problems highlights reasoning shortcuts. We show score inflation from knowledge is correlated with language resourcedness (Section 4.2) and that providing the correct reasoning logic mitigates it (Section 4.3)
A method for generating new uncontaminated reasoning problems. Experiments with then-unpublished UKLO 2025 problems show that perturbation-related performance drops persist, indicating that the effects not due solely to training-set overlap (Section 4.4
System for writing language. A glossary of linguistic terms used in this article included in Appendix
2

---

Published as a conference paper at ICLR 2026
Problem permutation
Original Problem
Permutation p
Preamble:
Kabyle is language spoken
Preamble: Language spoken
Generate mappings
Context: Here are several example translations Kabyle:
Context: Here are several example translations Language
Rulsowe
Ufgent
They flew ran
Eqcawg
They flew
Uzziegh
Effnar
ran
Tufeg
She flew did not fly A boy ran
Gagac
She flow
ufgegh ara
Em eqcar imi
fly
Yuzzel weqcic
Leffan ghaztut
boy ran
Ur yufeg
He did not fly
Em legac imi
He did not
Ur muqient ara Temuqel teqcict
They did not observe A girl observed
Em yeznawg Imi
They did not observe girl observed
Mapping te m
Gayezan gaztutg
Question 2:1: Translate the following: Sub-question Sub-question 10: They did not run
Question 2.1: Translate the following: Sub-question 9:...
Sub-question 10: They did not run
negation Ur... ara run ..uzzl
Simplified reasoning steps needed to answer
negation + Em .... imi run .eeffn They + ..awg
They ..ent
uzzlent ara
Correct answer
Em offnawg imi
Figure 2: Obfuscation example. Problem 160 (Kabyle) before (left) and after (right) obfuscation with the simplified inductive reasoning steps needed for answering sub-question 10. For each obfuscation, we sample character mapping (permutation) based on the ruleset (see Appendix[B.3 [B.3 to learn more about rulesets), then apply it to obfuscate the problem and the answer. Only the original problem can be solved with the aid of model's internalised knowledge.
RELATED WORK
2.1 REASONING IN LLMS
LLMs show rapid progress on dask-reasoning across domains, such as mathematical word problems
(Cobbe et al., | Hendrycks et al., 2021 Chen et al.] 2023 Shao al., 2024), visual pattern-
matching (Chollet] 2019). multi-step planning (Valmeekam et al.
2022 Kambhampati (2024), and
commonsense crestion-answering (Zelikman al.] 2022 Lin al.
2020 Rajani et | Talmor
ct al. 2019). It remains unclear to what degree these abilities generalise to novel domains and tasks.
Models fail in systematic ways, showing distractability (Shi et al. 2023), content effects (Lampinen
et al. 2024). and sharp drops under minor perturbations (Mirzadeh et al.) 2024). They also rely on superficial patterns (token bias) QJiang al., 2024). and this reliance persists despite inference-time scaling (McCoy et al.) 2024). This is further complicated by dataset contamination, where benchmark questions appear in training data (Zhou et al.] 2023; Jacovi et al. [023) Yang et al.] [002)). The
evidence suggests that high LLM benchmark scores are inflated through shortcuts (Bean et al. 2024).
2.2 LINGUISTIC REASONING WITH LLMS
Linguistic reasoning problems test the in-context learning and reasoning ability of LLMs by providing examples of a language and tasks to accomplish based on linguistic analysis (Bean et al. 2024 Tanzer ct al. 2024). These problems require a mix of deductive, abductive, and analogical reasoning. allowing broad relevance beyond linguistics (Bean et al. 2024). Linguistics Olympiads are a common source of these problems, where puzzles have been curated for student competitions, ensuring the quality of the puzzles as well as their solvability by someone with no prior knowledge of the languages in question (Şahin et al., 2020 Chi et al. 2024). Morphological complexity has been shown to be particularly challenging for LLMs as well as languages with lower similarity to English (Choudhary et al. 2025). Advanced problems from these competitions typically draw on low-resource languages where LLMs have minimal pre- pre-training exposure, and remain difficult for top models (Sánchez et al.) 2024; Lian et al. (025). However, models show signs of partial contamination even in low resource languages (Bean et al. 2024), indicating a need for further methods improve benchmark validity.
3

---

Published as a conference paper at ICLR 2026
3 LINGOLY-TOO BENCHMARK
LINGOLY- TOO is reasoning benchmark extending the LINGOLY benchmark (Bean et al. 2024). It consists of 1,203 questions and a total of 6,995 sub-question and answer pairs, generated through an obfuscation process designed by experts to preserve the intrinsic reasoning steps needed to solve each question, and to generate novel test data that measure the generalisable reasoning abilities of models.
3.1 LINGUISTICS TASKS
The benchmark is adapted from 82 problems from the UKLO. Linguisties Olympiad problems have several desirable features. All problems are grounded in natural languages, but require preliminary knowledge in linguistics or any other domain beyond what is expected from middle and high school students. Each problem self-contained and can be solved from the provided context using general reasoning and pattern matching skills. Answers vary in format, including free response and multiple choice, and can be evaluated with simple rule-based matching. Problems span five difficulty levels (Breakthrough, Foundation, Intermediate, Advanced, Round 2) sourced from UKLO
typical problem sheet (referred to hereafter as problem) consists of multiple questions, each question consists of multiple (sub-question, answer) pairs. Fach problem presents words or phrases in language of focus (termed Problemese) alongside their translations into the language in which the problem is written (termed Solverese) (Bozhanov & Derzhanski 2013). The solver should deduce the underlying patterns and structures and use them to produce a best answer for each sub-question in manner that resembles few-shot learning (Figure
3.2 REASONING- PERMUTATION
Although the original 82 problems include low-resource languages, these languages are increasingly represented in pre-training datasets, allowing models to rely on memorisation or knowledge rather than reasoning. We address this challenge by generating reasoning -equiiiriinivvvainnn permutations that obfuscate the original languages while preserving the underlying language mechanisms and solution steps. In Linguisties Olympiads, obfuscation describes the process that prevents cheating by removing metadata such as language names, language families, and geographic information that could identify the Problemese language (Asia Pacific Linguistics Olympiad [2024] United Kingdom Linguistics Olympiad 2021). We extend this approach by obfuscating the Problemese text itself, making unlikely that models can solve problems using knowledge or memorisation.
Because the problems require symbolic reasoning over subwords (both morphemes such as suffixes, and phonemes), common permutation techniques that operate on complete words such as synonym replacement, paraphrasing, word swapping not suitable. These would corrupt the symbolic units and invalidate the problems. Instead, we manually created a ruleset for each problem to generate valid permutations of the targeted tokens (Problemese text). Valid permutations treat graphemes (both single letters and letter-combination like English th and sh) as single units, and preserve any relationships between them which are relevant solving the problem. They also preserve any loanwords or English cognates that might aid in solving the problem. Names of people, deities, and sacred places were also kept unchanged.
It is important to note that naive permutation of graphemes can easily render the problem unsolv- able if it does not preserve the properties used in the underlying linguistic mechanism. We il-
Last vowel Suffix
or -siz
0 or -suz -süz
lustrate this with the example of minor vowel harmony highlighted in Problem 5 (Turkish, by Bozhidar Bozhanov). The suffix -sV (meaning 'lacking', similar to the English '-less') ) can take four different forms depending on the previous vowel (Table []. From a linguistic perspective, each of the four vowel pairs (e.i), (o.u), (ö.1), (a.1)
a or -s1z
Table Problem (Turkish) suffix form. Suffix form of -sVz changes depending on the last vowel.
For problems marked with multiple levels, we take the lower one. The distribution statistics of these difficulty levels can be found in Appendix [
4

---

Published as a conference paper at ICLR 2026
differ in the shape of the lips (roundedness) and position of the tongue (backness) when that sound is produced. For example, (e.i) are [ORUND] and [-BACK], while (0,u) are [+ROUND] and -BACk] Any permutation which did not preserve these pairs would render the problem much harder and possibly unsolvable, since the vowel the suffix would not correspond as naturally to the previous vowel. As such, all possible permutations must preserve the pairings, although the pairs may be permuted amongst themselves.
Our expert list of valid permutations takes into account the interactions between different segments maintain two principles: (i) ensuring the permutations are reasoning-equivariann and (ii) obtaining the maximum number of possible permutations. Further details can be found in Appendix
3.3 DATA GENERATION
We manually annotated 1,005 (sub-question, answer) pairs from 82 problems with special tags for later processing. Metadata such language names, language families, and geographic information, which had no impact on problem solvability but may provide clues for LLMs to use shortcuts were removed. A detailed description of the annotation process, including examples, is provided in Appendix
We randomly sampled up to 6 valid permutations per problem, and generated obfuscated versions of the problem by altering the Problemese text according to the selected permutations. Finally, we removed any special characters used for annotations. The original questions and the 6 permutations resulted in 6,995 (sub-question, answer) pairs across all problems
3.4 EVALUATION
Metries We define a correct prediction as exact match to the answer, with no credit given to partially correct answers. The correct answer often involves making small changes to words already present in the problem, scores that offer partial credit may reward incorrect answers resulting from wrong reasoning.
Let ji k, p) be the exact match score for problem i {L...., 82). question j E { .I...., n, sub- question & E {I...., mi,}) } after applying permutation p E {0..... 6). where n, the total number of questions in problem and m,j is the total number of uub-questions in problem question j. To simplify notation, permutation p = 0 the identity, i.e. the unpermuted case. We define M obf to be the average exact match score across all sub-questions in all (non-identity) permutations of problem question j. that is, Mai (1/(6m.,)) Σm E L(i. j. k p). In analyses where we compare results to those on the original (unpermuted) problems, we define M (1/m4,) E k= L(i,j, 0) be the average exact match across all sub-questions in the unpermuted problem question j. To evaluate a single model on the entire benchmark, we report the average across all problems and questions, that is
82
82
Mars
M
i.j obf
and
Mog
Mid
82
82
j=1
j=1
Evaluation pipeline We broke down problems into individual questions and prompted models on each question with all of its (sub-question, answer) pairs. Each prompt consisted of preamble (detailing background information about the language), context (key information the models need to reason over and infer the correct rules), all questions in the problem (some questions depend on previous ones) and the specific question to be answered. The prompt also specified the expected json structure response. The exact template used is provided in Appendix E Since our aim is reasoning rather than instruction-following, we applied rule-based post-processing to recover improperly formatted responses. Remaining unparsed responses were counted incorrect and detailed in Appendix
We refer to problems by their numhering on the UKL.O past problems database (United Kingdom Linguistics
Olympiad (2233..
"Tis less than 1, 005 033 the total possihle permutations for some problems less than 6.
5

---

Published as a conference paper at ICLR 2026
Bmakthrough
0.72
0:75
0/70
0:69
0.18
0.11
Claude 3.7 Sonnel (no fhinking)
0.45
0.32
0.17
0:55
0,7
0.04
Claude 3.7 Sonnet (Hinking)
Foundation
0.75
0.m
0.25
DespStek-U2 Sreppeek--3..com 1-Ternmua
0.22
0.1
0.41
0.29
0.30
Lama 3 70B Inc-riittttttt
Internediate
4.1
0.06
0.50
0.52
0.59
0.6
0:39
0,50
0,30
0.10
0.17
0.11
Advanced
Claude Opus
0.44
0.43
0.45
0.43
01-grevvan
o 0:31
03 mini (high)
0.31
Claude 3.5
0.28
0.5
0
0.12
o8-mini (lw)
0.12
0:1
0.09
Gomini
GPT44.5
Gemini
GPT-40
Roundz
GPT
0:20
0.27
0.25
0.66
0.18
0:06
Phs4
0.08
Figure 3: Breakdown of Maks scores by difficulty level. For the stronger models, we see clear
decreasing trend scores with increasing problem difficulty.
Models We evaluated fifteen reasoning and general-purpose model variations including open- source and closed-source models the benchmark. See Appendix F for further details about the models.
3.5 BENCHMARK VALIDATION
Several measures were taken to ensure the quality and correctness of the benchmark. All problem annotations were performed manually and subsequently validated by two team members with expertise in Linguistics Olympiads. In addition, automated checks were incorporated to detect issues. Any identified errors triggered re-annotation and validation. A second round of both manual expert review and automated validation was conducted on the final data.
Obfuscation designed preserve reasoning steps and verified by team members with Linguisties Olympiads expertise. Additionally, two International Linguistics Olympiad (IOL) medallists, familiar with the original UKLO problems, audited sample of obfuscations and independently confirmed that all problems remained solvable via the same reasoning steps.
Because obfuscated text no longer adhered the original languages, it could impose cognitive penalty in humans who may be more familiar with the original language orthographies. To estimate this effect, ran randomised controlled trial sample of six problems (chosen be relatively easy to ensure novices had a realistic chance of solving them) with 172 human subjects. Participants were randomly assigned to either an original or obfuscated problem. Performance decreased 5.70% on obfuscated problems, likely due to their unfamiliarity, as noted by one auditor. Details of the study and analysis are provided in Appendix
EXPERIMENTS AND ANALYSIS
4.1 OVERALL PERFORMANCE
Figure shows all models' results on LINGOLY-TOO. The Mag metric (no obfuscation), is reported for comparison, which posit an over-estimation of reasoning abilities. We see that frontier models achieve ~ 0.59 on Mog- However, using Mors metric (with obfuscation) this drops to maximum 0.48. Modf scores also reflect improved reasoming capabilities of reasoning models over general-pp purpose models. For example, GPT-5 scores higher than GPT-4.5, and Claude 3.7 (thinking) outperforms Claude 3.7 (no thinking) (0.44 vs. 0.30). addition, 03-mini (high) outperforms 03-mini (low) by larger margin (0.31 0.13), indicating that increasing inference-time reasoning budget is useful for the benchmark tasks. Performance also seems related to the difficulty level the best performing models but the pattern is not consistent in the weaker models (Figure 3 GPT-5 and Claude 3.7 Sonnet score highly (0.81 and 0.75, respectively) on Breakthrough problems but score below 0.31 at the Round 2 level, demonstrating that the benchmark is far from saturated and offers range difficulties valuable for tracking progress. Overall, no model scores over 0.5, suggesting that despite recent breakthroughs, multi-hop inductive reasoning remain open challenge.
general-purpose

---

Published as a conference paper at ICLR 2026
GPT-4o
Llama-3.3 70B
Answer (% of total)
Mog (SB)
Many (SB)
Mog (SB)
Mouy (SE)
All
0.08 (0.00)
0.02 (0.00)
0.05 (0.00)
0.03 (0.00)
Digit (13.9%)
0.16 (0.01)
0.08 (0.00)
0.19 (0.01)
0.11 (0.01)
Single Char (144%
0,03 (0,00)
0.03 (0.00)
0.05 (0.00)
0.04 (0.00)
Y/N (U.6%)
0,62 (0.04)
0.31 (0.04)
0.19 (0.04)
0.49 (0.04)
Other (71.6%)
0.07 (0.00)
0.01 (0.00)
0.02 (0.00)
0.01 (0.00)
Table 2: Performance in the no context senten the no context setting, prompts contain insufficient information to answer questions. Scores are categorised by answer type.
GPT-5
Claude Opus 4.1
Claude 3.7 Sonnet (thinking)
Gemini 2.5 Pro
Origind
Bootstapped
Prequency
0.00
0.29
0.4
DeepSeek-Va. V3.1-Terminus
-rreview
a3-mini (high)
Claude Sonnet (no thinking)
Original
Boxtsaaped
Prequency
Exact Match
Exact Match
Exact Match
Exact Match
Figure 4: Score distribution across bootstrapped samples. Distribution of scores across 500 bootstrapped samples of our data by model. Each consists of 82 problems. Open source models are shown in orange while proprietary models are in blue (for full results, see Appendix [
4.2 KNOWLEDGE VS. REASONING ABILITIES
Ability to score without reasoning A key goal of the benchmark to minimise the effects of models' knowledge reasoning estimates. We evaluate LLMs' ability to score without reasoning on obfuscated data by adopting the no context setting from Bean et al. (022)), where critical information is removed from context, rendering the problems unsolvable by reasoning alone. We conduct a small experiment using 30 permutations per problem with two models (Llama 3.3 70B and GPT-4o).
Table Sisows the results in the context setting. Marf for Llama 3.3 70B and GPT-4o is 0.02 and 0.03 respectively. When only considering questions with lower chance of random guessing (Other), Mors further drops to 0.01 in both models, demonstrating the effectiveness of obfuscation. This is not the case the unobfuscated questions where GPT-4o is more successful in circumventing reasoning, scoring Mo, 0.07 through knowledge and memorisation. Inspecting few examples where GP was successful reveals cases where GPT-4o correctly answers through direct translation, such generating the correct Welsh translation: Aeth meddyg i Gymru the English sentence A doctor went to Wales.
Reasoning gap To compare benchmark estimates with those from only the unobfuscated questions, we construct bootstrap-style distribution over permutations. Specifically, we create 500 bootstrap sets, each containing all 82 problems. For each problem, we independently and uniformly select one of its seven versions (original plus six permutations) and compute the score. Figure 4 shows the distribution of scores on bootstrapped sets. When using unobfuscated problems, the score (Mag) consistently appears at the right tail of the distribution, reflecting significant decrease in models'
7

---

Published as a conference paper at ICLR 2026
Score
Standard
Dush
Character
Original
0,087
0.051
0.053
Permuted
0.050
0.045
0.035
Table 3: Exact match score with varying tokenisation. 'Dash" tokenisation inserts dash between each character the Problemese data. 'Character' tokenisation forces the model tokenise each character individually.
performance with obfuscation. A similar gap is observed in the subset of problems used in the human study, where average model performance falls from 0.45 to 0.37 (8.6%) on obfuscated problems. However, reasoning models incur a smaller performance drop, on par with humans (5.8%), suggesting that reasoning models have reduced the gap in solving problems in their original orthography versus obfuscated versions compared with general-purpos models.
Tokenisation effect A possible explanation of the performance drop is that uncommon character sequences would affect LLMs through tokenisation by producing unuseful token representations. As tokenisers of LLMs are trained using frequency statistics on languages in their original orthography, sub-optimal tokenisation could explain the fall model performance. We conduct a small experi- ment multilingual model (Aya-23-35B) compare three variations of tokenisation: standard tokenisation, introducing dash between each character in the Problemese and tokenising each character separately. T that enforcing separation of the tokens does not improve model performance. For both the original and obfuscated versions, exact match score does not improve the tokenisation is altered, pointing towards explanations of failed reasoning other than the standard tokenisation. Additional tokenisation results are detailed in Appendix K
Variance across permutations Models score higher unobfuscated problems, possibly by utilising their knowledge from exposure to the original languages. Since obfuscation preserves solution logic, quantify the performance difference due to obfuscation. For fixed permutation p of problem i, we define A as the difference between the average score across all questions of problem under obfuscation p and the average score of all questions for the unobfuscated problem. That is
1 n2 j=1
M = 1/ns
MiJ. j=1
and
A.P obf
Ii M
MiJ
Then ALp. is negative when the model performs worse on a problem permutation, zero when there no difference, and positive otherwise. Careful inspection reveals that although on average these differences are negative, model performance gaps vary across problems as well as across permutations within each individual problem. In a few cases such as Problem 24 (Tadaksahak) and Problem 67 (Navajo), some permutations even led to improved performance. We also problems that had similar effects upon permutation across several models, indicating similarities in model failure modes. Appendix includes full analysis of A abf across all generated permutations of the 82 problems.
Language resourcedness effect Problems from higher resource languages generally show large negative A = 1/6 2, A values across all models such as Problem 74 (Japanese): -0.59, Prob- lem 76 (Finnish): -0.59 and Problem 178 (Italian): 0.57. We approximate language resourcedness using the number of speakers of the language (in logarithmic scale). To quantify its effect on the gap in reasoning estimates, we apply linear regression to predict A from resourcedness, separately for each problem difficulty and for models grouped into reasoning and remeral-purposs categories. We apply the Bonferroni correction to account for multiple-testing when reporting significance.
Results (Figure 5) show that language resourcedness is negatively correlated with reasoning perfor- mance gaps at the higher difficulty levels. In reasoning and general- purpose models, coefficients are negative (3 0, p 0.01 for Advanced and Round 2 problems). One possible explanation is that when models are unable to reason at higher difficulty levels, models are better at guessing the answer in languages with larger amount of data available for training.
8

---

Published as a conference paper at ICLR 2026
Breakthrough
Foundation
Intermediate
Advanced
Round2
80%
Reasoning General
40%
ma d ga
0
Bobr
0%
dof
-40%
80%
0-1
10A5
1048
0-1
10A5
10^8
0-1
1045
10^8
0-1
10A5
10*8
0-1
10A5
10^8
Speakers (log)
Speukers (log)
Speakers (log)
Speakers (log)
Speakers (log)
Figure 5: The effect of permutation is larger for high- languages. Each point represents all problems associated with a language of specific numbers of speaker. Solid lines are fitted regressions and shaded areas are the 95% confidence intervals.
4.3 EVIDENCE OF NASCENT REASONING
Let us define
82
M rob
min Σ ki.j,k;p) 1=1
Mrob 82
Micd
ma.
1=1
as a more robust metric of average model performance across all problems since measures the minimum score achieved any problem permutation. Under this metric, performance drops in all models (see Figure for more details). The best model, GPT-5, only achieves 0.29 (a decrease of over 0.1 Reasoning models, which score the highest, more consistent than general purpose models. An attempt for systematic analysis of reasoning traces using L.MSs was unsuccessful as LLMs were unable to identify erroneous reasoning steps, making both type I and type II errors. We manually inspected the reasoning traces of Claude 3.7 (thinking) on two randomly selected problems where the model correctly answered questions in all permutations. We found examples where the model can identify subwords and suffixes across all permutations, infer their meaning, and deduce the correct answers (e.g. excerpt from one trace: The verb prefixes seem to agree with the subject: "si-" "they"). However, these positive examples of a generalisable reasoning ability were among many more failed cases. Among them are traces with repeated analyses, sometimes drawing the same conclusions, others drawing different ones, even within the same permutation. More generally, M shows that all models perform noticeably poorly, underscoring that reasoning is brittle and remains an open challenge to frontier models.
Reasoning with expert guidance By prompting with guidance steps written by team members based on the original problems, we evaluated whether model performance on the permuted problems improved. Using two problems chosen to have concise solutions (Problems 69 and 164) and four models from several providers (Claude 3.7 (thinking), Claude 3.5, Gemini 1.5 Pro, 03-mini high), we find that all models improve with the mean score on permuted questions (Morr) increasing (0.66 to 0.76). This experiment suggests that, as in domains such as mathematies, multi-hop reasoning on unfamiliar language problems benefits from improved intermediate inductive reasoning steps.
4.4 KNOWLEDGE VS MEMORISATION
While we assume a possibility that models are exposed to some of the unpermuted problems during training, we do not think that this accounts for total performance gaps. To illustrate, we were able to access and evaluate a subset models on 5 problems from the UKLO 2025 that had not yet been published online at the time of the experiment and are of higher difficulty level. Results in Table 9 show a comparable performance drop between the unpermuted and permuted cases to that observed in the benchmark. The difference in scores illustrates that the performance decrease is not simply due to memorising answers since these problems have not been included in models training. The gap also highlights the effectiveness of our permutation control for knowledge even unseen problems.
9

---

Published as a conference paper at ICLR 2026
5 LIMITATIONS
Exact Match uniformly penalises all incorrect answers, including partially correct ones, and limits insights into failure modes. Yet, metrics such as BLEU Papineni et al., 2002) and ROUGE (Lin) 2004) are unsuitable for the very short answers common in our benchmark (note from Table 2 that 14% of our answers are single digits) and chrF (Popović 2015) is sensitive to repeating related words from the context, which is common behaviour in the models being tested. For example, Problem 164.1 (3) requires a translation of "your (plural) iguana" to Ulwa. The word for iguana in Ulwa is kahma, which is already given in the context. The correct answer is kahmanama, as your (plural) is -mana- Claude Opus 4.1 returned kahmana, which is the wrong form and placement of the possessive. If we assigned partial credit through e-g. edit distance, the answer from Claude will score highly, even though this is an incorrect answer based on incorrect reasoning. Assigning partial credit would unnecessarily inflate the baseline through repeating parts of words in the context. Future work could explore alternative evaluation metrics that capture partial solutions or help discern partially correct answers to ensure construct validity (Bean et al. 2025). Novel evaluation scores could credit correct intermediate reasoning steps alongside the final answers.
While problems in LINGOLY-TOO do not depend on any specific domain knowledge, we limit this work to inductive and deductive reasoning in the natural language modality. The benchmark is not a comprehensive evaluation of all reasoning abilities across all modalities but results suggest that improvements in certain reasoning abilities would translate improvements in others.
6 CONCLUSION
We introduced LINGOLY-TOO, benchmark for evaluating reasoning in language tasks. By applying expert-designed obfuscations to Linguistics Olympiad problems, we generated permutations that preserved solution logic while remaining novel to models. Our results showed that the benchmark effectively controls for model knowledge and memorisation, revealing that LLMs (including reasoning models) often rely on shorteuts rather than genuine reasoning. We further quantified how language resourcedness amplify these shortcuts. Finally, we observed that reasoning models were better able to apply symbolic and inductive reasoning and that progress reported in domains such as mathematics and coding appears to partially translate to language problems.
Our benchmark remains unsaturated, especially at the highest problem difficulty, leaving ample headroom. Our results support existing claims of model reliance on shortcuts, which biases reasoning ability estimates. Compared to the unobfuscated problems, deriving estimates from obfuscated problems paints more conservative picture of the reasoning ability of frontier models. We find that reasoning consistency and robustness remain limited frontier models and warrant further attention.
ETHICS STATEMENT
This paper aims to advance the field of machine learning. Constructing and publishing problems using low-resource languages to create a benchmark may lead to concerns about the exploitation of other harm caused to the communities who use and own these languages (Tanzer et al. [024 The problems in this benchmark are created either by native speakers or using published resources such as sketch grammars. In these cases, the relevant language communities have already given consent for a linguist to publish information on the language. In this paper, we transform existing puzzles, and do not create new content in the languages.
The obfuscations of the problem amount to a new orthography for the language, and do not change any other aspect of the language such as grammar or the meanings of words; not alter the names of people, deities, or sacred places; and are not publicly available. All participants in the human study were provided with information about the language they had studied and its speakers after completing the study.
REPRODUCIRILIIT STATEMENT
We release complete anonymised code and evaluation scripts for our benchmark for reproducibility at http:///ihhibbcoom///maallllllll
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors would like thank Riley Kong and Deeraj Pothapragada for their help with the external audit of the obfuscated problems. L.Y. was supported by EPSRC [EP/S026347/1] and the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA). H.M. is supported by ESRC [ES/P000649/1] and would like to acknowledge the London Initiative for Safe AL. A.M.B. supported by the Clarendon Fund Scholarships at the University of Oxford. A.M.B. and A.M. were partially supported by the Oxford Internet Institute's Research Programme funded by the Dieter Schwarz. Stiftung gGmbH. R.O.K. is supported by the Clarendon Fund Scholarships and the Jesus College Old Members' Scholarship at the University of Oxford.
REFERENCES
Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Cher Tan, al. Aya 23: Open weight releases to further multilingual progress. arXiv preprint, 2024. URL https: http:///xxxvvv.../x../
2405 15032
Asia Pacific Linguistics Olympiad. Problems by year, 2024. URL https://aplo...
probl embolemm/// hwtpsprrbllsssee ems by-arrrrbyyyrr////--
Andrew M Bean, Simeon Hellsten, Harry Mayne, Jabez Magomere, Ethan A Chi, Ryan Andrew Chi, Scott A Hale, and Hannah Rose Kirk. LINGOLY: A benchmark of olympiad-level linguistic rea- soning puzzles in low resource and extinct languages. In NeurlPS 2024 Datasets and Benchmarks Track, 2024,
http://arx....orr/////2400 abs/24/6..62. 06196
Andrew M Bean, Ryan Othniel Keams,, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, et al. Measuring what matters: Construct validity in large language model benchmarks. In NewrlPS 2025 Datasets and Benchmarks Track, 2025. httsst aatpiiiiiii....ii htsp//lss///11// 04703
Bozhidar Bozhanov and Ivan Derzhanski. Rosetta stone linguistic problems. Ivan Derzhanski and Dragomir Radev (eds.), Proceedings of the Fourth Workshop Teaching NLP and CL, pp. 1-8, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https 7/aclantho ogw.ooggyoooooooo...ooooooo -
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023, ISSN 2835-8856. URL htttts:///ppeneeeeee
net forum?
Id=YE24ZPt8zdl
Nathan Chi, Teodor Malchev, Riley Kong, Ryan Chi, Lucas Huang, Ethan Chi, R. McCoy, and Dragomir Radev. Model.ing A novel dataset for testing linguistic reasoning in language mod- els. In Michael Hahn, Alexey Sorokin, Ritesh Kumar, Andreas Shcherbakov, Yulia Otmakhova, Jinrui Yang, Oleg Serikov, Priya Rani, Edoardo M. Ponti, Saliha Muradoğlu, Rena Gao, Ryan Cotterell, and Ekaterina Vylomova (eds.). Proceedings of the 6th Workshop on Research in Com- putational Linguistic Typology and Multilingual NLP, pp. 113-119, St. Julian's ss Malta, March 2024. Association for Computational Linguistics. URL https:// https///aaaaattooo...ooooo
14/
François Chollet, On the measure of intelligence. arXiv preprint, 2019. URL http::/arxix
arg/abss/911 01547
Mukund Choudhary, KV Aditya Srivatsa, Gaurja Aeron, Antara Raaghavi Bhattacharya, Dang Khoa Dang Dinh, Ikhlasul Akmal Hanif, Daria Kotova, Ekaterina Kochmar, and Monojit Choud- hury. UNVEILING: What makes linguistics olympiad puzzles tricky for LLMs? In Second Conference on Language Modeling, 2025. URL bhttps://openreseee
net. forum?id
EcRc11EXc4]
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv preprint, 2021. URL
https::/erxiv.crm httpl//.orggbb/////o 14168

---

Published as a conference paper at ICLR 2026
Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2)233335 1994.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A sur- vey. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the As- sociation for Computational Linguistics: ACL 2023, pp. 1049-1065. Toronto, Canada, July 2023, Association for Computational Linguisties. doi: 10.18653/v1/2023.1 1223 URL https://acl www.acannnntooooo.ooooooo org/ 2023. f Findings acl. 677
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 5075-5084. Association for Computational Linguis- tics, 2023. doi:
00/33//1/222.emmmm .em..e-mlpppe.... m08. URLhttps: /ac:://acllla /atpa/aaanntto.......o...o...oa. org/ 2023. emn pw-mappmmnn 3008
Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang. Tanwi Mallick, Weijie Su, Camillo Jose Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 4722-4756, Mi- ami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 4024.emmlpp main..mmiin.7.. 272. URL https: http::::/aacaaatthoaoo
org/ 2224 www.pmmmainnn....m..ammmmmmmmawm 272/
Subbarao Kambhampati. Can large language models reason and plan? Annals of the New York Academy of Sciences, 1534(1):15 18, 2024. ISSN 0077-8923, 1749-6632. doi: 10.1111/nyas.. 15125
URL htattp://arxiae org/ abs /255//403322 4034343400 04121
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, San- ket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoe Le, and Orhan Firat. Big-bench extra hard, 2025. URL htt.h:tp/// www.xxx..rg/aab/255500 19187
Manfred Kohrt. The term 'grapheme" in the history and theory of linguisties. New trends in graphemics and orthography, pp. 80-96, 1986.
Philipp Koralus. Reason and Inquiry: The Erotetic Theory. Oxford University Press, 1st edition, 2022. ISBN 978-0-19-186254-.0 URL httpp:///a...rrrr//.0000////////
001.0000 0001
Andrew K Lampinen, Ishita Dasgupta, Stephanie C Y Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James McClelland, and Felix Hill. Language models, like humans, show content effects on reasoning tasks. PNAS Nexus, 3 pga) 2024. ISSN 2752-6542. doi: 00.1003/pnasnexux/gae233 URL htttttts::///oi wtt.:/o..oorrgo. wttpp//pp.pppppppeeusssooe/eea/e pggae233
Changmao Li and Jeffrey Flanigan. Task contamination: Language models may not be few- shot anymore. Proceedings of the AAAI Conference Artificial Intelligence, 38(16):18471- 18480, 2024. doi: 10. 10.16690aaaa...o v38iaaaiiiiiii1 338816.29808 URL.
php/ AAAI wat.a//aricee//iii hatpcl/evvee///88888888 29808
Da-Chen Lian, Ri-Sheng Huang, Pin-Er Chen, Chunki Lim, You-Kuan Lin, Guan- Yu Tseng, Zi-Cheng Yang, Zhen- Yu Lin, Pin-Cheng Chen, and Shu-Kai Hsieh. LingBench++:: linguistically-inno benchmark and reasoning framework for multi-step and cross-cultural inference with LLMs, 2025.
ats//bbs/25000 2507 16809
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Compu- fational Linguistics: EMNLP 2020, pp. 1823-18400 Association for Computational Linguistics.
12

---

Published as a conference paper at ICLR 2026
2020. doi: 10.18633/w1/2022..mmml
URL BtLL:ttssp///caaaaalloooyy
2020. www.iiiggsnemm....i.....
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization
Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL https: ac https///caaattoooooo..ooo
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy, and Thomas L. Griffiths. When language model optimized for reasoning, does still show embers of autoregression? An analysis of OpenAl o1. arXiv preprint, 2024. URL http::::t...... /ttppi.///avvvcc.oo org/ abs /21.//41..0000 01792
Iman Mirzadeh, Kcivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mchrdad Farajtabar. GSM-Symbolic: Understanding the limitations of mathematical reasoning in large
language models. arXiv preprint, 2024. URL htt.:tttpa/aa...coooc wbs/ 2410 05229
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318. Association for Computational Linguisties, 2002. URL httppp::/acclantoooooo..oo//////
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, and Mohamed Shaaban. Humanity's last exam, 2025. URL http:::///// petpss:/arxxvv.or
abs/ maa//2501.0 14249
Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Ondrej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), Proceedings of the Tenth Workshop Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015. Association for Computational Lin- guisties. doi: 10.18653/v I/W15-3049,
org/ /ww.o//gg/..........o/..../..../ /115- 3049/
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! Leveraging language models for commonsense reasoning. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting the Association for Computational Linguistics, pp. 4932-4942, Association for Computational Linguistics, 2019. doi: 10.18653/v1/ P19-1487, URL hww.https:.. http:////aaaaathholcoooooo
org/ P19 1487/
Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. AI and the everything in the whole wide world benchmark. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Truck on Datasets and Benchmarks, volume 1, 2021.
Anka Reuel-Lampann Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J Kochenderfer. BetterBench: Assessing AI benchmarks, uncovering issues, and establishing best practices. Advances Neural Information Processing Systems, 37:21763-21813, 2024.
Jenna Russell, Marzena Karpinska, and Mohit lyyer. People who frequently use ChatGPT for writing tasks are accurate and robust detectors of Al-generated text, 2025. URL https: /oos:///rrxiy
org/abs/2500 2aabb/22511.. 15654
Gözde Güil Şahin, Yova Kementchedjhieva, Phillip Rust, and Iryna Gurevych. PuzzLing Machines: A challenge learning from small data. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1241-1254, Online, July 2020. Association for Computational Linguistics. doi: allmaiin
10.18653/ 1/t0233/1/2222.ccc acl- main. 155. URL https: 1157]
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In International Conference on Learning Representations, 2023. URL
hittps://axe org/abs. 2210 01240
Dominick Scasserra. The influence of perceived task difficulty on task performance, 2008. URL https: / rdw. rowan, edw/ edw/etd/756
13

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang. Y. K. Li, Y. Wu, and Daya Guo. DeepSeckMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprinx arXiv:2402.03330, 2024. URL
httt:://axivv wwwp://wwwwww/b/ooteoooooo 2202 200bbb/300000030
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. 31210-31222. PMLR, 2023.
Aaditya K Singh and DJ Strouse. Tokenization counts: the impact of tokenization arithmetic in frontier Ilms. arXiv preprint wrxXi:22422114000 2024.
Eduardo Sánchez, Belen Alastruey, Christophe Ropers, Pontus Stenetorp, Mikel Artetxe, and Marta R. Costa-jussà. Linguini: A benchmark for language-agnostic linguistic reasoning. arXiv preprint, 2024.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.). Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pupers), pp. 4149-4158. Association for Computational Linguistics, 2019. doi: 10.18653/v1/N19-156 1421. URL htt.ttss//aaaaanholllooooooo//ee 1tt.aatttoooooooooo//11111111111
Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas Delas-Kyriazi. A benchmark for learning translate new language from one grammar book. In The Twelfth International Conference on Learning Representations, 2024. URL htttts:://opprreeeeeet
net nwt/forumm
id=tbVWugg9E2h
United Linguistess Olymppp Olympiad. Special arrangements for the 2021 competition, 2021. URL
United Lingussics Olympiad. Past challenge puzzles: How far you go?, 2023. URL
https::/www wklo. otttas/swwww...ll...ooooooorraaa paapess Accessed: 2025-01-28.
Karthik Valmeekam, Alherto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benehmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. URL https://
openreview wwt.:eetttteooee fort:/oormmmmmmm -imiiidddwwwl wUJ- 7UUU-X0
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint, 2023.
URL /ttttps:/arxxv.coo
abs 2211. 04850
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STaR: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL pittps://openreviee net/Eorum?idi 3EL
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your LLM an evaluation benchmark cheater. arXiv preprint, 2023. URL /bat::///rxiv. http://.////a...ooo/////////////
01964
14

---

Published as a conference paper at ICLR 2026
A GLOSSARY
Word
Meaning
cognate
Word or word-part with the same historical origin another. E.g. father, German Vater, Swedish far.
diacritic
Symbol added to letter other basic glyph; often called 'accent'. Group languages all derived from a common ancestor. E.g. the Romance language family is derived from Latin.
(language) family
grapheme
Letter or group of letters that typically represent a single sound or suprasegmental feature. See Kohrt (186)) for discussion of other definitions in common usage.
lexeme
Basic unit of the lexicon (vocabulary) of a language. E.g. sit, sits, sitring, sat are all inflected forms of the same lexeme.
loanword
Word borrowed from another language. E.g. Nepali pēnsil 'pencil'.
morpheme
Basic grammatical unit of meaning within word. E.g. help-less-ness consists of three morphemes.
morphology
Branch of linguistics dealing with the internal structure and formation of words.
orthography
System for writing a language, including the choice letters or other glyphs and spelling conventions.
phonetics
Branch of linguistics dealing with the production and perception of speech sounds signs.
phonology
Branch of linguistics dealing with the systematisation and organisation of sounds signs within language.
phonological
dis-
Distinction between two speech sounds that is treated as meaningful within language.
tinction
Problemese
Unknown language that is the focus of a Linguistics Olympiad problem. Branch of linguistics dealing with the study of meaning.
semantics
Solverese
Language that Linguisties Olympiad problem is written in; assumed to be the working language of the solver.
suprasegmental
Phonetic or phonological feature that extends beyond a single speech sound. E.g. stress, intonation, pitch.
syllabification
Procedure for forming valid syllables out of string of speech sounds.
synlux
Branch of linguistics dealing with the organisation of words into phrases, clauses and sentences.
Table Glossary of linguistic terms.
15

---

Published as a conference paper at ICLR 2026
B PERMUTATION RULESETS
The core of the obfuscation procedure is the altering of the Problemese data. This is done by creating a new orthography for the Problemese language, so that the underlying grammar remains the same, but the data looks unlike any other existing data in the language. It is therefore impossible to solve the problem by relying on prior knowledge instead of reasoning. However, since this "re-spelling" preserves the important structure of the problem data, it preserves the solution steps.
B.1 OVERVIEW
To create a new orthography, each graphemd the original orthography must be replaced with new grapheme. We choose to use the same graphemes in the original and obfuscated orthographies, so that each new obfuscated orthography is simply a permutation of graphemes in the original orthography. We represent these permutations mapping dictionaries from original graphemes to the new graphemes.
There are two possible effects of such obfuscation that could increase the difficulty of a problem or even render unsolvable:
Some problems rely on solvers recognising English cognates or loanwords, either explicitly implicitly. We give two examples.
Swedish (Problem 1 176, by David Hellsten) includes the following question. "One of these adjectives behaves slightly differently to the others. Which one? [..]] It is closely related an English adjective with the same meaning. Which one?" This relies on noticing that the Swedish adjective lite-, lill- (translated as 'small') is cognate to the English word little.
Taa (Problem 217, by Ethan Chi) does not strictly require the solver to notice any cognates or loanwords; however, when the problem featured in UKLO, many solvers broke into the problem by observing that bōkòsè was likely a loanword that meant 'box'.
both cases, an arbitrary new orthography would make it impossible to recognise the cognate loanword. This would render Swedish impossible and Taa significantly more difficult.
(ii) Many problems rely on solvers either deducing or already understanding the linguistic relationships between certain sounds. We gave one such example in Section 3.22 and give two more here, which are representative of the sorts of issues that can arise.
In Tadaksahak (Problem 24, by Bozhidar Bozhanov), specific form of a verb is formed by adding one of {s.2.5.5) to the end of the word; solvers are told that à is pronounced like 'sh' in 'shoe', and 5 like 's' in 'casual'. This is plausible observation to make, since these are the four 'sibilant" (hissing) consonants, a fact which is usually taught in high school English classes, if not earlier.
In Dinka (Problem 189, by Simi Hellsten), the solver must understand how vowels change in Dinka nouns and verbs to indicate different "grades" singular and plural nouns, and different subjects for verbs. They must observe that vowels do not change randomly, but only raise or lower in the mouth:
it ++ a and N 1++0+00
The problem designed under the assumption that the solver can differentiate the front unrounded vowels (left) from the back vowels (right). This would be expected knowledge of a student sitting the UKLO R2 paper.
In both cases, an arbitrary new orthography would render the problem nearly impossible to solve, since it would not visibly preserve the important linguistic properties of the sounds that the graphemes represent.
'System for writing a language, including the choice of letters or other glyphs and spelling conventions. Lettee group of letters that typically represent single sound other feature.
16

---

Published as a conference paper at ICLR 2026
To prevent issues of type (i), manually found all cognates in the Problemese data, and ensured that they were not respelled if it would affect the difficulty of the problem. In cases of cognates that had no impact on the problem, such as most problems about European languages, these cognates were still respelled. We also ensured that all names of people, deities, and sacred places were not re-spelled.
To prevent issues of type (ii), we created mroblem-speciffe rulesets describing which permutations of the graphemes would not affect the solvability or difficulty of the problem. We call such permutations valid. Our permutation rulesets are highly problem-specific, only taking into account linguistic properties of sounds that were relevant to solving that problem. Aside from problems where there were no such properties, no two problems had the same set of relevant linguistic properties, and so each problem's permutation ruleset is unique. Section B.5 permutation rulesets for two problems.
we detail the construction these
Sampling permutations according to these rulesets allows for the dynamic generation of new obfus- cated variations of the problems in the benchmark.
B.2 METHODOLOGY
To determine the permutation ruleset for a given problem, the following procedure was applied.
First, the set of graphemes present in the Problemese data determined. This involved using the authors' expert judgement on which strings should be analysed as single graphemes, and which could be split into multiple units. A common example was deciding when a combination of letter with diacritic should be treated as a single grapheme (as in Spanish fi or as multiple graphemes (as in Spanish á = a ). This sometimes involved consulting literature was sufficient to consult the commentary on the problem provided by UKLO, as well as the authors' own solutions to the problems. Note that a grapheme could be substring of another grapheme- consider English h as a substring of sh. In this situation, the permutation algorithm would scan the annotated text left-to-right, and exchange the longest matching grapheme with its replacement under the obfuscation in a greedy manner.
the language, but usually it
We then determined what phonetic and phonological data needed to be preserved in the orthography for the problem to remain solvable and the difficulty to remain unchanged. In many cases, more phonological data was preserved than strictly necessary, such as always keeping vowels and conso- nants separate. This was both to preserve alternative solving paths than the authors', which may have involved a different set of phonological observations, and allow for consistent syllabification across permutations. This data was then encoded in the permutation ruleset for the problem, for example by separating out certain sounds to only be permuted amongst themselves, such as with {5.2.3.5) in Tadaksahak, by fixing especially difficult graphemes.
Whenever valid permutations were sampled to generated obfuscated versions of problems, the permutation was chosen to have fixed points outside of the fixed set. This was achieved by selecting random cycle whenever a random permutation of a set of graphemes had to be chosen.
B.3 STRUCTURE OF PERMUTATION RULESET
permutation ruleset for particular problem partitions the graphemes in the Problemese data into four types of structured collection: sets, tables, free-tables and a fixed set. A permutation is choice of structure- preserrvig bijection of each of these collections, which combine give bijection of all of the graphemes in the data.
Sets A set is simply a collection of graphemes, with no extra structure that needs to be preserved. A permutation can therefore be any bijection of the graphemes in set.
The graphemes in table are partitioned into tuples, each of the same length, thought of as columns of a table. The permutation then freely permutes these 'columns', while preserving the "rows" (index in the tuple) where the graphemes appear. For example, a table
Tables
((p, b). (1. d). (k, g)) =
p b
k
d
g
17

---

Published as a conference paper at ICLR 2026
will have 3! ways to rearrange the columns, hence possible permutations:
k
obfuscations
P
k
P b
p b
P b
d
d
d
b
d
d
k t
g
d
Linguistically, this example represents a situation where plosives appear in voiceless/voiced pairs, and this data must be preserved, but the place of articulation of the plosives not relevant, so may be permuted. Compare this to situation where only the voicedness was relevant, but not the voiceless/voiced pairs, which could be obfuscated by a pair of sets {p. t. k). (b, g), allowing for 3! x 3! 36 possible obfuscations.
Free-Tables These generalisation of tables where some of the 'rows' consist of collections of more than one element, together forming set. A choice of permutation is first a choice of bijection of the columns, then choice of bijection from each original 'cell' to the obfuscated 'cell'. For example, free-table
{(,, (p, b.f)), (n, (r. d, s)) =
p. f 1,
has 2! x (3!)² 72 possible permutations. An example is shown below, where the non-identity bijection of the columns was chosen.
m+ n p f b b d+p 5
m
11 r, d,
obfuscation
11
111
p. f
I, S. d
b. f.
f+d
Linguistically, this represents situation where homorganic nasals are relevant, but otherwise place and manner of articulation are not relevant.
Fixed Set Any grapheme not in any set, table free-table fixed by all of our permutations. Additionally, other strings that should not be changed during obfuscation could be specified as single grapheme within the fixed set. Such strings primarily fell into two categories: transparent loanwords or cognates, and names. Where words in the Problemese data had noticeable resemblances to English another well-known language, such Nepali pênsil 'pencil' or Yawalapiti amiku 'friend' (cf. Portuguese amigo), and where these resemblances may have decreased the difficulty of the problem by providing starting point for reasoning, they that transparent cognates were not preserved was when the problem did not involve the meanings of words, so recognition of the cognate would provide no advantage. Names of people (both real and fictional), deities, and sacred places were also always preserved. The names of other places and languages were preserved where possible, but not if it would consequently indicate what the Problemese language was.
always preserved. The only times
B.4 COMPARISON TO OBFUSCATION IN LINGUISTICS OLYMPIADS
This method of obfuscation is more severe than what is typically used by Linguistics Olympiads. Most Linguistics Olympiads do not perform any intentional obfuscation, but where is done common notations are exchanged for other common notations as far as possible. For example, in UKLO 2021 A4 (Sauk by Ryan Chi), long vowels were changed from being written with circumflex (e.g. to being doubled (e.g. aq). This is typically sufficient to prevent search engines from returning relevant results, but has little effect on any advantage gained by having prior familiarity with the language.
B.5 EXAMPLES OF PERMUTATION RULESETS
We present the thought process behind the creation of the permutation rulesets for two problems, Somali (Problem 91, by Harold Somers) and Stodsde (Problem 218, by Simi Hellsten). These are representative of the process behind all of the more complicated permutation rulesets. We first.
18

---

Published as a conference paper at ICLR 2026
give simplified form of the grammar of the language that solver would be expected deduce, then explain how this solution could be used to determine a permutation ruleset. Note that expert judgement, coming from familiarity with both these specific problems and Linguistics Olympiads in general, required to construct these rulesets.
Problem 91, Somali (by Harold Somers). This problem involves determining the formation of Isg and 3sg forms of Somal verbs. The solution is as follows:
(i) The Isg form adds -ay to the root, while the 3sg form adds -tay.
(ii) After the consonants q, C, x, (the so-called guttural consonants), we have the change + d.
(iii) Everywhere, we have the following changes: yr - d: lr > sh; di + d: dhi - dh.
There are no names or loanwords in the Somali data that need to be preserved by the obfuscation, so the permutation ruleset will only depend on the phonology of the problem. All permutations should fix dd dh, sh, since they have phonological relationships which are relevant to the solution of the problem, and cannot be found in different family of consonants. The guttural consonants can be permuted amongst themselves but not with the other consonants, so form a set [g. C, x, '). / and are treated specially within the solution, but any liquids could fill those roles without substantially changing the difficulty of the problem, so we define a set [1. w, y). All other consonants featured in the Somali data form a set (b.f. j. h, k, n, s). The vowels form another set {a, e, i, o, u].
Problem 218, Stodsde (by Simi Hellsten). This problem involves understanding the formation of causative verbs in Stodsde 8 The solution involves understanding the structure of a Stodsde syllable, and the changes that occur in each part of the syllable. No changes occur in the vowel, or consonants after the vowel. The part of a syllable before the vowel has the following structure, divided into Slots 1-5. Each slot can only have consonants of certain type. and undergoes specific changes to form the causative.
Slot
Slot
Slot
Slot
Slot Any non-plosive + non-sib.
Nasal
Non-sib. fricatives
Liquid (1 r)
plosive/sibilant
+V n
m
+ if Slot sibilant
fricative + affricate, unless Slot is nasal
+ otherwise
Finally, everywhere in the verb, v. 5.- z f. f, s the following sound is voiceless. Several alternative and equivalent analyses are also possible.
Again, the dataset contains no loanwords or names, so the permutation ruleset fully determined by the relevant phonology. Since voicedness is relevant throughout, but not all voiced/voiceless pairs are given the data, any permutation must preserve voicedness. We also fix 5. hence Since we cannot permute 0. must fix all of Slot 1. Thus m, n, and are fixed, hence also f. For Slot 2, the remaining non-sibilant fricatives are the voiceless Z, which cannot be permuted with anything, and voiced B, Y, which can be permuted. In Slot 3, we can freely permute I, Slot needs distinguish between sibilants and sibilant affricates (which are paired), and simple plosives. Considering only those that appear in the Stodsde data, this means we must fix 15"; we can permute p. k, q. and b, d. 8. while fixing the aspiration h; and we can permute the pairs (z, dz), (3, d5). and the pairs (s, ts), (s", 1s)). Finally, we can permute the vowels, and the remaining nasals D. pt has nothing could permute with, so must also be fixed.
The final permutation ruleset thus has fixed set (m, n, 4, B. 2, f. 15"h j.: sets (8, 8). (1, r), (p. k q). (b, d, g). (g. n). {u, a, a, i); and tables ((z, dz).(3. d5)), ((s, 1s), (st, (5))..
7 Somali a Cushitic language spoken by approximately million people, primarily ethnic Somalis Fast Africa and in diaspora. Stodsde Gyalrongic language, spoken by approximately 4,000 people in Sichuan, China.
19

---

Published as a conference paper at ICLR 2026
Problem
Original
Annotated
67
This problem is about the way in which Navajo speakers build sen- tences out of a verb V.
This problem is about the way which $S$Language XS$S speakers build sentences out of a verb V.
16 Ulwa is a language spoken in Nicaragua. contains quite a few loanwords from English, which is spoken the Bluefields area of the country.
S$SLanguage XS$S &&& &&& contains quite few loanwords from English &&& &&&.
61
dinaldalusanda they were cleaning it
@ @ @ @inaldalusaadaa @ @@ they were clean- ing tinaktakawda ida
61
tii)ak-takaw ida
Table 5: Annotation examples. Each row is extract from problem in LINGOLY-TOO. On the left, the extract appears in the original UKLO problem sheet or solution file. On the right is the extract after annotation. Problem 67 (Navajo by Babette Verhoeven) is example of the language name annotation. Problem 16 (Ulwa by Richard Sproat) is example of the cultural context annotation. Problem 61 (Ilokano by Bozhidar Bozhanov) illustrates the annotation of Problemese preparation for further obfuscation, and is an example of removing grading guidelines from the dataset to prepare for checking exact matching.
C ANNOTATIONS
C.I ANNOTATION PROCESS
Annotation was the first step of the obfuscation process in LINGOLY-TOO, The primary task was removing metadata included in the problem text such as language names, language families, and geographic information that was not relevant for solving the problem via reasoning, but could have allowed models to ascertain facts about the language, even when obfuscated. Problemese data that would be altered in later stages of obfuscation were also highlighted during annotation.
All problem annotations were performed by a single annotator to minimise the inconsistency between different problems. Two other members of the team had expertise in Linguistics Olympiads. They validated each of the annotated problems, identifying and documenting any errors or inconsistencies parsed through automatic checks imple-
that needed correcting. In addition, all annotations mented as part of the data generation pipeline. Despite our best efforts to ensure all problems are all correct, mistakes may remain the result of undetected human error.
C.2 ANNOTATION CATEGORIES
Four categories of annotations were performed: language and place names, cultural context, Prob- lemese data, and grading guidelines. Rach involved bracketing strings in the problem files with triplets of symbol not present used elsewhere in the problems.
Language and Place Names The English name of the Problemese language was replaced with 'Language X' wherever it appeared in the problem, and was annotated using S$S tag (see row one in 5.. If multiple non-English languages were mentioned in the problem, the second was replaced with Language Y' and the third with 'Language Z'. maximum of three languages appeared in any given problem. Where related terms such as geographic regions people groups associated with those languages were relevant to problem, they were replaced with e.g. 'Region X', and also annotated using the $$$ tag. Where two historical varieties of the same language were featured in the same problem, their names were replaced with 'Language X' and 'Language Y', and a sentence clarifying their relationship was added.
Cultural Context Any other metadata or cultural references that could indicate the origin of a language but were not relevant to solving the problem were replaced with space, and annotated with
20

---

Published as a conference paper at ICLR 2026
&&& tag (see row two in Table 5). If it not clear whether some part of the metadata might be useful solving the problem, it was annotated with the S$S tag, and edited in accordance with the principles in the previous paragraph.
Problemese Data All Problemese data were annotated using an @ @ @ tag (see row three in TJble Where problems involved multiple languages, these were not distinguished in annotation. Instead, the permutations of graphemes used to alter the Problemese data were designed to preserve the distinctions between the languages. Many the problems included pronunciation guides, since students sitting UKLO exams are not expected to have any prior linguistic knowledge. These pronunciation guides were not annotated, since the permutations of the graphemes used to alter the Problemese always preserved any relevant phonetic and phonological distinctions.
Grading Guidelines Most UKLO answer sheets include marking guidelines for awarding partial eredit, which often accompanied by symbols such as '+', 'I' and brackets written inside the correct answers. These were simply removed from the answer files (see row four in Table 5
D PROBLEM DIFFICULTY BREAKDOWN
Difficulty Level
Unobfuscated
Obfuscated
Breakthrough
60 (6.0%)
320 (5.3%)
Foundation
(8.4%)
504 (8.4%)
Intermediate
136 (13.5%)
816 (13.6%)
Advanced
374 (37.2%)
2244 (37.5%)
Round2
351 (34.9%)
2106 (35.2%)
Table 6: Counts (percentages) by difficulty level. This is for the 1, 005 unobfuscated (sub-question, answer) pairs and the 5. 990 obfuscated (sub-question, answer) pairs. Note that the obfuscated percentages are slightly different compared to the unobfuscated as some problems allow for less than permutations.
E PROMPT TEMPLATE
Below the prompt used across models. In the no context setting, [context) is removed.
Below is a problem sheet from linguistics exam. You will first see the entire sheet, then be asked to respond to specific questions from the sheet. Your answers to the questions should roly only reasoning about the Information provided in the sheet. (question_body]
Now respond to the following questicns: (preamble]
coontett]
[all (all_subquestions]
|instructions]
(formattec[outpurt
The [formattedooupputt tag populated with an empty dictionary containing the keys of the expected answer for each part of the question. The (instructions) tag is populated based the setting, For default prompt:
Only respond with json output. De net include anything other than the json in your response. Format your response as jaon file with the keys as provided below:
21

---

Published as a conference paper at ICLR 2026
in closed models when applicable, preceded the prompt with the following generic system message:
You are helpful assistant.
F EVALUATION MODELS
In our experiments, we fixed prompts and system messages across models whenever possible. For closed models, we followed the recommendations of model providers and the temperature whenever possible. For open models, we loaded them in full precision when possible or in 8-bit for larger models. All evaluation experiments were conducted using nodes with either 4 A100 GPUs or 4 H100 GPUs for open source models except for Deepseek R1 / Model details are listed listed in
Table 7
Model
Version
Type
Quantised
Claude Sonnet
claude-3-53 3-0----o-ttttt22221202222
Proprietary
Claude 3.7 Sonnet (thinking)
claude-3-7-sonne Saaudd=377-ommme222225000
Proprictary
Claude 3.7 Sonnet (no thinking)
dluud=347-smmeee22255199999
Proprietary
Claude Opus 4.1
claude-opus-441-2255000
Proprietary
DeepSeek- 13.1-Terminus
deepseck/deepsscck-racanne
Open-source
Gemini 1.5 Pro
gemini-1.5-pro
Proprietary
Gemini 2.5
gemini-22.-pmn-exx--02-22
Proprictary
GPT-4o
gp1-4o-2024-05-13
Proprietary
GPT-4.5
gp:44.5-previee22223300222
Proprietary
GPT-5
gpt-5-2025-08-07
Proprictary
Llama 3.3 70B
mataallamm//lamaa333-770-
Open-source
8-bit
ol-preview
ol-preview
Proprietary
03-mini
03-mini-2025-011 microsoft/phi-4
Proprictary
Phi4
Open-source
8-bit
Table 7: Details of models evaluated.
G SUMMARY OF RESPONSE ERRORS
Table & contains the summary of responses for all prompts by model. A higher number of responses by 03-mini (low) were empty due to generated reasoning tokens exceeding the maximum token budget but excluding empty responses by 03-mini (low) did not affect its ranking.
For Deepseek R1, we used http://www.tteeehssssiieeeeeeeet
22

---

Published as a conference paper at ICLR 2026
Model
Total
Empty Response
Bad Parsing
Claude 3.5 Sonnet
6,995
Claude 3.7 Sonnet (no thinking)
6,995
Claude 3.7 Sonnet (thinking)
6,995
22
Claude Opus 4.1
6,995
DeepSeek- -e.eek--..--TTmiiiii -e-rmminnus
6,995
115
GPT-4.5
6,995
GPT-4o
6,995
176
GPT-5
6,995
225
Gemini 1.5 Pro
6,995
16
Gemini 2.5 Pro
6,995
Llama 3.3 I0B-Instruct
6,995
316
Phi4
6,995
287
ol-preview
6,995
148
03-mini (high)
6,995
38
11
03-mini (low)
6,995
1,208
10
Table 8: Summary of model responses for benchmark prompts.
H FULL RESULTS
0.
Maer
07
0.4
02
0.0
Breaichrough
GP14
Clack Cree
0.9
033
Crane 5.7 Sueeet (yo (ang))
Chande S-Sont
029
GPTE 40
07
COMNL (00)
033
037
Foundation
Cente
Zel
541
Gertet
E21
E31
11
Intermediate
0:22
0.5
61
010
Advanoed
2.47
3.4
543
011
0.3
0.2
0-3
512
0.9
Round2
825
517
6:14
6:11
Figure 6: Main results on LINGOLY-TOO. (a) Scores by model. Mag is based on the original problems and Moby is based the obfuscated problems. Moty (Robust) is calculated after taking the worst score across all permutations of the question. (b) Breakdown of Moss scores by difficulty level.
23

---

Published as a conference paper at ICLR 2026
Models
Models
Claude 3,7 Sarmat 68-mirs mhight
Claude 3.7 Sonnel (ho thinking)
Claude 3.7 Sonnat (thinkin (thinking)
Claude 3.7 Claude Opus 4.1 Someet (thinking)
Ceming 2.5 Pro DeepSeek-V> V3 1.
DeepSeak-V22 1- 1-
Liana Semeee 3.3 70B Instrutt
Liane 3.3 70B- 70B-instion insrrutt
Claude 3 thinkinsn)
Gemini Pro
Claude Opus 1.1
Gemini
Gemini 2.5 Pro
03 mini (low)
03-min
03-mini preview
preeees Terming
af TTrmiius
GPT-4o
GPT-5
GPT44 (low)
GPT
( (high)
GPT:5
Phi
Pro
Diffrrmnce oxaNEE matthh due to cofuscation problem
Exant matteh on origiial problems
-0.5
0.0
0.5
0.0
0.5
Mag
Figure 7: Heatmap of results on each problem for each model. Right: the scores on the unob- fuscated problems. Left: The differences between the score the unobfuscated problems and the average the obfuscated versions. Red indicates lower performance after obfuscation.
24

---

Published as a conference paper at ICLR 2026
1
SCORE DISTRIBUTION DETAILS
Claude 3.5 Scnnet
Clande 3.7 Sonnet (no thinking)
Claude 3.7 Sonnet (thinking)
Orighel Beotsrappol
Frequency
Claude Opus 4.1
DeepSeck-V33 13.rmmermsnus
GPT-4.5
Origkal Beotsrapoo
Froquescy
Bract
Exact March
Match
GPT-4o
GPT-5
Gemini 1.5 Pro
Origial
Bectenppe
Froquecyy
0.5
0.49
Exacl Match
Exact Match
Exact Match Phi4
Gemini 2.5 Pro
Llama 3.3 70B-Insrtttt
Origial
Bootsrappo
Frequncc
Exact Match
Exact March
Exact Match
ol-preview
03-mini (high)
03-mini (low)
Origial
Bootarappol
Frequecy
038
0.08
6.55
Exact Match
Exact March
Exact Match
Figure 8: Distribution of bootstrapped scores across 500 bootstrapped samples over 82 problems. Open source models are shown in orange while proprietary models are in blue. The performance on the original, unobfuscated problems consistently appear at the right tail.
25

---

Published as a conference paper at ICLR 2026
J
UKLO 2025 QUESTIONS
Model name
Unobfuscated
Obfuscated
Claude 3.7 Sonnet (thinking)
0.25
0.20
Claude 3.5 Sonnet
0.25
0.15
GPT 4o
0.16
0.09
o3-mini (high)
0.10
0.08
Table 9: Performance by model on unobfuscated obfuscated questions from the UKLO 2025 paper. Note that this paper was released after these models have been trained therefore the performance gap cannot be attributed training set contamination.
We were able to access, annotate, and evaluate a subset of models on problems from the UKLO 2025 papers that have not yet been published online by the time of the experiment and are of a higher difficulty level. Results in Table are comparable results the benchmark. The difference in scores illustrates that performance gap is not only due to memorising answers since these problems have not been included in models training. The gap also highlights the effectiveness of our permutation to control for prior knowledge even on unseen problems, which is the main aim of the benchmark.
K TOKENISATION EXPERIMENT
We conduct an experiment to explore if the impact on tokenisation due to permuting the graphemes of language explains the gap model performance.
K.1
SET-UP
We choose the Aya 23 35B model by Cohere (Aryabumi al. 2024), which was designed for multilingual applications (23 languages). The tokeniser of Aya 23 is based on the popular byte-pair encoding algorithm (BPE) (Gage) 1994). Since we report off-the-shelf model performance, we investigate changes in the performance we alter the tokenisation of the Problemese portion of the prompt.
When prompting this model, the Problemese sections are treated separately. In the first experiment, we break down the Problemese into unicode characters (with the NFD standardisation) and obtain separate tokenisation for each character that we then concatenate together. In the second experiment, we add dashes in in-between the characters of the Problemese force some separation in the interpretation of multiple characters. The rest of the question is tokenised in the standard way, and the problem context and the Problemese are then concatenated together (in the order that they appear in the question) before being fed into the model.
For this experiment, we used 10 obfuscations of all problems except Problem (which had long prompts taking up too much memory).
We limited the model outputs to four times the length of the correct response length. After cleaning, the number of incorrectly processed responses is 32 questions for the standard tokenisation, 289 questions for the input with dashes, and 292 questions for the single character tokenisation (out of a total of 10, 765 questions). This may partially account for the additional drop in the performance of the alternative tokenisations.
K.2 ADDITIONAL RESULTS
In Table 10 we see how the average model performance changes under alternative tokenisation for the unobfuscated problems. For the unobfuscated case (Table 10), the performance decreased for greater number of problems than for the obfuscated case (Table I1) This aligns with the greater drop in performance that observe in Table
Finally, we show how the exact match scores vary under alternative tokenisations in Figure by plotting (standard tokenisation score, alternative tokenisation score) for each problem. When the
26

---

Published as a conference paper at ICLR 2026
(a) Unobfuscated Dash
(b) Obfuscated Dash
0.4
Exact soore dash input.
0.8
0.6
0.2
0.4
0.1
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
0.0
0.1
0.2
0.3
0.4
(c) Unobfuscated
Character
(d) Obfuscated Character
0.4
Exact score character input
0.8
0.3
0.6
0.2
0.4
0.1
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
0.0
0.1
0.2
0.3
0.4
Exact score of standard input
Exact score of standard input
Figure 9: Problem-level comparison of exact match scores between standard and alternative tokenisation. The dashed line represents the threshold where the score remains unchanged after altering the tokenisation. Point above the dashed line indicate better performance with alternative tokenisation, while points below the line indicate worse performance. (a) The problems are unobfus- cated and we compare scores of standard tokenisation against dash tokenisation.
(b) The problems are obfuscated and we compare scores of standard tokenisation against dash tokenisation. (c) The problems are unobfuscated and we compare scores standard tokenisation against character-level tokenisation. (d) The problems obfuscated and compare scores of standard tokenisation against character-level tokenisation.
27

---

Published as a conference paper at ICLR 2026
Prompt type
Decreased
Unchanged
Increased
Dash
57
5
Character
58
Table 10: Performance change under alternative tokenisation for the unobfuscated problems. Score for each problem was averaged over all questions within the problem for the two alternative tokenisation methods (Dash and Character), then compared to the score obtained using standard tokenisation and categorised into Increased, Decreased, or Unchanged.
Prompt type
Decreased
Unchanged
Increased
Dash
43
25
Character
37
30
14
Table 11: Performance change under alternative tokenisation for obfuscated problems. For each alternative tokenisation, the score for each problem was averaged over all obfuscations then compared to the score obtained using standard tokenisation and categorised into Increased, Decreased, or Unchanged.
problems are not obfuscated (Figure I(a) and (c)), the alternative tokenisation more frequently reduced the score to zero, indicating that the tokenisation of the original order may be capturing useful information. For the obfuscated cases (Figure (b) and (d)), there appeared to be more spread in the scores differences, and no instances the catastrophic effect of changing from a high exact match under the standard tokenisation to score of zero.
K.3 SUMMARY
We explored whether the drop in performance could be explained by tokenisation of rare character sequences rather than reduced access to prior knowledge. Using multilingual model (Aya 23 35B) with BPE tokenisation, we compared standard tokenisation to (i) inserting dash between every character in Problemese and (ii) forcing single-character tokens. The exact-match score did not improve under either alternative: unobfuscated problems 0.087 against 0.051 or 0.053, and on obfuscated questions 0.050 compared with 0.045 or 0.035. This argues against tokenisation alone as the driver of the performance gap: for each type of tokenisation, there marked drop in the score after obfuscation.
Problem-level analyses show many problems, consistent with the view that original orthographies benefit from familiar segmentations while obfuscated orthographies remove these knowledge shorteuts. Taken together with "no-context" results, these findings support our interpretation that obfuscation of these linguistic problems mitigates predicting the answers from prior knowledge.
decreases than increases, especially unobfuscated
While is true that the tokenisation effect may not be the same across problem domains, tokenisation has also been shown to make a difference in other domains like arithmetic problems (Singh Strouse 2024). We expect that the direction of our finding to hold more generally: we introduce logically equivalent permutations and show that LLMs are not equivariant under these permutations in way that is unexplained by only tokenisation effects.
L HUMAN EVALUATION
Obfuscation does not change the reasoning steps required to solve the problem, and therefore it serves as a method to robustly test reasoning capabilities whilst mitigating memorisation bias. However, obfuscation may have an effect how humans solve the problems through means other than changing the reasoning process. To measure the effect of obfuscation on human performance, we carried out randomised controlled trial (RCT) with 172 human participants across 6 problems (all relatively easy). After controlling for random differences in problem frequency, obfuscation was found to be associated with 5.80 percentage points lower performance (p-value of 0.059). We speculate that
28

---

Published as a conference paper at ICLR 2026
this may be because the resulting character combinations unusual in naturally occurring languages, thus the problems are perceived as being harder.
L.1 METHODOLOGY
Experiment Design We used an RCT with 172 participants. Each participant was assigned to one of two groups: one group solving unobfuscated problems (86 people) and another solving obfuscated problems (86 people). Participants were then assigned one of six problems and given up to 45 minutes to complete the problem. They were required spend a minimum of 30 minutes on the problem before submitting their answers. Responses were scored using exact match with the answers. All data collection was carried out through a Qualtries survey.
Problem Inclusion Criteria and Design We used two criteria select problems. First, we only considered Breakthrough and Foundation-leve problems, designed to be taken by 10-14 year olds. Our intention was ensure participants had realistic chance of solving the problem, and thus the responses would provide better signal of the effect of obfuscation. Second, we only considered problems where the language was not available through Google Translate, since this would have enabled participants to easily solve the problems. Two exceptions to this rule were Karelian, which not available on Google Translate but was removed due to a high similarity with Finnish, and Ligurian, which is available on Google Translate without stress being indicated, which formed the basis of the problem. The selected six problems are below.
Language
Difficulty level
Problem
Original author(s)
Warlpiri
Breakthrough
207
Mary Laughren
Umbrian
Breakthrough
191
Michael Salter
Kabyle
Breakthrough
160
Kazune Sato, Simi Hellsten
Tariana
Foundation
210
Babette Verhoeven, Simi Hellsten
Ligurian
Foundation
147
Kevin Liang
Amele
Foundation
88
Babette Verhoeven
Table 12: Problems selected for the randomised controlled trial. All problems are Breakthrough and Foundation-lewe designed to be taken by 10-14 year olds. None of these languages are available on Google translate in manner that would aid the solvability of the problem. Problem is the listing of the problem in the UKLO past problems database (United Kingdom Linguistics Olympiad! 2023).
Since all six problems and their solutions are easily accessible on the UKLO website, we rewrote elements of each problem. First, we rephrased the problem instructions so that quick internet search did not return the solutions. Second, we changed some of the lexemes in the Problemese such that the linguistic rules required to answer the questions were identical, but the specific words to translate were different. For example, in Kabyle we substituted the verb azzel 'to run' with aker 'to steal', which follows the same conjugation pattern. All problems were minimally rewritten except from Umbrian, where the known vocabulary originates exclusively from inscriptions the Iguvine Tablets and thus is extremely small. Additionally, as the LLM experiments, all background and cultural information
removed, and the language name replaced by 'Language X'.
For each problem, we randomly sampled two permutations. Participants who were assigned to the obfuscated group were randomly assigned one of the obfuscated versions.
Participant Inclusion Criteria Participants were recruited online using Prolific, a crowdworker platform. First, all participants had to be English speaking and monolingual, reducing the risk that they may have had prior exposure to the Problemese language. Second, all participants were required to have a minimum education level of an undergraduate degree. This served as a proxy for ability and was chosen to ensure participants could reasonably complete the problems.
Participants were compensated line with the UK aged-21 and over minimum wage of f11.44 per hour (experiments conducted January 2025). Additionally, participants received performance bonuses of f2.00 if they scored above 50% but less than 100%. and 66.00 if they scored 100%. The incentive structure was explained to participants before they started the problem.
29

---

Published as a conference paper at ICLR 2026
40
Group means
Obruscated
Froquency
60
30
Unobruscated
Correlation: D.006
40
20
10
100
10
Score
Self-reported difficulty
Figure 10: Distributions of participant scores and self-reported difficulty. Left: Distribution of scores across all six problems and problem types. Scores are highly skewed towards The original group had a higher mean score compared to the permutated group. Right: Participant self-reported difficulty. Participants were asked the question 'On a scale from 0 (very easy) to 10 (impossible), how challenging did you find this problem?'. Responses are highly skewed towards 10 (impossible). The permutated and original means are near identical. Despite the symmetric distributions, correlation between score and self- self-reported difficulty is 0.006.
All ICML Publication Ethics protocols were followed and the experiment had prior approval by the research ethics committee at our institution. All participants were required to provide informed consent by signing the form shown Section L..5
To control for the possibility that participants search for the solutions, each question was presented in image form, making it harder to directly copy and paste questions into an internet browser. Additionally, we added a post-response trap question asking 'What do you think the language the problem (Language X) was?'. Given the obscurity of the chosen languages, we assumed a correct answer to this question implied that participants had either found the solutions or were using a language model, both of which were explicitly prohibited in the instructions.
Controlling for Internet and LLM Use
To mark responses to the trap question, we first automatically identified entries which identified the correct language. Next, two authors with significant language model experience independently identified all responses which qualitatively appeared AI generated. Evidence suggests that people who frequently use state-of-the-att LLMs are good detectors of AI generated text, often outperforming commercial detection software (Russell et al. 2025). These included responses containing common language model phrases, excessively long responses and those demonstrating unusually impressive knowledge of academic linguistics, such as claiming the Problemese was likely an agglutinative language. Each author independently marked responses, then discussed cases where they disagreed. Since suspected cheating in the trap question does not necessarily imply that the participant cheated in the main problem, these participants were left in the data for the main analysis. Results excluding these participants are reported as a robustness check.
L.2 RESULTS AND ANALYSIS
Overall Performance Participants found these problems hard, with mean performance across all problems at 18.85% (mean random guessing is 9.19%). Responses to the question 'On a scale from 0 (very easy) to (impossible), how challenging did you find this problem?' reveal that many participants found the problems near impossible, with 39.5% reporting difficulty of 9 or 10 out of 10. There no significant difference in the distributions of self-reported difficulty across the participants taking obfuscated and unobfuscated problems (Figure 10).
Table [3] shows the mean score by problem and obfuscation type for both human participants and the LLMs. The mean score for human participants solving the unobfuscated and obfuscated problems were 21.70% and 16.00% respectively. To assess the statistical significance of this change, we used a one-sided Mann-Whitney Whitney U test. This was chosen due to the skew in the participant scores (Figure Left). The data rejects the null hypothesis that the distribution of scores under each problem type are equal with p-vuuue of 0.041.
In addition to the one-sided Mann-Whitney U test, we employed a linear regression with setproosseedatiiiiyyyoossstseee standard emrors. This accounted for the fluctuations in problem fre-
30

---

Published as a conference paper at ICLR 2026
Human
LLM
Problem
Original
Obfuscated
Delta
Original
Obfuscated
Delta
Random
Warlpiri
39.88 (n=14)
44.64 (m=14)
+4.76
46.67
27.66
-19.01
20.83
Umbrian
18 18.67 (n=15)
11.54 (m=13)
-7.13
30.00
18.67
-11.33
0.00
Kabyle
27.69 (n=13)
12.00 (m=15)
-15.69
24.00
19.62
-4.38
5.00
Tariana
7.14 (n=14)
0.00 (n=14)
-7.14
14.07
4.13
-9.95
0.00
Ligurian
22.86 (n=15)
24.76 (n=15)
+1.90
44.76
16.94
-27.82
28.57
Amele
15.00 (n=15)
3.33 (n=15)
-11.67
7.50
2.98
-4.52
0.00
All
21.70
16.00
-5.70
27.83
15.00
-12.84
9.19
Table 13: Human and LLM performance. Scores represent the mean score. Sample size is shown in brackets for human performance. LLM scores are calculated as the mean score over all LLMs in the study and all permutations. Random represents the expected score from random answers, given the basic formatting instructions in each question followed.
Model 1
(SE)
Model 2 (SE)
Model 3
(SE)
Intercept
12.063
(3.59)
12.765
(3.81)
22.723
(5.63)
Obfuscation
-5.796"
(3.07)
-5.005*
(3.27)
-5.100
(3.94)
Kabyle
10.326
(5.50)
12.395
(6.09)
12.939
(8.17)
Ligurian
14.643**
(5.22)
17.276
(5.85)
25.715
(6.10)
Tariana
-5.595
(3.16)
-5.938
(3.88)
10.610
(7.85)
Umbrian
5.983
(1.46)
8.018
(5.38)
4.971
(6.66)
Warlpiri
33.095
(5.27)
35.553
(5.15)
29.213
(6.35)
Observations
172
146
86
R²
0.292
0.326
0.275
Residual Std. Error
20.010
(df=165)
20.249
(df=139)
19.017
(df=79)
FF Statistic
14.126
(df=6; 165)
16.926
(df=6; 139)
7.239
(df=6; 79)
Table 14: Linear regressions for participant score under different inclusion criteria. The dependent variable is participant score in all models. The participant inclusion criteria as follows. Model I: All participants (n=172). Model 2: Excluding participants who we suspect may have cheated, those who returned more than 50% missing data, and those with mean ChrF score below 10 (n=146). Model 3; Excluding participants who scored below random guessing (n=86). SE: Heteroskedassicityyconmn consistent standard errors. df: Degrees of freedom. "p 11; "*p .05; *** < .01
quency in the human data. This analysis provides consistent results, suggesting obfuscation is associated with a 5.80 drop in score (p-value of 0.059) (Table 14
We also examine the performance of LLMs on the same problems, averaging their scores over all permutations. Models average 27.83% and 15.00% on unobfuscated and obfuscated problems, respectively, yielding mean drop of 12.84 percentage points. This approximately double the decline found in the human group, supporting the claim that models benefit from prior language exposure when completing the unobfuscated problems.
Accounting for Suspected Cheating Since participants took our survey remotely, they had the possibility cheating. To account for this, our survey contained trap question designed to establish whether participants had discovered the solutions online or used an LLM. Responses to the trap questions were first automatically scored based on whether they correctly identified the Problemese language, then were evaluated qualitatively and independently by two of this paper's authors. The authors identified 15 and 17 responses, 14 of which were common, and, after discussion, it was decided that all 18 responses should be removed to carry out robustness checks. In addition, participants who had more than 50% missing data, and those with a mean chrF score below 10 (visibly irrelevant answers, e.g. inputting random letters) were also removed. Table [4] Model shows the results excluding these participant groups. The effect on the point estimate and standard emorr of obfuscation is negligible.
31

---

Published as a conference paper at ICLR 2026
Model 4
(SE)
Intercept
0.068
(0.40)
Obfuscation
-0.689*
(0.34)
Kabyle
0.597
(0.52)
Ligurian
0.139
(0.53)
Tariana
-1.890**
(0.70)
Umbrian
0.699
(0.54)
Warlpiri
1.840"*
(0.63)
Observations 172
Pscudo R
0.163
Table 15: A logistic regression predicting above random guessing score. The dependent variable is a binary variable indicating whether participant's score was above random. Obfuscated problems are more likely lead below random guessing performance with a coefficient significant the 5% level. SE: fiteerooskdaaiicctycconsssiieiei standard errors. df: Degrees of freedom. "p < .1; ""p 005; ***p " .011
Effect when Only Considering those Scoring Above Random We also consider whether the effect of obfuscation remains consistent when only include participants who scored above random, iec who likely understood some of the problem.
First, the likelihood of scoring above random depends on the problem type. In the group taking the unobfuscated problems, 57.0% of participants scored above random. In the group taking the obfuscated problems, only 43% scored above random. Furthermore, logistic regression predicting above random performance, suggests obfuscation is significant factor with p-value of 0.045 (Table 15).
Second, after excluding individuals who scored below random guessing, the data can no longer be treated as the outcomes of an RCT, thus non-stratified tests such as the Mann-Whitme U are no longer appropriate. A linear regression controlling for differences in problem frequency suggests that the coefficient on obfuscation is no longer significant in this subset (p-value of 0. 0.196), though the point estimate (-5.100) remains similar and this result could be an artifact of the small sample size.
Data Availability An anonymised version of the dataset and a notebook to generate results are available in the GitHub repository.
L.3 EXTERNAL AUDIT OF PROBLEMS
To verify that the problems selected for the RCT remained solvable when rewritten and obfuscated, we asked two International Linguistics Olympiad (IOL) medallists to audit one set of obfuscations each. Both auditors were external to this research project and acted independently. Both had prior familiarity with the original UKLO problems.
The auditors confirmed that all problems remained solvable via the same reasoning steps and were significantly harder under the orthographies. One auditor commented that problems appeared to be marginally harder since the languages appeared more 'unfamiliar' and the other commented that they did not believe made any difference. Both also mentioned that they personally found the problems harder since they knew the original orthographies and thus found the new versions 'disorienting'. however this is not issue that we would expect novice test takers experience.
L.4 DISCUSSION
The results suggest that obfuscation causes small but statistically significant drop in performance when taken by inexperienced test takers. This is despite the required reasoning steps and problem solvability remaining unchanged. The point estimate of the effect is -5.80 percentage points: approximately equivalent to half a question per problem. If LLMs behave similarly to participants in this study, then we may expect some performance decrease have prior exposure to the original Problemese language. Therefore, performance drops between unobfuscated and obfuscated problems should not be solely attributed to language exposure. However,
in cases where the model did not
32

---

Published as a conference paper at ICLR 2026
estimates for the impact of obfuscation are small; thus would not expect large changes in performance given robust reasoning skills.
We do not have strong evidence for why obfuscation appears to make problems harder. The external audit confirmed that the problems remained solvable via the same reasoning steps, thus any changes to the problem diffieulty are superficial only. We speculate that obfuscation may cause the orthography to appear less familiar or naturalistic relative to English, giving the perception that the problems are harder solve, thus causing participants to exert less effort (Scasserra) 2008). Whilst the mean self- reported difficulty level is marginally higher in the obfuscated group (7.39 versus 7.48, Figure Top, the distributions are not significantly different under Mann- Whitney U test. However, we do find that participants solving obfuscated problems spent less time on the problem before submitting their answers (p 0.071 under one-sided Mann-Whitey U test), which would support this hypothesis.
Alternatively, knowledge of English alone may have been sufficient to score marginally higher in the unobfuscated versions of some problems. In partieular, Ligurian and Umbrian are both European languages descending from Latin. In Ligurinn, this would have increased the familiarity of the language, e.g. vió:vet:a, 'violer' or pásta, 'pasta', but would have offered no assistance when solving the reasoning problem. In Umbrian, knowledge of English may have directly helped, especially the Latin translation were also provided (under the label of 'Language Y'). For example, one question asked for translation of the Language Y (Latin) words populum and urbs into both Language X (Umbrian) and English. Knowledge of the English words population and urban may have helped the participants locate the correct translations, community and town, from the text provided. While this may have offered marginal assistance in unobfuscated problems, the overall results are largely unchanged when Umbrian is excluded.
Limitations These results are subject to several limitations. First, participants generally found these problems extremely challenging with only 50% of participants scoring above random. The significance of the results is partially driven by differences in scoring below random and it is unclear whether they would hold under different distributions of scores. As with all human evaluations, the results are highly specific to our demographic of participants (inexperienced test takers) and may not generalise outside of this.
Second, we specifically chose relatively easy problems so that participants had realistic chance of solving them. As a result, these results may not be indicative of the effect of obfuscation across the full distribution of problems.
Third, despite our best efforts to prevent and identify cheating, there a risk these results are driven by the availability of online materials. This is problem with any form of online and unproctored assessment, which future studies might wish to address through in-person test taking.
L.5 INFORMED CONSENT
All participants were required to consent to participation in the study by signing the form below.
Informed consent
You are being invited to take part in a research project led by the University of Oxford. Before you decide whether to participate, it important for you understand why the research being done and what it will involve. The University of Oxford supports the practice of protecting human participants in research. Please take time to read the following information carefully and discuss it with others if you wish. We appreciate your interest in participating in this online task.
What is this study?
This research aims to measure how well people can solve linguistic reasoning problems. A linguistic reasoning problem is set of questions that requires problem-takers to study new language that they have likely never seen before. The aim of the problem to use example translations between English and the second language decipher how this language works. You will then have to apply your deciphered knowledge to translate new words and phrases. In this research study, the second language may be written in non-standard way.
33

---

Published as a conference paper at ICLR 2026
At the start of the survey, you will be provided with further details about the specific problem. These details contain all information required to solve the task and no background knowledge of languages other than English required.
When you start the survey, you will be given some basic instructions, then will click through to the main problem. You will have up 45:00 minutes solve the question and write down your answers. There is countdown on the page which shows time remaining, After completing the problem, there is series of fo questions about your experience that will take around minutes to complete. We will collect your responses to the reasoning problem and follow-up questions.
How will be compensated?
Compensation for this task is 27.63. In addition to the base compensation, this study has bonus system based on performance. Scoring 50% or higher will result in a f2.00 bonus (E9.63 total wage). Scoring 100% will result in a $6.00 bonus (f13.63 total wage). Please note that bonus payments will not be made automatically and may take up to a week to reach your Prolific account.
Further information and FAQs.
You have been invited to participate as you are over the age of 18. Please read through this information before agreeing to participate (if you wish to).
You may ask any questions before deciding to take part by contacting the researcher (details who a DPhil student
below). The research contact is Harry Mayne the Oxford Internet Institute the University of Oxford. This project is being completed under the supervision of Dr Adam Mahdi.
Do I have to take part?
No. Please note that participation is voluntary. If you do decide to take part, you may withdraw at any point for any reason before submitting your answers by pressing the 'Exit' button or closing the browser.
How will my data be used?
We will not collect any data that could directly identify you. Your IP address will not be stored. We will take all reasonable measures to ensure that data remains confidential.
The responses you provide will be stored a password prottcctdd electronic file on the University of Oxford's secure servers and may be used in academic publications, conference presentations, reports or websites. Research data with Prolific IDs will be stored internally for three years after publication public release of the work of the research. De-identified research data, without Prolific IDs, may be publicly released and therefore in the public domain.
The data that we collect from you may be transferred to, stored and/ or processed at a destination outside the UK and the European Economic Area. By submitting your personal data, you agree to this transfer. storing or processing. The results may be written in partial fulfilment of the requirements for DPhil degree.
Who will have access to my data?
The University of Oxford is the data controller with respect to your personal data and, as such, will determine how your personal data is used in the study. The University will process your personal data for the purpose of the research outlined above. Research is task that we perform
34

---

Published as a conference paper at ICLR 2026
in the public interest. Further information about your rights with respect to your personal data is available from https://oompliance.ddmi hwtp:/iiinnaaaamin..mmmm wwt.miinox..c.ukiie. ii/i
The data you provide may be shared with the researchers this project, Harry Mayne, Dr Adam Mahdi and any other author involved in the publication the research.
We would also like your permission to use the data in future studies and to share data with other researchers (e.g. in online databases). Data will be de- identifiiee (Prolific IDs removed) before shared with other researchers results are made public.
Can I withdraw my data?
Yes, your data can withdrawn up until 11:59 PM, May 2025. To withdraw your data please contact Harry Mayne (harry.may marry.mmmneeeee @oii aePoi.ox.ace ox.oc.uk)) or Prolific. Your participation in this study is entirely voluntary, and you have the right to withdraw any time before the deadline without penalty or negative consequences. If you choose to withdraw, any data collected from you will be deleted and not included in the final analysis.
Who has reviewed this study?
This project has been reviewed by, and received ethics clearance through subcommittee of the University of Oxford Central University Research Ethics Committee [Application 1037300J.
Who do contact if have concern or wish to complain?
If you have concern about any aspect of this study, please speak to Harry Mayne (harry.mayne ax.a.kk or his supervisor, Dr Adam Mahdi aadam.mahdis @ow..oo........kkkeo and we will do our best to answer your query. We will acknowledge your concern within 10 working days and give you an indication how it will be dealt with. If you remain unhappy or wish to make formal complaint, please contact the Chair of the Research Ethics Committee at the University of Oxford who will seek to resolve the matter as soon as possible:
Social Sciences & Humanities Interdivisional Research Ethics Committee; Email: ethics@ corss@@soccc....ee uk: Address: Research Services, University of Oxford, Boundary Brook House, Churchill Drive, Hcadington, Oxford OX3 7GB
Please note that you may only participate in this survey if you are 18 years of age older.
I confirm that:
have had the opportunity to ask questions and receive satisfactory answers.
understand participation in this study voluntary.
understand that withdraw my data from the study before 11:59 PM, 1st May 2025 without giving reason negative consequences.
understand who will have access to personal data provided, how the data will be stored and what will happen the data at the end of the project.
understand how raise concern make complaint.
have read and understood the information on this sheet.
consent
do not consent
L.6 PARTICIPANT INSTRUCTIONS AND COMPENSATION
35

---

Published as a conference paper at ICLR 2026
General Instructions
Welcome to this reasoning challenge.
When you click forward to the next page, you will be presented with linglustic reasoning problem. The problem concerns some unknown language called 'Language x, which might be language with very few or remaining speakers, and may be written in a non-standard way.
You will receive background information on Language along with example English translations. Your task to decipher how the language works and translate new examples, You are likely to need a pen and paper to work this out.
You have up 45 minutes to complete this task. Your remaining time will be displayed at the top of the page.
You must spend least minutes on task; you will be able proceed until this time has elapsed.
Please only submit your answers when you are confident as you be able to go back.
Good luck!
Figure 11: Participant general instructions. All participants who consented to the experiment were provided with these instructions.
Bonus Structure
Compensation for this task is based the following bonus
structure, where 'Score' is your final score the reasoning problem.
Score
Bonus
Total Wage
0%+
E7.63
50%+
f2.00
29.63
100%
26.00
213.63
As mentioned in the coneent form, bonue payments will not be madi automatically and may take up to week to reach your Prolific occount.
Figure 12: Participant compensation. Compensation was based on a bonus structure to incentivise effort. Participants guaranteed least the UK minimum wage.
36

---

Published as a conference paper at ICLR 2026
M DATASET METADATA
Below are details about the benchmark dataset following the Hugging naFa template.
license: ce-by-nc-nd-4.0
task__ateep catk_caaeeoiees
question-answerre
tags:
reasoning
linguistics
benchmark
pretty_ L2
size_ =za_categories:
1K<n<10K
source_datasets:
configs:
config_name: default data_files:
split: test
path:
extra extra_gatee pated__poopts >-
### LingOly-T00 LICENSE AGREEMENT
The Lingoly-T00 dataset is distributed under a CC-BY-NC-ND 4.0 license.
All questions in the Lingoly-T00 dataset have been used with the permission of
the original authors. The original authors and the United Kingdom
Linguistics
Olympiad may retain rights to control the use, and users of this dataset will liability they use the dataset beyond the terms of indicated by the benchmark.
The authors with time.
take responsibility for any licenses that change
In addition to this license, we ask that uses of the dataset are in line with the Acceptable Use policy described below.
### Acceptable Use Pelicy
This dataset is exclusively intended benchmark for evaluating language
models subject to the terms of the license. For the integrity the benchmark, users should not:
Re-distribute the questions or answers of the benchmark in formats (such plain text) which leak the benchrark to web-scraping. Train language models directly using the content this benchmark. iitrr__gated_fieee
By clicking Submit belew I accept the terms of the license and Acceptable Use policy: checkbox extra_gaatebuuuttooooonttt submit
37


Reviews:
[{"review_id": "P8PWulVdu2", "reviewer": "Reviewer_eZVp", "summary": "This paper introduces LingOly-TOO, a challenging reasoning benchmark that obfuscate Linguistics Olympiad Problems to avoid advantaging models that are using shortcuts such as memorisation and knowledge. The obfuscations preserve the underlying solution logic while removing orthographic clues that could trigger patterns from memorisation or knowledge. Without surprise, the performance of models drastically decrease.\n\nThe authors defined knowledge as information stored in model parameters after training, which captures linguistic, factual, and commonsense patterns useful for downstream tasks, and memorisation as when models exploit contaminated datasets, reporting answers previously seen in training\n\nThe authors adapted 82 problems from the UKLO, and obfuscate the problem to avoid models relying on linguistic patterns. More specifically, the authors manually created a ruleset for each problem to generate valid permutations of targeted tokens. They apply extra-care to keep names of people, sacred places, etc intact. Overall, the authors generate 6 valid permutations per problem and generate obfuscated versions by altering the problem text. Overall, there are 6995 question-answer pairs.\n\nThe authors propose two metrics: the average exact match score across all questions in all permutations and the average exact match across all questions in the unpermuted problem.\n\nIn terms of experiments, the authors evaluate 15 reasoning models. The performance varies between the original and obfuscated variants, with GPT and Claude being the most robust models. The authors do a detailed analysis to measure the gap between reasoning and knowledge. It would be great to add a case study to support the findings. In terms of metrics, it would be good to compare human performance on the task and include more details (e.g., how do they compare with LLMs) - I'm aware of Appendix L but would like to see more. Moreover, it would be fairer to assess the performance of models with pass@k.\n\nOverall, I am lukewarm about this paper. Creating obfuscating variants of problems does not seem to relate to real-life tasks, even for measuring reasoning. I have the feeling that this benchmark is focusing on a unrealistic problem / not a real problem. I appreciate the analysis and experiments of the authors, and the paper is very well written and structured.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Various models are used in the benchmark.\n- Detailed analysis.", "weaknesses": "- I have the feeling that this benchmark focuses on an unrealistic problem / does not represent real life tasks. We cannot expect LLMs to necessarily perform good on those.\n- Please add human evaluation + pass@k metrics", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "681ofGtCMr", "reviewer": "Reviewer_SdFn", "summary": "This paper introduces a challenging reasoning benchmark of 7k question answer pairs, built by applying grammeme-level obfuscations to Linguistic Olympiad problems.  The motivation for building a new dataset is that models rely on prior language knowledge learnt via pre-training. This dataset is built to test out models' reasoning capabilities and removes any cues that could trigger memorized translations.", "questions": "The authors manually created rulesets for this dataset, which makes it harder to extend this work to other datasets. Did the authors try to use LLMs as annotators to see how feasible it is to extend them to other domains/ datasets?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Release of a large-sized (7k) benchmark dataset that clearly separates the reasoning abilities from memorized knowledge.\n2. In-depth analysis: The authors conducted various analyses, such as the ability of the model to reason, the effect of tokenization on uncommon characters, and the various across different permutations. \n3. Release of dataset and code", "weaknesses": "1. In the no-context setting, the difference between the original and the obfuscated dataset is very less; how does the author come to a concrete conclusion?\n2. To check the effect of performance drop with uncommon characters, why don't we replace the context with random but real tokens that are not part of the training set? Similar to the ProntoQA dataset, where entities are replaced with false ontology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TyzRzZiaWw", "reviewer": "Reviewer_cqTp", "summary": "This paper introduces LingOLY-TOO, a new benchmark for evaluating reasoning abilities in LLMs. It is built from Linguistics Olympiad problems. The key innovation is the use of \"reasoning-equivariant permutations\" to obfuscate the problem text. This process changes the orthography but is carefully designed to keep the underlying logical structure and solution steps unchanged. The authors show that while models perform reasonably well on the original problems, their performance drops significantly on the obfuscated versions. This suggests that models rely on prior knowledge and memorization rather than pure reasoning on the original tasks. The paper also includes detailed analysis on the effect of language resourcefulness and a human study.", "questions": "1. Have you considered any evaluation metrics other than exact match (e.g., edit distance or partial credit schemes) that could capture instances where the model's reasoning is mostly correct but the final output has a minor error? What are the potential challenges in implementing such metrics for this benchmark?\n\n2. The human study shows a 5.7% performance drop due to obfuscation. Could you discuss a bit more how we should interpret the model's performance drop in light of this? Specifically, how much of the model's drop might be attributed to the increased difficulty of processing the unfamiliar orthography, versus the removal of knowledge-based shortcuts?\n\n3. The permutation rulesets are designed by experts. Do you think this method could be applied to other domains like mathematical reasoning or code generation? What would be the main challenges in designing \"reasoning-equivariant permutations\" for those domains?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The core idea of the paper is excellent. Using orthographic obfuscation to create a \"knowledge-free\" test for reasoning is a very clever and direct way to tackle the problem of data contamination. This is a timely and important contribution.\n\nThe process for creating the permutations is very rigorous. I am impressed by the careful, manual design of the rulesets by experts to ensure the problems remain solvable. The validation by IOL medallists adds strong credibility to the method.\n\nThe experimental section is very comprehensive. The authors evaluated a wide range of models, including the latest reasoning-specific ones. The analysis goes beyond just overall scores to include \"no-context\" tests, robustness checks, and the correlation with language resources. The human study is also a valuable addition.", "weaknesses": "The use of exact match is simple but might be too strict. Sometimes, a model might have the correct reasoning but make a small mistake in formatting the final answer. Using only exact match could penalize such cases.\n\n\nThe human study shows a small but noticeable performance drop (5.7%) for humans on obfuscated problems. This suggests that the obfuscation itself might add some cognitive load, making the problems slightly harder to parse, even for humans who don't rely on prior knowledge of the language.\n\nThe process of creating permutation rulesets relies heavily on manual expert work. This might make it difficult to scale the benchmark to a much larger size or to adapt it quickly to other domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HGau02bEoT", "reviewer": "Reviewer_afLH", "summary": "The paper introduces LINGOLY-TOO, a 6,995-QA benchmark built by applying expert, grapheme-level orthographic obfuscations to 82 UKLO problems, aiming to disentangle reasoning from knowledge/memorization. The authors define clear metrics (Mog vs. Mobf, plus robust variants), run broad model evaluations, and provide validation via auditors and a human RCT. Results show sizable drops under obfuscation (e.g., top model ≈0.60→0.48), correlations with language resourcedness, and modest gains from guided reasoning.", "questions": "1. **On Failure Attribution (related to Weakness 1)**\n\n   Can the authors extend the analysis to *quantify* the causes of reasoning failure? For instance, by labeling dominant error types and reporting their prevalence across models and difficulty levels.\n\n2. **On Ruleset Comprehensibility (related to Weakness 2)**\n\n   While we appreciate the authors’ effort on the permutation design, could the paper include auxiliary tables or diagrams that summarize the rules and constraints at a glance, to help non-linguist researchers quickly understand how reasoning equivariance is enforced?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Clear problem framing: measuring reasoning when shortcuts (knowledge, contamination) are minimized is timely and important.\n* Methodological originality: the reasoning-equivariant, linguistically-informed permutations are thoughtful and non-trivial; the Turkish vowel harmony example nicely motivates rule design.\n* Strong experimental design: multiple families of models, bootstrap analysis, no-context control, tokenization controls, and human study make the case persuasive.", "weaknesses": "1. **Insufficient Failure-Mode Analysis**\n\n   The paper documents performance declines but does not deeply probe *why* models fail on obfuscated problems (e.g., difficulty inferring morphemic patterns, inconsistent multi-step reasoning, or fallback to guessing). Please add a qualitative analysis of model outputs—contrasting common errors on obfuscated vs. original items—to connect performance gaps to specific reasoning deficits. Building on this, apply statistical tools (e.g., clustering) to categorize and quantify linguistic reasoning failures.\n\n2. **Limited Accessibility of the Permutation Ruleset**\n\n   The permutation rules (Appendix B) are dense and lack a high-level summary in the main text. A concise overview—such as a table of key constraints and invariances, or a concept diagram illustrating how reasoning equivariance is preserved—would make the method more accessible, especially to readers without a linguistics background.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)