LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation

## Abstract

Abstract Assessing the reasoning capabilities of Large Language Models (LLMs) is susceptible to overestimation due to data exposure of evaluation benchmarks. We introduce a framework for producing linguistic reasoning problems that reduces the effect of memorisation in model performance estimates and apply this framework to develop LingOly-TOO , a challenging benchmark for linguistic reasoning. By developing orthographic templates, we dynamically obfuscate the writing systems of real languages to generate numerous question variations. These variations preserve the reasoning steps required for each solution while reducing the likelihood of specific problem instances appearing in model training data. Our experiments demonstrate that frontier models, including Claud 3.7 Sonnet, o1-preview and DeepSeek R1, struggle with advanced reasoning. Our analysis also shows that LLMs exhibit noticeable variance in accuracy across permutations of the same problem, and on average perform better on questions appearing in their original orthography. Our findings highlight the opaque nature of response generation in LLMs and provide evidence that prior data exposure contributes to overestimating the reasoning capabilities of frontier models. 1 1 1 Code and data: https://huggingface.co/spaces/jkhouja/lingoly-too

## 1 Introduction

Recent advancements in Large Language Models (LLMs) have resulted in impressive performance on reasoning benchmarks (Kojima et al., 2022; OpenAI et al., 2024), achieving strong results on mathematical word problems (Cobbe et al., 2021), competition mathematics (Hendrycks et al., 2021) and various symbolic and algebraic reasoning tasks (Wei et al., 2022). However, the claim that LLMs can truly reason draws scepticism (Kambhampati, 2024; Mirzadeh et al., 2024; Jiang et al., 2024), in part because the scientific validity of benchmark findings is controversial. Cognitive agents can be said to reason when they apply abstract rules to prior information to obtain novel judgements or decisions (Koralus, 2022; Huang & Chang, 2023; Lampinen et al., 2024). Such rules may take the form of logical arguments or mathematical equivalences as well as more informal `commonsense' reasoning. Specific benchmark datasets can only partially represent general-purpose capabilities and benchmark evaluations in general struggle with construct validity, that is, measuring what they intend to measure (Raji et al., 2021).

For a benchmark to measure reasoning, reasoning must be both a necessary and a sufficient condition for high performance. However, it is difficult to establish the necessity of reasoning for a task when models may have prior exposure to similar or identical tasks in pre-training and can rely on a form of memorisation in solving the task. Several popular benchmark datasets are fully or partially exposed in pre-training, impeding their effectiveness in performance estimation (Li & Flanigan, 2024; Zhou et al., 2023). Taken together, these challenges raise questions around the mechanisms behind LLMs' improved performance in reasoning tasks and the genuine abilities of frontier models.

A number of evaluation techniques can reduce the potential effect of memorisation. Prior works have used out-of-distribution tasks, such as reasoning about low-resource languages to better approximate the reasoning abilities of LLMs (McCoy et al., 2023; Tanzer et al., 2024; Bean et al., 2024; Sánchez et al., 2024). Others have used synthetic templates to assess LLMs' robustness to minor perturbations of popular benchmark problems (Jiang et al., 2024; Mirzadeh et al., 2024), or reported no context baselines (Bean et al., 2024) to identify instances when LLMs draw on information stored in their parameters and not presented in-context.

We combine these two approaches, beginning from linguistic reasoning tasks and applying orthographic(^2^22For a glossary of terms, see Appendix [A](#A1).) perturbations to ensure that the problems are out-of-distribution. Linguistic reasoning is a subset of general reasoning and has been previously assessed in LLMs via Linguistics Olympiad problems (Bean et al., 2024; Sánchez et al., 2024; Şahin et al., 2020). Typical problems require a combination of deductive, abductive and analogical reasoning for the purpose of understanding some aspect of the grammar of an unknown language. This domain offers a parallel to mathematical reasoning, requiring similar generalised reasoning capabilities in a less saturated context.

In this paper, we introduce a framework for the dynamic generation of obfuscated evaluation benchmarks for linguistic reasoning. By creating problems outside the distribution of LLM training sets, we aim to mitigate the effects of data exposure in assessing reasoning abilities of LLMs and enable scalable and controllable experiments. We apply this framework to extend LingOly, a linguistic reasoning benchmark utilising problems from the UK Linguistics Olympiad (UKLO) in over 90 low-resource languages (Bean et al., 2024). LingOly-TOO is a challenging reasoning benchmark that sheds light on current limits and gaps in the capabilities of state-of-the-art models in solving linguistics problems.

Our main contributions are: (i) an orthography-based templatisation method for generating unseen linguistics problems based on natural human languages; (ii) LingOly-TOO, a benchmark for evaluating the linguistic reasoning abilities of LLMs; and (iii) an empirical evaluation of reasoning consistency in frontier models on unseen data.

## 2 The LingOly-TOO Benchmark

LingOly-TOO consists of $27,325$ linguistic
reasoning questions, which we dynamically generated through an obfuscation process that we explain in the following sections. To minimise the risk of contamination in pre-training, we used orthographic rulesets to obfuscate linguistics problems. These problems are drawn from UKLO past problems, which were standardised in the LingOly benchmark (Bean et al., 2024). We reused and adapted 82 of these standardised problem sheets.

A typical single problem in a Linguistics Olympiad provides examples of words or phrases in a language of focus (termed Problemese) alongside their translations into the language in which the problem is written (termed Solverese) (Bozhanov & Derzhanski, 2013). Each problem includes multiple questions and answers. Through logical reasoning, the solver is expected to deduce the underlying linguistic rules and structures and use them to determine a single best answer to each question. Problem authors, who are volunteer linguists, research the Problemese languages before including them. We obtained permission from individual authors to include their work in this benchmark and discuss permissions from language communities in Section [Impact Statement](#Sx1).

Although these problems heavily feature low-resource languages, previous work has shown that examples of these languages have appeared in pre-training datasets, potentially allowing models to predict the answers without reasoning ability (Bean et al., 2024; Tanzer et al., 2024). To address this, we construct orthographic variations that obfuscate the original languages without changing the reasoning steps of the problems. This approach allows us to ensure that models have not memorised the correct responses, making the benchmark a more robust measure of reasoning ability. Figure [1](#S0.F1) provides an overview of our framework. In the rest of this section, we explain orthographic templatisation and describe the annotation details and process of generating LingOly-TOO benchmark.

### 2.1 Orthographic Templatisation

In Linguistics Olympiads, obfuscation refers to any process by which a problem is modified to prevent cheating–in most cases, by preventing internet searches (Asia Pacific Linguistics Olympiad, 2024; United Kingdom Linguistics Olympiad, 2021). It typically consists of removing the language name(s) and other metadata that could identify or point towards it (e.g. language family, number of speakers, area where it is spoken).

For each problem, we created a ruleset to generate acceptable permutations of the Problemese graphemes present in the data. To ensure that the problems' solvability is not affected, acceptable permutations preserve phonetic and phonological distinctions relevant to solving the problem, as well as any loanwords or transparent English cognates that might aid in solving it. Names of people or deities are also preserved in the obfuscation process.

### 2.2 Annotation

To prepare the text for obfuscation, we manually annotated $82$ problems with special tags for later processing. We removed metadata such as language names, families and geographic regions that have no impact on solvability of problems. Annotations were performed manually and subsequently validated by two members of the team with expertise in Linguistics Olympiads. We also incorporated automated checks to validate the outputs. A detailed description of the annotation process, including examples, is provided in Appendix [B](#A2).

### 2.3 Data Generation

Starting with the specific orthographic template for each problem, we randomly sampled permutations of the Problemese graphemes according to the rules of the template. We prevented identity mappings of the graphemes unless explicitly specified. We then obfuscated the problem by substituting the graphemes according to these mappings, and removed any special tokens used for annotations. We conducted a second round of both human-expert manual inspection and automatic validation tests to ensure the correctness of the final data. Most problems permit far more permutations than is necessary for robust evaluations. Due to resource limits, we generated $6$ obfuscations per problem to evaluate all models resulting in $27,325$ questions. On subsequent experiments involving a subset of the models, we generated a larger instance of the benchmark using $30$ obfuscations per problem.

Figure: Figure 2: LingOly-TOO benchmark main results. (a) Mean exact match scores by model on LingOly-TOO $M_{og}$ is on nonobfuscated problems and $M_{obf}$ is on obfuscated problems. (b) $\Delta_{obf}$ for the $6$ obfuscations of $57$ problems (for brevity), showing performance changes by model. Red indicates a performance drop for that particular obfuscation, while blue indicates an improvement. Results for all problems are in Appendix [H](#A8).
Refer to caption: /html/2503.02972/assets/x2.png

### 2.4 Evaluation

Metrics  We adopt exact match, a binary metric calculated for each question and answer in a problem. exact match takes the value $1$ if the response text matches the correct answer text and 0 otherwise. Our goal is to capture all failure cases, including those where the permutation of the correct answers was minimal, following the approach of Bean et al. (2024) and Sánchez et al. (2024). The nature of the tasks is such that the correct answer often involves making small changes to words already present in the problem, so scores that offer partial credit may reward repetition of in-context words rather than actual reasoning.

We define $M_{og}$ as the average exact match score across all questions in a single unobfuscated (original) problem and $M_{obf}$ as the average exact match score for all questions in a single obfuscated problem. We also define

$$ $\text{$\Delta_{obf}$}=\text{$M_{obf}$}-\text{$M_{og}$}$ $$

as the difference in exact match due to obfuscation. We note that $\Delta_{obf}$ is positive when the model improves on an obfuscated problem and negative when the model's performance decreases compared to the original problem. To evaluate a single model on the entire benchmark, we report the average score $M_{og}$ for the 82 original (unobfuscated) problems and $M_{obf}$ for all obfuscated versions of the 82 problems.

Our pipeline breaks down problems into individual questions and generates prompts for each question in a zero-shot setting. Each prompt consists of a preamble (detailing background information about the language), context (providing examples from the language), all questions in problem sheet and finally the specific question to be answered. The exact template used is provided in Appendix [D](#A4). Similar to Bean et al. (2024), we found that chain of thought prompting (Kojima et al., 2022; Wei et al., 2022) did not improve the performance (Kojima et al., 2022; Wei et al., 2022) and adopted a zero-shot setting for our experiment (see Appendix [F](#A6) for details ).

To focus on reasoning performance rather than instruction-following, we applied post-processing steps to recapture improperly formatted responses. After post-processing, about $0.2\%$ of the responses remained unreadable and were scored as incorrect. Blank answers, which constituted $0.8\%$ of responses from proprietary models and $5.4\%$ from open models, were also considered incorrect (see Appendix [G](#A7)).

Models  For our main experiment on the smaller benchmark, we evaluated eleven LLMs: Claude 3.7 Sonnet, Claude 3.5 Sonnet, Gemini 1.5 Pro, GPT-4.5, GPT-4o, o1-preview, o3-mini, DeepSeek R1, Phi-4, Llama 3.3 70B, and Aya 23 35B. For experiments on the larger benchmark, we evaluated eight models: Claude 3.5 Sonnet, Gemini 1.5 Pro, GPT-4o, Phi-4, Llama 3.3 70B, and Aya 23 35B, Llama 3.2 3B, Llama 3.2 1B. Further details about the models used in our experiments can be found inAppendix [E](#A5).

## 3 Results

### 3.1 Model Performance

Figure [2](#S2.F2) shows mean exact match scores for eight models on both the original and obfuscated problems. Claude 3.7 Sonnet, which achieved the highest score on the obfuscated problems, scored mean $M_{obf}$$=0.43$, followed by o1-preview ($0.32$) and o3-mini ($0.31$). The best open source model is DeepSeek R1 which scored $0.27$. All remaining models scored below $0.30$ indicating that the benchmark is challenging even for frontier models.

Compared to the problems in their original orthographies, all models drop in average performance on the obfuscated versions. Figure [2](#S2.F2) shows $\Delta_{obf}$ across all obfuscations of 57 problems of the 82 problems (full scores can be found in Appendix [H](#A8)). Problems about higher resource languages generally show the largest negative average $\Delta_{obf}$ across all models such as Problem 76 (Finnish): $-0.36$, Problem 178 (Italian): $-0.32$, Problem 144 (Dutch): $-0.21$. However, within each individual problem, not all obfuscations had the same impact. In cases such as Problem 24 (Tadaksahak) and Problem 67 (Navajo), different obfuscations had varying effects, and in some cases, even led to improved model performance. The distribution of scores for a sample of individual problems is found in Appendix [I](#A9).

Figure: Figure 3: Performance by problem difficulty level for o1-preview. The relative difficulty level of problems is largely preserved by the obfuscation process. Categories as defined by UKLO.
Refer to caption: /html/2503.02972/assets/x3.png

Figure: Figure 4: The effect of obfuscation is larger for high-resource languages. Scores for each problem are averaged across all eight models in Figure [2](#S2.F2). Active speaker numbers serves as a proxy for language resourcedness. Speaker numbers are transformed by $\log(x+1)$ to emphasise differences between low-resource languages.
Refer to caption: /html/2503.02972/assets/x4.png

We also find that the relative difficulty level of problems is largely maintained by the obfuscation process. UKLO categorises all problems into five difficulty levels. When we consider the strongest performing models, the easier problems (Breakthrough and Foundation) are the highest scoring problems both before and after obfuscation. Similarly, the most challenging problems (Round 2) remain the hardest. This is illustrated for OpenAI o1-preview in Figure [3](#S3.F3). However, this trend does not hold for the weaker models, potentially since all scores are nearer zero. Additional results are displayed in Appendix [I](#A9).

Figure: Figure 5: Score distribution across bootstrapped samples. Distribution of scores across 500 bootstrapped samples of our data by model. Each consists of 82 problems. Open source models are shown in orange while proprietary models are in blue.
Refer to caption: /html/2503.02972/assets/x5.png

**Table 1: Model performance in the no context setting. The table shows mean scores in the no context setting (with insufficient information to answer questions). Scores are categorised by answer type. `Other' includes all remaining questions which do not fit the other response types.**
| Model | Answer | $M_{og}$ | $M_{obf}$ |
| --- | --- | --- | --- |
| GPT-4o | Digit | 0.16 | 0.08 |
|  | Other | 0.07 | 0.01 |
|  | Single Char | 0.03 | 0.03 |
|  | Y/N | 0.62 | 0.31 |
| Llama-3.3-70B | Digit | 0.19 | 0.11 |
|  | Other | 0.02 | 0.01 |
|  | Single Char | 0.05 | 0.04 |
|  | Y/N | 0.19 | 0.49 |

### 3.2 Benchmark Validation

Contamination
To assess the impact of memorisation in the original problems, we generated a bootstrapped distribution of scores across all versions of each problem. We sampled $500$ sets, each containing one randomly selected version of each problem out of seven (including the original version). Figure [5](#S3.F5) shows the empirical distribution of scores for the models on the bootstrapped sets. The performance of models on the original set of problems, mean $M_{og}$, consistently appears at the extreme right of the distribution, indicating that models are disproportionately better at solving the problems in their original orthography. Since we have constructed the obfuscations to preserve the logic of the problems, we attribute this difference to prior exposure to the original orthographies allowing the models in many cases to complete the problems without reasoning.

We also evaluated LLMs' ability to score on our benchmark by guessing the answer with incomplete information present in context. By adopting the no context setting from Bean et al. (2024), we conduct a small experiment using our larger benchmark ($30$ obfuscations) with two models (Llama 3.3 70B and GPT-4o) due to resource constraints. We removed the context from the prompt, rendering the problems unsolvable by reasoning alone.

Table [1](#S3.T1) shows a summary of the results in the no context setting for both tested models. When information is insufficient to solve the problem, mean $M_{og}$ and $M_{obf}$ for both models drop ($M_{obf}$ in particular is low). For instance, GPT-4o achieves $M_{og}$$=0.08$ vs. $M_{obf}$$=0.02$. When excluding cases with higher chance of guessing correctly (answers which are Yes/No, single character or numerical), $M_{obf}$ drops to $\sim{0.01}$ for both models. This confirms that obfuscation effectively counteracts the effects of memorisation.

In addition, we investigated the effect of language resourcedness on model performance by approximating the former with number of speakers of the language (in logarithmic scale) of the problem. We apply linear regression with resourcedness as a predictor of the average score for the models grouped into open-source and closed-source. We repeat the process for $M_{og}$, $M_{obf}$ and $\Delta_{obf}$. While $M_{og}$ improves for languages with more speakers ($\beta=0.056$, $p<0.01$) for closed models and ($\beta=0.021$, $p<0.01$) for open models, $M_{obf}$ shows no clear relationship for the obfuscated problems ($\beta=-0.003$, $p=0.26$) for open models and ($\beta=0.002$, $p=0.726$) for closed models. The performance drop $\Delta_{obf}$ also widens for problems with more speakers ($\beta<0$, $p<0.01$ for both open-source and closed-source models). These results provide additional evidence of the effectiveness of obfuscation in substantially reducing the effects of exposure to high-resource languages. Figure [4](#S3.F4) shows scatter plots of scores for original and obfuscated problems averaged for eight models evaluated on the larger benchmark.

Impact of Obfuscation on Humans
To ensure that the reasoning element of the problems is preserved under obfuscation, we investigate the impact on human performance. We performed a randomised controlled trial (RCT) with $172$ participants recruited through Prolific. All participants were required to speak English, have no familiarity with other languages, and have at least an undergraduate degree. Participants were randomly assigned to two groups (unobfuscated and obfuscated), then presented with one of six problems, which were all chosen to be relatively easy to ensure novice test takers had a realistic chance of solving them. There were two possible permutations of each obfuscated problem.

The mean scores for those solving unobfuscated and obfuscated problems were $0.22$ and $0.16$, respectively. After fitting a linear regression controlling for random variation in problem frequency, obfuscation is associated with a $5.80$ percentage point decrease in performance ($p=0.059$). Furthermore, these results are supported by a one-sided Mann–Whitney U test, which rejects the null hypothesis that the distributions of scores are identical ($p=0.041$). This suggests obfuscation causes a small drop in scores when the problems are taken by human participants without prior linguistics experience.

In addition, the selected problems were externally audited by two International Linguistics Olympiad medallists, each given one set of obfuscations. The auditors confirmed that all problems remained solvable and that problem difficulty appeared consistent under obfuscation. The combined analysis implies that obfuscation does not change the reasoning steps required to solve the problem, but does cause a small drop in human performance. We speculate that this may be because the resulting orthographies seem less familiar or naturalistic relative to English, causing the problems to be perceived as more challenging, and participants to exert less effort (Scasserra, 2008). This hypothesis is weakly supported by the finding that participants solving obfuscated problems spent less time on problems before submitting their answers ($p=0.071$ under a one-sided Mann–Whitney U test).

If LLMs reason in a similar way to the participants in this RCT, we may expect small drops in performance under obfuscation, even in the absence of prior exposure to the Problemese language. However, we have no reason to expect large deterioration as indicated by the mean $\Delta_{obf}$ across models. Full methodological details, statistical analysis, and discussion are presented in Appendix [K](#A11).

Tokenisation
As tokenisers of LLMs were trained on languages in their original orthography, we explored whether tokenisation can explain the variance (and drop) in model performance. Byte Pair Encoding (Gage, 1994), a common tokenisation approach, takes into account transition frequency between characters. Since obfuscation produces less common sequences of characters, we hypothesise this may affect the tokenisation step. We select Aya-23-35B to explore alternate tokenisation strategies given it was explicitly trained for multilingualism. We compare three variations of tokenisation: standard tokenisation, introducing a dash between each character in the Problemese and tokenising each character separately. Since the problems span several low-resource languages which coule be tokenized sub-optimally, we suspect that the dense representations of the tokens are unlikely to be useful for solving the problems. These variations break up the tokenisation of the Problemese words to ensure that tokens do not cross semantically meaningful boundaries in the words which would make the problems more difficult.

**Table 2: Exact match results with varying input to the tokeniser. Performance of Aya-23-35B decreases when alternative tokenisation techniques are used, both on obfuscated and unobfuscated problems. `Dash' tokenisation inserts a dash between each character in the Problemese data. `Character' tokenisation forces the model to tokenise each character individually.**
| Prompt type | Unobfuscated | Obfuscated |
| --- | --- | --- |
| Standard | 0.087 | 0.050 |
| Dash | 0.051 | 0.045 |
| Character | 0.053 | 0.035 |

Table [2](#S3.T2) shows that enforcing separation of the tokens does not improve model performance. For both obfuscated and original versions, exact match score decreases if the tokenisation of Problemese is altered. When comparing at the question level, using alternative tokenisations improved performance in less than one fifth of the cases. This suggests that the standard tokenisation of the obfuscated problems is not making the problems more difficult or is the sole explanation of failed reasoning in the model. For more details see Appendix [J](#A10).

### 3.3 Impact of Data Exposure

To explore the potential impact of data contamination on model performance, we conducted a small study with our larger benchmark using LLama 3.2 1B and LLama 3.2 3B. We used a subset of the obfuscated problems as training data to run supervised fine-tuning on both models. In order to inspect the effects of contamination by the degree of exposure as well as the type of generalisation affected, we split the data into the following disjoint sets: (1) Training and validation datasets ($n\!=\!12,110$ for train and $n\!=\!3,054$ for validation); (2) Unseen questions and exposed obfuscations ($n\!=\!6,864$); (3) Unseen obfuscations ($n\!=\!7,926$); (4) Unseen obfuscations and questions ($n\!=\!2,100$); (5) Unseen problems ($n\!=\!20,537$).

**Table 3: Performance change after fine-tuning. Difference in mean exact match after fine-tuning on each of the held out splits. Answer changed refers to when obfuscation altered the correct answer, same refers to when the answer after obfuscation was not altered. Q: Questions, O: Obfuscations, Q/O: Questions and Obfuscations, P: Problems.**
| Model | Answer | Train | Val | Held out Q | Held out O | Held out Q/O | Held out P |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama 3.2 1B Inst | All | 0.58 | 0.02 | 0.04 | 0.32 | 0.03 | 0.03 |
|  | Same | 0.88 | 0.03 | 0.10 | 0.60 | 0.06 | 0.06 |
|  | Changed | 0.15 | 0.00 | 0.00 | 0.06 | 0.01 | 0.01 |
| LLama 3.2 3B Inst | All | 0.71 | 0.01 | 0.02 | 0.38 | 0.02 | 0.01 |
|  | Same | 0.89 | 0.01 | 0.04 | 0.61 | 0.03 | 0.00 |
|  | Changed | 0.45 | -0.01 | 0.01 | 0.17 | 0.02 | 0.02 |

We fine-tuned two models using the training and validation splits and evaluate them on the remaining unseen subsets. We use the same prompts as in our evaluation experiments for both fine-tuning and evaluation phases. For multiple-choice questions, we randomly shuffle the order of the answer choices to minimise the risk of over-fitting. We fine-tune the instruction-tuned Llama 3.2 1B and 3B models for 1 epoch using AdamW with $\mathrm{lr}\!=\!1e\!-\!5$, a weight decay of $0.01$ and a batch size of $8$ for the 1B model and $4$ for the 3B model. Results were comparable using other hyperparameter choices, though we did not carry out extensive hyperparameter search. We report the difference of exact match scores $M_{og}$ and $M_{obf}$ before and after fine-tuning to measure improved performance after fine-tuning.

Table [3](#S3.T3) shows the difference after fine-tuning for each split. As expected, both models improve significantly on the questions included in the training ($0.71$ for the 3B model and $0.58$ for the 1B model). Both models also improve on unseen obfuscations split ($0.38$ and $0.32$ respectively). Except for unseen obfuscations, both models show limited improvement on all other unseen splits (mean difference in scores for both models $\leq 0.4$). The modest improvement on unseen questions is surprising given that these questions are continuations of the same training problems in their same obfuscations. This is more clearly evident when only considering answers which are altered by the obfuscation procedure with mean difference $\leq 0.02$ in both models. The exception is the 3B model's ability to improve $0.17$ on unseen obfuscations of questions included in training. Overall, the results suggest overfitting on obfuscations present in training data. The naive inclusion of few examples of reasoning questions does not necessarily improve reasoning abilities in general. Yet, the positive change in score in unseen obfuscations suggests that a level of transfer occurs when seeing multiple permutations of the same type of question.

## 4 Related Works

### 4.1 Reasoning in LLMs

Prior works assess reasoning in LLMs within specific problem domains, such as mathematical word problems (Cobbe et al., 2021; Hendrycks et al., 2021; Chen et al., 2023; Shao et al., 2024), visual pattern-matching (Chollet, 2019), multi-step planning (Valmeekam et al., 2022; Kambhampati, 2024), and commonsense question-answering (Zelikman et al., 2022; Lin et al., 2020; Rajani et al., 2019; Talmor et al., 2019). Inference-time techniques, like chain-of-thought (CoT) prompting, present a promising direction for reasoning tasks (Wei et al., 2022; Kojima et al., 2022). These techniques bear superficial resemblance to Dual Process Theory's System 2 model of human decision-making (Evans & Stanovich, 2013; Kahneman, 2013), as LLMs expend additional deliberative effort to `think for a long time before responding' (OpenAI et al., 2024). Recent work utilising reinforcement learning to optimise CoT strategies (Xu et al., 2025; Deepseek-AI, 2025; OpenAI et al., 2024) shows impressive promise on relevant tasks (Valmeekam et al., 2024; McCoy et al., 2024; Guan et al., 2025).

However, other research shows that LLMs do not reason perfectly, exhibiting human-like failures such as distractability (Shi et al., 2023) and content effects (Lampinen et al., 2024) on reasoning tasks. Unlike humans, LLMs exhibit varying performance across minor perturbations of reasoning problems, suggesting to some that they do not reason at all (Mirzadeh et al., 2024). LLMs also exhibit `token bias' (Jiang et al., 2024)–a sensitivity to the probability of outputs, even in non-probabilistic settings (McCoy et al., 2019, 2023; Razeghi et al., 2022), which is not improved by inference-time scaling (McCoy et al., 2024).

### 4.2 Contamination in Reasoning Benchmarks

Dataset contamination, when benchmark questions occur directly in training data, can damage the reliability of reasoning benchmarks (Zhou et al., 2023; Jacovi et al., 2023; Yang et al., 2023). Contamination allows LLMs to achieve high performance through memorisation, undermining the construct validity of the task (Bean et al., 2024).

Prior efforts to mitigate and contextualise the effects of contamination in reported performance have utlized questions perturbations and incorporating synthetic variations to prompts. Perturbation can be achieved through the introduction of handcrafted adversarial content, like irrelevant context (Shi et al., 2023) or structural variations of the original questions (Patel et al., 2021). Closer to our work, other approaches generate synthetic variants of questions using templatisation (Jiang et al., 2024; Kumar et al., 2021). Mirzadeh et al. (2024) uses synthetic templates to produce variants of problems in the GSM8K dataset (Cobbe et al., 2021), taking unreasonably high performance on the original questions as evidence of memorisation. To measure memorisation on LingOly, Bean et al. (2024) report accuracy relative to a no context baseline–a score achieved on problems lacking the requisite context and thus only solvable by memorisation. Similarly, Patel et al. (2021) report accuracy on the MAWPS benchmark (Koncel-Kedziorski et al., 2016) where final questions, instead of prior context, are removed. Deng et al. (2024) introduce `testlet-slot guessing,' in which models are asked to fill out a masked incorrect answer for a multiple choice question, as an indication of memorised answers.

An alternative approach to combat contamination makes use of linguistics problems in low-resource settings (Tanzer et al., 2024; Chi et al., 2024; Bean et al., 2024). Low-resource languages are low-probability settings for LLMs (Vaduguru et al., 2021; McCoy et al., 2023), making them effective settings for testing in-context reasoning (Tanzer et al., 2024; Bean et al., 2024). Prior benchmarks utilise problems from national (Bean et al., 2024) and international (Vaduguru et al., 2021; Sánchez et al., 2024) Linguistics Olympiads.

## 5 Limitations

Coarse Metrics
Exact Match uniformly penalises all incorrect answers, including partially correct ones, and limits insights into various failure modes. Yet, metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) are unsuitable for the very short answers common in our benchmark and chrF (Popović, 2015) is sensitive to repeating related words from the context. In reality, Linguistics Olympiads typically award partial credit for answers which show partial understanding of the problems. Future work could explore alternative evaluation metrics that capture partial solutions or help discern partially correct answers. By operating on the solution steps in addition to final answers, novel evaluation scores could credit correct reasoning steps and allow deeper insights into prediction pathways in models.

Limited Model Improvements
Aside from our fine-tuning experiment around data contamination, we do not show clear ways in which a model could be improved on this benchmark. Exploring other approaches such as few-shot-learning (Brown et al., 2020) or incorporating translation as an auxiliary task may prove more effective at developing systems with improved and generalisable capabilities.

## 6 Conclusion

In this work we introduced LingOly-TOO, a challenging benchmark of linguistic reasoning problems. We apply orthographic templatisation to Linguistics Olympiad problems to produce obfuscated variants that preserve the same reasoning steps. We demonstrate through numerous experiments that obfuscation helps alleviate measurement bias due to data exposure and provides reasoning estimates that correlate with the ability to solve linguistic reasoning problems. We further show that the reasoning abilities of state-of-the-art models remain inconsistent and that simple fine-tuning does not necessarily endow models with context-free and robust problem solving skills. This work establishes a measure of reasoning that is robust to effects of data exposure and contributes to ongoing efforts targeting the inherent difficulty of fully understanding response generation in frontier models.

## Impact Statement

This paper aims to advance the field of machine learning.

Constructing problems around the features of low-resource languages to create a benchmark may lead to concerns about the benefit to the communities who speak these languages (Tanzer et al., 2024). For Linguistic Olympiads, problems are created from sources in the public domain and where the communities have already given a linguist permission to publish information about the language. In this paper, we transform existing puzzles, and do not create new content in the languages. We restrict the dataset from training or redistribution so that new uses of these languages must come from the original sources.

## References

- Aryabumi et al. (2024)
Aryabumi, V., Dang, J., Talupuru, D., Dash, S., Cairuz, D., Lin, H., Venkitesh, B., Smith, M., Campos, J. A., Tan, Y. C., et al.
Aya 23: Open weight releases to further multilingual progress.
*arXiv preprint*, 2024.
URL [https://arxiv.org/abs/2405.15032](https://arxiv.org/abs/2405.15032).
- Asia Pacific Linguistics Olympiad (2024)
Asia Pacific Linguistics Olympiad.
Problems by year, 2024.
URL [https://aplo.asia/problems/problems-by-year/](https://aplo.asia/problems/problems-by-year/).
- Bean et al. (2024)
Bean, A. M., Hellsten, S., Mayne, H., Magomere, J., Chi, E. A., Chi, R. A., Hale, S. A., and Kirk, H. R.
LINGOLY: A benchmark of olympiad-level linguistic reasoning puzzles in low resource and extinct languages.
In *The 38th Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024.
URL [https://openreview.net/forum?id=cLga8GStdk](https://openreview.net/forum?id=cLga8GStdk).
- Bozhanov & Derzhanski (2013)
Bozhanov, B. and Derzhanski, I.
Rosetta stone linguistic problems.
In Derzhanski, I. and Radev, D. (eds.), *Proceedings of the Fourth Workshop on Teaching NLP and CL*, pp.  1–8, Sofia, Bulgaria, August 2013. Association for Computational Linguistics.
URL [https://aclanthology.org/W13-3401/](https://aclanthology.org/W13-3401/).
- Brown et al. (2020)
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D.
Language models are few-shot learners.
*arXiv preprint*, 2020.
URL [https://arxiv.org/abs/2005.14165](https://arxiv.org/abs/2005.14165).
- Chen et al. (2023)
Chen, W., Ma, X., Wang, X., and Cohen, W. W.
Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.
*Transactions on Machine Learning Research*, 2023.
ISSN 2835-8856.
URL [https://openreview.net/forum?id=YfZ4ZPt8zd](https://openreview.net/forum?id=YfZ4ZPt8zd).
- Chi et al. (2024)
Chi, N., Malchev, T., Kong, R., Chi, R., Huang, L., Chi, E., McCoy, R., and Radev, D.
ModeLing: A novel dataset for testing linguistic reasoning in language models.
In Hahn, M., Sorokin, A., Kumar, R., Shcherbakov, A., Otmakhova, Y., Yang, J., Serikov, O., Rani, P., Ponti, E. M., Muradoğlu, S., Gao, R., Cotterell, R., and Vylomova, E. (eds.), *Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP*, pp.  113–119, St. Julian's, Malta, March 2024. Association for Computational Linguistics.
URL [https://aclanthology.org/2024.sigtyp-1.14/](https://aclanthology.org/2024.sigtyp-1.14/).
- Chollet (2019)
Chollet, F.
On the measure of intelligence.
*arXiv preprint*, 2019.
URL [http://arxiv.org/abs/1911.01547](http://arxiv.org/abs/1911.01547).
- Cobbe et al. (2021)
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J.
Training Verifiers to Solve Math Word Problems.
*arXiv preprint*, 2021.
URL [https://arxiv.org/abs/2110.14168](https://arxiv.org/abs/2110.14168).
- Deepseek-AI (2025)
Deepseek-AI.
DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.
*arXiv preprint*, 2025.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- Deng et al. (2024)
Deng, C., Zhao, Y., Tang, X., Gerstein, M., and Cohan, A.
Investigating data contamination in modern benchmarks for large language models.
In Duh, K., Gomez, H., and Bethard, S. (eds.), *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp.  8706–8719. Association for Computational Linguistics, 2024.
doi: 10.18653/v1/2024.naacl-long.482.
URL [https://aclanthology.org/2024.naacl-long.482/](https://aclanthology.org/2024.naacl-long.482/).
- Evans & Stanovich (2013)
Evans, J. S. B. T. and Stanovich, K. E.
Dual-process theories of higher cognition: Advancing the debate.
*Perspectives on Psychological Science*, 8(3):223–241, 2013.
ISSN 1745-6916, 1745-6924.
doi: 10.1177/1745691612460685.
URL [https://journals.sagepub.com/doi/10.1177/1745691612460685](https://journals.sagepub.com/doi/10.1177/1745691612460685).
- Gage (1994)
Gage, P.
A new algorithm for data compression.
*The C Users Journal*, 12(2):23–38, 1994.
- Guan et al. (2025)
Guan, X., Zhang, L. L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang, M.
rStar-Math: Small llms can master math reasoning with self-evolved deep thinking.
*arXiv preprint*, 2025.
URL [https://arxiv.org/abs/2501.04519](https://arxiv.org/abs/2501.04519).
- Hendrycks et al. (2021)
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J.
Measuring mathematical problem solving with the MATH dataset.
In Vanschoren, J. and Yeung, S. (eds.), *Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks*, 2021.
- Huang & Chang (2023)
Huang, J. and Chang, K. C.-C.
Towards reasoning in large language models: A survey.
In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), *Findings of the Association for Computational Linguistics: ACL 2023*, pp.  1049–1065, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-acl.67.
URL [https://aclanthology.org/2023.findings-acl.67/](https://aclanthology.org/2023.findings-acl.67/).
- Jacovi et al. (2023)
Jacovi, A., Caciularu, A., Goldman, O., and Goldberg, Y.
Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks.
In Bouamor, H., Pino, J., and Bali, K. (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp.  5075–5084. Association for Computational Linguistics, 2023.
doi: 10.18653/v1/2023.emnlp-main.308.
URL [https://aclanthology.org/2023.emnlp-main.308/](https://aclanthology.org/2023.emnlp-main.308/).
- Jiang et al. (2024)
Jiang, B., Xie, Y., Hao, Z., Wang, X., Mallick, T., Su, W. J., Taylor, C. J., and Roth, D.
A peek into token bias: Large language models are not yet genuine reasoners.
In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp.  4722–4756, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.272.
URL [https://aclanthology.org/2024.emnlp-main.272/](https://aclanthology.org/2024.emnlp-main.272/).
- Kahneman (2013)
Kahneman, D.
*Thinking, Fast and Slow*.
Psychology/Economics. Farrar, Straus and Giroux, first paperback edition edition, 2013.
ISBN 978-0-374-53355-7.
- Kambhampati (2024)
Kambhampati, S.
Can large language models reason and plan?
*Annals of the New York Academy of Sciences*, 1534(1):15–18, 2024.
ISSN 0077-8923, 1749-6632.
doi: 10.1111/nyas.15125.
URL [http://arxiv.org/abs/2403.04121](http://arxiv.org/abs/2403.04121).
- Kohrt (1986)
Kohrt, M.
The term ‘grapheme’in the history and theory of linguistics.
*New trends in graphemics and orthography*, pp.  80–96, 1986.
- Kojima et al. (2022)
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y.
Large language models are zero-shot reasoners.
*Advances in neural information processing systems*, 35:22199–22213, 2022.
- Koncel-Kedziorski et al. (2016)
Koncel-Kedziorski, R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H.
MAWPS: A math word problem repository.
In Knight, K., Nenkova, A., and Rambow, O. (eds.), *Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp.  1152–1157. Association for Computational Linguistics, June 2016.
doi: 10.18653/v1/N16-1136.
URL [https://aclanthology.org/N16-1136/](https://aclanthology.org/N16-1136/).
- Koralus (2022)
Koralus, P.
*Reason and Inquiry: The Erotetic Theory*.
Oxford University Press, 1st edition, 2022.
ISBN 978-0-19-186254-0.
URL [https://doi.org/10.1093/oso/9780198823766.001.0001](https://doi.org/10.1093/oso/9780198823766.001.0001).
- Kumar et al. (2021)
Kumar, V., Maheshwary, R., and Pudi, V.
Adversarial examples for evaluating math word problem solvers.
In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2021*, pp.  2705–2712. Association for Computational Linguistics, November 2021.
doi: 10.18653/v1/2021.findings-emnlp.230.
URL [https://aclanthology.org/2021.findings-emnlp.230/](https://aclanthology.org/2021.findings-emnlp.230/).
- Lampinen et al. (2024)
Lampinen, A. K., Dasgupta, I., Chan, S. C. Y., Sheahan, H. R., Creswell, A., Kumaran, D., McClelland, J. L., and Hill, F.
Language models, like humans, show content effects on reasoning tasks.
*PNAS Nexus*, 3(7):pgae233, 2024.
ISSN 2752-6542.
doi: 10.1093/pnasnexus/pgae233.
URL [https://doi.org/10.1093/pnasnexus/pgae233](https://doi.org/10.1093/pnasnexus/pgae233).
- Li & Flanigan (2024)
Li, C. and Flanigan, J.
Task contamination: Language models may not be few-shot anymore.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 38(16):18471–18480, 2024.
doi: 10.1609/aaai.v38i16.29808.
URL [https://ojs.aaai.org/index.php/AAAI/article/view/29808](https://ojs.aaai.org/index.php/AAAI/article/view/29808).
- Lin et al. (2020)
Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., and Ren, X.
CommonGen: A constrained text generation challenge for generative commonsense reasoning.
In Cohn, T., He, Y., and Liu, Y. (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2020*, pp.  1823–1840. Association for Computational Linguistics, 2020.
doi: 10.18653/v1/2020.findings-emnlp.165.
URL [https://aclanthology.org/2020.findings-emnlp.165/](https://aclanthology.org/2020.findings-emnlp.165/).
- Lin (2004)
Lin, C.-Y.
ROUGE: A package for automatic evaluation of summaries.
In *Text Summarization Branches Out*, pp.  74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL [https://aclanthology.org/W04-1013/](https://aclanthology.org/W04-1013/).
- McCoy et al. (2019)
McCoy, R. T., Pavlick, E., and Linzen, T.
Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.
*arXiv preprint*, 2019.
URL [https://arxiv.org/abs/1902.01007](https://arxiv.org/abs/1902.01007).
- McCoy et al. (2023)
McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T. L.
Embers of autoregression: Understanding large language models through the problem they are trained to solve.
*arXiv preprint*, 2023.
URL [http://arxiv.org/abs/2309.13638](http://arxiv.org/abs/2309.13638).
- McCoy et al. (2024)
McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., and Griffiths, T. L.
When a language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAI o1.
*arXiv preprint*, 2024.
URL [https://arxiv.org/abs/2410.01792](https://arxiv.org/abs/2410.01792).
- Mirzadeh et al. (2024)
Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M.
GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models.
*arXiv preprint*, 2024.
URL [http://arxiv.org/abs/2410.05229](http://arxiv.org/abs/2410.05229).
- OpenAI et al. (2024)
OpenAI, El-Kishky, A., Selsam, D., Song, F., Parascandolo, G., Ren, H., Lightman, H., Chung, H. W., Akkaya, I., Sutskever, I., Wei, J., Gordon, J., Cobbe, K., Yu, K., Kondraciuk, L., Schwarzer, M., Rohaninejad, M., Brown, N., Zhao, S., Bansal, T., Kosaraju, V., and Zhou, W.
Learning to reason with LLMs, 2024.
URL [https://openai.com/index/learning-to-reason-with-llms/](https://openai.com/index/learning-to-reason-with-llms/).
- Papineni et al. (2002)
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.
BLEU: a method for automatic evaluation of machine translation.
In *Proceedings of the 40th Annual Meeting on Association for Computational Linguistics*, ACL '02, pp.  311–318, USA, 2002. Association for Computational Linguistics.
doi: 10.3115/1073083.1073135.
URL [https://doi.org/10.3115/1073083.1073135](https://doi.org/10.3115/1073083.1073135).
- Patel et al. (2021)
Patel, A., Bhattamishra, S., and Goyal, N.
Are NLP models really able to solve simple math word problems?
In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp.  2080–2094. Association for Computational Linguistics, June 2021.
doi: 10.18653/v1/2021.naacl-main.168.
URL [https://aclanthology.org/2021.naacl-main.168/](https://aclanthology.org/2021.naacl-main.168/).
- Popović (2015)
Popović, M.
chrF: character n-gram F-score for automatic MT evaluation.
In Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Hokamp, C., Huck, M., Logacheva, V., and Pecina, P. (eds.), *Proceedings of the Tenth Workshop on Statistical Machine Translation*, pp.  392–395, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
doi: 10.18653/v1/W15-3049.
URL [https://aclanthology.org/W15-3049/](https://aclanthology.org/W15-3049/).
- Rajani et al. (2019)
Rajani, N. F., McCann, B., Xiong, C., and Socher, R.
Explain yourself! Leveraging language models for commonsense reasoning.
In Korhonen, A., Traum, D., and Màrquez, L. (eds.), *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pp.  4932–4942. Association for Computational Linguistics, 2019.
doi: 10.18653/v1/P19-1487.
URL [https://aclanthology.org/P19-1487/](https://aclanthology.org/P19-1487/).
- Raji et al. (2021)
Raji, D., Denton, E., Bender, E. M., Hanna, A., and Paullada, A.
AI and the everything in the whole wide world benchmark.
In Vanschoren, J. and Yeung, S. (eds.), *Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks*, volume 1, 2021.
- Razeghi et al. (2022)
Razeghi, Y., Logan IV, R. L., Gardner, M., and Singh, S.
Impact of pretraining term frequencies on few-shot numerical reasoning.
In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2022*, pp.  840–854, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
doi: 10.18653/v1/2022.findings-emnlp.59.
URL [https://aclanthology.org/2022.findings-emnlp.59/](https://aclanthology.org/2022.findings-emnlp.59/).
- Russell et al. (2025)
Russell, J., Karpinska, M., and Iyyer, M.
People who frequently use chatgpt for writing tasks are accurate and robust detectors of ai-generated text, 2025.
URL [https://arxiv.org/abs/2501.15654](https://arxiv.org/abs/2501.15654).
- Şahin et al. (2020)
Şahin, G. G., Kementchedjhieva, Y., Rust, P., and Gurevych, I.
PuzzLing Machines: A challenge on learning from small data.
In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pp.  1241–1254, Online, July 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.acl-main.115.
URL [https://aclanthology.org/2020.acl-main.115/](https://aclanthology.org/2020.acl-main.115/).
- Scasserra (2008)
Scasserra, D.
The influence of perceived task difficulty on task performance, 2008.
URL [https://rdw.rowan.edu/etd/756](https://rdw.rowan.edu/etd/756).
- Shao et al. (2024)
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D.
DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprinx arXiv:2402.03300*, 2024.
URL [http://arxiv.org/abs/2402.03300](http://arxiv.org/abs/2402.03300).
- Shi et al. (2023)
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, D.
Large language models can be easily distracted by irrelevant context.
In *International Conference on Machine Learning*, pp.  31210–31227. PMLR, 2023.
- Sánchez et al. (2024)
Sánchez, E., Alastruey, B., Ropers, C., Stenetorp, P., Artetxe, M., and Costa-jussà, M. R.
Linguini: A benchmark for language-agnostic linguistic reasoning.
*arXiv preprint*, 2024.
URL [https://arxiv.org/abs/2409.12126](https://arxiv.org/abs/2409.12126).
- Talmor et al. (2019)
Talmor, A., Herzig, J., Lourie, N., and Berant, J.
CommonsenseQA: A question answering challenge targeting commonsense knowledge.
In Burstein, J., Doran, C., and Solorio, T. (eds.), *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pp.  4149–4158. Association for Computational Linguistics, 2019.
doi: 10.18653/v1/N19-1421.
URL [https://aclanthology.org/N19-1421/](https://aclanthology.org/N19-1421/).
- Tanzer et al. (2024)
Tanzer, G., Suzgun, M., Visser, E., Jurafsky, D., and Melas-Kyriazi, L.
A benchmark for learning to translate a new language from one grammar book.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=tbVWug9f2h](https://openreview.net/forum?id=tbVWug9f2h).
- United Kingdom Linguistics Olympiad (2021)
United Kingdom Linguistics Olympiad.
Special arrangements for the 2021 competition, 2021.
URL [https://archives.uklo.org/covid2021](https://archives.uklo.org/covid2021).
- United Kingdom Linguistics Olympiad (2023)
United Kingdom Linguistics Olympiad.
Past challenge puzzles: How far can you go?, 2023.
URL [https://www.uklo.org/past-exam-papers/](https://www.uklo.org/past-exam-papers/).
Accessed: 2025-01-28.
- Vaduguru et al. (2021)
Vaduguru, S., Sathe, A., Choudhury, M., and Sharma, D. M.
Sample-efficient linguistic generalizations through program synthesis: Experiments with phonology problems.
*arXiv preprint*, 2021.
URL [https://arxiv.org/abs/2106.06566](https://arxiv.org/abs/2106.06566).
- Valmeekam et al. (2022)
Valmeekam, K., Olmo, A., Sreedharan, S., and Kambhampati, S.
Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change).
In *NeurIPS 2022 Foundation Models for Decision Making Workshop*, 2022.
URL [https://openreview.net/forum?id=wUU-7XTL5XO](https://openreview.net/forum?id=wUU-7XTL5XO).
- Valmeekam et al. (2024)
Valmeekam, K., Stechly, K., and Kambhampati, S.
LLMs still can't plan; can LRMs? A preliminary evaluation of OpenAI's o1 on PlanBench.
In *NeurIPS 2024 Workshop on Open-World Agents*, 2024.
URL [https://openreview.net/forum?id=Gcr1Lx4Koz](https://openreview.net/forum?id=Gcr1Lx4Koz).
- Wei et al. (2022)
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D.
Chain-of-thought prompting elicits reasoning in large language models.
In *36th Conference on Neural Information Processing Systems*, 2022.
- Xu et al. (2025)
Xu, F., Hao, Q., Zong, Z., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., Ouyang, T., Meng, F., Shao, C., Yan, Y., Yang, Q., Song, Y., Ren, S., Hu, X., Li, Y., Feng, J., Gao, C., and Li, Y.
Towards large reasoning models: A survey of reinforced reasoning with large language models.
*arXiv preprint*, 2025.
URL [http://arxiv.org/abs/2501.09686](http://arxiv.org/abs/2501.09686).
- Yang et al. (2023)
Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., and Stoica, I.
Rethinking benchmark and contamination for language models with rephrased samples.
*arXiv preprint*, 2023.
URL [https://arxiv.org/abs/2311.04850](https://arxiv.org/abs/2311.04850).
- Zelikman et al. (2022)
Zelikman, E., Wu, Y., Mu, J., and Goodman, N.
STaR: Bootstrapping reasoning with reasoning.
In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), *Advances in Neural Information Processing Systems*, 2022.
URL [https://openreview.net/forum?id=_3ELRdg2sgI](https://openreview.net/forum?id=_3ELRdg2sgI).
- Zhou et al. (2023)
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., and Han, J.
Don't make your LLM an evaluation benchmark cheater.
*arXiv preprint*, 2023.
URL [http://arxiv.org/abs/2311.01964](http://arxiv.org/abs/2311.01964).

## Appendix A Glossary

**Table 4: Glossary of linguistic terms.**
| Word | Meaning |
| --- | --- |
| cognate | Word or word-part with the same historical origin as another. E.g. English father, German Vater, Swedish far. |
| diacritic | Symbol added to a letter or other basic glyph; often called `accent'. |
| (language) family | Group of languages all derived from a common ancestor. E.g. the Romance language family is derived from Latin. |
| grapheme | Letter or group of letters that typically represent a single sound or suprasegmental feature. See (Kohrt, 1986) for a discussion of other definitions in common usage. |
| lexeme | Basic unit of the lexicon (vocabulary) of a language. E.g. sit, sits, sitting, sat are all inflected forms of the same lexeme. |
| (transparent) loanword | Word borrowed from another language. A loanword is transparent if it is obviously a loanword. E.g. Nepali pēnsil `pencil'. |
| morpheme | Basic unit of meaning within a word. E.g. help-less-ness consists of three morphemes. |
| morphology | Branch of linguistics dealing with the internal structure and formation of words. |
| $n$-graph | Collection of $n$ letters representing a single grapheme. E.g. th and sh are English digraphs. |
| orthography | System for writing a language, including the choice letters or other glyphs and spelling conventions. |
| phonetics | Branch of linguistics dealing with the production and perception of speech sounds or signs. |
| phonology | Branch of linguistics dealing with the systematisation and organisation of sounds or signs within a language. |
| phonological distinction | Distinction between two speech sounds that is treated as meaningful within a language. |
| Problemese | Unknown language that is the focus of a Linguistics Olympiad problem. |
| semantics | Branch of linguistics dealing with the study of meaning. |
| Solverese | Language that a Linguistics Olympiad problem is written in; assumed to be the working language of the solver. |
| suprasegmental | Phonetic or phonological feature that extends beyond a single speech sound. E.g. stress, intonation, pitch. |
| syllabification | Procedure for forming valid syllables out of a string of speech sounds. |
| syntax | Branch of linguistics dealing with the organisation of words into phrases, clauses and sentences. |

## Appendix B Annotations

### B.1 Annotation Process

Annotation was the first step of obfuscation process in LingOly-TOO. The primary task was removing metadata included in the problem text such as language names, families, and geographic regions that were not relevant for solving the problem via reasoning, but could have allowed models to ascertain facts about the language, even when obfuscated. Problemese data that would be altered in later stages of obfuscation were also highlighted during annotation.

All problem annotations were performed by a single annotator to minimise the inconsistency between different problems. Two other members of the team had expertise in Linguistics Olympiads, one being Vice-Chair & Chair of Test Development at UKLO, and the other Secretary General & International Jury Chair of the Asia Pacific Linguistics Olympiad. They validated each of the annotated problems, identifying and documenting any errors or inconsistencies that needed correcting. Despite these checks, mistakes may remain as the result of human error.

### B.2 Annotation Categories

**Table 5: Annotation examples. Each row is an extract from a problem in LingOly-TOO. On the left, the extract appears in the original UKLO problem sheet. On the right, the extract after annotation. Problem 67 (Navajo by Babette Verhoeven) is an example of the language name annotation. Problem 16 (Ulwa by Richard Sproat) is example of the cultural context annotation. Problem 61 (Ilokano by Bozhidar Bozhanov) illustrates the annotation of Problemese in preparation for further obfuscation, and is an example of removing grading guidelines from the dataset to prepare it for checking exact matching.**
| Problem | Original | Annotated |
| --- | --- | --- |
| ${67}$ | This problem is about the way in which Navajo speakers build sentences out of a verb V. | This problem is about the way in which $$$Language X$$$ speakers build sentences out of a verb V. |
| ${16}$ | Ulwa is a language spoken in Nicaragua. It contains quite a few loanwords from English, which is spoken in the Bluefields area of the country. | $$$Language X$$$ &&& &&& contains quite a few loanwords from English &&& &&&. |
| ${61}$ | dinaldalusanda they were cleaning it | @@@dinaldalusanda@@@ they were cleaning it |
| ${61}$ | t(in)ak+takaw+da ida | tinaktakawda ida |

Four categories of annotations were performed: language and place names, cultural context, Problemese data and grading guidelines. Each involved bracketing strings in the problem files with triplets of a symbol not used elsewhere in the problems.

Language and Place Names
The English name of the Problemese language was replaced with `Language X' wherever it appeared in the problem, and was annotated using $$$ tag (see row one in Table [5](#A2.T5)). If multiple non-English languages were mentioned in the problem, the second was replaced with `Language Y' and the third with `Language Z'. A maximum of three languages appeared in any given problem. Where related terms such as geographic regions or people groups associated with those languages were relevant to the problem, they were replaced with e.g. `Region X', and also annotated using the $$$ tag. Where two historical varieties of the same language were featured in the same problem, their names were replaced with `Language X' and `Language Y', and a sentence clarifying their relationship was added.

Cultural Context  Any other metadata or cultural references that could indicate the origin of a language but were not relevant to solving the problem were replaced with a space, and annotated with &&& tag (see row two in Table [5](#A2.T5)). If it was not clear whether some part of the metadata might be useful in solving the problem, it was annotated with the $$$ tag, and edited in accordance with the principals in the previous paragraph.

Problemese Data  All Problemese data were annotated using an @@@ tag (see row three in Table [5](#A2.T5)). Where problems involved multiple languages, these were not distinguished in annotation. Instead, the permutations of graphemes used to alter the Problemese data were designed to preserve the distinctions between the languages. Many of the problems included pronunciation guides, since students sitting UKLO exams are not expected to have any prior linguistic knowledge. These pronunciation guides were not annotated, since the permutations of the graphemes used to alter the Problemese always preserved any relevant phonetic and phonological distinctions.

Grading Guidelines  Most UKLO answer sheets include marking guidelines for awarding partial credit, which are often accompanied by symbols such as `+', `|' and brackets written inside the correct answers. These were simply removed from the answer files (see row four in Table [5](#A2.T5)).

## Appendix C Obfuscation Rulesets

The core of the obfuscation procedure is the altering of the Problemese data. This is done by creating a new orthography for the Problemese, which is a simple permutation of the graphemes present in the original data. We created problem-specific rulesets governing acceptable permutations, which ensure that the solvability of the problem is preserved, and allow for the dynamic generation of new obfuscated variations of the problems in the benchmark.

### C.1 Methodology

To determine the obfuscation ruleset for a given problem, the following procedure was applied.

Firstly, the set of graphemes present in the Problemese data was determined. This involved using the authors' expert judgement on which strings should be analysed as n-graphs and therefore single graphemes, and which could be split into multiple units. A common example was deciding when a combination of a letter with a diacritic should be treated as a single grapheme (as in Spanish ~n) or as multiple graphemes (as in Spanish á). This sometimes involved consulting literature on the language, but usually it was sufficient to consult the commentary on the problem provided by UKLO, as well as the authors' own solutions to the problems. Note that a grapheme could be a substring of another grapheme–consider English h as a substring of sh. In this situation, the obfuscation algorithm would scan the annotated text left-to-right, and exchange the longest matching grapheme with its replacement under the obfuscation in a greedy manner.

We then determined what phonetic and phonological data needed to be preserved in the orthography. In many cases, more phonological data was preserved than strictly necessary, such as always keeping vowels and consonants separate. This was both to preserve alternative solving paths to the authors', which may have involved a different set of phonological observations, and to allow for consistent syllabification across obfuscations. This data was then encoded in the obfuscation ruleset for the problem.

### C.2 Structure of an Obfuscation Ruleset

An obfuscation ruleset for a particular problem partitions the graphemes in the Problemese data into four types of structured collection: sets, tables, free-tables and a fixed set. An obfuscation is a choice of structure-preserving permutation of each of these collections, which combine to give a permutation of all of the graphemes in the data.

Sets  A set is simply a collection of graphemes, with no extra structure that needs to be preserved. An obfuscation can therefore involve any permutation of the graphemes in a set.

Tables  The graphemes in a table are partitioned into tuples each of the same length, thought of as columns of a table. The obfuscation then freely permutes these `columns', while preserving the `rows' (index in the tuple) where the graphemes appear. For example, a table

{(p, b), (t, d), (k, g)} =
p
t
k
b
d
g

will have $3!=6$ ways to permute the $3$ columns, hence 6 possible obfuscations:

| p | t | k |
| --- | --- | --- |
| b | d | g |

$\xrightarrow{\text{obfuscations}}$
p
t
k
b
d
g
,
t
p
k
d
b
g
,
k
t
p
g
d
b
,
p
k
t
b
g
d
,
t
k
p
d
g
b
,
k
p
t
g
b
d

Linguistically, this represents a situation where plosives appear in voiceless/voiced pairs, and this data must be preserved, but the place of articulation of the plosives is not relevant, so may be permuted. Compare this to a situation where only the voicedness was relevant, but not the voiceless/voiced pairs, which could be obfuscated by a pair of sets {p, t, k}, {b, d, g}, allowing for $3!\times 3!=36$ possible obfuscations.

Free-Tables  These are a generalisation of tables where some of the `rows' consist of collections of more than one element, together forming a set. An choice of obfuscation is first a choice of permutation of the columns, then a choice of bijection from each original `cell' to the obfuscated `cell'. For example, a free-table

{(m, {p, b, f}), (n, {t, d, s})} $=$
m
n
p, b, f
t, d, s

has $2!\times(3!)^{2}=72$ possible obfuscations. An example is shown below, where the non-identity permutation of the columns was chosen.

| m | n |
| --- | --- |
| p, b, f | t, d, s |

$\xrightarrow{\text{obfuscation}}$
n
m
t, s, d
b, p, f
$\qquad\overset{\cong}{\longleftrightarrow}\qquad$
m $\mapsto$ n
n $\mapsto$ m
p $\mapsto$ t
t $\mapsto$ b
b $\mapsto$ s
d $\mapsto$ p
f $\mapsto$ d
s $\mapsto$ f

Linguistically, this represents a situation where homorganic nasals are relevant, but otherwise place and manner of articulation are not relevant.

Fixed Set  Any grapheme not in any set, table or free-table was fixed by any permutation. Additionally, other strings that should not be changed during obfuscation could be specified as a single grapheme within the fixed set. Such strings primarily fell into two categories: transparent loanwords or cognates, and names. Where words in the Problemese data had noticeable resemblances to English or another well-known language, such as Nepali pēnsil `pencil' or Yawalapiti amiku `friend' (cf. Portuguese amigo), and where these resemblances may have decreased the difficulty of a the problem by providing a starting point for reasoning, they were always preserved. The only times that transparent cognates were not preserved was when the problem did not involve the meanings of words, so recognition of the cognate would provide no advantage. Names of people (both real and fictional) and deities were also always preserved. The names of places and languages were preserved where possible, but not if it would consequently indicate the what the Problemese language was.

### C.3 Comparison to Obfuscation in Linguistics Olympiads

This method of obfuscation is more severe than what is typically used by Linguistics Olympiads. Although most Linguistics Olympiads do not perform any intentional obfuscation, where it is done common notations are exchanged for other common notations as far as possible. For example, in UKLO 2021 A4 Sauk by Ryan Chi, long vowels were changed from being written with a circumflex (e.g. ^a) to being doubled (e.g. aa). This is typically sufficient to prevent search engines from returning relevant results, but has little effect on any advantage gained by having prior familiarity with the language.

### C.4 Effects of Obfuscation Across Linguistic Subdisciplines

Given the difficulty of preserving phonological structure when altering Problemese data, there is a risk that problems with a stronger emphasis on phonology are accidentally rendered unsolvable during obfuscation. We demonstrate that this is not the case, at least on a large scale, by categorizing the problems by their primary linguistic focus and comparing the effects of obfuscation.

Linguistic Subdisciplines  We partition the problems into six categories based on their primary linguistic focus as described by the `Linguistic Subject' classification provided by UKLO. Compounding problems focus on the meaning of lexemes given their structure and cultural context; morphology problems focus on how morphemes (word-parts) combine to form grammatical words; numbers problems focus on the structure and use of numeral phrases; phonology problems focus on the classification and structure of speech sounds, and their effects on grammar; semantics problems focus on how meaning impacts grammar; and syntax problems focus on how words combine to form grammatically valid phrases and sentences.

Results  Models perform comparably on the original problems across all subdisciplines, with the exception of numbers, where performance is significantly lower, although this may be due to numbers problems typically only appearing at higher difficulties in UKLO. The effect of obfuscation is also roughly uniform across subdisciplines (with the exception of numbers), suggesting that obfuscation does not especially impact phonology problems. Compounding problems may be more affected by obfuscation; we hypothesise that dictionaries and lexicons (which are typically the source material for compounding questions) are more likely to occur in model training data than full or sketch grammars (the typical sources for other problem types), giving models an advantage on the original versions of many compounding problems.

Figure: Figure 6: Performance by linguistic subdiscipline. Left: Averaged across all models. Models perform comparably across linguistic subdisciplines, with the exception of numbers problems. The effect of obfuscation is relatively even across subdisciplines. Right: Averaged across the four highest-scoring models (o1 Preview, Claude 3.5 Sonnet, GPT 4o, Gemini 1.5 Pro). Compounding problems are affected particularly severely by obfuscation.
Refer to caption: /html/2503.02972/assets/x6.png

### C.5 Examples of Obfuscation Rulesets

Problem 91, Somali (by Harold Somers).
This problem involves determining the formation of 1sg and 3sg forms of Somali(^3^33Somali is a Cushitic language spoken by approximately 24 million people, primarily ethnic Somalis in East Africa and in diaspora.) verbs. The solution is as follows:

- (i)
The 1sg form adds -ay to the root, while the 3sg form adds -tay.
- (ii)
After the guttural consonants q, c, x, ', the so-called guttural consonants, we have the change t $\to$ d.
- (iii)
Everywhere, we have the following changes: y $\to$ d; lt $\to$ sh; dt $\to$ d; dht $\to$ dh.

There are no names or loanwords in the Somali data that need to be preserved by the obfuscation, so the obfuscation ruleset will only depend on the phonology of the problem. All permutations should fix t, d, dh, sh, since they have phonological relationships which are relevant to the solution of the problem, and cannot be found in a different family of consonants. The guttural consonants can be permuted amongst themselves but not with the other consonants, so form a set {q, c, x, '}. l is treated specially within the solution, but any liquid could fill that role without substantially changing the difficulty of the problem, so we define a set {l, r, w, y}. All other consonants featured in the Somali data form a set {b, f, g, j, h, k, n, s}. The vowels form another set {a, e, i, o, u}.

Problem 218, Stodsde (by Simi Hellsten).
This problem involves understanding the formation of causative verbs in Stodsde.(^4^44Stodsde is a Gyalrongic language, spoken by approximately 4,000 people in Sichuan, China.) The solution involves understanding the structure of a Stodsde syllable, and the changes that occur in each part of the syllable. No changes occur in the vowel, or consonants after the vowel. The part of a syllable before the vowel has the following structure, divided into Slots 1-5. Each slot can only have consonants of a certain type, and undergoes specific changes to form the causative.

| Slot 1 | Slot 2 | Slot 3 | Slot 4 | Slot 5 |
| --- | --- | --- | --- | --- |
| Nasal | Non-sib. fricatives | Liquid (l or r) | plosive/sibilant | Any non-plosive + non-sib. |
| m $\to$ v |  | $\to$ ⁢z if Slot 4 is sibilant | fricative $\to$ affricate, |  |
| n $\to$ $\emptyset$ |  | $\to$ z otherwise | unless Slot 5 is a nasal |  |

Finally, everywhere in the verb, v, ⁢z, z $\to$ f, ⁢l, s if the following sound is voiceless.

Again, the dataset contains no loanwords or names, so the obfuscation ruleset is fully determined by the relevant phonology. Since voicedness is relevant throughout, but not all voiced/voiceless pairs are given in the data, any permutation must preserve voicedness. We also fix ⁢z, hence ⁢l. Since we cannot permute $\emptyset$, we must fix all of Slot 1. Thus m, n, and v are fixed, hence also f. For Slot 2, the remaining non-sibilant fricatives are the voiceless X, which cannot be permuted with anything, and voiced K, G, which can be permuted. In Slot 3, we can freely permute l, r. Slot 4 needs to distinguish between sibilants and sibilant affricates (which are paired), and simple plosives. Considering only those that appear in the Stodsde data, this means we must fix t^h; we can permute p, k, q, and b, d, g, while fixing the aspiration ^h; and we can permute the pairs (z, dz), (Z, dZ), and the pairs (s, ts), (s^h, ts^h). Finally, we can permute the vowels, and the remaining nasals N, ⁢n; j has nothing it could permute with, so must also be fixed.

The final obfuscation ruleset thus has fixed set {m, n, ⁢l, ⁢z, X, v, f, t^h, ^h, j}; sets {K, G}, {l, r}, {p, k, q}, {b, d, g}, {N, ⁢n}, {u, @, 2, o, a, æ, i}; and tables {(z, dz),(Z, dZ)}, {(s, ts), (s^h, ts^h)}.

## Appendix D Prompt Template

Below is the prompt used across models. In the no context setting, {context} is removed.

{formatted_output} tag is populated with an empty dictionary containing the keys of the expected answer for each part of the question. The {instructions} tag is populated based on the setting. For default prompt:

in chain-of-thought setting, {instructions} is set to:

in closed models when applicable, we precede the prompt with the following generic system message:

## Appendix E Evaluation Models

In our experiments, we fixed prompts and system messages across models whenever possible. For closed models, we followed the recommendations of model providers and set the temperature to 0 whenever possible. For open models, we loaded them in full precision when possible or in 8-bit for larger models. All experiments were conducted using A100 and H100 GPUs for open source models except for Deepseek R1 (^5^55https://www.together.ai/models/deepseek-r1) for inference.

| Model | Version | Model Type | Quantization |
| --- | --- | --- | --- |
| Aya 23 35B | CohereForAI/aya-23-35B | Open-source | 8-bit |
| Claude 3.5 Sonnet | claude-3-5-sonnet-20241022 | Proprietary |  |
| Claude 3.7 Sonnet | claude-3-7-sonnet-20250219 | Proprietary |  |
| Deepseek R1 | deepseek/deepseek-r1 | Open-source |  |
| Gemini 1.5 Pro | gemini-1.5-pro | Proprietary |  |
| GPT 4o | gpt-4o-2024-05-13 | Proprietary |  |
| GPT 4.5 | gpt-4.5-preview-2025-02-27 | Proprietary |  |
| Llama 3.2 3B Inst | meta-llama/Llama-3.2-3B-Instruct | Open-source |  |
| Llama 3.2 1B Inst | meta-llama/Llama-3.2-1B-Instruct | Open-source |  |
| Llama 3.3 70B | meta-llama/Llama-3.3-70B-Instruct | Open-source | 8-bit |
| o1-preview | o1-preview | Proprietary |  |
| o3-mini | o3-mini-2025-01-31 | Proprietary |  |
| Ours 1B | Checkpoint will be shared in published version | Open-source |  |
| Ours 3B | Checkpoint will be shared in published version | Open-source |  |
| Phi4 | microsoft/phi-4 | Open-source | 8-bit |

## Appendix F Results for Chain-of-Thought

During early iterations, we explored chain-of-thought (CoT) prompting technique as potentially beneficial for improving reasoning abilities in LLMs. However, we find that it performs worse than without CoT on both models that were tested. For models designed for reasoning, we do not explore CoT following recommendations of providers (^6^66https://platform.openai.com/docs/guides/reasoning). We follow the same evaluation setup as our experiments except using the CoT prompt (Appendix [D](#A4)).

| Model | Obfuscated | Exact Match | Exact Match (CoT) |
| --- | --- | --- | --- |
| GPT 4o | False | $0.330$ | $0.238$ |
|  | True | $0.149$ | $0.114$ |
| Llama 3.3 70B | False | $0.128$ | $0.013$ |
|  | True | $0.077$ | $0.014$ |

## Appendix G Summary of Response Errors

Below is summary of responses for all prompts by model. Llama 3.2 1B Inst performed particularly poorly with many responses that did not include json structure. It also resulted in $49$ inference errors that were excluded from the analysis with negligible impact on the estimates for the model.

**Table 6: Summary of responses on larger benchmark ($30$ obfuscations).**
| Model | Total | Empty Response | Bad Parsing |
| --- | --- | --- | --- |
| Aya 23 35B | 27,325 | 1,602 | 0 |
| GPT 4o | 27,325 | 536 | 169 |
| Gemini 1.5 Pro | 27,325 | 0 | 56 |
| Llama 3.2 1B Inst | 27,276 | 10,687 | 0 |
| Llama 3.2 3B Inst | 27,325 | 1,022 | 0 |
| Llama 3.3 70B- Instruct | 27,325 | 1,202 | 0 |
| Ours 1B | 27,325 | 3,073 | 0 |
| Ours 3B | 27,325 | 2,809 | 0 |
| Phi4 | 27,325 | 2,086 | 0 |
| Claude 3.5 Sonnet | 27,325 | 7 | 140 |

**Table 7: Summary of responses on smaller benchmark ($6$ obfuscations).**
| Model | Total | Empty Response | Bad Parsing |
| --- | --- | --- | --- |
| Aya 23 35B | 6,995 | 370 | 0 |
| Claude 3.5 Sonnet | 6,995 | 8 | 0 |
| Claude 3.7 Sonnet | 6,995 | 22 | 0 |
| Deepseek R1 | 6,995 | 128 | 0 |
| GPT 4.5 | 6,995 | 0 | 6 |
| GPT 4o | 6,995 | 176 | 0 |
| Gemini 1.5 Pro | 6,995 | 16 | 0 |
| Llama 3.3 70B-Instruct | 6,995 | 316 | 0 |
| Phi4 | 6,995 | 287 | 0 |
| o1-preview | 6,995 | 148 | 0 |
| o3-mini | 6,995 | 38 | 11 |

## Appendix H Full Results

[Uncaptioned image]: /html/2503.02972/assets/x7.png

## Appendix I Score Distribution Details

Below is the distributions of scores for top performing models on sample of problems across their obfuscated versions.

[Uncaptioned image]: /html/2503.02972/assets/x8.png

## Appendix J Tokenisation Experiment

### J.1 Set-up

A key question when permuting the graphemes of a language without further refining the models is the impact of the tokeniser. We explore this with the Aya 23 35B model by Cohere (Aryabumi et al., 2024), which was designed for multilingual applications (23 languages). The tokeniser of Aya 23 is based on the popular byte-pair encoding algorithm (BPE) (Gage, 1994). Since we report off-the-shelf model performance, we investigate changes in the performance if we alter the tokenisation of the Problemese portion of the prompt.

When prompting this model, the Problemese sections are treated separately. In the first experiment, we break down the Problemese into unicode characters (with the NFD standardisation) and obtain separate tokenization for each character that we then concatenate together. In the second experiment, we add dashes in-between the characters of the Problemese to force some separation in the interpretation of multiple characters. The rest of the question is tokenised in the standard way, and the problem context and the Problemese are then concatenated together (in the order that they appear in the question) before being fed into the model.

For this experiment we used 10 obfuscations of all problems except Problem 34 (which had long prompts taking up too much memory).

We limited the model outputs to four times the length of the correct response length. After cleaning, the number of incorrectly processed responses is $32$ questions for the standard tokenisation, $289$ questions for the input with dashes, and $292$ questions for the single character tokenisation (out of a total of $10,765$ questions). This may partially account for the additional drop in the performance of the alternative tokenisations.

Figure: Figure 7: Problem-level comparison of exact match scores between standard and alternative tokenisation. The dashed line represents the threshold where the score remains unchanged after altering the tokenisation. Point above the dashed line indicate better performance with alternative tokenisation, while points below the line indicate worse performance. (a) The problems are unobfuscated and we compare scores of standard tokenisation against dash tokenisation. (b) The problems are obfuscated and we compare scores of standard tokenisation against dash tokenisation. (c) The problems are unobfuscated and we compare scores of standard tokenisation against character-level tokenisation. (d) The problems are obfuscated and we compare scores of standard tokenisation against character-level tokenisation.
Refer to caption: /html/2503.02972/assets/x9.png

### J.2 Additional Results

In Table [8](#A10.T8) we see how the average model performance changes under alternative tokenisation for unobfuscated problems. For the unobfuscated case (Table [8](#A10.T8)), the performance decreased for a greater number of the problems than for the obfuscated case (Table [9](#A10.T9)). This aligns with the greater drop in performance that we observe in Table [2](#S3.T2).

**Table 8: Performance change under alternative tokenisation for unobfuscated problems. Score for each problem was averaged over all questions within the problem for the two alternative tokenisation methods (Dash and Character) then compared to the score obtained using standard tokenisation and categorized into Increased, Decreased or Unchanged.**
| Prompt type | Decreased | Unchanged | Increased |
| --- | --- | --- | --- |
| Dash | 57 | 19 | 5 |
| Character | 58 | 19 | 4 |

**Table 9: Performance change under alternative tokenisation for obfuscated problems. For each alternative tokenisation, the score for each problem was averaged over all obfuscations then compared to the score obtained using standard tokenisation and categorized into Increased, Decreased or Unchanged.**
| Prompt type | Decreased | Unchanged | Increased |
| --- | --- | --- | --- |
| Dash | 43 | 25 | 13 |
| Character | 37 | 30 | 14 |

Finally, we show how the exact match scores vary under alternative tokenisations in Figure [7](#A10.F7) by plotting (standard tokenisation score, alternative tokenisation score) for each problem.
When the problems are not obfuscated (Figure [7](#A10.F7)(a) and (c)), the alternative tokenisation more frequently reduced the score to zero, indicating that the tokenisation of the original order may be capturing useful information. For the obfuscated cases (Figure [7](#A10.F7)(b) and (d)), there appeared to be more spread in the scores differences, and no instances of the catastrophic effect of changing from a high exact match score under the standard tokenisation to a score of zero.

## Appendix K Human Evaluation

Linguistic obfuscation does not change the reasoning steps required to solve the problem, and therefore it serves as a method to robustly test reasoning capabilities whilst mitigating memorisation bias. However, obfuscation may change the difficulty level of problem through means other than changing the reasoning process. To measure the effect of obfuscation on problem difficulty, we carried out a randomised controlled trial (RCT) with $172$ human participants across $6$ problems (all relatively easy). After controlling for random differences in problem frequency, obfuscation was found to be associated with $5.80$ percentage points lower performance ($p$-value of $0.059$). We speculate that this may be because the resulting character combinations are more unusual in naturally occurring languages, thus the problems are perceived as being harder.

### K.1 Methodology

Experiment Design  We used a RCT with $172$ participants. Each participant was assigned to one of two groups: one group solving unobfuscated problems ($86$ people) and another solving obfuscated versions ($86$ people). Participants were then assigned one of six problems and given up to $45$ minutes to complete the problem. They were required to spend a minimum of $30$ minutes on the problem before submitting their answers. Responses were scored using exact match with the answers. All data collection was carried out through a Qualtrics survey.

Problem Inclusion Criteria and Design  We used two criteria to select problems. First, we only considered Breakthrough and Foundation-level problems, designed to be taken by 10-14 year olds. Our intention was to ensure participants had a realistic chance of solving the problem, and thus the responses would provide better signal of the effect of obfuscation. Second, we only considered problems where the language was not available through Google Translate, since this would have enabled participants to easily solve the problems. Two exceptions to this rule were Karelian, which is not available on Google Translate but was removed due to a high similarity with Finnish, and Ligurian, which is available Google Translate without stress being indicated, which formed the basis of the problem. The selected six problems are below.

**Table 10: Problems selected for the randomised controlled trial. All problems are Breakthrough and Foundation-level, designed to be taken by 10-14 year olds. None of these languages are available on Google translate in a manner that would aid the solvability of the problem. UKLO problem is the listing of the problem in the UKLO past problems database (United Kingdom Linguistics Olympiad, 2023).**
| Language | Difficulty level | Problem | Original author(s) |
| --- | --- | --- | --- |
| Warlpiri | Breakthrough | 207 | Mary Laughren |
| Umbrian | Breakthrough | 191 | Michael Salter |
| Kabyle | Breakthrough | 160 | Kazune Sato, Simi Hellsten |
| Tariana | Foundation | 210 | Babette Verhoeven, Simi Hellsten |
| Ligurian | Foundation | 147 | Kevin Liang |
| Amele | Foundation | 88 | Babette Verhoeven |

Since all six problems and their solutions are easily accessible on the UKLO website, we rewrote elements of each problem. First, we rephrased the problem instructions so that a quick internet search did not return the solutions. Second, we changed some of the lexemes in the Problemese such that the linguistic rules required to answer the questions were identical, but the specific words to translate were different. For example, in Kabyle we substituted the verb azzel `to run' with aker `to steal', which follow the same conjugation pattern. All problems were minimally rewritten except from Umbrian, where the known vocabulary originates exclusively from inscriptions on the [Iguvine Tablets](https://en.wikipedia.org/wiki/Iguvine_Tablets), and thus is extremely small. Additionally, as in the LLM experiments, all background and cultural information was removed, and the language name was replaced by `Language X'.

For each problem, we randomly sampled two obfuscation types. Participants who were assigned to the obfuscated group were randomly assigned one of the two obfuscated versions.

Participant Inclusion Criteria  Participants were recruited online using Prolific, a crowdworker platform. First, all participants had to be English speaking and monolingual, reducing the risk that they may have had prior exposure to the Problemese language. Second, all participants were required to have a minimum education level of an undergraduate degree. This served as a proxy for ability and was chosen to ensure participants could reasonably complete the problems.

Participants were compensated in line with the UK aged-21 and over minimum wage of £$11.44$ per hour (experiments conducted January 2025). Additionally, participants received performance bonuses of £$2.00$ if they scored above $50\%$ but less than $100\%$, and £$6.00$ if they scored $100\%$. The incentive structure was explained to participants before they started the problem.

All ICML Publication Ethics protocols were followed and the experiment had prior approval by the research ethics committee at our institution. Further details about ethical approval have been removed from this paper for the peer review process but are available upon request. All participants were required to provide informed consent by signing the form shown in Section [K.5](#A11.SS5).

Controlling for Internet and LLM Use  To control for the possibility that participants search for the solutions, each question was presented in image form, making it harder to directly copy and paste questions into an internet browser. Additionally, we added a post-response trap question asking `What do you think the language in the problem (Language X) was?'. Given the obscurity of the chosen languages, we assumed a correct answer to this question implied that participants had either found the solutions or were using a language model, both of which were explicitly prohibited in the instructions.

To mark responses to the trap question, we first automatically identified all entries which identified the correct language. Next, two authors with significant language model experience independently identified all responses which qualitatively appeared AI generated. Evidence suggests that people who frequently use state-of-the-art LLMs are good detectors of AI generated text, often outperforming commercial detection software (Russell et al., 2025). These included responses containing common language model phrases, excessively long responses and those demonstrating an unusually impressive knowledge of academic linguistics, such as claiming the Problemese was likely an agglutinative language. Each author independently marked responses, then discussed cases where they disagreed. Since suspected cheating in the trap question does not necessarily imply that the participant cheated in the main problem, these participants were left in the data for the main analysis. Results excluding these participants are reported as a robustness check.

### K.2 Results and Analysis

Overall Performance  Participants found these problems hard, with mean performance across all problems at $18.85\%$ (mean random guessing is $9.19\%$). Furthermore, the responses to the question `On a scale from 0 (very easy) to $10$ (impossible), how
challenging did you find this problem?' reveal that many participants rates the problems near impossible, with $39.5\%$ reporting a difficulty of $9$ or $10$ out of $10$. There is no significant difference in the distributions of self-reported difficulty across the participants taking obfuscated and unobfuscated problems. The total distributions of participant scores and self-reported difficulty are reported in Figure [8](#A11.F8).

Figure: Figure 8: Distributions of participant scores and self-reported difficulty. Left: Distribution of scores (%) across all six problems and problem types. Scores are highly skewed towards 0. The unobfuscated group had a higher mean score. Right: Participant self-reported difficulty. Participants were asked the question `On a scale from 0 (very easy) to $10$ (impossible), how challenging did you find this problem?'. Responses are highly skewed towards $10$ (impossible). The obfuscated and unobfuscated means are near identical. Despite the symmetric distributions, correlation between score and self-reported difficulty is $0.006$.
Refer to caption: /html/2503.02972/assets/x10.png

Table [11](#A11.T11) shows the mean score broken down by problem language and obfuscation type. The mean score for those solving the unobfuscated and obfuscated problems were $21.70\%$ and $16.00\%$, respectively. Given the skew in the participant scores (Figure [8](#A11.F8) Left), we primarily used nonparametric tests to assess statistical significant. A one-sided Mann-Whitney U test to establish whether the distribution of scores under the original problems was greater than the distribution of obfuscated scores, rejects the null hypothesis of equal distributions, with a $p$-value of $0.041$.

**Table 11: Human performance across original and obfuscated problems. Sample size is in brackets. All scores are out of $100\%$. Random guessing is the expected score from random answers, given the basic formatting instructions in each question are followed.**
| Problem | Mean score: | Mean score: | Change in | Random |
| --- | --- | --- | --- | --- |
|  | original | obfuscated | score | guessing |
| Warlpiri | $39.88$ (n=$14$) | $44.64$ (n=$14$) | $+4.76$ | $20.83$ |
| Umbrian | $18.67$ (n=$15$) | $11.54$ (n=$13$) | $-7.13$ | $0.00$ |
| Kabyle | $27.69$ (n=$13$) | $12.00$ (n=$15$) | $-15.69$ | $5.00$ |
| Tariana | $7.14$ (n=$14$) | $0.00$ (n=$14$) | $-7.14$ | $0.00$ |
| Ligurian | $22.86$ (n=$15$) | $24.76$ (n=$15$) | $+1.90$ | $28.57$ |
| Amele | $15.00$ (n=$15$) | $3.33$ (n=$15$) | $-11.67$ | $0.00$ |
| All | $21.70$ (n=$86$) | $16.00$ (n=$86$) | $-5.70$ | $9.19$ |

To control for small perturbations in problem frequency caused by random allocation, we employed a linear regression with heteroskedasticity-consistent standard errors to partially control for the skew. Table [11](#A11.T11) shows the results, which suggest obfuscation is associated with a $5.80$ drop in performance ($p$-value of $0.059$). Furthermore, using a bootstrapping method with $10,000$ samples, we estimate a $95\%$ confidence interval of $[-11.849,0.210]$.

**Table 12: Linear regressions for participant score under different inclusion criteria. The dependent variable is participant score in all models. The participant inclusion criteria are as follows. Model 1: All participants (n=$172$). Model 2: Excluding participants who we suspect may have cheated, those who returned more than $50\%$ missing data, and those with mean ChrF score below $10$ (n=$146$). Model 3. Excluding participants who scored below random guessing (n=$86$). SE: Heteroskedasticity-consistent standard errors. df: Degrees of freedom. ${}^{*}p<.1$; ${}^{**}p<.05$; ${}^{***}p<.01$**
|  | Model 1 | (SE) | Model 2 | (SE) | Model 3 | (SE) |
| --- | --- | --- | --- | --- | --- | --- |
| Intercept | $12.065^{***}$ | $(3.59)$ | $12.765^{***}$ | $(3.81)$ | $22.723^{***}$ | $(5.63)$ |
| Obfuscation | $-5.796^{*}$ | $(3.07)$ | $-5.805^{*}$ | $(3.27)$ | $-5.100$ | $(3.94)$ |
| Kabyle | $10.326^{*}$ | $(5.50)$ | $12.395^{**}$ | $(6.09)$ | $12.939$ | $(8.17)$ |
| Ligurian | $14.643^{***}$ | $(5.22)$ | $17.276^{***}$ | $(5.85)$ | $25.745^{***}$ | $(6.40)$ |
| Tariana | $-5.595$ | $(3.46)$ | $-5.938$ | $(3.88)$ | $10.610$ | $(7.85)$ |
| Umbrian | $5.983$ | $(4.46)$ | $8.018$ | $(5.38)$ | $4.971$ | $(6.56)$ |
| Warlpiri | $33.095^{***}$ | $(5.27)$ | $35.553^{***}$ | $(5.15)$ | $29.213^{***}$ | $(6.35)$ |
| Observations | $172$ |  | $146$ |  | $86$ |  |
| $R^{2}$ | $0.292$ |  | $0.326$ |  | $0.275$ |  |
| Residual Std. Error | $20.010$ | (df=$165$) | $20.249$ | (df=$139$) | $19.017$ | (df=$79$) |
| F Statistic | $14.126^{***}$ | (df=$6$; $165$) | $16.926^{***}$ | (df=$6$; $139$) | $7.239^{***}$ | (df=$6$; $79$) |

Accounting for Suspected Cheating  Our survey contained a trap question designed to establish whether participants had discovered the solutions online or used an LLM. Responses to the trap questions were first automatically scored based on whether they correctly identified the Problemese language, then they were evaluated qualitatively by two of this paper's authors. The two authors independently identified $15$ and $17$ responses, $14$ of which were common between both lists. After discussion, it was decided that all $18$ likely indicated internet or LLM use and should be removed for robustness checks. In addition, participants who had $50\%$ missing data, and those with a mean chrF score below $10$ (visibly irrelevant answers, e.g. inputting random letters) were also removed. Table [12](#A11.T12), Model 2 shows the results excluding these participant groups. The effect on the point estimate and standard error of obfuscation is negligible.

Effect when Only Considering those Scoring Above Random  Exactly $50\%$ of participants scored below random guessing, highlighting the difficulty of these problems to an inexperienced test taker. For supplementary analysis, we considered whether the effect of obfuscation remained consistent when we only considered those who scored above random and therefore likely understood some of the problem.

First, the likelihood of scoring above random depends on the problem type. In the group taking the unobfuscated problems, $57.0\%$ of participants scored above random. In the group taking the obfuscate problems, only $43\%$ scored above random. Furthermore, a logistic regression predicting above random performance, suggests obfuscation is a significant factor with a $p$-value of $0.045$ (Table [13](#A11.T13)).

Second, after excluding individuals who scored below random guessing, the data can no longer be treated as the outcomes of an RCT, thus non-stratified tests such as the Mann-Whitney U test are no longer appropriate. A linear regression controlling for differences in problem frequency suggests that the coefficient on obfuscation is no longer significant in this subset ($p$-value of $0.196$), though the point estimate ($-5.100$) remains similar and this result could be an artifact of the small sample size.

**Table 13: A logistic regression predicting above random guessing score. The dependent variable is a binary variable indicating whether a participant's score was above random. Obfuscated problems are more likely to lead to below random guessing performance with a coefficient significant at the $5\%$ level. SE: Heteroskedasticity-consistent standard errors. df: Degrees of freedom. ${}^{*}p<.1$; ${}^{**}p<.05$; ${}^{***}p<.01$**
|  | Model 4 | (SE) |
| --- | --- | --- |
| Intercept | $0.068$ | $(0.40)$ |
| Obfuscation | $-0.689^{**}$ | $(0.34)$ |
| Kabyle | $0.597$ | $(0.52)$ |
| Ligurian | $0.139$ | $(0.53)$ |
| Tariana | $-1.890^{***}$ | $(0.70)$ |
| Umbrian | $0.699$ | $(0.54)$ |
| Warlpiri | $1.840^{***}$ | $(0.63)$ |
| Observations | $172$ |  |
| Pseudo $R^{2}$ | $0.163$ |  |

Data Availability  An anonymised version of the dataset and a notebook to generate results are available in the GitHub repository.

### K.3 External Audit of Problems

To verify that the problems selected for the RCT remained solvable when rewritten and obfuscated, we asked two International Linguistics Olympiad (IOL) medallists to audit one set of permutations each. Both auditors were external to this research project and acted independently. Both had prior familiarity with the original UKLO problems.

The auditors confirmed that all problems remain solvable via the same reasoning steps and were not significantly harder under the new orthographies. One auditor commented that problems appeared to be marginally harder since the languages appeared more `unfamiliar' and the other commented that they did not believe it made any difference. Both also mentioned that they personally found the problems harder since they knew the original orthographies and thus found the new versions `disorienting', however this is not a problem we would expect novice test takers to experience.

### K.4 Discussion

The results suggest that obfuscation causes a small but statistically significant drop in performance when taken by inexperienced test takers. This is despite the required reasoning steps and problem solvability remaining unchanged. The point estimate of the effect is $-5.80$ percentage points: approximately equivalent to half a question per problem. If LLMs behave similarly to participants in this study, then we may expect some performance decrease even in cases where the model did not have prior exposure to the original Problemese language. Therefore, performance drops between unobfuscated and obfuscated problems should not be solely attributed to language exposure. However, our estimates for the impact of obfuscation are small; thus we would not expect large changes in performance given robust reasoning skills.

We do not have strong evidence for why obfuscation appears to make problems harder. The external audit confirmed that the problems remained solvable via the same reasoning step, thus any changes to the problem difficulty are superficial only. We speculate that obfuscation may cause the orthography to appear less familiar or naturalistic relative to English, giving the perception that the problems are harder to solve, thus causing participants to exert less effort (Scasserra, 2008). Whilst the mean self-reported difficulty level is marginally higher in the obfuscated group ($7.39$ versus $7.48$, Figure [8](#A11.F8)), the distributions are not significantly different under a Mann-Whitney U test. However, we do find that participants solving obfuscated problems spent less time on the problem before submitting their answers (p = 0.071 under a one-sided Mann Whitey U test), which would support this hypothesis.

Alternatively, knowledge of English alone may have been sufficient to score marginally higher in the unobfuscated versions of some problems. In particular, Ligurian and Umbrian are both European languages descending from Latin. In Ligurian, this would have increased the familiarity of the language, e.g. vió:vet:a, ‘violet’ or pásta, ‘pasta’, but would have offered no assistance when solving the reasoning problem. In Umbrian, knowledge of English may have directly helped, especially as the Latin translation were also provided (under the label of `Language Y'). For example, one question asked for a translation of the Language Y (Latin) words populum and urbs into both Language X (Umbrian) and English. Knowledge of the English words population and urban may have helped the participants locate the correct translations, community and town, from the text provided. While this may have offered marginal assistance in unobfuscated problems, the overall results are largely unchanged when Umbrian is excluded.

Limitations  These results are subject to several limitations. First, participants generally found these problems extremely challenging with only $50\%$ of participants scoring above random. The significance of the results is partially driven by differences in scoring below random and it is unclear whether they would hold under different distributions of scores. As with all human evaluations, the results are highly specific to our demographic of participants (inexperienced test takers) and may not generalise outside of this.

Second, we specifically chose relatively easy problems so that participants had a realistic chance of solving them. As a result, these results may not be indicative of the effect of obfuscation across the full distribution of problems.

Third, despite our best efforts to prevent and identify cheating, there is a risk these results are driven by the availability of online materials. This is a problem with any form of online and unproctored assessment, which future studies might wish to address through in-person test taking.

### K.5 Informed Consent

All participants were required to consent to participation in the study by signing the form below. Information breaking the anonymity of peer review has been redacted.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: See above.
soundness: 2
strengths: - Various models are used in the benchmark.
- Detailed analysis.
summary: This paper introduces LingOly-TOO, a challenging reasoning benchmark that obfuscate Linguistics Olympiad Problems to avoid advantaging models that are using shortcuts such as memorisation and knowledge. The obfuscations preserve the underlying solution logic while removing orthographic clues that could trigger patterns from memorisation or knowledge. Without surprise, the performance of models drastically decrease.

The authors defined knowledge as information stored in model parameters after training, which captures linguistic, factual, and commonsense patterns useful for downstream tasks, and memorisation as when models exploit contaminated datasets, reporting answers previously seen in training

The authors adapted 82 problems from the UKLO, and obfuscate the problem to avoid models relying on linguistic patterns. More specifically, the authors manually created a ruleset for each problem to generate valid permutations of targeted tokens. They apply extra-care to keep names of people, sacred places, etc intact. Overall, the authors generate 6 valid permutations per problem and generate obfuscated versions by altering the problem text. Overall, there are 6995 question-answer pairs.

The authors propose two metrics: the average exact match score across all questions in all permutations and the average exact match across all questions in the unpermuted problem.

In terms of experiments, the authors evaluate 15 reasoning models. The performance varies between the original and obfuscated variants, with GPT and Claude being the most robust models. The authors do a detailed analysis to measure the gap between reasoning and knowledge. It would be great to add a case study to support the findings. In terms of metrics, it would be good to compare human performance on the task and include more details (e.g., how do they compare with LLMs) - I'm aware of Appendix L but would like to see more. Moreover, it would be fairer to assess the performance of models with pass@k.

Overall, I am lukewarm about this paper. Creating obfuscating variants of problems does not seem to relate to real-life tasks, even for measuring reasoning. I have the feeling that this benchmark is focusing on a unrealistic problem / not a real problem. I appreciate the analysis and experiments of the authors, and the paper is very well written and structured.
weaknesses: - I have the feeling that this benchmark focuses on an unrealistic problem / does not represent real life tasks. We cannot expect LLMs to necessarily perform good on those.
- Please add human evaluation + pass@k metrics

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: The authors manually created rulesets for this dataset, which makes it harder to extend this work to other datasets. Did the authors try to use LLMs as annotators to see how feasible it is to extend them to other domains/ datasets?
soundness: 3
strengths: 1. Release of a large-sized (7k) benchmark dataset that clearly separates the reasoning abilities from memorized knowledge.
2. In-depth analysis: The authors conducted various analyses, such as the ability of the model to reason, the effect of tokenization on uncommon characters, and the various across different permutations. 
3. Release of dataset and code
summary: This paper introduces a challenging reasoning benchmark of 7k question answer pairs, built by applying grammeme-level obfuscations to Linguistic Olympiad problems.  The motivation for building a new dataset is that models rely on prior language knowledge learnt via pre-training. This dataset is built to test out models' reasoning capabilities and removes any cues that could trigger memorized translations.
weaknesses: 1. In the no-context setting, the difference between the original and the obfuscated dataset is very less; how does the author come to a concrete conclusion?
2. To check the effect of performance drop with uncommon characters, why don't we replace the context with random but real tokens that are not part of the training set? Similar to the ProntoQA dataset, where entities are replaced with false ontology.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Have you considered any evaluation metrics other than exact match (e.g., edit distance or partial credit schemes) that could capture instances where the model's reasoning is mostly correct but the final output has a minor error? What are the potential challenges in implementing such metrics for this benchmark?

2. The human study shows a 5.7% performance drop due to obfuscation. Could you discuss a bit more how we should interpret the model's performance drop in light of this? Specifically, how much of the model's drop might be attributed to the increased difficulty of processing the unfamiliar orthography, versus the removal of knowledge-based shortcuts?

3. The permutation rulesets are designed by experts. Do you think this method could be applied to other domains like mathematical reasoning or code generation? What would be the main challenges in designing "reasoning-equivariant permutations" for those domains?
soundness: 3
strengths: The core idea of the paper is excellent. Using orthographic obfuscation to create a "knowledge-free" test for reasoning is a very clever and direct way to tackle the problem of data contamination. This is a timely and important contribution.

The process for creating the permutations is very rigorous. I am impressed by the careful, manual design of the rulesets by experts to ensure the problems remain solvable. The validation by IOL medallists adds strong credibility to the method.

The experimental section is very comprehensive. The authors evaluated a wide range of models, including the latest reasoning-specific ones. The analysis goes beyond just overall scores to include "no-context" tests, robustness checks, and the correlation with language resources. The human study is also a valuable addition.
summary: This paper introduces LingOLY-TOO, a new benchmark for evaluating reasoning abilities in LLMs. It is built from Linguistics Olympiad problems. The key innovation is the use of "reasoning-equivariant permutations" to obfuscate the problem text. This process changes the orthography but is carefully designed to keep the underlying logical structure and solution steps unchanged. The authors show that while models perform reasonably well on the original problems, their performance drops significantly on the obfuscated versions. This suggests that models rely on prior knowledge and memorization rather than pure reasoning on the original tasks. The paper also includes detailed analysis on the effect of language resourcefulness and a human study.
weaknesses: The use of exact match is simple but might be too strict. Sometimes, a model might have the correct reasoning but make a small mistake in formatting the final answer. Using only exact match could penalize such cases.


The human study shows a small but noticeable performance drop (5.7%) for humans on obfuscated problems. This suggests that the obfuscation itself might add some cognitive load, making the problems slightly harder to parse, even for humans who don't rely on prior knowledge of the language.

The process of creating permutation rulesets relies heavily on manual expert work. This might make it difficult to scale the benchmark to a much larger size or to adapt it quickly to other domains.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. **On Failure Attribution (related to Weakness 1)**

   Can the authors extend the analysis to *quantify* the causes of reasoning failure? For instance, by labeling dominant error types and reporting their prevalence across models and difficulty levels.

2. **On Ruleset Comprehensibility (related to Weakness 2)**

   While we appreciate the authors’ effort on the permutation design, could the paper include auxiliary tables or diagrams that summarize the rules and constraints at a glance, to help non-linguist researchers quickly understand how reasoning equivariance is enforced?
soundness: 3
strengths: * Clear problem framing: measuring reasoning when shortcuts (knowledge, contamination) are minimized is timely and important.
* Methodological originality: the reasoning-equivariant, linguistically-informed permutations are thoughtful and non-trivial; the Turkish vowel harmony example nicely motivates rule design.
* Strong experimental design: multiple families of models, bootstrap analysis, no-context control, tokenization controls, and human study make the case persuasive.
summary: The paper introduces LINGOLY-TOO, a 6,995-QA benchmark built by applying expert, grapheme-level orthographic obfuscations to 82 UKLO problems, aiming to disentangle reasoning from knowledge/memorization. The authors define clear metrics (Mog vs. Mobf, plus robust variants), run broad model evaluations, and provide validation via auditors and a human RCT. Results show sizable drops under obfuscation (e.g., top model ≈0.60→0.48), correlations with language resourcedness, and modest gains from guided reasoning.
weaknesses: 1. **Insufficient Failure-Mode Analysis**

   The paper documents performance declines but does not deeply probe *why* models fail on obfuscated problems (e.g., difficulty inferring morphemic patterns, inconsistent multi-step reasoning, or fallback to guessing). Please add a qualitative analysis of model outputs—contrasting common errors on obfuscated vs. original items—to connect performance gaps to specific reasoning deficits. Building on this, apply statistical tools (e.g., clustering) to categorize and quantify linguistic reasoning failures.

2. **Limited Accessibility of the Permutation Ruleset**

   The permutation rules (Appendix B) are dense and lack a high-level summary in the main text. A concise overview—such as a table of key constraints and invariances, or a concept diagram illustrating how reasoning equivariance is preserved—would make the method more accessible, especially to readers without a linguistics background.


Decision: Accept (Poster)