UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?
Authors: Mukund Choudhary
KV Aditya Srivatsa
Gaurja Aeron
Antara Raaghavi Bhattacharya
Dang Khoa Dang Dinh
Ikhlasul Akmal Hanif
Daria Kotova
Ekaterina Kochmar
Monojit Choudhury
Venue: COLM 2025 (2025)
Keywords: linguistic reasoning
metalinguistics
LLM evaluation
morphology
linguistics olympiad
interpretability
low resource languages
annotation
TL;DR: The study presents a linguistic features-based annotation of Linguistics Olympiad puzzles to find LLM weaknesses; LLMs are bad at puzzles with higher morphological complexity, dissimilar to English, and when the puzzle is data constrained.

Abstract: Large language models (LLMs) have demonstrated potential in reasoning tasks, but their performance on linguistics puzzles remains consistently poor. These puzzles, often derived from Linguistics Olympiad (LO) contests, provide a minimal contamination environment to assess LLMs' linguistic reasoning abilities across low-resource languages. This work analyses LLMs' performance on 629 problems across 41 low-resource languages by labelling each with linguistically informed features to unveil weaknesses. Our analyses show that LLMs struggle with puzzles involving higher morphological complexity and perform better on puzzles involving linguistic features that are also found in English. We also show that splitting words into morphemes as a pre-processing step improves solvability, indicating a need for more informed and language-specific tokenisers. These findings thus offer insights into some challenges in linguistic reasoning and modelling of low-resource languages.

Published as a conference paper at COLM 2025
UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?
Mukund Choudhary! KV Aditya Srivatsa* Gaurja Aeron Antara Raaghavi Bhattacharya Dang Khoa Dang Dinh Ikhlasul Akmal Hani Daria Kotova', Ekaterina Kochmar', Monojit Choudhury' Mohamed Bin Zayed University of Artificial Intelligence IIT Gandhinagar Haavard University "VinUniversity Universitas Indonesia
Abstract
Large language models (LLMs) have demonstrated potential reasoning tasks, but their performance on linguistics puzzles remains consistently poor. These puzzles, often derived from Linguistics Olympiad (LO) con- tests, provide a minimal contamination environment to assess LLMs' lin- guistic reasoning abilities across low-resource languages. In this work, we analyze LLMs' performance on 629 problems across 41 low-resource languages by labelling each with linguistically informed features to unveil weaknesses. Our analyses show that LLMs struggle with puzzles involving higher morphological complexity and perform better on puzzles involving linguistic features that are also found in English. We also show that splitting words into morphemes as a pre- pre-processing step improves solvability, indi- cating need for more informed and language-specific tokenisers. These findings thus offer insights into some challenges in linguistic reasoning and modelling of low-resource languages.
1
Introduction
Human proficiency in analysing languages and linguistic reasoning contributes significantly to efficient learning from limited linguistic input. Following frameworks and goals established by Mahowald al. (2024); McClelland et al. (2019); Marcus (2020); Beguš et al. (2023), improvements in LLM metalinguistic processing capabilities can enhance general performance across diverse tasks. Linguistics Olympiad (LO) puzzles pro- vide us with rich data to probe into these LLM abilities, especially in extremely low resource languages (LRL), as past research has shown (Chi et al,, 2024). Typically, LO puzzles have par- allel data between an unknown language and corresponding forms in a language known to the solver (Bozhanov Derzhanski, 2013). The solver (typically pre-collegiate level) is expected to deduce the mapping between the linguistic structures of the unknown and the known lan- guages, which involves observing syntactic pat- terns, semantic correspondences, phonological rules, or a combination of these. We shall refer to them features akin to linguistic features (Dryer & Haspelmath, 2013). Fig. shows
In- context Data
English
Bangime
The person hits
yiwere myū dars nnc kw ne/ no
We hear you
the
nnc dere ne/ daza
The fish drinks the
ny Vye mnye
see the bone
myū nęere shura
Problems
Answers
bird
nie dgrg ne/no
nX nigrre shura
The fish sess the bone
Colors ore morteed by oncoretors for better solvabiity and readabilty osnoloted morpheme boundory that was not of paztle
pazzle
Annotated
ft
cf
eng (sim)
Broad
Class
Broakdown
Puzzle
SBV perdfis
0'0
0.00
Syntax Morphology
Figure 1: An LO puzzle with UNVEILING annotations. In-context data are transla- tion pairs, and problems are to be trans- lated on the basis of the data above.
"Equal contribution. *Correspondence to mukund. cwwudhaarr@bbzuui..cc.ee
Work done during an internship at MBZUAT.

---

Published as a conference paper at COLM 2025
Datasel
Source
Puzziles (Problems)
Classes
Difficulty
Langwages
MUDELLIN
(Chi et al, 2024)
made
46 (272)
Nour-Adleccive (d775)) (a Semantic (10140 Word Order (40%), Poseession (10%)
TINN 2.N5s. 1250) 4(11N)
19
(Shin al,, 2020)
PUZZLING
Olympiadds
Inguistic
National
96 (752)
81
(Bean et 2024)
LiNGOty
Linguistics
Olyrpladd
UK
(1133)
Compounding, Moephology.
Numbers, Phonooogyy Somantee, Syntax
Boonkthoough (6.5%) Foundation (1335%) Intermsdiiato (33,5%) Advansed (32.9)).
Round 2 (33.7%)
LINGUINE
(Sinches al, 2004
Intersabooaa
Lingusatics
Olymsiind
(894)
Number Transhtecation
Sequenze transcuction, Blanks,
Table 1: Summary of datasets with Olympiad style puzzles. % proportion in data
example of puzzle set in Bangime (an unknown language) with the corresponding English (a known language) translations. The puzzles are similar to in-context few-shot learning tasks (Brown et al., 2020) where context contains all the data needed to solve the problems.
Generally, these are built from data in endangered languages or LRLs to ensure that partici- pants do not already know the patterns in the data. Therefore, we expect these languages to be absent from the LLM training data as well. This makes LO puzzles challenging and minimal contamination setting to assess an LLM's linguistic reasoning abilities, and, for that reason, they have been incorporated into various benchmarks (Kazemi et al., 2025).
Several researchers have used LO puzzles to study language models' reasoning abilities (summarized in Tab. 1). LLMs (while being impressive in application-based language tasks) perform consistently poorly in these Olympiads, whereas high school students can solve most of the puzzles fully. While these studies show that LLMs struggle, they do not explain why. Since these puzzles reflect the complex interplay of linguistic features in natural languages, pinpointing the causes of poor performance is challenging. On the other hand, a feature-level analysis of LLM weaknesses can enhance our understanding of their linguistic reasoning and can provide insights into bridging the gap between LLM abilities and core component of human cognitive flexibility to identify patterns, extract rules, and generalise from minimal data across unfamiliar languages.
To answer the research question of what linguistic features make LO puzzles challenging for LLMs, we have annotated 64 puzzles (629 problems, 41 LRLs) from existing datasets (Sec. 2.1) with 50 linguistic features (from WALS, also including other attributes such as similarity to English see Sec. 2.2). We discover that LLM performance (Sec. 2.3) is significantly negatively correlated with the number of morphological features of language present in an LO puzzle (Sec. 3.1) and the number of features in puzzle that have low coverage of exemplifying data in the puzzle (Sec. 3.3). LLM scores also positively correlated with the average number of features in a puzzle that are present in English (Sec. 3.2). Further experiments showed that breaking down words in LRL into morphemes in morphologically complex puzzles helped LLMs (Sec. 3.5), seemingly indicating the need for more language- specific handling of tokenisation of words in LRI. Findings from the study could inform LLM-based research on sample-efficient LRL learning, metalinguistic reasoning, and more.
2 Setting the Stage for UNVEILING
In this section, we set the stage for unveiling LLM weaknesses in solving LO puzzles by:
1. Curating data and finding unseen puzzles.
2. Establishing an annotation scheme to break down puzzles into linguistic features.
3. Obtaining LLM responses to evaluate performance against the feature annotations.
2.1 Data
Existing Datasets An overview of the four datasets that are based on LO-style puzzles can be found in Tab. 1. All of these reveal that LLMs are poor at solving LO problems based
World Atlas of Languages by Dryer & Haspelmath (2013)
2

---

Published as a conference paper at COLM 2025
Finding Features
English
Bangimo
panee
fiecre açd
Broad classes
WALS classes
In English? for similar??
Context Constrained?
features
cenfect Data
The Bi otta
Eaglish
Bangime
Syntax Morphology
WardOrdar SimpleClause
0 (2)
Pistle
The pince talo
years eys dare
de
persfix
0 (1)
Whtch Uhe contost
Jal
1el more than
Solving the
su beao
agga seure dieae
Mapphng frotures Cleases besed on araabngenttiiggee
[1) thene a samilrr
coctrtatoo "guir bununn
winkh are colie the problame??
3 Marking Attributts
Puzzle
Figure 2: UNVEILING annotation pipeline and attributes marked in sample from the data.
on Exact-Match Accuracy, which measures the percentage of problems per puzzle where the predicted final answer exactly matches the ground truth. LLMs performed the best (40-60%) in solving puzzles from MODELING (Chi et al., 2024) (author-made, easier LO-style puzzles) and Foundation-level (younger students) puzzles in LINGOLY (Bean et al., 2024), while performing the worst (25%) on International Linguistics Olympiad (IOL) puzzles from LINGUINI (Sánchez et al., 2024).
Modsling
Eoacc waakkiioo wow.eaaaaa Secooeee (O)
Filtering out Solvable Puzzles LLMs struggle with LINGUINI's set of IOL puzzles as they are tougher and are designed for par- ticipants who have more experience with LO puzzles. Thus we only consider puz- zles from MODELING (M) and LINGOLY (L). To maintain uniformity across datasets and select the simplest tasks, we retained only Dosetta-Stona style (Bozhanov & Derzhan- ski, 2013) puzzles which had paired transla- tions context and problems that required participants to either translate from an LRL to English or vice versa. This filtering step led to an initial pool of puzzles consisting of all 48 M puzzles? and 45 L. puzzles out of 90.
Lingtly (1)
AEw Peatre
CPF--4
Stama2-77 PM
La Th0
Figure 3: Results on the NoContext prompt to find puzzles with potential contamination.
Contamination Checks and Pruning MODELING (M) and LINGOLY (L) employ measures to account for data contamination, i.e. being privy to the exact puzzles or being familiar with the languages involved, which may lead to inflated results. M puzzles are based on extremely low-resource languages that were reported to have 0% data leakage to OpenAI models up to GPT-4. Bean et al. (2024) (L) report all analyses with a NoContext baseline for querying problems without the puzzle in-conlext data and improvements over it.
We employ similar measure by using a NoContext prompt (App. A) followed by a manual inspection of all model responses and drop the contaminated puzzles from both datasets (Fig. 3): 7 out of 48 M puzzles (answered by the GPT-403 without any in-context data) and 22 out of 45 L puzzles were pruned. This is expected as there are more high-resouree languages (e.g., Romani and Nepali) in the L puzzles set, and corresponding puzzles are available online via UKLO website.
The final dataset contains 64 puzzles (629 problems) 41 (272) M + 23 (357) L.5
AAll M puzzles were first solved and checked by the authors for inconsistencies (L puzzles were already checked by UKLO participants). Issues included insufficient data, absence of affixes in answers, detailed account of minor changes made the data provided in App. B.
3This model was not public at the time of MODELING's publication.
4 Faah puzzle is made of multiple translation problems, which are evaluated independently. "Remaining contamination (S 4%) was where models matched answers by picking random words and piecing together possible solution from both languages' context and questions.
3

---

Published as a conference paper at COLM 2025
2.2 Feature & Attribute Annotations
Background & Motivation LLM performance explanations have been limited to using broad external difficulty metrics (e.g. participant scores Likert ratings see Tab. 1). As LLMs might model language differently (McCoy et al., 2024; Warstadt et al., 2020), such mea- sures conflate linguistic reasoning with puzzle-specific quirks and offer low interpretability.
A more comprehensive probe of LLM linguistic abilities in English was done by Warstadt et al. (2020) and Waldis et al. (2024). They craft linguistic textbook-style tasks to find that linguistic competence correlates with the model size, but also that different features have different learning curves and instruction tuning negatively influences performance and natural quality - i.e., they mimic linguistic features instead of generalising productively like a native speaker. They also find that formal linguistic abstractions like syntax in English are more easily modelled by LLMs as compared to less explicitly lexically structurally represented abstractions like pragmatics.
Attribute
Classes
y
LO puzzles offer an opportunity to conduct similar studies on LLM abilities in LRLs. However, they are based on real lan- guage (not artificially controlled) data, each puzzle involves simultaneously changing features from different linguistic abstrac- tions like phonology, and semantics (Fig. 1).
b_Syntax
0.519
0.381
Proportion of Broad Attributes per Puzzle
b_Morphology
0.307
0.322
Semantics
0,132
0.243
b. P.Ponolloyy
0.042
0.11
Worrdrrder
0.437
0.399
*.ComplexSentences
0.004
0.023
Thus, the following sections describe how we analyse LO puzzles by building on top of Bozhanov & Derzhanski (2013)'s insight that LO puzzles are crafted "to be solved by deducing linguistic patterns, with other sources of technical complexity... ... not mak- ing them linguistically interesting". As Fig. indicates, we break them down into fine- grained, linguistically grounded proxies for complexity, which we call "Atfributes" from this point in the paper. Attributes include 50 linguistic "Features" " from abstractions like morphology, synlax, etc., and other derived attributes such as how many features are similar English features, and if there are redundant context pairs.
Proportion WALS Attributes per Puzzle
*.NominalSyntax
0.096
0.186
SimpleClauses
0,131
0.192
Verballateegries
0.03
0.078
NominalCategories
0.164
0.244
Morpphlogy
0.075
0.181
w_lexicon
0.023
0.131
P.hooology
0.039
0.109
Proportion of Data Constrained
Features per Puzzle (cf)
0.474
0.435
Proportion of Features also Present in English per Puzzle (eng)
0.296
0.36
Proportion of Features
Similar to English per Puzzle (eng sin)
0.498
0.381
Table 2: Summary Statistics of all attributes means (1) and standard deviations (o)).
Methodology ex-LO participants- with training in Linguistics and LO puzzle design independently annotated all puzzles with WALS linguistic features and classified them in three ways (outlined below). For an overview of the annotation pipeline, see Fig. 2; the annotation guidelines presented in App. F).
2.2.1 Solving Puzzles
All puzzles were first solved independently by the annotators. They also marked the context data used translate the problems to help with the next steps.
2.2.2 Finding Features (ft)
Broken-down contexts were then used to extract features. A feature here (defined by Dryer & Haspelmath (2013), similar to Warstadt et al. (2020)'s phenomenon) is structural property of language that describes one aspect of cross-linguistic diversity. Annotations were based on a set of 50 WALS features (App. D.) comprising various grammatical and semantic affixation, vowel harmony, abstract concepts like moiety, and more. Feature description with examples and citations can be found in App. D.6
éCode and annotations are publicly available at https::/ts / wtt...co/mmkknndck//eeeiei......
4

---

Published as a conference paper at COLM 2025
2.2.3 Marking for Attributes
(a) Linguistic Attributes We cluster features into linguistic abstractions defined by WALS and UKLO? for more generalized but still linguistically grounded analysis of LLM abilities. Based on UKLO, we define Broad classes (b ): Syntax (phrasal/ /clausal rules), Phonology (sound units), Morphology (subword units), Semantics (meaning units), and based on WAL.S, we define WALS classes (w.):
Nominal/VerbalCat Caggaaa Relating nouns and verbs such as case and tense.
WordOrder: Order of sentence constituents like adjective-noun subject-object.
Phonology: Features based on units of sound, tones, vowel harmony, and so on.
Lexicon: Lexical items representing number systems, alienability, and so on.
ComplexSentennces: Relative clauses and other multi-clause constructions.
NominalSyyytaa//immeelllssssss Minor parts like possessive or voice markers.
Morphology: Reduplication and other features that use morphemes.
(b) Similarity to English (eng): To unveil biases via typological distance from English, marked no. of features that were present in English or showed similar behavior (sim).
(c) Constrained features (cf): It is generally easier for solver uncover a feature and its rules in an LO puzzle when it appears in different contexts with redundant minimal pairs of translations. A puzzle is tougher if another feature changes in the same evidence pairs. Thus to unveil if LLMs face similar difficulties, we mark how many features have exactly enough translation pairs to uncover the respective feature from the context data. 8
2.2.4 Resolving Annotation Differences
70-
Oeercl
Finally, after comparing the independent annotations, we found that the annotators agreed almost perfectly on which WALS features were present in puzzle (Cohen's 's k=0.893), substantially on how redundan- cies per puzzle (x=0.631), and on the count of features per puzzle that were present in English as well (x=0.761).
60.
Trommatt on from Eng lsh Trnedbo on to English
Maat Acuurayy (Pg)
60x
00
10-
They resolved the disagreements by relabel- ing the puzzles with newly agreed-upon reasoning. Specifically, for finding con- strained features, they independently came up with pairs in contexts relevant to uncov- ering the feature and agreed on a union. For English similarity, they referred WALS and other research in English linguistics (cited for all features in App. D).
Gm1-4
Get- & Rurba
Lanal 44500
Uanna Liana 208
Watreil Sx70
Uleana F0B
Uleaa2 126
Ueeaa
Figure 4: Benchmarking results overall and language direction-specifi avg. Exact-Match accuracies for all models across prompts.
Tab. 2 provides statistical summary of all attributes used for analyses (Sec. 3).
2.3 Benchmarking Experiments
We evaluate 10 L GPT 4, 4o, 3.5-Turbo (OpenAI, 2024), Llama2 7B, 13B, 70B (Touvron et al., 2023), Llama3 8B, 70B, 3. 1-405B (Dubey et al., 2024), and Mixtral 7x8B (Jiang et al., 2024). These were prompted with 6 styles and strategies from previous wo
7 A quick reference for WALS classes and for UKLO classes
&There can never be fewer than enough contexts by design.
9 "Moeels were selected to be representative of various parameter sizes, families, and contain both open & closed models for balance and comparability. Refer to App. A for more details.
M marked prompts are borrowed from Chi al. (2024), borrowed from Bean al. (2024).

---

Published as a conference paper at COLM 2025
Null: prompt to gauge model performance without explicit instructions.
MinimalM: similar to Null but with task description.
Hand-Tuned prompt fine-tuned by IOL medalist.
Basic CoTM: Chain-of-Thought prompt asking the model answer step-by-step.
Full CoT similar to above but with an example of reasoning in a dummy puzzle.
LingOly A more detailed prompt similar to the Minimal prompt above.
We find that our results on the curated dataset are in line with existing literature (Fig. 4):
Larger and closed-source models are better (Srivastava et al., 2023). Smaller mod- els (Mixtral-8x77 and L1ama3-88) outperform larger ones 5-Turbo) rarely, which can be explained by Waldis et al. (2024)'s finding that linguistic skills vary depending on architecture and tuning parameters.
Models have a translation direction bias and score better when translating from the low-resource language to English compared to from (Sahin et al., 2020).
For more detailed breakdown and other obser- vations, please refer to App. C. Note that we use a model's average performance across prompts for the following analyses.
In Contont Transfation Transttiion Poirs
hexp
mtuup
Juan wetk ye veexyeeipy
Juan watches him
wir yauk yexpl sees the slog
Analyses UNVEIL that:
Ejts yol moxurunk nexyejiou
watch the baby
To unveil patterns behind LLM performances we compute the correlations between the attributes and exact match scores across prompts. Analy- ses are only based on significant correlations.
Isg. 2sg.. 3agi noun (obi)
verbs +for long verbal affix
QuestionnAArswer Pairs
Wll moxirunk yexp
boby sees
Mejts Juan mntuup
You work (for) Juan
3.1 Morphologically complex puzzles are difficult for LLMs
work (for him
Ejts ye's ntunpy
The baby watches dhg
yel maxufunk yill ui yexyyeipy
We find that the count of morphological features (b Morphology) in an LO puzzle strongly pre- dicts how well model can solve the puzzle. Tab. 31l shows that higher morphological feature count correlates with lower exact match scores across all LLMs (all significant p-values < .001). These scores are averaged over all prompting strategies; however, this observation holds over most individual model-prompt combinations as well. For all combinations, please refer to Tab. 12 in Appendix.
Figure 5: An annotated Ayutla puzzle. Colors mark equivalent units.
To provide some illustrative examples, puzzle in Ayutla Mixe (Mixe-Zoque family, Fig. 5) encoded many morphological fea- tures, and no LLMs solved more than one question. In contrast, Bangime (language isolate from Mali, Fig. 1) had only one such feature (affix for person), most LLMs solved except for the one that involved the affix. Finally, a puzzle with no morphological complexity, based on Engenni (Niger-Congo family, Fig. 7), was solved fully by all models.
Mapudungan
kümé wentru good
kümě pafu good turkey
the problems
Mixtepec Zapotec
guits nquits
white paper
Note that smaller and older models that, on average, score lower across all puzzles also showed lower correlations with b Morphology. Accurate answers by these models were mostly random or chance matches and were not consistent across ques- tions in the same puzzle (better scoring models would either consistently not apply morphological rule or it productively across answers).
mEEd nquits
white child
Figure 6: Sample of con- text from two puzzles.
845 and "*s*" imply that corresponding p-value less than 0.05, 0.01, and 0.001, respectively.

---

Published as a conference paper at COLM 2025
1lama2 (7B)
Ilama2
Ilama2
Ilama3.1
GPT
GPT
(13B)
(70B)
(8x7B)
(SB)
(70B)
(405B)
(3.5-Turbo)
(4o)
(4)
Average Exact-Match Score
0.074
0.143
0.224
0.271
0.269
0,381
0:405
0248
0.488
0:408
Count of b Morphology
-2219
-0.36
-0496"
-0.46"
0.522"
0.583"
0.572"
0.486"
0.677
0.553
Average
0.031
0.053
0.368"
0.3"
04491 TM
0.364"
0.276'
0.399
0.58
0.381
Count of of
<0.14
40.188
-0.44""
-0.466
-1.667"
-0.55"
"0.92""
-0.43"
-0.52""
-0.577
Table Correlation values (Pearson) between exact-match scores and attribute value aggregates per puzzle for selected attributes against all models across prompts.
In Sec. 3.5, we actively test our hypothesis of LLMs being weak in splitting morphemes into the right tokens in morphologically richer languages. Results show that explicit morpheme boundaries increase the scores in better seatforperforming models.
eni dhi we eat
bhu dhi eat In- In-Context Translation Pairs
3.2 LLMs have English bias
abhwa mise
the dog sleeps
We find that a puzzle's average number of features that are present in English (avg eng) strongly predicts how well an LLM scores on that puzzle. Tab. 3 shows that a higher aver- age similarity positively correlates with higher exact match scores. Note that the average model-prompt combinations have a slightly lower significance level (p-value S .01); however, many individual model-promp combinations are consistent with this observation (Tab. 12).
mi bidha - walk
afeni fyani
the bird flies
abtwa dhi
the dog eats
complee
Redundonn
afeni bidha
the bird walks
nouns, verbs
Question Answer Pairs
As illustrative examples (Fig. 6), p potec (Oto-Manguean family) required the solver to figure out Noun-Adjectin order (opposite to English); however, LLMs could not solve more than 30% the questions this puzzle. In contrast, Mapudungan (Araucanian family) had the same setup as Mixtepec Zapotec, except that the word order in noun-adjective phrases in this language is the same as in English, so most LLMs fully solved this puzzle.
puzzle in Mixtepec Za-
fly - bhu fyani
eni mise we sleep
mi mise sleep
afeni dhi - the bird eats
Figure 7: Engenni Annotated
Note that most correlations are in the same range and consistent across prompts and models, all model types are similarly biased towards English feature-like puzzles.
3.3 More constrained evidence for feature makes a puzzle difficult for LLMs
We find that the count of features with exactly enough pairs needed to uncover them (cf) in an LO puzzle strongly predicts how well a model can solve the puzzle. Tab. 3 shows that higher exact feature count correlates with lower exact match scores across most LLMs (all significant p-value < ,001 and negative correlations). Note that these scores are averaged over different prompting strategies however, this observation holds over most individual mooep- combinations as well (Tab. 12).
The Engenni puzzle (Fig. 7) shows that the last two examples in the puzzle are redundant and are not required to uncover either the feature (Subject-Veri) Verb order) or vocabulary. Thus, there is more than enough evidence to figure out all features. LLMs scored higher on such puzzles - i.e., puzzles where both annotators found ample evidence for features in them.
Note that smaller and older models lack these correlations due to low performance in general (especially on more data-constrained puzzles). Further, Null and Minimal prompts (which did not explain the task) weaken correlations for other mid-sized models.
3.4 Other Observations
Some other statistics and significant correlations that were only localised to certain model prompt combinations are summarised below. 12
12Other attributes that were not mentioned in the above analyses were highly correlated with the features that were not represented in more than half of the dataset, make any generalised claims.
7

---

Published as a conference paper at COLM 2025
w.Wrrdrrerr showed significant and neg- ative correlations with the smaller and older models (Llama2-7B & 138) for spe- cific prompts. Note that these are also the
Morpheme Boundary
Average Evact-Harch Soore
0.5-
Wibbaspcs
9.5
models that perform poorly overall and show no correlation with other attributes. Semantics- and phonology-based attributes across the two classification systems were scarce in the dataset, leading to low sig- nificance scores. However, at slightly lower significance level (p-value S .01), the models from OpenAT showed a nega- tive correlation, implying that puzzles with more semantics- phonology-based features were tougher for OpenAl models (consis- tent with Waldis et al. (2024); Warstadt et al. (2020)). We also extended our experimental results to DeepSeek-R1, a model tuned for Inference-Tinee Compute (DeepSeek-Al et al., 2025) see Appendix C.1 for more details.
9.4-
0.2-
Ulmoad
Lianed 2roa
Llanad 278
Uloaa Tob
Dianat-i 18
UlmaL 4000
Figure 8: Performance change in models upon making morpheme boundaries explicit
3.5 Breaking morphemes down helps LLMs
Insightful findings from Sec. 3.1 unveiled that the morphological complexity of an LO puzzle strongly predicted LLM scores. We further probe subset of 9 puzzles of varying morphological complexity and feature types on which models did not perform well.
As illustrated with pink bars in Fig. 1 and with boxes in Fig. 5, we first annotate the morpheme boundaries in the context (based on grammar (App. D) and answers), prob- lems, and answer data of these puzzles.
Puzzle
Exact-Match Score (+Gain)
T_Hokano
0.333 (+0.333)
L. Koreliam
0.769 (+0.154)
Lardil
0.333 (+0.167)
Then we run inference using selected mod- els (with prompt styles that led to best scores respectively) again, with two vari- ations of morpheme boundary representa- tions: whitespace-separatte
M. M_Ayutia Mixe
0.0 (+0.0)
M. Bangine
0.4 (+0.2)
M. Bangime 5
0.6 (+0.2)
M M.Gaagu_YYmiin Yimithir
0.7 (+U.2)
(which made morpheme boundaries the same the word boundaries), and @-separated. We ob- serve that there is substantial increase in scores, both quantitatively and qualitatively (Fig. and Fig. 20). Note that maintain the original tokenizer for all experiments and add separation characters (i.e., whitespace or @) to the input text before tokenization. Although there is overlap between the resultant word-piece tokens and true morphemes, it is incidental and low (with GPT-4, we observed only 0.62 Jaccard similarity between the morphemes and the word-pieces).
M. Kutenai
1.0 (+0.6)
M. Tolonac
0.333 (+0.167)
Table 4: Llama3-708 puzzle-wise score gain marking morpheme boundaries explicitly
The results thus show that all models that were performing above 15% on the subset solved more problems accurately after splitting. While 1ama2-7B, 13B, and Llama3-8B were mostly unaffected by the splits, other models like Llama3-70B (Tab. 4) reached 50% accuracy, solving some puzzles better than the baseline scores of much bigger models. The substantial score improvements after morphological splitting demonstrate that morphology was indeed the primary bottleneck, while the remaining performance gaps in some puzzles reveal the contribution of non-linguistic reasoning components.
Furthermore, separating morpheme boundaries by a whitespace or by '@' did not lead to very different results. Fig. 9 shows Llama3-708's response change after split an extended version of the Bangime puzzle from Fig. 1. The response was formatted and worded before and after splitting in exactly the same way by the LLM (we have added ellipses for simpler presentation), except that after marking the morpheme boundaries, it identified the right morphemes and their translations after splitting.
8

---

Published as a conference paper at COLM 2025
Finally, we also find that LLMs could not solve some puzzles with morphemes that packed a lot of semantics (e.g,, kinship in Lardil) or with lot of morphemes (e.g, Ayutla Mixe - Fig. 5).
Lat's analyze the given exaeples find patterns and relationships betneen Bangline phrases and their English
Fros the exaeples,
observe the folloning:
""ayy"" seces to be related to "op-
""inne appears to be related to "*ke"*
"'nang" contains *no"", a.d
However, for most cases (Tab. 11), we observe that just splitting LRL context data into mor- phemes can lead to better translations.
""ne"" 15 a suffix that appears in phroses Aith "sa"" or *ape ane
"'nejind"" associated ""svie"
""dong" associated with ""hee bird"
Now, lat's try to translate "hhe fish swins"".
4 Discussion & Related Work
Fros the axaeplas, we know that "ogee is "*the fish"*
and ine i5 ""sii""n
"iinnt* means ""ani""
As interest grows in understanding how LLMs'
processing of language aligns with linguistic the-
tesical combination English: ** fish swims"
ory (Futrell & Mahowald, 2025), researchers find
Bangine: "age nejind"
Bangine iim:::: "age jiné"
that LM abilities pertain specific linguistic features rather than languages in the context of
Figure 9: Llama3-70B response change
multilingual tasks (Acs al., 2024). The UNVEIL- ING method probes LLMs in a similar fashion.
(before to after) recognises the per- son affix after splitting morphemes and reaches the correct final answer.
We find that breaking LO puzzles down into lin- guistic attributes to study LLMs is reliable (i.e., confirms some existing findings from an intrinsic perspective of linguistic features in the puzzle instead of Likert-based difficulty ratings (Chi et al., 2024)) and can be expanded (i.e,, allows us to unpack more about LLM performance).
We show that data constraint adds to a puzzle's difficulty for LLMs by annotating redundancy for each feature in a puzzle. Other research shows extrinsically that puzzles are made easier for LLMs by adding natural language data across linguistic abstractions (Zhu et al., 2025), prompting models to generate data from typologically close language (Ramji & Ramji, 2024), or by augmenting data through character replacements (Watanabe, 2024). 13
Data:
English bias: We show feature-level typological bias towards linguistic features in En- glish. Past research in LM performance on LO puzzles and results from Sec. 2.3 arrive at this conclusion extrinsically by showing that there is also translation directionality bias towards English, where all models showed that LLMs are better at translating into English as com- pared to translating into an LRL. Outside of LO puzzles, in a cross-linguistic generalization ability probing of LLMs through a Wug Test (Anh et al., 2024), authors found that most LLMs do not add affixes nonce words, add them incorrectly, or even use English affixes in another lower-resourced language and context.
Interestingly, Warstadt et al. (2020) find LMs to have concentrated robust knowledge of specific domains of grammar in English (based on supervised training on acceptability data esdy of minimal pairs). They note that "these sludies cover a limited set of phenomena, excluding ellipsis, quantification, and countless others. This is likely due to the labour-intense collecting such targeted minimal pairs". " We believe that LÔ puzzles are by design positioned as neat set of minimal pairs in extremely les-resdd languages, addressing this problem.
nature of
Morphology& & Tokenisation: While our analyses show significant correlation between poorer LLM performance and an increasing number of morphological features in an LO puzzle (based on unseen natural language), most other LO research confirms this very superficially. Upon making morpheme boundaries explicit, we found that LLMs performed better generally. This implies need for deeper focus revamping tokenisation ap- proaches by either researching ways for tokenisers to adapt to language-specific contexts or using tokeniser-free approaches.
Much like our experiments, Ismayilzada et al. (2024) investigate the morph generalization abilities of LLMs through the lens of compositionality and find a major gap in their perfor-
For most of the work cited here, the data creation strategy is either not fully explained, or synthetic data is either linguistically infeasible or not validated by native speakers.

---

Published as a conference paper at COLM 2025
mance compared to humans (in agglutinative languages). Weissweiler et al. (2023) also apply a Wug test in typologically diverse languages (English, German, Tamil, and Turkish) to arrive at the same conclusion.
Haslett (2025) finds that this is because LLMs do not directly perceive units like affixes and semantic radicals. Here, subword tokens determined by frequency, not semantics, and end up obscuring meaningful units. Larger vocabularies of longer units hide mob-morpeemic units from LLMs.
However, the tokenisation problem has been recently proven to be NP-complete (Whit- tington et al., 2024), and various new solutions, such as Greedy tokenisation (Lim et al., 2025), Byte-free tokenisation (where dedicated local transformers identify dynamically sized patches of bytes) (Pagnoni et al., 2024), or trigram-based hashing of vocabulary (Deiseroth et al., 2024), might help with the morphology gap.
Another line of relevant work to use language-specific tokenisers. Dang et al. (2024); Bayram et al. (2025) find that subword-based based and morphologically informed tokenization works better for Turkish, whereas character- tokenization benefits Hindi morphology. New methods like MorphBPE (BPE with an additional step of blocking pair-merging apart from morpheme boundaries) (Asgari et al., 2025; Lerner & Yvon, 2025) showed faster convergence in training and more alignment with morphologies of languages with different richness (Fanar et al., 2025).
Consequently, there is need for more intrinsic approaches to evaluation of model abilities in morphological processing. Ali et al. (2024) show that low fertility scores are a necessary but not sufficient criterion for this. Arnett & Bergen (2024) also claim that tyte-premiums and scaling data need to be factored in with morphology-tokenisee to explain performance degradation in LRLs. Based on our analyses of LLMs,, we believe that designing translation puzzles in unseen languages (like LÔ puzzles), controlled for specific and rare morphological features, is a starting point for that.
alignment differences
Conclusions & Future Work
We presented linguistic feature-based breakdown of Linguistics Olympiad style puzzles and a correlation study between various features of linguistics puzzles and the performance of LLMs on these puzzles. Our analyses show that LLMs are better at solving puzzles with more English-like features, struggle to solve puzzles with more features that have constrained data to uncover them, and puzzles that have a higher number of morphological features in low-resource and unseen languages. This shows that LLMs can be studied in more depth in terms of their linguistic reasoning and puzzle-solving abilities and weaknesses in Linguistics Olympiad
setting.
We further probed models by making morpheme boundaries explicit unveil that LLMs performed better with this small change in the puzzles. This motivates a deeper look into tokenisation techniques used in multi-lingual settings, specifically for extremely low- resource languages or unseen data.
We believe that a linguistic feature-based analysis of LO puzzles can provide better insight into how LLMs unpack new language data, and that we could design more targeted LO- style puzzles to unveil specific aspects. Other analyses could include addressing different kinds of morphology, e.g. inflectional as well as derivational, to better understand the observed effects. We could also explore how different prompting strategies change LLM responses and their consistency in solving LO puzzles (Lin et al., 2023).
Finally, we hope that this work may also inform other tasks such as studying multi-modal LLM capabilities in deciphering low-resource languages' orthographies (Shih et al., 2025) or in making educational language learning apps (Vasselli et al., 2024) annotated with hand-crafted features to bolster LLM abilities in modelling low resource languages.
10

---

Published as a conference paper at COLM 2025
Ethics Statement
With regard to the datasets used in our work, we comply with all available licenses and/ have sought authors' explicit permission to use their datasets. We have made minimal changes and will publish only mappable annotations and list of fixes for full reproducibility. The annotators volunteered for the task and are co-authors of the paper. We foresee no other major ethical concerns that could arise from this work or the data annotated.
Reproducibility Statement
All code and data are publicly released https://githhe com//mukknddd2/uueeillll.. http::://ggth.b...om/uuuudd//d/d suring that our experiments can be independently reproduced. To guarantee consistency, all experiments involving LLMs were performed with a temperature setting of 0. Detailed information about the specific model variants used and the complete querying specifications can be found in App. A. We also note that the contamination check performed in this study does not rule out the possibility that models released after the study may have been trained on the publicly available subset puzzles used herein.
en-
References
Judit Acs, Endre Hamerlik, Roy Schwartz, Noah A Smith, and Andras Kornai. Morphosyn- tactic probing of multilingual BERT models. Natural Language Engineering,3000 30(4):753-792, 2024.
Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Jo- hannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, et al. Tokenizer choice for LLM training: Negligible or crucial? In Findings of the Association for Computa- tional Linguistics: NAACL 2024, pp. 3907-3924, 2024.
Dang Anh, Limor Raviv, and Lukas Galke. Morphology Matters: Probing the Cross- linguistic Morphological Generalization Abilities of Large Language Models through Wug Test. In Tatsuki Kuribayashi, Giulia Rambelli, Ece Takmaz, Philipp Wicke, and Yohei Oseki (eds.), Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, PP. 177-188, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 /00653/vl/// /12/2224.c2lm11..1 URL hattp:://aclannthoooggoo http://cllanttoooooorrg////2/2/0 org/2024. cm
Catherine Arnett and Benjamin K Bergen. Why do language models perform worse for morphologically complex languages? ar preprint ar SrXiv:2411.14188, 2024.
Ehsaneddin Asgari, Yassine El Kheir, and Mohammad Ali Sadraei Javaheri. MorphBPE:
A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies. ar Xiv preprint ar Xrix:2502.0088/ 2025.
M Ali Bayram, Ali Arda Fincan, Ahmet Semih Gümüş, Sercan Karakaş, Banu Diri, and Savaş Yıldirı. Tokenization Standards for Linguistic Integrity: Turkish as Benchmark. Xr preprint arXio:2502.07057 2025.
Andrew M Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan Chi, Ryan Chi, Scott A Hale, and Hannah Rose Kirk. LINGOLY: A Benchmark of Olympiad-Len Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages. arXiv preprint arXiv:2406.0613 2024.
Gasper Beguš, Maksymilian Dabkowski, and Ryan Rhodes. Large linguistic models: Ana- lyzing theoretical linguistic abilities of LLMs. arXiv preprint arXiv:2305.00943 2023.
Bozhidar Bozhanov and Ivan Derzhanski. Rosetta stone linguistic problems. In Proceedings of the Fourth Workshop on Teaching NLP and CI. pp. 1-8, 2013.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini
11

---

Published as a conference paper at COLM 2025
Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Nathan Chi, Teodor Malchev, Riley Kong, Ryan Chi, Lucas Huang, Ethan Chi, R. McCoy, and Dragomir Radev. MODELING: A Novel Dataset for Testing Linguistic Reasoning in Lan- guage Models. In Michael Hahn, Alexey Sorokin, Ritesh Kumar, Andreas Shcherbakov, Yulia Otmakhova, Jinrui Yang, Oleg Serikov, Priya Rani, Edoardo M. Pontii, Saliha Mu- radoğlu, Rena Gao, Ryan Cotterell, and Ekaterina Vylomova (eds.), Proceedings of the 6th Workshop on Research in Computational Linguistic Typpoogy and Multilingual NLP, pp. 113-119, St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL wrg.///....//.4/..// sigtyp-1.14
Thao Anh Dang, Limor Raviv, and Lukas Galke. Tokenization and Morphology in Mul- tilingual Language Models: A Comparative Analysis of mT5 and ByT5. arxiv preprint ar arXit:2410.11627 2024.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W.L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y.K. Li, Y. Q. Wang, Y.X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DéepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. URL https://arxxt wwwpaaawwi....raabba. orgg/bs//501111 12948.
Björn Deiseroth, Manuel Brack, Patrick Schramowski, Kristian Kersting, and Samuel Weinbach. T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Represen- tations for Mefory-Efficient Embeddings. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), ,), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 21829-21851. Miami, Florida, USA, November 2024. Associ- ation for Computational Linguistics. doi: 10.18653 10////112222..m/// 20224.emnlp-main 1217. URL https://aclantholoogy org/2024 w0w.244.llppmai.nn...m... 1217/.
12

---

Published as a conference paper at COLM 2025
Matthew S. Dryer and Martin Haspelmath (eds.). WALS Online (e2020.3). Zenodo, 2013. doi: 10.5281//emood..785533 URL htwps:
http//.....11///1 wtw.///eeoo.. 7385533.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela Fan, et al. The Llama 3 Herd of Models, 2024. URL htttp:://arxiv.or//abaaa http:/aabb220..21725 21783.
Team Fanar, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowd- hury, et al. Fanar: An Arabic-Centric Multimodal Generative AT Platform. arXiv preprint arXit:2501.13942 2025.
Richard Futrell and Kyle Mahowald. How Linguistics Learned to Stop Worrying and Love the Language Models. Xiv preprint arXiv:2501.17047 2025.
David A Haslett. Tokenization changes meaning in large language models: Evidence from Chinese. Computational Linguistics, pp. 1-30, 2025.
Sälevä, Mete Ismayilzada, Defne Circi, JänveSä Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman, and Lonneke van der Plas. Evaluating morphological compositional generalization large language models. arXiv preprint arXto:2410.1265, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of Experts, 2024. URL https://arxiv. org/abs/2401 04088.
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q Tran, Quoc V. Le, and Orhan Firat. BIG-Bench Extra Hard. arx preprint arXiv:2502.19187, 2025.
Paul Lerner and François Yvon. Unlike "Likely", "Unlike" Unlikely: BPE-based Segmen- tation hurts Morphological Derivations in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 5181-5190, 2025.
Jia Peng Lim, Davin Choo, and Hady W Lauw. A partition cover approach to tokenization. ar Xiv preprint ar xrXt::2501.0624 2025.
Zheng-L. Miing-LLin Lin, Chiao-Han Yen, Jia-Cheng Xu, Deborah Watty, and Shu-Kai Hsieh. Solving Linguistic Olympiad Problems with Sree-of-Thoughe Prompting. In Jheng-Long Wu and Ming-Hsiang Su (eds.), Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023), pp. 262-269, Taipei City, Taiwan, October 2023. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP). URL / https:///aclanthoolggy anthology. org 2rgg/223... rocling-1. 33.
Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models. Trends in cognitive sciences, 2816):117-540, 2024.
Gary Marcus. The next decade in Al: four steps towards robust artificial intelligence. arv preprint ar xrXiv:2002.06177, 2020.
James L McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze. Extending machine language models toward human-level language understanding. Xiv preprint ar Xit:1912.058)) 2019.
13

---

Published as a conference paper at COLM 2025
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy, and Thomas L. Grif- fiths. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(::): e2322420121, 2024. doi: 10.1073/pnas 232..2222220111. URL https: http:::/www..pmt pnas. org htrg//oi/ie /abs/ 10. 10.10/3/pnas. http3//p2aa.22222122222 2322420121.
OpenAL. Hello GPT-4o, May 2024. URL https:///ppenaa.commiin/e///////
Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Mar- garet Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, et al. Byte latent transformer:
Patches scale better than tokens. arXiv preprint ar Xfi::2412.09871,, 2024.
Raghav Ramji and Keshav Ramji. Inductive Linguistic Reasoning with Large Language Models. arXiv preprint arXi:2441..17819, 2024.
Gözde Gül Şahin, Yova Kementchedjhieea, Phillip Rust, and Iryna Gurevych, PUZZLING Machines: A challenge on learning from small data. arXiv preprint ar Xiv:2004.13166 2020.
Eduardo Sánchez, Belen Alastruey, Christophe Ropers, Pontus Stenetorp, Mikel Artetxe, and Marta R Costa-jussà, LINGUINI: A benchmark for language-agnostic linguistic reasoning. arv preprint arXim:2409.12122 2024.
Yu-Fei Shih, Zheng-Lin Lin, and Shu-Kai Hsieh. Reasoning Over the Glyphs: Evaluation of LLM's Decipherment of Rare Scripts. arv preprint arXi:::501.17775, 2025.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, pp. 1-95, 2023. ISSN 2835-8856.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, and Shruti Bhosale, et al. Llama Chat Models, 2023. URL
2: Open Foundation and 2307. 09288.
org//bs//
Justin Vasselli, Arturo Martínez Peguero, Junehwan Sung, and Taro Watanabe. Applying lin- guistic expertise to LLMs for educational material development in indigenous languages. In Proceedings of the 4th Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP 2024), pp. 201-208, 2024.
Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, and Iryna Gurevych. Holmes: Benchmark the Linguistic Competence of Language Models. ar preprint ar dxXie:2404.18925, 2024.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8:377-392, 2020. doi: 10.1162/ a_0321, URL hutps:tts//aaaaannoooo
org/2020. tacl-1 25/.
Junehwan Sung Hidetaka Kamigaito Taro Watanabe. Exploring Metalinguistic Awareness in Pre-trained Language Models through the International Linguistics Olympiad Challenges. anlp, 2024.
Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, et al. Count- ing the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model. arXio preprint ar arXio:2310.15113, 2023.
Philip Whittington, Gregor Bachmann, and Tiago Pimentel. Tokenisation NP-Complete. ar Xiv preprint arXit:2412.152100 2024.
14

---

Published as a conference paper at COLM 2025
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, An- thony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Hug- gingFace's Transformers: State-of-the-art Natural Language Processing, 2020. URL https://arxiv.oor htt../orgg/abb/11..007777
Hongpu Zhu, Yuqi Liang, Wenjing Xu, and Hongzhi Xu. Evaluating Large Language Models for In-Context Learning of Linguistic Patterns In Unseen Low Resource Languages. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pp. 414-426 2025.
15

---

Published as a conference paper at COLM 2025
A Querying Details
A.1
Models
Tab. 5 presents the corresponding model versions for the LLMs used in our experiments and analyses. All GPT-* models were accessed from dedicated OpenAl API and all other models were imported from HuggingFace (Wolf al., 2020) and loaded and inferred using vLLM.
A.2 Computational Resources
We query all models with temperature of 0 and max output token length of 1024 for each problem response. The exact prompt templates used for querying are as follows:
LLM
Model Version
GPT-4
gpt-4-0613
GPI-4o
got-4-turboo22224-44999
GJP--3.5-Turbo
1pt-3.5-turbo--1125
Llama3. 1-405B
Llama3-70B
Llama3-8B
Llama2-70B
mete-1lam//LLam=222266hhhhhhhhh
Llama2-11
metar-1Lam/LLmmm222-1111hhhhhhh
Llama2-7B
Mixtral-7x8B
mistralaaiMMixtraa-ak----
Table 5: Model versions for the LLMs used in our experiments and analyses.
A.3 Prompts
In this section, we present the various prompts used in experiments and analyses.
1. Null Prompt (Only data from the puzzle and one problem at time, without any explicit instructions.) - See Figure 10.
2. MODELING Minimal Prompt (Chi et al., 2024) See Figure 11.
3. MODELING Hand-Tuned Prompt (Chi et al., 2024) See Figure 12.
4. MODELING Basic CoT Prompt (Chi et al., 2024) See Figure 13.
5. MODELING Full CoT Prompt (Chi et al., 2024) See Figure 14.
6. LINGOLY Std Prompt (Bean et al., 2024) See Figure 15.
7. No Context Prompt (Context pairs from data in the puzzle withheld for detecting contamination.) See Figure 16.
In each of the below prompt templates, the span "<<DATA>>" corresponds to the translation pairs of the source language to the target language provided as data for puzzle. This is presented in Figure 17. The exact layout of individual problem ("<<PROBLEM>>>) and all the problems ("<<ALL PROBLEMS>> in puzzle are presented in Figure 18.
16

---

Published as a conference paper at COLM 2025
Null Prompt
<<DATA>>
<<PROBEME>>
Figure 10: Prompt template for Null prompt
ModeLing Minimal Prompt
Here are some expressions in <<LANG>> (a never- seen-before foreign language) and their translations in English:
<<DATA>>
Given the above examples, please translate the following expression.
<<PROBEMM>>
Figure 11: Prompt template for ModeLing ing Minimal prompt
ModeLing HandTuned Prompt
This is translation puzzle. Below are example phrases in <<LANG>> (a never-seen-before foreign language) as well as their English translations. Some test phrases follow them. Your task to look closely at the example phrases and use only the information from them to transiate the test phrases.
<<DATA>>
Given the above examples, please translate the following expression.
<ROBLEM
Figure 12: Prompt template for ModeLing HandTuned prompt
17

---

Published as a conference paper at COLM 2025
ModeLing BasicCoT Prompt
This is translation puzzle. Below are example phrases in <<LANG>> (a never-seen-before foreign language) as well as their English translations. Some test phrases follow them. Your task is to look closely at the example phrases and use only the information from them to transiate the test phrases.
<<DATA>>
Given above examples, please translate following statements. Let's think step by step a logical way, using careful analytical reasoning get the correct result.
<<POBEEM>>>>>
Figure 13: Prompt template for ModeLing BasicCot prompt
B MODELING corrections
Here we present list of changes to the MODELING dataset that we made, along with our rationale. Each problem is referred to with language name and number if necessary." 14
Ainu: We removed "=" symbols: they are used to help beginner readers understand morphology. 15 However, it is not clear, whether those symbols would help an LLM confuse it.
Ayutla Mixe: We changed "Ejts yē'e nexpy" in answers "Ejts yē'ē ntunpy" translation for "I work for him". "Tun" is the root for "work", while "ex" is the one for "see". Similarly, we changed "Yě' maxu maxu'unk yë' uk yexpy." " "Ye' maxu' maxu'unk yê' uk yexyejtpy." in the solutions for consistency.16
Bangime We changed "nnE dçrç nEnonnE" in answers "nnE derę nEno" as translation to "we hear the bird", because the "nnE" suffix is not present in the examples in the question or even in the reference!" !7
Bangime 7: We changed "bad food" in the questions to "bad land" (answer: "gwyE begaū") because "bad food" is already in the given parallel sentences. We also changed "big house" to "big food" (answer: "jyEkE kanyoro") to maintain the same number for each noun in the data and questions.
Dogon: We changed "ú fl6" in solutions to "á flö" because the High-High (HH) pattern was not used anywhere in the examples (even in Dogon 2). The rule, as we understand it, is that the possessed noun tones are LL in the examples with non-pronominal noun possessors and HL in the pronominal noun possessors. This can be cross-checked in section 6.2.1.1 (pg. 141-143) of the reference." 18
We changed the question "kapkap yib ag" to "kamiket yib ag" and the corresponding answer from "speak very slowly" to "speak very stealthily" the word/root for "slowly" is not in the data.
Kalam:
Kutenai: We replaced "stomach" in the questions with "belly" and "wum" in answers with "wumnana". The original question requires inferring from "wumnana
we use E in Bangime to represent the mid-open front vowel sound
15 Ainu for beginners
A reference grammar for Ayutla Mixe
17 Bangime reference
A Grammar of Toro Tegu (Dogon) Tabi mountain dialect
18

---

Published as a conference paper at COLM 2025
ModeLing FullCoT Prompt
This is transiation puzzle. In moment, you will use logic and analytical reasoning translate from hever-seen-before language <<LANG>> to English. As a training example, here are some expressions Spanish and their translations in English.
1. Spanish: ventana roja red window
English:
2. Spanish: ventana azul
English: blue window
3. Spanish: manzana azul
English:
blue apple
Using the above examples, translate the following. Spanish: manzana roja
ANSWER: English:
red apple
EXPLANATION:
The first step notice that the
word "ventana" must mean window because (1) "ventana" appears twice between sentences and and (2) the only word that appears twice English translation "window." Next, we that "roja" must "red" and "azul" must "blue" by process of elimination. Next, we guess that Spanish, the noun precedes the adjective because "ventana" comes before "roja" and "azul." Therefore, the noun in sentence ("apple") must correspond to the word preceding the adjective ("manzana") in the Spanish translations. Putting this together, "manzana roja" must mean "red apple" in English.
Using similar logical and analytical reasoning to understand the grammar of the foreign languages step step, look closely
at the following example <<LANG>> phrases and only the information from them to translate the following test phrases.
<<DATA>>
Given the examples, please translate the following statements.
<<ROBEEM>>>
Figure Prompt template for ModeL ing FullCoT prompt
19

---

Published as a conference paper at COLM 2025
LingOly Std Prompt
Below is problem sheet from lingusitics exam. You will first see the entire mapped data and questions, then be asked respond the questions individually. Your answers the questions should rely only reasoning about the information provided in sheet. Data:
<<DATA>>
Questions:
<<ALL_ SAALL_PROLEMS>>>
Now respond to the following question from the set:
<<PROBLEM>>
Figure 15: Prompt template for Ling0ly prompt
NoContext Prompt
This is translation puzzle from linguistics
Below test phrases to translated <<LANG>> never-seen-before foreign language) some English. You will asked to translate specific phrase from the
<<ALL PROBLEMS>>
Now translate the following phrase:
<<PROBLEM>>
Figure 16: Prompt template for NoContext prompt
// Context Pair #1 <<SRC LANG>>; ><SRC_LANG>>> "<<SRC "<<SRC_TEXTT TEXT>> <<TRG LANG>>; "<xTRG TEXT>>
Context Pair #2
<<SRC LANG>>; "c<SRC TEXT>> <<TRG LTTRG_LANG>>; "<<TRG T
Context Pair #N
<<SRC LANG>>: "<<SRC "<<SRC_TEXT>>>>>>>> <xTRG_LANG>> "<<TRG_TEXT>>>"
TEXT>>"
Figure 17: Prompt template for <<DATA>> span. Lines starting with "//" are comments only for reference and not part of the actual span.
20

---

Published as a conference paper at COLM 2025
Single Problem (<<PROBLEM>>)
<<SRC "<NSR> T <<TRG -
All Problems in Puzzle
Problem #1
<<SRC BONGG_LAG>>; "NNSRC <<TRG LANG>>
Problem
<<SRC LANG>> "<=S<C=SSR____ "<<_C__EXT>>>"
<<TRG <<TRG_LANG>>>
Problem #N
<<SRC <<SRC_LANG>>; LANG>> "<<SRC_TEXT>>> <<TRG LANG>>>
Figure 18: Prompt template for "<<PROBLEM>>" and BECALL_PROBLENSS spans. Lines start- ing with "//" are comments only for reference and not part of the actual span.
belly" that belly a "small stomach", however official definitions state that "belly" is the general area of the stomach and adjuncts. In addition, "nana small" is being tested in another question.
Mapudungan 1: We changed "pichi mansun" to "pichi pafu" (answer: "small turkey") as the original phrase was translated already in the data.
Mixtepec Zapotec We changed "zhåb rôsåd" (pink clothing) to "yåg ròsåd" (pink stick) for the same reason in the previous task.
Ngadha 1: We changed "big leaf" in examples to "big fruit" as there is not enough data in the problem to establish that "fruit" ("l'ee" and "leaf" are the same (if they are) in the answers.
C Benchmarking details
We report the average exact match accuracy of each across dataset (O M+L) and across the M & subsets.
combination Tab.
Here an overview our benchmarking experiments, which show results in line with previous literature (Tab. 1 and BIG-Bench (Srivastava et al., 2023)), demonstrating that souure,, larger models are generally better solvers, and puzzles set low-resource languages are generally more difficult solve.
Along Models and Prompts We find that GPT-4o performs the best, and while GPT-4 scores slightly lower, shows more consistency across prompts than other models. In general, larger models within model family perform better. However, for suitable prompts, relatively smaller models Llama3 and Mixtral-8x7B outperform larger models like Llama2-70B and GPT-3. OPT-3.5-Tures 5-Turbo. This aligns with Waldis et al. (2024)'s observation that, while parameter size generally correlates with linguistic competence, specific architecture and tuning parameters can impact models' capabilities, particularly in the domains of morphology and syntax.
Consistent with the results of Sánchez al. (2024), who find that OpenAl models work better with fewer shot prompting, we also find that GPT-4o and Mixtral-8x1 perform better with more concise prompts (i.e., Null and Minimal).
21

---

Published as a conference paper at COLM 2025
Average Accuracy
Model
M
L
GPT-4
56.1
13.5
12.55
DTT-3.5-Turbee
35.1
6.6
9. 9,99M*
GPT-4o
65.7
18.6
15.77
Llama2-78
9.3
4.1
3.45**
L1ama2-13B
19.4
5.1
6,33
L1ama2-70B
31.9
5.3
10.7744
Llama3-88
38.4
6.6
11.05
L1ama3-70B
53.2
11.2
13.03
Llama3.
1-405B
55.3
14.1
11.93***
Mixtral-8xPP
39.1
5.8
12.13
Table 6: Average accuracy on M puzzles is significantly higher than L puzzles for all models. t-statistics marked with indicates p-value S .001.
Full CoT provides models with a descriptive solution to a short linguistic puzzle in Spanish noun phrase construction and proceeds to instruct the models solve the actual linguistic puzzle similarly. Thus, it is possible that many of the models perform well with the Full CoT prompt provides short demonstration in known language.
Along Dataset Sources All models score higher on M puzzles than on L puzzles, As described earlier, this is because M puzzles are made the style LO puzzles but involve lower number of linguistic features per question and have more data per feature to uncover in the puzzle.
Along Translation Direction Consistent with Şahin et al. (2020)'s findings for LLMs before 2021, we find that most models also score better when translating from the low- resource language to English compared to when translating, from English (see Figure 19). This difference is most pronounced in GPT-4o's ' results on M puzzles.
C.1 Extended Results for DeepSeek-R1
Table presents the exact match accuracy on the entire evaluation set (O) for Re which has been tuned for inferenceTTime Compute (DeepSeek-A) Al et al., 2025). We compare these scores with the best scores for each prompt setting as shown Table We observe that R1 demonstrates superior performance while maintaining our observed correlations between model performance and morphological features (Pearson correlation -0.74, p- value 0.001). To better determine the impact of ITC on this significant gain, future work must extend our study to a wider set of ITC models.
Prompt
ModeLing Minimal
ing Hand-Tuned
Modelinng
ModeLing
LINGOLY
Abergpe
Model
M
M
M
Q
M
Q
M
D
M
Grr-4
41.7
98.2
12.3
41.5
96.
11.4
6012
54.5
15.0
41.1
55.7
152
42.0
56.9
15.5
38-4
33.6
11.4
4015
56.1
13.5
3.7.5.. Tucbo
5.2
5:4
3.1
25.5
37.6
4.8
33.4
47.3
8.6
25.3
34.6
32.5
47.2
7.3
26.5
37.4
7.1
24.8
35.1
566
47.4
62.1
12.2
71.5
19.8
49.2
66.0
47.6
60.9
21.8
47.0
67.4
16.5
48.1
65.4
17.8
46.8
65.7
18A
11.5
5.9
10.2
5.9
58
277
7.4
9.3
4.
Lama2-130
8.6
26.8
14.4
190
26.2
6.3
9.1
12.
24
14.
194
Unmaz-700
186
26.6
4.2
23.5
547
35
23.3
36.1
6.1
22.4
308
7.5
25.9
36.2
7.3
18.6
27.1
34
72.4
31,9
53
17.5
27.8
38.5
43.5
7.6
56
Lomaat-700
37.0
60.5
53.4
347
11.7
36.1
53.2
11.2
Thmmch 148068
207
27.5
8.5
31.7
11.6
48.6
66.5
45.7
641
21.3
46.0
07.8
156
44.3
63.0
108
40.5
55.3
141
Mistral-R-78
23.9
385
3.3
32.5
45.3
9.8
31.8
46.0
6.5
2400
34.6
522
23.1
33.0
53
27.2
39.9
4.6
27.1
39.1
5.6
Prumpe Avegy
22.5
31.7
43.9
39.8
29.1
Table 7: Summary of an average exact match accuracy of each model-prompt combination across our dataset (O) and subsets of puzzles from MODELING (M) and LINGOLY (L). Models are divided by family and arranged by parameter size. Bold scores mark the best model for each prompt, underlined scores mark the best prompts per model.
22

---

Published as a conference paper at COLM 2025
ModeLing (M)
Date English From English
Accuracy
20
GPT-4
GPT-3.
GPT-4p
Ulema?
Ulama?
Liema?
Llama3
Llema3
Llama
Mixtral
LingOly (L)
Exact Motch Accuracy (%)
60
To English
English
0-
GPT-4
GPT-3.5 Turbo
GPT
Liamaz-7p
Llama2-13B
Llama2-1 2-8
Ulam3--p -88
Llama3-708 3-88
Llama3, 1-4058
Mixtral-Ba Bx 78
Figure 19: Models scores across translation directions.
Previous Best
Score for DeepSeek-R1
Prompt
NullPrompt
47.4
60.7
ModeLing Minimal
Modeling Hand-Tuned
49.2
64.7
ModeLingg Basic-CoT
ModeLing Full-Col
49.0
60.5
LingOly Std
Table 8: Extended results for DeepSeek-R1 compared against the previous best scores across both datasets (O) for each prompt setting (see Table 7).
D Linguistic Features, Definitions, and Classification
See Tab. 9.
E Correlation Between Attributes and Model Scores
See Tab. 12.
Annotation Guidelines & Inter-Annotator Agreement
Annotation guidelines are as follows:
Solve the problem yourself if the solution is not detailed enough to list out all the features used in the problem
Identify the linguistic features that are being tested or borrow them from the solution (if present), map them to the finer features list (add if it is not already there).
If a new feature was added in the last step, map it into WALS categorisation and broad categorisation (based on general definition).
Mark 0 for the feature in the data column if there are data points that can be removed from the problem without affecting the solvability for that feature, otherwise
23

---

Published as a conference paper at COLM 2025
WALS Feature
Broad category
Delinition
Example Problem
Example Problem
Other Research
Cilation
Powestecve Clasiffaation
Nomiinll By nttax
Semuntion
the possoserd déject could possessor (kinship,
MlAbun
Description Abburs
WALS- Poss,
Papuan Language
57 Postion of Poes. Markers (peslinkers
Nominal Syntus
Sy Syntas
proeereivve particle that between two nouns
Mlhuun
Description of Abure A
Linkers and aggemment WALS l'ass. Markees
Visst I'apaan Language of
Irian Jaya (531 and 5.22)
13. Torw Bons)
Phonology
Phenology
pilch pattem
Muggon
Graammr of Degun
WALS- Tone
101 Exprreston of Parceeinall
Simple Clauses
Syntax
phonologcal rales or lopresentation
M.Dogon
A Grammar Degan (6221.1)
WALS- Expe of Pron.
Subjects (yren)
changed by proneun
33 Nomizul
Naminal
Syntax
muriar indicating plarality of noun
M, Nui,5
of Rupa Nui- Pasahs Keevet S.5
WALS- Num. Pl.
Pturality (pli
Categories
Simple Clouses
Morphology
Ayutla
referenio graenmar
Per.
Marking
tranetiivamess (agent-paticnt varb aftixes to mark penson (only aggnnt)
(trnnsfi))
synujk (63.31 8.24)
102 Verbal
Simple Clouses
Mosphology
Ayutla Mise
A reliernnoe grammnar of Ayutla (Tukyor'
WALS Vers Pass Marking
Persom Marking
(penffic)
syunjk)
81-82 Order of SV
Word Order
Syntax
order of subject, object. and verb in sentenoe. onders for clauses without obfects and
MLAyuta,MMvv
of Ayutia Mise (Tukyo'aa sysngi) 9.3.133
reference
WAIS Order of SOV WALS Order of SV
Woed Order
Systas
Burgieme
of
WALS Andser N-A
Adi. Noun fenunaljsejuunn
adjective
unknown allitiation
NA Dimiznativs
Nominal Catggorics
Semantics
marker to express the modiifeer maoun
K.Kutennt
Kutenai lextoo
Intro: Dtm, across lengaageo, theuries
clessfy
112 Negalive Morphemes
Simple Clauses
Semanties
particl expecss negation aabsence a
M Kutenzz
Kutcnal Lextoon
WALS Neg. Morph.
(nng)
mouen
Nominal
Meephology
MUGesppyYimith
Lang
Semantic Affiser (aneffa)
Calegorics
losical cuteguriees
Englishs Study
Spatial Dewmpttion
Cognitive Linguittics
Guagu Yamithirr
Semantics
clisoliirr foe
Ngadha
NGADHA TEXT
Classilless (els)
Categories
TRADITION Stuphanas 2.2.5
WALS Num. Class.
131 Numeral
Lesicon
Semantie
involves counting, numerall,, and hov positive number expressioms ane
M. M.Ngadha.2
NGADHA TEXT TRADITIEN -Saphanas
Numeral Systems WALS Nuut. Bases
Bases (nua)
Dywcans 3.1.3'
NA Compounds orep
Morphology
Semantics
compostiinnaltt:: mesningg pacts expression combine to convery something bigger
M.Kalam
On the of serial verb constructions in Kalam
Altributve Compounds (WALS Aasn't cissiyy Hi6))
125 Purpsow Classes
Complex
ntta
origins
vorbs combine single clause without morphony nlactic marking
Classe
Kalam
linding subundiaation
Syntax
Syntax
of possessive marker phrase
Plevses
Order of
Word Order
Syntax
order of noun and prepeeition propoiitional plaziso
M.Sen.2
Sert Grammar
WALS WALS--Odeerdd Ad
Adpusition
Noun
(nouggng
60 Adjectival
Notiinal Syntax
Syntax
order of noun and odpecical cliose in noun phrase being noun then rebectival clause
M.Ser.2
Sent Grammar 244
WALS Gen., Adj and Rel. Cls.
Claises
(scunaije)
24 Possessive
Nominal Syntax
Morphology
possesatves marked affix
M.Jolmnac
and murphackgy of Filommnoo Mata Tofonac 5.2
WALS Poss. Noun Phr.
Noun Pieroats
(psseti))
Moephology
gender marking allives
L.ja
Bgja
Categories
24

---

Published as a conference paper at COLM 2025
Simple Clntsee
Syntox
presessive purticles
1,Cophic
UKLO-Cophic
WAI5 Pene.
69 Po-Poshoonn of
Vebul Categories
Merphology
tense marking affisee
T_Kiche
UKTA Kiiche
WATS- Fox of Temee-Ap. All.
Tense Aspect
Affiwes 1a Shneeffa)
47 Intersifiers
Nominal Categories
Syntax
particle for intensiffees
L_Kiche'
UKLO Kiiche
"Thet's proper cool
Reflesivee
emerging intensifier
Pronouns (antra)
proper in Briziaa English
WAT.S Int. Rell Pro.
Morphology
Phonology
I_Kabyle
procoss, hoophuphmmoooilally
Sogmental
Deletion Gestural
Overlap? Wol S. dower's
classly thts
Kinstap (suuddy)
Semuntics
cuncapt melsty
Semantios Australian
Categories
Langrages IML
34 Nomiinll
Nominal
Moephulogy
number of subject ubjject marking alfines
Mayangga
UX1O- UKLO-Mayangu
WALS -ua. of Nom. Pl
Flurality
Categaries
(munffi)
Number
Nominal
Morphology
LNhanda
UKLO Nhanda
WALS No. of Cases
Cavss fiswe)
Catggurie
37
Nominal
Meephology
definitivemeee marking
NJhanda
UKLO- Nhanda
WAIS- The (the
rigft
Categories
112 Nogative
Simple Clauses
Moephulogy
regation marking afficss
Tadakaahak
UKLO Tadakahaik
WAIS Neg. Marph.
Morphemes
(arghe)
Simple
Syntax
active/ passire morpho-syntactic
UXLO-Taaakkahak
Corstractione
(rete))
32 Systems of
Nominal Catggriie
Semanties
whether noun has animacy not
L.Tacs
UKLO Taos
Cog, anim. and its to ling anim. Syx.
Girsler
Assigument
27 Redhplication frefap)
Marphology
Morphology
grammaticaal /ull
L Tawala
UKLO Tawala
WALS Reduplicatio
reduplication of llabbes
Other Syntux (misage!
Complex
Syntax
other syntactic
Tawala
UKLO Tawala
semantic placement negation particless, havingg aspect marking purticle at the und wach sentianee
Morpholkegy
Meephology
wifiowe "normalizce"
Teetal
UKLD-Twlbal
Morphology
Rules Emesiia!
Other
Phonology
Phenelogy
other phonoooopcca features like acsimilation
Wianyi
Phanology
interppaat
78 lipistemic
Verbal Categones
Morphology
modality marking affives
L_Zeu
UKLO Zeu
WALS Ept. Poss.
Poesbility
Gaadhts
Position
Numinal
Moephulogy
preposctional aftinus
UKLO Zou
Pos, Case chitics bet)
Case Affines
Categories
(prmiii)
Simple Classess
Meephology
interropation affixes
L.Zou
UKLO-
WALS Pol. Ques.
Quastions (queçfii
95 Order of
Word Order
Systax
prepositional syntax with reppct to head clause
M Ainu
(PDE) Aira applicatives in typulegioll prospective
WALS Relation bw Order of Clty, & Verb and Adp. &
Chi & Verb and
Adp. NN
(pnpe)
Nowel
Phonology
Phooology
phoeological rules based on quality closs
LUlwa
Escentials Linguistics:
WALS Newel Qual.
Quality foourd)
Desembing vewrlss
10 Vowel
Phonology
Phonology
phonological roles based comserants
LoCppic
Nasals and Nasalizabon
WALS- VaSS--Vwwelll Nasal.
Nasalisation
Table 9: Linguistic Feature tags used, their respective categorisations, definitions, example problem in the dataset, citation for the feature's 's presence in the example problem language, and related research on it.
25

---

Published as a conference paper at COLM 2025
#
Feature
#
Feature
K
Feature
Feature
Feature
adjnoun
1.00
11
genfix
0.49
21
nounadji
1.00
31
possfix
1.00
41
1.00
alien
1.00
12
intens
0.85
nounposs
1.00
posslinker
1.00
1.00
amim
misefix
1.00
nounprep
prepe
0.66
tensefix
0.84
case
1.00
14
miscphon
0.79
24
num
1.00
34
prepfix
1.00
44
D50
1.00
cls
0.90
15
miscsym
1.00
25
numfix
0.94
35
preproun
1.00
45
tone
1.00
comp
1,00
16
modfix
1.00
26
persfix
0.93
36
quesfix
0.66
46
transfix
0.91
7
def
0.88
17
moicty
1.00
27
bonpel
0.49
37
redup
0.73
47
voice
1.00
rasal
0.66
18
neg
1.00
28
pl
1.00
38
semfix
1.00
1.00
dim
negfix
1.00
pron
s0v
0.95
215
1.00
10
eli
1.00
20
nounadj
1.00
30
poss
1.00
40
s0
1.00
50
0.00
Table 10: Pre-adjudication inter-annotator agreement (Cohen's k) for marking feature pres- ence in a puzzle. Note that after adjudication, all feature labelings have perfect agreement (x=1.0).
Puzsde
Exaci-Match Score (+Gain)
Llama2-18
Llama2-138
Llara2-786
1laro3.1-1053
GPT-4
Holone
(-0.0)
(+0.0)
(+0.0)
(-0.167)
(+0.333)
(+0.167)
(+0.167)
(+0.167)
1_Koneion
(-0.877)
0.21 (-0.077)
0.462 (+0.231)
0231 (-0.077)
0.769 (40.154)
0.769 (-0.154)
0.692 (-0.77)
0.846 (-0.77)
Landdi
0.0 (-0.0)
0:167 (-0.167)
0.167 (+0.0)
0.333 (+0.333)
0.333 (+0.167)
0.167 (+0.)
0.167 (-0.167)
0.167
M.Ayutia Mixe
0.0
0.0 (+0.0)
0.0 (-0.25)
0.0 (+0.0)
0.0 (+0.0)
(-0.25
(.000000
(.00000)
M 3ungime
0.0 (-0.0)
0.0 (+0.0)
0.0 (-0.0)
0.0 (-0.0)
0.8 (+02)
(41()22)
0.6 (602)
0.6
M. Rangine 5
0.0 ( 0.0)
0.4 (10.4)
0.4 (-0.2)
0.2 (-0.2)
0.6 (+0.2)
08(104)
0.6
088 (802)2)
M...gnyyyiiiiii Yimitlir
(-0.1)
(+1.0)
(-3.))
0.1 (-0.2)
0.7 (+0.2)
(+0.5)
(.0
(-0
Kutenii
0.2 (-0.0)
(-0.4)
0.4 (+0.2)
0.4 (+0.0)
1,0 (+0.6)
(.800)
0.6 0.6
0.6 0-0
M_Jolone:
(-0.167)
(40.167)
0.167 (40.167)
0.333 (40.167)
0.333 (+0.167)
0.333
(40.333)
Table 11: Puzzle-wise score gain on marking morpheme boundaries with whitespaces
If the feature is something that English also las/s mark English has similar feature to the same end, mark there instead
Use references linked in the last two columns for any confusion (here look at App. D)
See Tab. 10. for inter-annotator agreement scores.
G Impact of Morpheme-Separated Input
See Tab. 11.
26

---

Published as a conference paper at COLM 2025
Morpheme Boundary Unmarked Marked by "g"
Average Exact-Match Score
GPT-4
GPT-4o
Ulemaz- 135
Lierat 70l
Llama2-11 78
Uamma 708
Utema3-- -88
Llema3. 4-405
LLM
Figure 20: Performance change in models on making morpheme boundary explicit using "@"
Aserage Esact-Machh Scoee
0,74
0..43
0224
0371
0.299
0.38L
0.455
0,38
0.425
DAR6
Altibbul
Setting
0380"
0.427
01211
01365"
6216.
6299"
0.433"
03317
536
Madeieg th
063
Bay
Baa
03325
0.00"
028
0334
0.37
137"
336
1L310
D280
03'
0.433
03364
0255
-300
2.76
4016
41191
403-5
Makiin weweerrrooolieeoooottttto
0211
0.45
2.50
4.25
0.355
0.054
4179
-215
2304
-217
40272
4.21
00.00
0118
0301
0021
2366
41545
40:08
-1469"
4U8
456"
4156""
4351"
-317
-036"
-2496"
4351""
414""
4311"
-U271
4139y"
8422"
-0.5""
4L3"
435""
A1554"
40561
2.00
0-8""
436"
-337
4135T
"154""
-3.00"
41548"
4362""
0.33
.038
601"
051""
0.6
4035
-3564"
3.511
0.5
0.45"
<035
-.11"
-1637"
4 4
634""
4135
400"
4336"
-0-
4477
-317
-0.89
4.156
0225"
0.88"
0.554
0314"
0355
0,5
Maklingoeii Thase Buable
00201
0377"
0.00"
0.404
362""
4
0415"
Maphadagy
"38""
41362
-1435
-2834
-3565
4.15
4154""
4210"
4094
40:06
4L377"
-063""
.3603"
JU8
I1301
-016
4035
41314
41301
41315
4156
4504
48017
-225
-223
41716
4131
41336
413
-13W
-0227
Madeley whweaaatttottttttttt
4.11
41315
41715
Naasigg
Sullaeessaaaler Oueel!
0053
4002
4.17
0.1
10.55
0351
0355
4009
0355"
-.34
00011
416
016
035
0221
0227
420
0233
40351
0053
-0.15
4335"
41111"
A377"
Makkiny FellCoTToopp
Madiing Moriieggoiiilin Proneee Baabler
0027
-300
40.36
0.001
2.0
2.09
4.14
4.61
0.28
0.221
0299
0002
01.5
025
0391
433"
40334
Makiing
227
NullPoepBaaall aller Cheral
0142
0.78
4.56
10.5P
0335
405
0.34"
40055
11234
4131
4225
4025
4333
432
41354
41311
4330"
40333
-.00%
4003
-0557
-315
Maksieepoocee Phespsdaalde
4225
40:37
2.00
-180
40.33
11554
41355
461
41351
-1187
Syytter
40126
40.19
-.12
2 1.05
40.00
019
41165
4006
40200
-3117
40222
Cheral
40063
40:43
-2055
4.09
4U274
4U99)
40338
40033
40087
-.134
41381
<300
40166
40.8
4124
2.10
41
4113
-319
Maddtigg www. firlorr
40125
41
401
2.16
40.90
4330
40183
-316
-0185
4.18
-31%
-0537
-0088
40885
4079
Oceeal
00211
4..01
4014
4.10
4.551
0.99
4.18
41.65
4215
413
27

---

Published as a conference paper at COLM 2025
.11
-0.183
-0.172
-0.00
-0.147
46.13
-0166
-0168
MiirlingBoii CaPloeeaateee
4.121
-0187
-0.194
40.14
-0164
-0.17
49.91
-0154
Matelingeullloo
0.85
41182
-015
4112
0.382
8.15
-116
atlention
Modellngg InsTtood
40387
4.151
-0158
9.221
4.19
MistalingMtuweel Proeeppeeeeeeeeeelllle
NuilPPraupt Builler
4,064
40.15
.0004
4.15
-2277
Ouaraii
011
-0172
40,382
40.17
0.168
41168
4.035
40.215
0,409
-0.355
003122
40401"
433
Midelinggaoii Butter
-,300
Modeling ullCa PraaaaaPaattilliin
40.09
-03000
40.88
4,966"
-.395"
-0415"
-0437"
0.383
a Marphalogy
Modeling lnndTsndPhhmptttaai
4.136
-0.253
-0.30"
-0.35
0.305
-0,79"
-0.96"
-..18"
0.AUS
+0.31
MaralligggutttaPesette Bulllee Ovenaii
.0.37
-0.232
-0.272
-0.36""
0.369
-0.65"
4.17
0.356"
0..24"
-0.05
0.038
0,015
00229
0,33""
00.46"
-0,59"
-.255"
0.235
0.401"
00419"
40.085
-0.135
-02445
-0.306
-0.298'
-0.409
-0.395"
-03122
-0,001
-0,309
Li Sedyl
-0.13
-0,039
<0 399
-0.77"
-0,374
-0.977
-0426"
-0.42"
-0.464 T
-0413"
Modeling PramprDuuddrr
-0.66
-0,282
-0362"
-0328"
-0.315
-0.33"
0.33""
0.45"
-0,35"
Modelligg all Ceaeling BrruiptBullerr
-0.94"
an N_NNimiiul, Categerios
0,038
-0.275
0.34"
0,429"
0,337
00083
-0.292
-0,352"
-0.10"
0.382
-0,89"
0221
0.377
0,522
0.418"
KullPhonptten Suibler
0.022
0.027
-0.77'
-0406
-0.15
-0.416
..138.
0.239
.0.566
-0478"
Onenai
-0.13
-0.033
-0.39"
-0377"
-0.34"
-0.397
-0.266
-0.22"
-0,464
-0413"
0.255
0.184
.0053
-0,57
40087
-0.11
..151
0.182
..122
-153
MadtllinglonnCC Paraneae
0,03
0.019
-0100
0004
00.94
-0144
-0201
0.16
-0.228
-114
Mideling F Rusiler
0
0.002
-0033
-0024
00.86
-0.122
-0.265
0..21
2 164
-019
nyNinminal Syntee
Medeelingiane Bosthtoottttttetiiiie
0.125
-0.013
-0.16
-0009
0.065
-0.13
-0213
0226'
0.111
-0119
Mistalingghtunn Pewtetaeeeeeeeeeileee
00086
-0.85
-0019
+0.09
4.17
-0.129
0.00
0.196
-0166
Nall Suibler
0.349
0.115
-0142
-0,029
0.089
-0.167
.0.25
0.157
-U17I
Oversii
0.255
0.184
.0053
.0.257
4.087
.0.111
.0.151
0.182
.0.12
.0152
0.211
-0027
.01522
.0.221
.0.19
-0188
-02144
4.243
40327"
4.34
MedelingBonit PareneeCeeeee Brutee
-0002
0.011
-0.197
-0.203
-0.166
-0181
-024
-0.175
-0273
-0218
Modeling ullt TPhamppt
-0.016
-0,05
-0.15
-0.158
40.137
-0216
-0.226
-0.263
-0.26
-.299
ayPhnnlyggy
Medelingl Pompppprrer
0.009
0.135
+0.186
0,229'
-0.19
0,251
-0224
wewelingggeteeeeeeeeeeeeee NullProupp BuuprBullle Overdl
+0.177
-0.2W
00.46
0.149
.0.22
.0.233
0/21
-0133
-0.221
40.19
1018
-0.044
40343
403227
2024
40207
-033"
-0518"
-0.533
4.566
-0.25
-0.443
-0.47"
40436"
MistlingBostt Bosi CaT Ruildrr
4316''
0.117"
-0.5"
-0.4%
-0.466
.0.51
-0.77
-0 1,389"
-0,412"
-0.78"
Mosteling Prouupptuuleer
0.27
0:457
-0,005
-0.855
0155g
-0.583
-0.59"
-0.29"
-0.572
-0508
olStmple.lmmes
Mastelingeäaaa Pawtsaasttttttttttttaeoo
0.367"
0,309
0.502"
-0.51"
40.56
00.774
-0.22"
-1,496
40.566
-0 06499
0.322
0.447
0.536"
-0.299
0.618
0.526
-0.356
-0.523
0..03
Null Suilder Ovenali
-0.405
-0.7%
-0422""
-0,59
-0,553
"0025"
-0.33"
"0.22""
ModelingBonii CaPaannea
-0224
-0,278
-0.44"
-0.36"
-0.322
-0,72"
-0421
-0247
0035""
-.303"
Medeling Prouppppttttttteee
60.235
-0.295
0.38
0.32""
-0.35
0.38
-0.00"
0.357"
0,404
0,389
Webis Categoris
Mosilling indeTuneePProptttoiii
-0.6
-0.3"
-0.26"
0.31"
0.355
-0.62"
-0.17" "M
0.30""
0.366
-0299
-02522
-0.279
-0385"
-1341"
-0.355
-0345
-0.308
402922
-0.55
0,373
NullPhomppt Builder Overll
-0.15
46.55
-0.28"
-0345"
+0.33"
-0.36
-0244
40,003
-0,443"
-0 35.5
-0.33"
-0.385
40.306
Fing MoSSeePaanntBiiil
-0254
-0.271
-0.019
0313"
0,082
00081
00038
0.022
0,022
0001
D...Nrd.OOd Order
Madeling Buttire
-0.207
00092
00011
-0066
0,02
00019
-0319'
-0.166
-0027
.0016
4.007
-005
0034
0,099
.0087
with "4" Mewgiiigiiieeeeeeeeeeeeeo respectively.
<0.54
-0.05
.0.84
0.66
.0008
NullPhenpt Builler Ovenlll
-043
-0432"
0.02
0051
4.159
0.087
003
0.047
0.166
0.085
-0254
-0.271
-0084
-0.099
0.02
0.003
0
0.301
0.062
-0006
Table 12: Pearson correlation values between exact-match scores and feature values for all features against all model and prompt-setting combinations. Correlation values marked and have a corresponding p-value less than 0.05, 0.01, and 0.001,
28


Reviews:
[{"review_id": "nJZbBnawyW", "reviewer": "Reviewer_D2sE", "summary": "This paper presents a fine-grained analysis into LLM limiation on Linguistics Olympiad (LO) puzzles. The paper looks into why models struggle. The authors curate and annotate a dataset of 629 problems across 41 low-resource languages with 50 WALS-derived linguistic features and other attributes like similarity to English and data-constrainedness. Their correlation analysis reveals that LLMs perform worse on puzzles with higher morphological complexity and more data-constrained features, while performing better on puzzles exhibiting features also found in English. They also demonstrate that pre-splitting words into morphemes improves solvability, suggesting that tokenization of low-resource, morphologically rich languages is a key issue.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Presents thorough experiments and extract important insights into the linguistic reasoning limitations.\n- Contributes to improvements for low-resource languages.\n- The created dataset is a valuable contribution (pending its release by the authors).\n- The paper provides convincing support for the choice of studying linguistic Olympiad puzzles, citing their utility in analyzing LLM performance on low-resource languages.\n- The paper consistently reports significance levels and provides sufficient discussion on the results.", "reasons_to_reject": "- There are numerous (minor) presentation issues. Including the following:\n  - Line 46 references the research question, which is never explicitly stated.\n  - Line 47 references WALS before it is defined.\n  - The caption of Figure 1 is non-descriptive and confusing for readers unfamiliar with the problem.\n  - Line 131 is redundant. \n  - \"Feature\" vs. \"Attribute\" are used inconsistently.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xcpOU3UNo8", "reviewer": "Reviewer_iDKA", "summary": "This paper investigates how several LLMs can cope with linguistic puzzles  (typically on very very low resources languages) used in Linguistics Olympiad Competitions. \n\nOn carefully constructed data set of 629 problems over 41 low resource languages, the models mostly struggle with complex morphology (unless explicit morphological segmentation is given).  On the other hand, LLMs perform better with puzzles involving linguistic features that are also found in English.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall nice work with insightful analysis.\n\nProvides another carefully curated dataset for future use.", "reasons_to_reject": "Not much", "questions_to_authors": "I wonder if you made distinctions between inflectional vs derivational morphology (or does NA Compound /comp -- cover this I think not but just wanted to check):\n\nCan we assume that the tokenization dictionary before and after morpheme splitting were the same? If so, how much of an overlap were there between the morphemes and word-pieces (I am guessing not much?)\n\nMinor comments\n\n-- Pgh 234 it seems the language names have been swapped in Figure 6. No?\n-- 194 -- font problem ?", "extra_scores": {}}, {"review_id": "cdYbwPOD8z", "reviewer": "Reviewer_Qiv2", "summary": "This paper looks at the performance of Large Language Models (LLMs) on language puzzles from Olympics Olympiads. The basic goal is to score puzzles for various linguistic properties and see which of those properties correlate with performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The basic methods and results seem solid. If a reader is interested in these puzzles or in prompt engineering more generally, this paper will be of interest.", "reasons_to_reject": "There are way too many acronyms; these don't help the reader and made things hard to understand.\n\nNot clear why is this an interesting/relevant problem. That is, what actual NLP task is this connected to?\n\nThe annotation is a bit mysterious. It looks like there was a lot of inter-rater reliability, but the parameters used were not fully explained.\n\nThere were some unexplained things. For example, the authors say they nnly consider puzzles from \"MODELING (M) and LINGOLY (L)\", but what are those?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "op3t8vRCMK", "reviewer": "Reviewer_TCNB", "summary": "### **Edit:** Score increased after rebuttal\n\n---\n\n**Summary:**\n\nThe authors analyze the failings of LLMs on linguistic reasoning problems based on the linguistic content of the puzzles. They find that puzzles sharing features with English are easier, and puzzles with more morphological complexity are more difficult. This indicates the potential for better tokenizers as a step towards better performance.\n\n---\n\n**Originality:**\n\nThis is a new method for understanding the performance of LLMs on linguistic reasoning problems. Applying linguistic concepts to the problems and identifying patterns offers a new insight on the poor performance of most models.\n\n**Quality:**\n\nThe paper seems to be methodologically sound. The core of the work is the re-annotation of the LO problems, which is accomplished by two expert annotators with good inter-rater agreement.  \n\n**Clarity:**\n\nThe writing is passable but not excellent. At many points, it is difficult to understand what the authors are referring to (e.g. L97 - “LLM performance explanations in literature” could apply to both a wide and narrow scope. WALS is also never defined/explained.) The writing is also imprecise in many places (e.g. L56 “and more”, L212 “morph”) and has grammatical errors which could confuse the reader (e.g. L270 is not a sentence). \n\nThe Figures should also be clarified. I don’t understand why the authors separated Figures 8 and 20. It would seem natural to add a third bar for @ separated morpheme in Figure 8. Readers should not need to refer to the appendix for results which are discussed in the main body (L298). Figures 5,6,7,9 would probably make more sense as a single larger figure which shows the process of annotation on a single puzzle. \n\nThe rendering of the Figures is also blurry on my screen. I appreciate this may not be the authors’ fault, but it is worth checking.\n\n**Significance:** \n\nThe results in this work have moderate significance, but the paper lacks somewhat in extension beyond implications for solving LO puzzles. The result about English bias is unsurprising, and would be more useful if it was extended to consider models with a higher emphasis on multi-linguality to show whether the bias is ameliorated (e.g. Aya/Command A or Gemma). The tokenization result is exciting and has the potential to be impactful, but it would be useful to connect to LLM reasoning/performance more generally, as linguistic reasoning itself is not a particularly common application. This connection seems quite possible with examples such as “counting the ‘r’s in ‘strawberry” coming from potentially similar issues.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This is a methodologically-sound and principled paper, which connects the abilities of LLMs on linguistic reasoning tasks to established categories of linguistic phenomena.\n- The implications, particularly the relevance of tokenization for task performance, are useful for improving technical methods.", "reasons_to_reject": "- It is unclear whether the authors have controlled for known causes of task difficulty (number of reasoning steps) when identifying new areas (number of morphological features). The newly identified features may be highly correlated, or not a separate phenomenon at all.\n- The significance is limited to a relatively small area of research with limited discussion of wider implications.\n- The experiments do not include state of the art models when trying to explain patterns in model performance.\n- The writing of the paper itself is casual and borders on sloppiness.", "questions_to_authors": "**Critical Issue:**\n- Do the authors control for problems simply having more features of any kind? We know that LLMs struggle with multi-hop reasoning, are morphemes specifically a separate phenomenon? (A solid answer to this question would suffice for me to weakly accept the paper.)\n\n**Additional Experiments:**\n- Why do the authors not consider Claude models, which are the highest scoring on both LingOly and Linguini by a considerable margin?\n- Would inference-time compute models show similar patterns in performance?\n- For the tokenization experiment, introducing whitespace means that the resulting tokens will mostly be beginning of word tokens (e.g. “_lucky” vs “_un”+”lucky”). Would it be better to manually tokenize the words with the boundaries in the correct places? (At least for the open models).\n- Does the English bias reproduce when using models with a deliberate focus on multilingual training data and tokenizers?\n\n**Interpretation:**\n\n- For the tokenization experiment, a similar approach was attempted in Khouja et al. (LingOly-TOO) which reported little benefit for the models. What explains the higher success in this case?\n- What would the authors consider to be the primary benefit of this work? If we assume that solving LO puzzles is not intrinsically useful, and that the puzzles are probably not the most efficient way for LLMs to learn LRLs, what larger area of research will benefit from these results?\n\n**Artefacts:**\n- Is/will the annotation data be available for future use?\n- The ethical statement should state that the authors complied with all relevant licenses and intellectual rights. “Publicly-available” does not provide a presumption of ethical access.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper identifies properties of Linguistics Olympiad puzzles that cause particular difficulty for LLMs.

Pros:
1. The paper provides an extensive set of experiments
2. The experiments are carefully run and well-presented
3. The paper involves a dataset that will be a useful contribution
4. The problem is an important one with applications to improving technology for low-resource languages

Cons:
1. Reviewers noted some presentational issues, such as fairly many acronyms and lack of clarity in stating the research question. I encourage the authors to revise the paper in the ways they have proposed to address these points.
2. Reviewers note that more discussion of broader implications would be helpful.
Decision: Accept