Venue: NAACL 2025

# Evaluating Morphological Compositional Generalization

in Large Language Models

Mete Ismayizada1,2, Defne Circi3, Jonne Saleva4, Hale Sirin7, Abdullatif Koksal5,6, Bhuwan Dhingra3, Antoine Bosselut1, Duygu Ataman9, Lonneke van der Plas 12,8

###### Abstract

Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. However, their linguistic generalization capabilities remain questionable, raising doubts about whether these models learn language similarly to humans. While humans exhibit compositional generalization and linguistic creativity in language use, the extent to which LLMs replicate these abilities, particularly in morphology, is under-explored. In this work, we systematically investigate the morphological generalization abilities of LLMs through the lens of compositionality. We define morphemes as compositional primitives and design a novel suite of generative and discriminative tasks to assess morphological productivity and systematicity. Focusing on agglutinative languages such as Turkish and Finnish, we evaluate several state-of-the-art instruction-finetuned multilingual models, including GPT-4 and Gemini. Our analysis shows that LLMs struggle with morphological compositional generalization particularly when applied to novel word roots, with performance declining sharply as morphological complexity increases. While models can identify individual morphological combinations better than chance, their performance lacks systematicity, leading to significant accuracy gaps compared to humans.

1EPFL, 2Idiap Research Institute, 3Duke University, 4Brandeis University,

5LMU Munich, 6University of Cambridge, 7Johns Hopkins University,

8Universita della Svizzera Italiana, 9New York University mete.ismayilzada@idiap.ch

Footnote †: Equal contribution

+
Footnote †: Equal supervision

## 1 Introduction

Large language models (LLMs) have recently achieved remarkable advances in the broad domain of natural language generation and understanding tasks Gemini (2024); Zhao et al. (2023); Bubeck et al. (2023); Wei et al. (2022); Brown et al. (2020). However, these models have also been shown to lack strong linguistic generalization capabilities Weissweiler et al. (2023); McCoy et al. (2023); Goldman et al. (2022); Wilson et al. (2023); Linzen (2020); Baroni (2019). This discrepancy casts doubt on whether language models learn a language the same way as humans do. When learning a language which is essentially a finite set of words and rules, humans exhibit linguistic creativity Chomsky (1965); Bergs (2019) and compositional generalization through _productivity_ and _systematicity_Fodor and Pylyshyn (1988); Chomsky (1957). These abilities allow humans respectively to produce and understand novel combinations of familiar grammar units. While compositional generalization abilities of language models have been extensively studied Lake and Baroni (2018); Keysers et al. (2019); Kim and Linzen (2020), the extent to which language models employ this ability in morphology however remains largely under-explored. Recent works evaluating morphological generalization in language models have only focused on the productivity aspect with a limited coverage of inflectional forms Weissweiler et al. (2023); Anh et al. (2024).

Figure 1: **Our morphological generalization tasks illustrated with an example in Turkish. ID and OOD refer to in-distribution and out-of-distribution respectively. English translations are not part of the task and only shown here for illustrative purposes.**

In this work, we address this gap by systematically investigating the morphological generalization abilities of LLMs through the lens of compositionality. Following Keysers et al. (2019), we define the morphemes (smallest meaningful units in a language)1 as the compositional primitives and design a novel suite of generative and discriminative language tasks based on the morphological combinations of these primitives. These tasks aim to test **morphological productivity** (ability to produce novel well-formed combinations of morphemes) and **systematicity** (ability to systematically understand novel combinations) respectively. Figure 1 illustrates an example of both tasks.

Footnote 1: In linguistics, a distinction is made between _free_ morphemes that can stand alone such as words “cat” and “come” and _bound_ morphphemes that can only appear as part of a larger expression (e.g. ”cats”, ‘coming’) such as affixes “s“ and “ing”.

We evaluate several state-of-the-art instruction-finetuned large multilingual models on these tasks: GPT-4, Gemini-1.5, Aya-23 and Qwen-2.5. To ensure our findings are not language-specific, we experiment with two morphologically rich (i.e. characterized by a large number of inflectional and derivational forms) languages: Turkish and Finnish. Both languages share typological features (e.g. agglutination) despite being unrelated.

We find that LLMs lack human-like morphological compositional generalization ability in agglutinative languages despite their high performance on various tasks in these languages. Our analysis shows that morphological productivity, especially when applied to novel word roots is highly challenging for LLMs. Moreover, as the morphological complexity of words increases the model performance sharply decreases (to nearly zero) while human performance is not consistently affected. On the systematicity task, while models perform much better than chance in identifying the validity of _individual_ morphological combinations, however, this behaviour is not robust or systematic i.e. models fail to _consistently_ determine validity of several compositions made up of the same set of morphemes.

In summary, our contributions are as follows: **1)** We design novel morphological generalization tasks that require compositional processing. **2)** We prepare specific test suites in both Turkish and Finnish to measure morphological generalization and make these available for future research2. **3)** Using our novel tasks and test suites, we conduct a systematic analysis of morphological compositional generalization abilities of LLMs. **4)** Our findings reveal a systematic gap in LLVM's ability compared to humans concerning morphological generalization in agglutinative languages that also requires compositionality.

Footnote 2: [https://github.com/mismayil/morph-gen](https://github.com/mismayil/morph-gen)

## 2 Related Work

### Compositional Generalization

Compositional generalization is the capacity to understand and produce novel compositions of seen primitives and is typically characterized by systematicity and productivity (Fodor and Pylyshyn, 1988a; Keysers et al., 2019). Systematicity refers to the ability to understand different combinations that are made up of the same known components such as _John loves Mary_ and _Mary loves John_. Productivity, on the other hand, is the ability to produce potentially infinite novel combinations of a finite number of known building blocks such as using conjunctions to construct sentences _Mary knows that John loves Mary_ and _John heard that Mary knows that John loves Mary_. Past work has developed several benchmarks to measure compositional generalization abilities of neural models both in fine-tuning and in-context learning settings and has shown this task to be highly challenging (Yang et al., 2024; Lake and Baroni, 2018; Keysers et al., 2019; Kim and Linzen, 2020; An et al., 2023; Dziri et al., 2023). These benchmarks have mainly focused on synthetic sequence matching, semantic parsing, question-answering and problem-solving tasks. Our work however, investigates compositional generalization in the context of morphology.

### Morphological Generalization

Morphological generalization is the ability to understand words based on their constituent parts known as _morphemes_ and combine them to derive new words (Wysocki and Jenkins, 1987). Morphemes are the smallest meaningful units of language that typically correspond to word roots and affixes (i.e. prefixes, infixes and suffixes). Composing these units to construct new words can be done through _inflection_ and _derivation_ tasks in morphology where derivation often changes the syntactic category of the words and inflection does not. These tasks have gained considerable attention as part of the SIGMORPHON's shared tasks (Cotterell et al., 2016, 2018; Vylomova et al., 2020; 

Kodner and Khalifa, 2022; Goldman et al., 2023) and efforts to create a universal morphology (McCarthy et al., 2020). While transformer-based models have been shown to achieve near-perfect accuracy on these tasks (Canby et al., 2020), recent work has also found that these results are inflated due to lemma overlap pointing to a lack of generalization (Goldman et al., 2022). Other works have recently investigated the morphological capabilities of LLMs using inflection tasks and reported similarly weak performance results (Anh et al., 2024; Weissweiler et al., 2023). Similar to our study, both of these works use the popular Wug test (Berko Gleason, 1958) to evaluate the morphological generalization, however, they only focus on the productivity aspect, and their coverage of inflectional and derivational forms is limited. For example, Weissweiler et al. (2023) considers only a handful of specific inflectional forms (e.g. first person singular agreement and past tense, second person plural agreement etc.) for each language and Anh et al. (2024) translates the original Wug test suite which is very small in size (23 samples) into different languages. On the other hand, we cast the inflection and derivation tasks into the form of a compositional generalization task and evaluate models on both productivity and systematicy aspects. While focus of other works is breadth (languages from different families), we instead conduct an in-depth analysis of morphological generalization in typologically similar but unrelated languages with a large test suite covering a wide and diverse range of inflectional and derivational combinations.

## 3 Methodology

### Background

The important role of compositional processing in language understanding and generation has been extensively studied (Carnap, 1947; Chomsky, 1965; Fodor and Pylyshyn, 1988b; Zadrozny, 1994; Bauer, 2001; Aronoff and Lindsay, 2014). Past works have shown that new word formation is often a multi-level process that requires identifying the correct order of primary and secondary morphemes (Kiparsky, 1982a, b; Hockett, 1954), and while humans might memorize some frequent words and phrases as a whole, most of the expressive language generation relies on productive rules of grammar (O'Donnell, 2015). However, not all languages are equally productive, and more productive languages (e.g. agglutinative) tend to have complex inflectional morphology (Cotterell et al., 2019; Ackerman and Malouf, 2013). Moreover, these languages have been shown to be harder to model for \(n\)-gram and recurrent language models (Cotterell et al., 2018; Czarnowska et al., 2019). Inspired by these works, we focus our study on two highly agglutinative languages and compositional tasks which we describe in detail below.

### Tasks

Similar to works studying compositional abilities of neural networks (Goodwin et al., 2020; Lake and Baroni, 2018; Keysers et al., 2019), we design two novel and simple compositional probing tasks to test morphological abilities of models. First, a **morphological productivity task** which we define as a generative task where the model is given a word root, a list of affixes (not necessarily in the correct order) and is asked to derive a meaningful word by composing the root with the affixes in the correct order. Second, a **morphological systematicity task** which we define as a binary discriminative task where the model is again given a word root, a list of affixes and a word derived from the root using the given affixes (not necessarily a meaningful word) and is asked to determine the grammatical validity of the derived word. Figure 1 illustrates these tasks with an example in Turkish.

Additionally, to measure the morphological generalization capabilities of LLMs, we take inspiration from Berko's Wug test (Berko Gleason, 1958) that is typically used to probe the inflectional and derivational morphological knowledge of children and design out-of-distribution (OOD) versions of our tasks using nonce word roots. More specifically, for each in-distribution (ID) word root in our test suite, we automatically generate a nonce word (i.e. word that does not exist in the given language) and use it in both tasks as the word root in place of the original one. However, since the model has never seen these words, to make sure the model understands the meaning of this new word, we provide the model with the original word root as a definition of the novel word root. Our generation of nonce words relies on the underlying morphophonological features and the frequency of each letter in a given language to make sure these words are plausible and inflected in the same way as the original root. Further details on nonce word generation can be found in Appendix B 



### Data

We focus our study on two highly agglutinative languages, Turkish and Finnish, and prepare test suites specific for our tasks in these languages. We particularly choose these languages because they are characterized by a large number of morphemes and hence require a high degree of compositional generalization ability.

TurkishTurkic languages are well-known to be highly agglutinative where the word is composed of several morphemes in addition to a root. We select Turkish as a representative of this language family in our study. To prepare our test suite we use the Bilkent Turkish Writings Dataset3 as our base corpus which contains \(6,844\) creative writings of Turkish 101 and Turkish 102 courses between 2014-2018 and hence, is full of morphologically complex words. Data statistics can be found in Appendix Table 1. We preprocess this dataset to extract words and the sentences they are found in. Then we employ a morphological analyzer for Turkish Ozturel et al. (2019) to segment these words into a root and surface-level morphemes. To create a diverse and balanced test suite, we sample \(\approx 150\) examples per morpheme length \(1\) to \(7\) while maximizing the number of unique roots and morphemes (in total \(1,049\) samples). Finally, we automatically generate a nonce word for each word in our test suite by relying on the fact that surface realizations of morphemes in Turkish are characterized by deterministic morphophonological processes such as vowel harmony, consonant assimilation and elision. Final data statistics and examples can be found in Appendix Tables 2 and 5 respectively. Further details on data collection can be found in Appendix D.

Footnote 3: [https://github.com/selimfirat/bilkent-turkish-writings-dataset](https://github.com/selimfirat/bilkent-turkish-writings-dataset)

FinnishWe first collect a \(\sim\)1,000,000 sentence subsample of the Finnish mC4 corpus Xue et al. (2021). We then extract unique words from the text and morphologically segment them using omfori Pirinen (2015) and UralicNLP Hamalainen (2019). After excluding words that analyzers did not cover, we manually annotate the segmentations to identify prefixes, lemmas, and affixes among the segments. We then perform stratified sampling based on the number of affixes to ensure an even range of morphological complexity in our data set. Finally, we extract sentences corresponding to each analyzed word from mC4 and validate whether they make sense. In a significant portion of cases, we notice that the raw sentences are noisy; in these cases, we opt to generate synthetic sentences using ChatGPT, which we (authors) then manually validate to be grammatical. Final data statistics and examples can be found in Appendix Tables 6 and 7.

## 4 Experiments

SetupWe treat the productivity task as an open-ended task in which the model is asked to derive a word from the given root and affixes and the systematicity task as a binary classification task in which the model is asked to determine whether the given derivation is grammatically correct. For the systematicity task, we generate negative examples by producing all the combinations of morphemes attached to the same root and choosing the top four compositions (two for morpheme lengths of 1 and 2)4 that are closest to the original valid combination measured by the Levenshtein distance. We do this to ensure our incorrect combinations are challenging enough for the model as they will be deceptively close to a plausible derivation. We also experiment with other negative example selection strategies such as random selection and a heuristic selection based on the linguistic characteristics of the given language. We describe these settings in more detail and compare the results in Section 5.5. Finally, we (authors) manually verify all the generated negative examples and fix the label of false negatives.

Footnote 4: For 1-morpheme words, we manually annotate a negative morpheme to generate one negative option.

ModelsWe evaluate several state-of-the-art multilingual instruction-finetuned LLMs, namely, two open-weights models, **Aya-23**Aryabumi et al. (2024) and **Qwen-2.5**Team (2024), and two closed-source models, **Gemini-1.5**Gemini (2024) and **GPT-4**OpenAI (2024). We evaluate all models on all languages except for Aya-23 which officially supports Turkish, but not Finnish.5. We also report the performance of a _random_ baseline that generates a derivation with a random combination of given morphemes (productivity task) and randomly decides whether the deriv

correct, and a _majority_ baseline, which selects the most frequent label (in our case _"No"_) for the systematicity task (not applicable for the productivity task). All models are evaluated using few-shot (1, 3, and 5) in-context learning and greedy decoding since our tasks are deterministic by nature6. Unless otherwise specified, prompt instructions are in English, and number of shots is set to \(5\) for reported results7. Further details on model evaluation can be found in Appendix C.

Footnote 6: We also experiment with other decoding strategies, however, find no significant difference in performance. Results for different decoding strategies can be found in Appendix A.5

Footnote 7: We also experiment with paraphrased version of our prompt instructions, but find no significant difference in performance. Results for paraphrased prompt instruction can be found in Appendix A.6

Evaluation MetricsFor the productivity task, we use **Exact Match** accuracy against the correct derivations. For the systematicity task, we report an average of **Macro-F1** scores for each sample and a **Coherence** score that measures whether the model correctly and consistently identifies the validity (or invalidity) of all derivations for a given set of morphemes. Hence, coherence is defined as a binary score where the model gets a score of \(1\) for a given sample if and only if it correctly guesses the validity of all derivations pertinent to that sample, otherwise \(0\). We employ this stringent metric to test the robustness of model performance similar to Storks and Chai (2021).

Human EvaluationWe evaluate human performance on both tasks using two native speakers8 per language, who annotate \(70\) and \(60\) samples from the Turkish and Finnish test suites, respectively. To ensure our evaluation sample is a representative sample of the entire test suite, we randomly select \(10\) examples per morpheme length for each test distribution. Human annotators follow the same task instructions used for model prompts and were shown five examples. We report almost perfect or substantial inter-annotator agreement measured by Cohen's kappa score Cohen (1960) for both tasks, languages, and test distributions Appendix Tables 3, 4). Finally, for each task metric, we report the average score of annotators as the final human score.

Footnote 8: Annotators were recruited from a Turkish and Finnish researcher community and were not compensated as they volunteered

ResultsFigure 2 and 3 summarize all model results for both morphological productivity and systematicity tasks evaluated respectively on the Turkish and Finnish data. We see that on the productivity task, all models except GPT-4 barely crack the random performance. While GPT-4 performs the best for both languages, it significantly lags behind the human performance (\(-43\%\) and \(-51\%\)

Figure 3: **Morphological productivity and systematicity task results for Finnish**. Detailed results for all shots are in Appendix Table 9.

Figure 2: **Morphological productivity and systematicity task results for Turkish**. Detailed results for all shots are in Appendix Table 8.

 

in Turkish and \(-40.8\%\) and \(-48.9\%\) for Finnish respectively for ID and OOD data). Moreover, the GPT-4 performance gap between the ID and OOD test suites for both languages is much larger than the human gap (\(\approx 10\%\) vs. \(3\%\) in Turkish and \(1.7\%\) in Finnish). These results indicate that humans are much more compositionally productive in morphology and generalize more robustly to novel unseen words.

From the systematicity task results, we see that models perform much better than random and majority baselines with GPT-4 again in lead, however, the performance gap compared to humans is still significant, especially, on robustness as measured by coherence score (\(-19.1\%\) and \(-46.5\%\) in Turkish and \(-8.8\%\) and \(-25.2\%\) in Finnish respectively for ID and OOD data). The ID and OOD performance gap is also significant for all models, especially when measured by coherence score (ranging from \(-9.5\%\) to \(-23.3\%\) in Macro-F1 and from \(-13.5\%\) to \(-37.2\%\) in Coherence) while this gap is very low (\(\approx 2\%\)) for humans when measured by both metrics. These results show that humans are much more compositionally systematic and consistent in discriminating between correct and incorrect morphological forms made up of the same set of morphemes.

## 5 Analysis

### Effect of Morphological Complexity

Recent works have shown that morphological complexity plays a crucial role in the morphological generalization abilities of LLMs Anh et al. (2024); Czarnowska et al. (2019); Cotterell et al. (2018b). Morphological complexity is typically categorized into _integrative_ (I-complexity) which refers to the predictability of inflected form and _enumerative_ (E-complexity) complexity which refers to the number of cases and inflectional paradigms in language grammar Ackerman and Malouf (2013). While both languages we study are morphologically complex, our test suites include inflectional and derivational forms of varying length in the number of morphemes (1-7 in Turkish and 1-6 in Finnish). This allows us to study the effect of within-language E-complexity on the performance of our models. Figure 4 summarizes the GPT-4 performance for both tasks stratified by the number of bound morphemes on the Turkish data. On the productivity task, we observe a sharp downward trend (plummeting to nearly zero) in performance as the number of morphemes increases for both ID and OOD test suites with a relatively constant gap between ID and OOD performance while humans exhibit no such dependence on complexity Appendix Tables 16, 25). This shows that humans learn their native language robustly and can easily produce and identify long novel words while models are quite sensitive to the morphological (E-) complexity.

On the systematicity task, Macro-F1 scores for ID and OOD remain mostly unchanged as complexity increases, but coherence scores show a negative correlation with the increasing morphological complexity. We also observe a surprisingly low performance on 1-morpheme OOD words which we attribute to the varying number of negative options by morpheme length and potential shortcuts in longer morpheme words, as discussed in Appendix A.3.

### Effect of Context

While our core tasks are somewhat synthetic in nature, we do also experiment with more realistic versions where we provide the model a sentence as an additional context. Specifically, we frame them as sentence completion tasks where a sentence with a blank is provided and the model is asked to fill in the blank with the correct word derived from the given word root and affixes (productivity task) or determine if the given derivation is the correct option for the blank (systematicity task).

Figure 5 summarizes the results for both productivity and systematicity tasks evaluated on the Turkish data where we provide a sentence with a blank to the model as a context (i.e. sentence completion task). This results in some improvement on the productivity task, however, we observe significant decrease in performance on the systematicity task especially for smaller models such as Aya-23 and Qwen-2.5 series and in OOD setting. This could be due to the additional complexity introduced by the extra context, however, we should note that worse performance on this task implies even stronger generalization failure since this task is more real-world and closer to the next word prediction task compared to the original context-free setup.

### Effect of Tokenization

Past work has shown that suboptimal tokenizers, especially byte-pair encoding Sennrich et al. (2016) used in GPT-4 have generally a negative effect on the morphological abilities of language mod 

els (Meyer and Buys, 2023; Bostrom and Durrett, 2020; Hofmann et al., 2021). Whether the low performance of the model on the productivity task can be attributed to the suboptimal nature of the tokenization is of interest in particular because our tasks rely on the morphologically segmented morphemes while the model utilizes byte-level tokens that are mostly English. To measure the effect of the tokenization, we ran a version of the productivity task where the morphemes provided to the model are obtained by segmenting the final derivation based on the model's own tokenizer instead of the morphologically-aligned units. Figure 6 compares the performance of the tokenizer-aligned morphemes with the morphologically-aligned morphemes on the ID test set.9 We see that the performance in both cases is very similar to each other which points to a possibility that tokenization may not be the underlying issue behind the low performance. This finding is also consistent with some past work on exploring morphological capabilities of ChatGPT (Weissweiler et al., 2023)10

Footnote 9: Since we use the word root as a definition for the nonce root and the tokenizer tends to break the words into meaningless chunks, we skip this experiment on the OOD test set.

Footnote 10: We note that we perform this analysis only with subword-level tokenizers, but not character-level tokenizers for two reasons: 1) To the best of our knowledge, at the time of writing this paper, there were no instruction-tuned multilingual language models for Turkish and Finnish that uses character-level tokenizers; 2) Past work has shown that character-level tokenizers do not offer any significant advantages over subword-level tokenizers in morphological generalization Libovický et al. (2021); Toraman et al. (2022).

### Effect of Morpheme Order

Since our goal is to study the ability of LLMs to combine the morphological units in the correct order, in all of our experiments we shuffle the order of the units in the prompts. However, given that models are sensitive to small prompt changes Pezeshkpour and Hruschka (2023); Zhu et al. (2024); Wang et al. (2023); Zhao et al. (2021), we also analyze the effect of changing the morpheme order on

Figure 5: **Morphological productivity and systematicity task results for Turkish showing the effect of additional context**. Detailed results are in Appendix Table 36. Results for Finnish are in Appendix Figure 13.

Figure 6: **GPT-4 productivity task results on the ID test suite for Turkish stratified by number of bound morphemes showing the effect of tokenization**. Detailed results are in Appendix Table 42.

 

the performance of the model. To this end, we run our main experiments with all the morphemes in their correct order and report the results in Figure 7. We can see that this small change improves the performance across both tasks and models and especially, in the productivity task, the improvement can be up to \(20\%\). This shows that models understand the tasks and can provide a correct answer by simply copying the morphemes when they are given in the correct order, however, they struggle to compose the correct order themselves. This further indicates that LLMs lack the necessary robust compositional generalization in morphology.

### Effect of Negative Sample Selection

In our systematicity task, we generate negative samples (i.e. derived combinations that are not grammatically correct) by permuting the order of morphemes attached to the root. While the number of permutations is manageable for 2 or 3 morphemes (e.g., 2!=2, 3!=6), it grows rapidly with more morphemes (e.g., 6!=720). Evaluating all permutations would be ideal for robust systematicity testing, but this is infeasible due to high computational costs. Instead, we can select a subset of reasonable size to be a representative sample of all possible negative options. However, the strategy for which samples and how many to select can be somewhat arbitrary. Therefore, we experiment with three different selection strategies, and set the number of selections to four for simplicity: 1) **random** where we randomly select four negative options; 2) **language-agnostic heuristic** where we select the top four negative options that are closest to the positive option measured by Levenshtein distance (our default strategy); and 3) **language-specific heuristic** where we employ linguistic features of the tested language to filter out options that may be "too easy" for the model. We found one such heuristic for Turkish test suite based on the fact that Turkish phonology does not allow two adjacent vowels in morpheme combinations which we describe in Appendix E. We report the results of these different negative sample selection experiments in Figure 8. We see that the random selection has the highest performance on both ID and OOD test sets, followed by the language-agnostic and language-specific strategies. This implies that all our previous model results might be an upper bound and the true performance gap compared to humans is even larger than what we observe.

### Error Analysis

In order to understand the limitations of language models on our tasks, we manually analyze \(30\) Turk

Figure 8: **Morphological systematicity task results for Turkish showing the effect of different negative sample selection strategies**. Detailed results are in Appendix Table 52.

Figure 7: **Morphological productivity and systematicity task results for Turkish showing the effect of the morpheme order**. Detailed results are in Appendix Table 46. Results for Finnish are in Appendix Figure 14.

 

ish word derivations for each morpheme combination length (1-7) and for both productivity ID and OOD test sets resulting in a total of \(178\) and \(185\) derivations from GPT-4 that are incorrect. We annotate each generation on three criteria: 1) whether the generation is an **invalid** word (i.e. grammatically incorrect word) 2) whether the generation is **unfaithful** (i.e. generation does not follow the productivity task constraints) and 3) whether the generation includes any **hallucinations** (i.e. whether the generation has extra morphemes not mentioned in the task prompt). Our analysis shows that while on the OOD test set, GPT-4 generates a grammatically incorrect word most of the time (\(79\%\)), this proportion is significantly lower for the ID test set (\(31\%\)). However, on the ID test set, we observe a high unfaithfulness and hallucination ratio (\(91\%\) and \(67\%\)) meaning that most of the valid generations do not follow the task constraints. On the other hand, we see lower unfaithfulness and hallucination ratios on the OOD test (\(75\%\) and \(52\%\) respectively) which points to a _real word bias_ also reported by Weissweiler et al. (2023) where the model is biased toward generating frequent words for word roots existing in a given language irrespective of the underlying task. In other words, OOD setting forces the model to perform the true morphological generalization task which it fails as indicated by the higher percentage of invalid derivations. To identify the root causes of some of these errors, we analyze the GPT-4 chain-of-thought answers on the Turkish data and reveal several failure modes such as **sequential dependency errors, semantic misinterpretations, lack of grammatical knowledge**, and **unfaithful reasoning**, all of which we detail with examples in Appendix A.4. Finally, we also analyze the few errors human annotators made and find that these errors are either trivial typos or failure to notice an extra letter in a long word.

## 6 Conclusion

In this paper, we proposed a novel experimental paradigm to test morphological generalization abilities of large language models through compositionality. Our tasks target measuring morphological productivity and systematicity in a given language. We applied these tasks on the morphologically complex languages of Turkish and Finnish and evaluated morphological compositional generalization abilities of several state-of-the-art large language models. Our experimental results and analysis reveal a significant gap in the performance of LLMs compared to humans with respect to generalization in morphology of agglutinative languages.

### Limitations

While our novel tasks are language, dataset, and model-independent, our study only focused on two agglutinative languages and a few large language models. Therefore, the applicability of our findings in other languages and models should be further studied. We also mainly focused on the grammatical validity of the words, whereas it would be equally interesting to study the capacity of LLMs to produce and understand novel semantically and pragmatically valid derivations. While we have also optimized our prompts to be as simple and maximally instructive and tested in multiple languages and in chain-of-thought setting, whether a different set of prompts would produce the same results is not clear. Finally, we mainly evaluate models using greedy decoding due to the deterministic nature of our tasks and additionally only experiment with temperature and top-p sampling, however, the effect of different decoding strategies needs to be explored.

## Acknowledgments

We gratefully acknowledge the support of the Swiss National Science Foundation (grant 205121_207437: C - LING) and the Microsoft Accelerating Foundation Models Research Program. Define would also like to acknowledge the support of the National Science Foundation under grant DGE-2022040. We also thank Mammad Hajili, Osman Batur Ince, Omer Goldman, members of the NLP Lab at EPFL and the CCL Group at Idiap Research Institute for their valuable feedback in the early stages of this project and Raghav Mantri for his help with the Gemini experiments.

## References

* Ackerman and Malouf (2013) Farrell Ackerman and Robert Malouf. 2013. Morphological organization: The low conditional entropy conjecture. _Language_, 89:429-464.
* An et al. (2023) Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Jian-Guang Lou, and Dongmei Zhang. 2023. How do in-context examples affect compositional generalization? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11027-11052, Toronto, Canada. Association for Computational Linguistics 

Dang Anh, Limor Raviv, and Lukas Galke. 2024. Morphology matters: Probing the cross-linguistic morphological generalization abilities of large language models through a way test. In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_, pages 177-188, Bangkok, Thailand. Association for Computational Linguistics.
* Aronoff and Lindsay (2014) Mark Aronoff and Mark Lindsay. 2014. 67 productivity, blocking, and lexicalization. In _The Oxford Handbook of Derivational Morphology_. Oxford University Press.
* Aryabumi et al. (2024) Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzil Fadaee, Ahmet Ustin, and Sara Hooker. 2024. Aya 23: Open weight releases to further multilingual progress. _Preprint_, arXiv:2405.15032.
* Baroni (2019) Marco Baroni. 2019. Linguistic generalization and compositionality in modern artificial neural networks. _Philosophical Transactions of the Royal Society B_, 375.
* Bauer (2001) Laurie Bauer. 2001. _Morphological Productivity_. Cambridge Studies in Linguistics. Cambridge University Press.
* Bergs (2019) Alexander Bergs. 2019. What, if anything, is linguistic creativity? _Gestalt Theory_, 41:173-183.
* Gleason (1958) Jean Berko Gleason. 1958. The child's learning of english morphology. _Word_, 14.
* Bostrom and Durrett (2020) Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4617-4624, Online. Association for Computational Linguistics.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Praffulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bernner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Darto Amodei. 2020. Language models are few-shot learners. _Preprint_, arXiv:2005.14165.
* Bubeck et al. (2023) Sebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. _Preprint_, arXiv:2303.12712.
* Canby et al. (2020) Marc E. Canby, Aidana Karipbayeva, Bryan Lunt, Sahad Mozaffari, Charlotte Yoder, and J. Hockennaier. 2020. University of illinois submission to the sign-morphon 2020 shared task 0: Typologically diverse morphological inflection. In _Special Interest Group on Computational Morphology and Phonology Workshop_.
* Carnap (1947) Rudolf Carnap. 1947. _Meaning and necessity: A study in semantics and modal logic_, volume 30. University of Chicago Press.
* Chomsky (1957) N. Chomsky. 1957. _Syntactic Structures_. Janus linguarum (Mouton, Paris).: Series Minor. Mouton.
* Chomsky (1965) Noam Chomsky. 1965. _Aspects of the Theory of Syntax_, 50 edition. The MIT Press.
* Cohen (1960) Jacob Cohen. 1960. A coefficient of agreement for nominal scales. _Educational and Psychological Measurement_, 20(1):37-46.
* Cotterell et al. (2019) Ryan Cotterell, Christo Kirov, Mans Hulden, and Jason Eisner. 2019. On the complexity and typology of inflectional morphological systems. _Transactions of the Association for Computational Linguistics_, 7:327-342.
* Cotterell et al. (2018a) Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Arya D. McCarthy, Katharina Kann, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Jason Eisener, and Mass Hulden. 2018a. The CoNLL-SIGMORPHON 2018 shared task: Universal morphological reinflection. In _Proceedings of the CoNLL-SIGMODPHON 2018 Shared Task: Universal Morphological Reinflection_, pages 1-27, Brussels. Association for Computational Linguistics.
* Cotterell et al. (2016) Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The sigmorphon 2016 shared task--morphological reinflection. In _Special Interest Group on Computational Morphology and Phonology Workshop_.
* Cotterell et al. (2018b) Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018b. Are all languages equally hard to language-model? In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 536-541, New Orleans, Louisiana. Association for Computational Linguistics.
* Czarnowska et al. (2019) Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, and Ann Copestake. 2019. Don't forget the long tail! a comprehensive analysis of morphological generalization in bilingual lexicon induction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 974-983, Hong Kong, China. Association for Computational Linguistics 

* Dziri et al. (2023) Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. Faith and fate: Limits of transformers on compositionality. _ArXiv_, abs/2305.18654.
* Fodor and Pylyshyn (1988a) Jerry A. Fodor and Zenon W. Pylyshyn. 1988a. Connectionism and cognitive architecture: A critical analysis. _Cognition_, 28(1):3-71.
* Fodor and Pylyshyn (1988b) Jerry A Fodor and Zenon W Pylyshyn. 1988b. Connectionism and cognitive architecture: A critical analysis. _Cognition_, 28(1-2):3-71.
* Gemini (2024) Team Gemini. 2024. Gemini: A family of highly capable multimodal models. _Preprint_, arXiv:2312.11805.
* Goldman et al. (2023) Omer Goldman, Khuyagaatar Batsuren, Salam Khalifa, Argyman Arora, Garrett Nicolai, Reut Tsarfaty, and Ekaterina Vylomova. 2023. Sigmorphbon-unimorph 2023 shared task 0: Typologically diverse morphological inflection. In _Special Interest Group on Computational Morphology and Phonology Workshop_.
* Goldman et al. (2022) Omer Goldman, David Guriel, and Reut Tsarfaty. 2022. (un)solving morphological inflection: Lemma overlap artificially inflates models' performance. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 864-870, Dublin, Ireland. Association for Computational Linguistics.
* Goodwin et al. (2020) Emily Goodman, Koustuv Sinha, and Timothy J. O'Donnell. 2020. Probing linguistic systematicity. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1958-1969, Online. Association for Computational Linguistics.
* Hamalainen (2019) Mika Hamalainen. 2019. Uralicnlp: An nlp library for auralic languages. _Journal of Open Source Software_, 4(37):1345.
* Hockett (1954) Charles F Hockett. 1954. Two models of grammatical description. _Word_, 10(2-3):210-234.
* Hofmann et al. (2021) Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schutze. 2021. Superbizarre is not superb: Derivational morphology improves BERT's interpretation of complex words. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3594-3608, Online. Association for Computational Linguistics.
* Keysers et al. (2019) Daniel Keysers, Nathanael Scharli, Nathan Scales, Hyllke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Denila Smopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2019. Measuring compositional generalization: A comprehensive method on realistic data. _ArXiv_, abs/1912.09713.
* Kim and Linzen (2020) Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9087-9105, Online. Association for Computational Linguistics.
* Kiparsky (1982a) Paul Kiparsky. 1982a. Lexical morphology and phonology. _Linguistics in the Morning Calm/Hanshin_.
* Kiparsky (1982b) Paul Kiparsky. 1982b. Word formation and the lexicon. In _1982 Mid America Linguistics Conference Papers/Dept. of Ling., Univ. of Kansas_.
* Kodner and Khalifa (2022) Jordan Kodner and Salam Khalifa. 2022. Signormbon-unimorph 2022 shared task 0: Modeling inflection in language acquisition. _Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_.
* Lake and Baroni (2018) Brenden M. Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. _Preprint_, arXiv:1711.00350.
* Libovicky et al. (2021) Jindrich Libovicky, Helmut Schmid, and Alexander M. Fraser. 2021. Why don't people use character-level machine translation? In _Findings_.
* Linzen (2020) Tal Linzen. 2020. How can we accelerate progress towards human-like linguistic generalization? In _Annual Meeting of the Association for Computational Linguistics_.
* Louukkonen et al. (2024) Risto Louukkonen, Jonathan Burdeg, Elaine Zosa, Aarne Talman, Ville Komulainen, Vaino Hatapnaa, Peter Sarlin, and Sampo Pyssalo. 2024. Poro 34b and the blessing of multilinguality. _Preprint_, arXiv:2404.01856.
* McCarthy et al. (2020) Arya D. McCarthy, Christo Kirov, Matteo Grella, Amrit Nidhi, Patrick Xia, Kyle Gorman, Ekaterina Volomova, Sabrina J. Mielke, Garrett Nicolai, Mikka Silfverberg, Timofey Arkhangelskiy, Nataly Krizhanovsky, Andrew Krizhanovsk, Elena Klyachko, Alexey Sorokin, John Mansfield, Valls Entreits, Yuval Pinter, Cassandra L. Jacobs, Ryan Cotterell, Mans Hulden, and David Yarowsky. 2020. Unimorph 3.0: Universal morphology. In _International Conference on Language Resources and Evaluation_.
* McCoy et al. (2023) R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asil Celikjilmaz. 2023. How much do language models copy from their training data? evaluating linguistic novelty in text generation using RAVEN. _Transactions of the Association for Computational Linguistics_, 11:652-670.
* Meyer and Buys (2023) Francois Meyer and Jan Buys. 2023. Subword segmental machine translation: Unifying segmentation and target sentence generation. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 2795-2809, Toronto, Canada. Association for Computational Linguistics 

Timothy J O'Donnell. 2015. _Productivity and reuse in language: A theory of linguistic computation and storage_. MIT Press.
* OpenAI (2024) OpenAI. 2024. Gpt-4 technical report. _Preprint_, arXiv:2303.08774.
* Ozturel et al. (2019) Adnan Ozturel, Tolga Kayadelen, and Demirsahin I. 2019. A syntactically expressive morphological analyzer for turkish. In _Proceedings of the 14th International Conference on Finite-State Methods and Natural Language Processing_, pages 65-75, Dresden, Germany. Association for Computational Linguistics.
* Pezeshkpour and Hruschka (2023) Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of options in multiple-choice questions. _Preprint_, arXiv:2308.11483.
* Pirinen (2015) Tommi A Pirinen. 2015. Development and use of computational morphology of finnish in the open source and open science era: Notes on experiences with omorfi development. _SKY Journal of Linguistics_, 28:381-393.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
* Storks and Chai (2021) Shane Storks and Joyce Chai. 2021. Beyond the tip of the iceberg: Assessing coherence of text classifiers. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3169-3177, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Team (2024) Qwen Team. 2024. Qwen2.5: A party of foundation models.

* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izcaard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLama: Open and efficient foundation language models. _Preprint_, arXiv:2302.13971.
* Vylomova et al. (2023) Ekaterina Vylomova, Jennifer White, Elizabeth Salesky, Sabrina J. Mielke, Shijie Wu, Edoardo Maria Ponti, Rowan Hall Maudslay, Ran Zmigrot, Josef Valvoda, Sevetana Toldova, Francis Tyers, Elena Klyachko, Ilya Yegorov, Natalia Krizhanovsky, Paula Czarnowska, Irene Nikkarainen, Andrew Krizhanovsky, Tiago Pimentel, Lucas Torroba Hennigen, Christo Kirov, Garrett Nicolai, Adina Williams, Antonios Anastasopoulos, Hilaria Cruz, Eleanor Chodroff, Ryan Cotterell, Miika Silfverberg, and Mans Hudlen. 2020. SIGMORPHON 2020 shared task 0: Typologically diverse morphological inflection. In _Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_, pages 1-39, Online. Association for Computational Linguistics.
* Wang et al. (2023) Jingxiao Wang, Zichen Liu, Keun Hee Park, Zhuotijun Jiang, Zhaoheng Zheng, Zhuofeng Wu, Muhao Chen, and Chaowei Xiao. 2023. Adversarial demonstration attacks on large language models. _Preprint_, arXiv:2305.14950.
* Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. _Preprint_, arXiv:2206.07682.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. _Preprint_, arXiv:2201.11903.
* Weissweiler et al. (2023) Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anuba Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schuettez, Kemal OhZare, and David Mortensen. 2023. Counting the bugs in ChatGPT's wugs: A multilingual investigation into the morphological capabilities of a large language model. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 6508-6524, Singapore. Association for Computational Linguistics.
* Wendler et al. (2024) Chris Wendler, Venimim Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamass work in English? on the latent language of multilingual transformers. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15366-15394, Bangkok, Thailand. Association for Computational Linguistics.
* Wilson et al. (2023) Michael Wilson, Jackson Petty, and Robert Frank. 2023. How abstract is linguistic generalization in large language models? experiments with argument structure. _Transactions of the Association for Computational Linguistics_, 11:1377-1395.
* Wysocki and Jenkins (1987) Katherine Wysocki and Joseph R. Jenkins. 1987. Deriving word meanings through morphological generalization. _Reading Research Quarterly_, 22(1):66-81.
* Xue et al. (2021) Liting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 483-498, Online. Association for Computational Linguistics 

Haoran Yang, Hongyuan Lu, Wai Lam, and Deng Cai. 2024. Exploring compositional generalization of large language models. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop)_, pages 16-24.
* Zadrozny (1994)Wlodek Zadrozny. 1994. From compositional to systematic semantics. _Linguistics and philosophy_, 17:329-342.
* Zhang et al. (2023) Ruochen Zhang, Samuel Chayawijaya, Jan Christian Blaise Cruz, Genta Winata, and Alham Aji. 2023. Multilingual large language models are not (yet) code-switchers. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12567-12582, Singapore. Association for Computational Linguistics.
* Zhao et al. (2021) Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. _Preprint_, arXiv:2102.09690.
* Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. _Preprint_, arXiv:2303.18223.
* Zhu et al. (2024) Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie. 2024. Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts. _Preprint_, arXiv:2306.04528.

## Appendix A Additional Analysis

### Effect of Instruction Language

Since most LLMs are pre-trained on significantly more instruction data in English than other languages, we base most of our results on experiments where we use English as the prompt instruction language. However, as our data is in a different language, this results in a code-switched language which has been shown to be a challenge for large language models (Zhang et al., 2023). To measure the effect of the instruction language on the morphological generalization tasks, we run our experiments with Turkish and Finnish as the instruction language and report results for both tasks in Figure 9. We mostly observe a drop or no change in performance when the instruction language is other than English.

### Effect of Chain-of-thought Reasoning

Chain-of-thought prompting has been shown to be effective in eliciting strong reasoning capabilities from LLMs (Wei et al., 2023). In order to measure the effect of this reasoning technique on LLMs' performance on our tasks, we evaluate GPT-4 (the best performing model) on both productivity and systematicity tasks in zero-shot and 5-shot chain-of-thought settings. We report the results of these experiments compared with the 5-shot standard prompting in Figure 10. We observe that while 5-shot chain-of-thought performance is better than the zero-shot chain-of-thought, it is slightly worse than or similar to the 5-shot standard prompting. To identify the causes of these errors, we manually analyze the several chain-of-thought answers which we describe in Appendix A.4.

### Further details on the effect of morphological complexity

In Figure 4, we observe a surprisingly low performance (\(\approx 40\%\) drop from ID performance) on the 1-morpheme OOD words, but we attribute this behaviour to the varying number of negative options available for each morpheme length and possible presence of shortcuts in larger morpheme words. We should note that we have different number of total options to discriminate for a given sample depending on the number of morphemes (for 1 and 2 morphemes, we have 2 options, for 3-7, we have 5 options). Hence, a single mistake is penalized more in the former case than in the latter. However, within the former category, we see a 

much higher performance for 2-morpheme examples which might seem surprising, however, we hypothesize that this could be due to the presence of potential shortcuts for the model to exploit in the 2-morpheme case. Indeed, if we analyze the proportion of errors in both cases, we find that in the 1-morpheme case, a significant portion of errors (\(64\%\)) is false negative i.e. the model identifies a nonce root with a valid morpheme as grammatically incorrect, while this is not the case in the 2-morpheme case. However, in the 2-morpheme case, the model might be exploiting the correct order of morphemes as sole evidence for the validity of the derivation while in the 1-morpheme case, there is no such shortcut and the model should understand the applicability of the given morpheme to the given word root.

### Chain-of-thought Error Analysis

We randomly sample \(10\) examples from the 5-shot chain-of-thought experiments on the Turkish evaluation data (per morpheme length and test distribution) where GPT-4 made an error and manually analyze its answers across both tasks. Our analysis reveals the following primary types of errors:

1. **Sequential Dependency Errors** One common error we observe in the productivity task is due to the sequential processing of the given affixes by GPT-4. It typically starts applying the given affixes in the order they are given, however, since the affixes are typically given in shuffled order, this often results in an invalid word early on. The model, however, does not seem to realize its mistake and continues with the generation often confidently assigning meaning to the intermediate erroneous words. For example, given the word root "hedef" and affixes "-in", "-dig" and "-le", it considers the affixes sequentially in this order by first producing "hedefin" which is valid, then "hedefindig" which is invalid, however, it interprets the generation as "which is the target" and finally produces "hedefindigle" which it interprets as "with what is the target".
2. **Semantic Misinterpretations** Another set of errors stems from GPT-4 misinterpreting the meaning of the individual morphemes or the whole derivation. For instance, in one example, where the given morphemes are "bagir" ("to shout"), "-sa" and "-k" and GPT-4 is asked to determine the validity of the combination "bagirsak", it misinterprets this derivation as meaning "intestine" (which is also written as "bagirsask") and argues that this derivation can not be made up of the given affixes. While this reasoning is correct, the

Figure 10: **GPT-4 morphological productivity and systematicity task results for Turkish showing the effect of chain-of-thought reasoning. Detailed results are in Table 43.**

Figure 9: **Morphological productivity and systematicity task results for Turkish showing the effect of the instruction language. Detailed results are in Table 30. Results for Finnish can be found in Figure 12.**



model misses the other plausible meaning of this derivation ("as if we shout") that can be derived from the given morphemes. In another example, where the model is given the morphemes "oyna" and "-sana" and asked to produce a valid word, it misinterprets the meaning of the morpheme "-sana" as "to you" and argues that it can not be applied to the root "oyna" whereas "-sana" is a valid suffix added to verbs.
3. **Lack of Grammatical Knowledge** Another common pattern we see can be attributed to the lack of proper grammatical knowledge. In one example, the model is given the morphemes "uyum", "-suc", "-luk" and "-ta" and asked to determine the validity of the derivation "uyumluktasuz" which is invalid, however, the model assesses the validity of each morpheme and concludes that the combination should also be valid. In another example, it tries to add a verb suffix to a noun ("yargiyoruz"). Yet in other examples, it argues that valid affixes do not exist in the language or a valid morphological combination is not possible.
4. **Unfaithful Reasoning** Finally, we also observe a large set of reasoning errors due to inconsistent reasoning chains, hallucinations or unfaithful instruction following. For instance, in one example, GPT-4 concatenates the morphemes "unut" and "-ali" and yet derives "untuluyno". In another example, it auto-corrects an invalid word ("kaldrrnizdagda") to a valid word "kaldrndginnzda" and argues that the original derivation is correct.

### Effect of Decoding Strategies

We mainly experiment with greedy decoding (e.g. temperature is set to 0 and top_p is set to 1) in all of our experiments as the nature of our tasks is deterministic. However, to check the sensitivity of our findings across diverse decoding settings, we additionally run our study with GPT-4 (the best performing model) on both tasks and languages with varying temperature and top_p values and report the results in Tables 53, 54, 55 and 56 respectively. We find no significant or systematic differences across different decoding strategies which strengthens the robustness of our findings.

### Effect of Prompt Instructions

Due to the cost of LLM evaluation, we mainly experiment with one set of prompt instructions that we have found to be simple and effective through a moderate level of prompt engineering. However, to check the sensitivity of our findings across different prompt instructions, we additionally run our study with GPT-4 (the best performing model) on both tasks and languages with a paraphrased version of the original prompt instructions (found in Appendix F) and report the results in Tables 57 and 58. We find no significant or systematic differences across different prompts which strengthens the robustness of our findings.

## Appendix B Nonce word generation

TurkishTo automatically generate novel nonce words in Turkish (out-of-distribution words that do not exist) that are inflected the same as the original word roots, we leverage the deterministic morphphonological features of Turkish. In particular, vowel harmony and consonant assimilation in Turkish completely determines which surface forms of the meta level morphemes would apply. Furthermore, these features depend only on the last vowel and the consonant. Hence, for a given word root in Turkish, we keep its last vowel and the consonant and randomly modify the other vowels and consonants with other vowels and consonants based on the frequency of each letter in Turkish to make sure we obtain words that would be plausible in this language. For example, if the given word root is _"sanat"_, we keep the suffix _"at"_ as is and modify the prefix _"san"_ by randomly replacing each vowel in it with another vowel and consonant with another consonant. This makes sure that the words inflect the same and they are of the same length. However, if the word is too short (only two letters), and there is no prefix, we generate a random prefix of length three with vowels and consonants alternating (Turkish typically doesn't allow dense consonant clusters)

FinnishThe Finnish nonce word generation is done similarly to the Turkish nonce word generation, where we alter only the word root. All consonants are replaced with other consonants and vowels with other vowels that conform to the rules of Finnish vowel harmony 



## Appendix C Model Evaluation

We evaluate the following state-of-the-art multilingual instruction-finetuned LLMs:

* **Aya-23**[14] a powerful open-weights multilingual LLM serving 23 languages including Turkish. We evaluate both 8B and 35B sizes of this model series, but only on Turkish dataset as Aya-23 does not officially support Finnish yet.
* **Qwen-2.5**[6] recent open-weights multilingual LLM that has shown impressive results across various benchmarks and supports over 29 languages. We evaluate both 7B and 32B sizes of this model series in both languages.
* **Gemini-1.5**[10] a closed-source multilingual LLM that supports over 40 languages including Turkish and Finnish. We evaluate the gemini-1.5-flash version in both languages.
* **GPT-4**[11] a closed-source multilingual LLM that supports many languages including Finnish and Turkish. We evaluate the 2024-02-15-preview version in both languages.

Models are evaluated using in-context few-shot learning where number of shots take values in {1,3,5}. We make sure each shot has the same number of morphemes as its corresponding task example. By default, all our prompt templates are in English since LLMs are quite proficient in following instructions in this language Wendler et al. (2024), however, we also experiment with instruction templates in Turkish and Finnish which generally show worse performance (Appendix A.1). Similarly, while by default we use the standard prompting for all experiments, we also experiment with chain-of-thought prompting Wei et al. (2023), but find very little difference in performance (Appendix A.2). Prompts for all tasks and languages can be found in Appendix F.

## Appendix D Data

TurkishSince the morphological analyzer we use to process the Turkish dataset Oztarvel et al. (2019) is based on a finite state machine relying on purely syntactic rules, it produces several alternative decompositions for some words (e.g. analyzer

\begin{table}
\begin{tabular}{|l|c|} \hline \#words & 3,775,470 \\ \#unique words & 348,173 \\ \#unique roots & 9,576 \\ \#unique meta affixes & 103 \\ \#unique affixes & 372 \\ \#unique meta affix compositions & 21,930 \\ \#unique affix compositions & 37,853 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of BTWD dataset in Turkish. Meta affixes refer to the bound morphemes that are not surface-realized.

\begin{table}
\begin{tabular}{|l|l|c|} \hline
**Task** & **Test Distribution** & \(\kappa\) \\ \hline Productivity & ID & 0.94 \\ Productivity & OOD & 0.91 \\ Systematicity & ID & 0.94 \\ Systematicity & OOD & 0.99 \\ \hline \end{tabular}
\end{table}
Table 3: Human inter-annotator agreement on Turkish test suite measured by Cohen’s \(\kappa\) score. We note that since the productivity task is an open-ended generative task, the chance agreement would be close to 0, hence \(\kappa\) score is equal to the raw agreement.

\begin{table}
\begin{tabular}{|l|l|c|} \hline
**Task** & **Test Distribution** & \(\kappa\) \\ \hline Productivity & ID & 0.77 \\ Productivity & OOD & 0.78 \\ Systematicity & ID & 0.75 \\ Systematicity & OOD & 0.84 \\ \hline \end{tabular}
\end{table}
Table 4: Human inter-annotator agreement on Finnish test suite measured by Cohen’s \(\kappa\) score. We note that since the productivity task is an open-ended generative task, the chance agreement would be close to 0, hence \(\kappa\) score is equal to the raw agreement.



produces both decompositions "an+la+dig+imz" and "anla+dig+imz" for the word "anladigmizrze". Hence, we further apply some language-specific heuristics to automatically filter out invalid decompositions. This preprocessing still leaves some words with multiple decompositions that can only be validated using semantics, hence, as a last step, we (authors) manually verify and determine the final segmentation of a word.

## Appendix E Heuristic Negative Sample Selection For Turkish

Turkish phonology does not allow two vowels to occur together and typically employs "buffer" letters such as _"y", "s"_ in between these vowels, however, blindly permuting the order of Turkish morphemes inevitably results in negative samples where two vowels may occur next to each other. We hypothesized that models might easily identify these options by exploiting the "no-two-vowel" shortcut and without considering the semantic order of morphemes. To check this hypothesis, we counted the number of GPT-4 mistakes corresponding to options that both have and don't have two vowels occurring together and found that while the model makes a mistake in around \(8\%\) (in-distribution) and \(16\%\) (out-of-distribution) of all the negative options that do not have two vowels occurring together, these ratios are only \(1\%\) and \(4\%\) when we look at the negative options that have two adjacent vowels. Motivated by this discrepancy, we designed our third heuristic-based selection strategy for Turkish such that after ranking the options by their distance to the positive option, we select the top four negative options that do not have two adjacent vowels in their morpheme composition wherever possible.

## Appendix F Prompts

This section lists the instruction prompts for all tasks and language templates. We present examples in one-shot setting, templates for different shots are the same with more examples. For the English language template, we provide examples in Turkish, the templates are the same for Finnish with examples in Finnish.

### Templates in English

**Productivity task prompt [ID root]**

You are given a word root and a list of affixes (separated by comma) in Turkish and your task is to generate a grammatically correct word from this root using all the given affixes. You are allowed to use only the given affixes and each affix only once. Answer with only the generated word.

Example 1:

Word root: bulas

Affixes: ma, sa, tur, ydi, k

Answer: bulastrmasaydkk

Example 2:

Word root: bekle

Affixes: me, di, z, n, c

Answer:

**Productivity task prompt [OOD root]**

You are given a novel word root with its definition and a list of affixes (separated by comma) in Turkish and your task is to generate a grammatically correct word from this root using all the given affixes. You are allowed to use only the given affixes and each affix only once. Answer with only the generated word.

Example 1:

Word root: lldjs

Definition: lldjs means karis in Turkish.

Affixes: sa, ydi, k, ma

Answer: lldsmasaydk

Example 2:

Word root: ihek

Definition: ihek means emek in Turkish.

Affixes: in, imiz,ler, ci

Answer: 



\begin{table}
\begin{tabular}{l|p{142.3pt}|p{85.4pt}} \hline
**ID root (OOD root)** & **Affixes** & **ID Derivations** \\ \hline sohbet (sakset) & -ler or -yin & sohbetler \(\checkmark\) sobbetyin \\ \hline stra (youva) & -dan, -mis & sradanmism



**Systematicity task prompt [ID root]**

You are given a word root, a list of affixes (separated by comma) and a word in Turkish that is derived from the given affixes. Your task is to determine whether the derived word is grammatically correct. Answer only with Yes or No.

Example 1:

Word root: kucuk

Affixes: umuz, lug, den

Derived word: kucuklugumtzden

Answer: Yes

Example 2:

Word root: even

Affixes: sel, e, lig

Derived word: evrenessellig

Answer:

**Systematicity task prompt [OOD root]**

You are given a novel word root with its definition, a list of affixes (separated by comma) and a word in Turkish that is derived from the given word root using the given affixes. Your task is to determine whether the derived word is grammatically correct. Answer only with Yes or No.

Example 1:

Word root: enesilvote

Definition: enesilvote means universite in Turkish.

Affixes: niz, yse, de

Derived word: enesilvotedeyseniz

Answer: Yes

Figure 11: GPT-4 morphological productivity and systematicity task results for Finnish stratified by number of bound morphemes. Detailed results are in Tables 25, 26, 27.

\begin{table}
\begin{tabular}{|l|c|} \hline \#samples & 480 \\ \#unique roots & 406 \\ \#unique affixes & 386 \\ \#unique affix compositions & 365 \\ \hline \end{tabular}
\end{table}
Table 6: Statistics of our final test suite in Finnish.





\begin{table}
\begin{tabular}{l|p{142.3pt}|p{142.3pt}} \hline
**ID root (OOD root)** & **Affixes** & **ID Derivations** \\ \hline yópaikka (áydainca) & -nne or -ksi & yöpaikkanne \textbackslash{}�iakkasi \\ \hline sano (tato) & -taan, -pas & sanotaanpas \textbackslash{} Sanopastaan \\ \hline petoks (seloks) & -i, -ne, -en & petoksnieen \\  & -i, -ne, -en & petoksneeni \\  & & petoksienne \\  & & petoksennei \\ \hline olosuhte (olanzike) & -kuvaus, -i, -lia, -an & kuvausolosuhteiltaani \\  & -kuvasolsuhetianelta & kuvausolosuhteinalta \\  & & kuvausolosuhteanilta \\ \hline palvelu (sapsevu) & -laina, -n, -valitys, -j, -a & lainanvalityspalveluja \\  & -laina, -n, -valitys, -j, -a & nlainavalityspalveluja \\  & & lainavalitysnpalveluja \\ \hline salaisuuks (noraekauks) & -motivaatio, -n, -notstatus, -i, -a, -ni & motivaationnosatussalaisuvuksinia \\  & & motivaatinonostaussalaisuuksinai \\  & & motivaationnoostaussalaisuuksinai \\  & & motivaationnotastassalaisuuksiani \\ \hline \end{tabular}
\end{table}
Table 7: Examples from our test suite in Finnish for each morpheme length from 1 to 6. OOD derivations can be obtained by replacing the ID root with the corresponding OOD root. Correct derivations are marked with \(\checkmark{\mathsf{\char 50}}\).





**Productivity task prompt [ID root] (with context)**

You are given a word root, a list of affixes (separated by comma) and a sentence with a blank (__) in Turkish and your task is to fill in the blank by generating a grammatically correct word from this root using all the given affixes. You are allowed to use only the given affixes and each affix only once. Answer with only the generated word. Example 1: Word root: kal Affixes: an, lar Sentence: giden geminin yokluguna bir tutlu inandramaz kerndilerini limanda ____Answer: kalanlar Example 2: Word root: kurtar Affixes: ccek, abil Sentence: goc ettikten sonra diger hemserlieri gibi mal, mulk pesinde olsa belki annesini paraszliktan ___belki de kizi bir fabrika kosesinde calismak zorunda kalmayip daha uzun yasayabilecekti Answer:

**Systematicity task prompt [ID root] (with context)** You are given a word root, a list of affixes (separated by comma), a sentence with a blank (__) and a word in Turkish that is derived from the given word root using the given affixes. Your task is to determine whether the derived word is the correct option to fill in the blank. Answer only with Yes or No. Example 1: Word root: kucuk Affixes: umuz, den, lug Sentence: ____kalma bir oyuna donusturduk hayatumzi Derived word: kicukli@umizden Answer: Yes Example 2: Word root: akl Affixes: lan, iz, acag Sentence: bir seyler ysadiktan sonra mi hep Derived word: aklacaglan Answer:

Figure 14: Morphological productivity and systematicity task results for Finnish showing the effect of the morheme order. Detailed results are in Table 49. Figure



**Productivity task prompt [ID root] (CoT)**

You are given a word root and a list of affixes (separated by comma) in Turkish. Your task is to construct a grammatically correct word by appending the given affixes to the root. Use each affix exactly once. After forming a word, list each affix used in the construction of that word to verify adherence to the rules. Check the following: Ensure no affix is used more than once, confirm that all provided affixes are used, verify that no extra affixes outside the provided list are included. Think step by step tags <Answer>correctword</Answer>.

Example 1: Word root: kuru

Affixes: t, mus

Answer: First, let's append the affixes to the root "kuru" in a grammatically correct order:

...<<explaining the correct order of morphemess>...

Example 2: Word root: mana

Affixes: sz, dr

Answer:

**Productivity task prompt [OOD root] (CoT)**

You are provided with a novel word root with its definition, and a list of affixes (separated by comma) in Turkish. Your task is to construct a grammatically correct word by appending the given affixes to the root. Use each affix exactly once. After forming a word, list each affix used in the construction of that word to verify adherence to the rules. Check the following: Ensure no affix is used more than once, confirm that all provided affixes are used, verify that no extra affixes outside the provided list are included. Think step by step tags <Answer>correctword</Answer>.

Example 1: Word root: kuru

Affixes: t, mus

Answer: First, let's append the affixes to the root "kuru" in a grammatically correct order:

...<<explaining the correct order of morphemess>...

Example 2: Word root: comaa

Affixes: sz, dtr

Answer:

**Productivity task prompt [OOD root] (CoT)**

You are provided with a novel word root with its definition, and a list of affixes (separated by comma) in Turkish. Your task is to construct a grammatically correct word by appending the given affixes to the root. Use each affix exactly once. After forming a word, list each affix used in the construction of that word to verify adherence to the rules. Check the following: Ensure no affix is used more than once, confirm that all provided affixes are used, verify that no extra affixes outside the provided list are included. Think step by step tags <Answer>correctword</Answer>.

Example 1: Word root: duru

Definition: doru means kuru in Turkish.

Affixes: t, mus

Answer: ...<<explanation>...

Example 2: Word root: cokan

Definition: cokan means yalan in Turkish.

Affixes: la, lar

Answer:

\begin{table}
\begin{tabular}{l r r r r r r r} \hline \hline \multirow{3}{*}{**Models**} & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**majority** & 0.0 / 0.0 / 0.0 & 0.0 / 0.0 / 0.0 & 41.2 / 41.2 / 41.2 & 41.2 / 41.2 / 41.2 & 0.0 / 0.0 / 0.0 & 0.0 / 0.0 / 0.0 \\
**random** & 24.6 / 25.0 / 25.0 & 24.7 / 24.6 / 24.2 & 41.8 / 41.8 / 43.5 & 43.0 / 42.3 / 42.3 & 9.1 / 9.0 / 9.4 & 8.5 / 9.6 / 9.0 \\ \hline
**aya-23-sb** & 12.8 / 13.7 / 13.3 & 8.8 / 11.5 / 12.3 & 62.0 / 64.6 / 67.5 & 53.9 / 49.3 / 51.5 & 27.9 / 31.4 / 36.0 & 19.1 / 15.7 / 18.4 \\
**aya-23-3eb** & 12.7 / 19.8 / 21.0 & 14.6 / 17.7 / 19.3 & 69.9 / 80.1 / 81.8 & 64.6 / 71.0 / 72.1 & 36.8 / 52.6 / 55.8 & 29.2 / 39.9 / 41.8 \\
**qwen-2.5-fb** & 15.0 / 14.9 / 15.8 & 13.2 / 12.9 / 12.9 & 71.1 / 73.6 / 74.6 & 65.7 / 66.8 / 66.0 & 40.5 / 44.3 / 45.1 & 33.5 / 33.9 / 33.1 \\
**qwen-2.5-32b** & 22.6 / 23.7 / 24.1 & 21.7 / 21.8 / 21.8 & 77.3 / 84.7 / 85.9 & 53.1 / 71.3 / 75.3 & 56.7 / 76.3 / 66.8 & 18.5 / 45.7 / 48.3 \\
**gemini-1.5-flash** & 28.8 / 30.5 / 30.7 & 24.9 / 25.7 / 25.1 & 60.8 / 80.8 / 85.4 & 41.4 / 52.8 / 62.1 & 32.2 / 63.6 / 70.7 & 0.4 / 19.3 / 33.3 \\
**gpt+d** & 49.0 / 52.1 / 54.2 & 36.7 / 40.5 / 43.9 & 85.5 / 90.2 / 91.6 & 61.9 / 77.7 / 78.8 & 71.4 / 76.8 / 76.6 & 33.5 / 55.9 / 51.4 \\ \hline
**human*** & 97.1 & 95.0 & 98.8 & 99.1 & 95.7 & 97.9 \\ \hline \hline \end{tabular}
\end{table}
Table 8: **1-shot / 3-shot / 5-shot results for Turkish in English template for all examined models across tasks. *Due to the cost of evaluation, our human study is only evaluated on \(70\) randomly sampled instances per task and test distribution.**

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**majority** & 0.0 / 0.0 / 0.0 & 0.0 / 0.0 / 0.0 & 40.7 / 40.7 / 40.7 & 40.7 / 40.7 / 40.7 & 0.0 / 0.0 / 0.0 / 0.0 / 0.0 / 0.0 \\
**random** & 29.4 / 32.3 / 29.8 & 32.7 / 30.2 / 30.2 & 42.4 / 43.3 / 42.4 & 42.4 / 43.7 / 42.5 & 10.2 / 11.7 / 10.8 & 10.8 / 10.8 / 9.6 \\ \hline
**qwen-2.5-fb** & 13.5 / 13.5 / 16.0 & 10.2 / 11.7 / 14.4 & 61.3 / 65.4 / 68.3 & 54.6 / 57.3 / 59.4 & 31.2 / 35.8 / 39.2 & 21.9 / 25.8 / 27.7 \\
**qwen-2.5-32b** & 22.5 / 21.9 / 22.3 & 19.2 / 19.8 / 21.3 & 52.0 / 65.9 / 69.0 & 43.6 / 54.7 / 62.2 & 19.0 / 39.8 / 42.5 & 5.2 / 22.1 / 33.1 \\
**gemini-1.5-flash** & 22.5 / 26.9 / 28.1 & 20.6 / 22.9 / 24.0 & 49.4 / 71.2 / 77.7 & 40.7 / 50.3 / 56.8 & 14.2 / 48.1 / 52.3 & 0.0 / 15.4 / 25.0 \\
**gpt-d** & 37.7 / 40.6 / 44.2 & 31.5 / 35.0 / 34.4 & 70.0 / 83.1 / 85.2 & 42.2 / 65.6 / 74.8 & 47.5 / 65.4 / 66.2 & 2.7 / 39.8 / 50.6 \\ \hline
**human*** & 85.0 & 83.3 & 89.4 & 91.7 & 75.0 & 75.8 \\ \hline \hline \end{tabular}
\end{table}
Table 9: **1-shot / 3-shot / 5-shot results for Finnish in English template for all examined models across tasks. s–Due to the cost of evaluation, our human study is only evaluated on \(60\) randomly sampled instances per task and test distribution.**



**Systematicity task prompt [ID root] (CoT)**

You are given a word root, a list of affixes (separated by comma) and a word in Turkish that is derived from the given word root using the given affixes. Your task is to determine whether the derived word is grammatically correct. First, analyze how the affixes interact with the word root. Then, assess the order in which the affixes are applied and verify that this order adheres to the language's rules. Think step by step and then provide your final answer within the tags <Answer>Yes/No</Answer>.

Example 1: Word root: kuru

Affixes: t, mus

Derived word: kurutmus

Answer: To analyze the derived word "kurutmus," we need to look at the affixes and how they interact with the word root "kuru."

...<<explaining the correct order of morphemes>>...

Example 2: Word root: etki

Affixes: yici, le

Derived word: etkileyici

Answer:

**Systematicity task prompt [OOD root] (CoT)**

You are given a novel word root with its definition, a list of affixes (separated by comma) and a word in Turkish that is derived from the given word root using the given affixes. Your task is to determine whether the derived word is grammatically correct. First, analyze how the affixes interact with the word root. Then, assess the order in which the affixes are applied and verify that this order adheres to the language's rules. Think step by step and then provide your final answer within the tags <Answer>Yes/No</Answer>.

Example 1: Word root: doru

Definition: doru means kuru in Turkish.

Affixes: t, mus

Derived word: dortumus

Answer: ...<<explain the correct order of morphemes based on the definition>...

Example 2: Word root: imli

Definition: imli means etki in Turkish.

Affixes: yici, le

Derived word: lmilleyici

Answer:

\begin{table}
\begin{tabular}{l r r r r r r r} \hline \hline  & \multicolumn{6}{c}{**Number of morphemes (excl. root)**} \\
**Models** & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**majority** & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 \\
**random** & 100.0 / 100.0 & 48.0 / 46.7 & 20.0 / 18.0 & 3.3 / 6.7 & 0.0 / 1.3 & 0.7 / 0.0 & 0.0 / 0.0 \\ \hline
**aya-23-8b** & 60.0 / 52.7 & 22.7 / 8.7 & 5.3 / 0.0 & 0.7 / 0.0 & 0.7 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 \\
**aya-23-35b** & 72.7 / 69.3 & 35.3 / 20.7 & 7.3 / 6.7 & 4.0 / 4.0 & 2.0 / 1.3 & 0.0 / 0.0 & 0.7 / 0.0 \\
**qwen-2.5-7h** & 63.3 / 64.0 & 26.7 / 22.0 & 13.3 / 6.0 & 0.0 / 0.7 & 0.0 / 0.0 & 0.7 / 0.0 & 0.7 / 0.0 \\
**qwen-2.5-32b** & 82.0 / 85.3 & 46.3 / 42.3 & 18.7 / 15.3 & 6.0 / 6.7 & 3.3 / 1.3 & 3.3 / 0.7 & 0.7 / 0.0 \\
**gemini-1.5-flash** & 86.7 / 80.0 & 52.7 / 44.7 & 36.0 / 30.7 & 12.0 / 10.7 & 7.3 / 7.3 & 4.7 / 0.7 & 2.0 / 0.0 \\
**gpt-4** & 95.3 / 96.7 & 80.7 / 65.3 & 62.7 / 43.3 & 43.8 / 31.3 & 27.3 / 17.6 & 19.3 / 0.7 & 13.8 / 2.1 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Morphological productivity 1-shot ID / OOD accuracy results for **Turkish** in **English** template for all examined models.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{6}{c}{**Number of morphemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**majority** & 33.3 / 33.3 & 33.1 / 33.1 & 44.3 / 44.3 & 44.4 / 44.4 & 44.4 / 44.4 & 44.4 / 44.4 \\
**random** & 44.9 / 42.4 & 37.6 / 39.3 & 40.2 / 41.4 & 43.5 / 44.5 & 42.6 / 46.4 & 41.9 / 42.4 & 42.3 / 44.8 \\ \hline
**aya-23-8b** & 72.9 / 54.7 & 68.0 / 48.7 & 66.1 / 50.0 & 58.8 / 54.8 & 59.0 / 56.5 & 54.3 / 55.3 & 55.1 / 57.5 \\
**aya-23-35b** & 70.4 / 60.2 & 82.7 / 70.2 & 83.1 / 73.0 & 65.0 / 58.9 & 63.5 / 65.5 & 61.5 / 61.9 & 63.2 / 62.6 \\
**qwen-2.5-7h** & 71.8 / 53.1 & 72.9 / 64.7 & 73.7 / 64.4 & 77.7 / 74.0 & 73.0 / 72.3 & 65.5 / 66.9 & 63.3 / 64.4 \\
**qwen-2.5-32b** & 65.6 / 34.2 & 57.0 / 35.1 & 75.6 / 57.6 & 87.4 / 61.5 & 86.8 / 62.1 & 84.2 / 57.3 & 84.7 / 63.9 \\
**gemini-1.5-flash** & 62.4 / 33.3 & 58.2 / 34.0 & 60.1 / 44.2 & 59.0 / 44.8 & 61.9 / 44.4 & 56.9 / 44.3 & 67.2 / 44.7 \\
**gpt-4** & 86.7 / 36.2 & 69.1 / 43.8 & 82.5 / 61.8 & 88.4 / 64.2 & 92.2 / 78.7 & 88.7 / 72.0 & 90.6 / 76.9 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Morphological systematicity 1-shot ID / OOD macro-F1 results for **Turkish** in **English** template for all examined models.





**Productivity task prompt [ID root][paraphrased]**

You are provided with a word root and a set of affixes (comma-separated) in language. Your task is to create a grammatically correct word using this root and all the provided affixes. You must use only the given affixes, and each affix can be used only once. Respond with the final word only. Example 1: Word root: bulas Affixes: ma, sa, tur, ydh, k Answer: bulastrmnasaydk

Example 2: Word root: bekle Affixes: me, di, z, n, e Answer:

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{8}{c}{**Number of morphemes (excl. root)**} \\
**Models** & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**majority random** & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 \\
**productivity task prompt [poraphrased]** & You are given a new word root along with its definition, and a set of affixes (comma-separated) in language. Assuming that the new word root is a valid language word, your task is to form a grammatically correct word using this root and all the provided affixes. You must use only the given affixes, and each one can be used just once. Provide only the generated word as your answer. Example 1: Word root: lldis Definition: lldis means karls in Turkish. Affixes: sa, ydh, k, ma Answer: lldismmasaydk

Example 2: Word root: ihek Definition: ihek means emek in Turkish. Affixes: in, imiz, ter, ci Answer:



**Systematicity task prompt [ID root][paraphrased]**

You are provided with a word root, a set of affixes (comma-separated), and a word in language that is derived from the given root using the provided affixes. Your task is to verify whether the derived word is grammatically correct. Respond with only Yes or No.

Example 1: Word root: kucuk Affixes: umuz, lug, den Derived word: kuciklugumtizden Answer: Yes Example 2: Word root: evren Affixes: sel, e, lig Derived word: evrenesellig Answer:

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{8}{c}{**Number of morphemes (excl. root)**} \\
**Models** & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**majority** & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 \\
**random** & 25.3 / 29.3 & 23.3 / 24.7 & 2.7 / 2.7 & 0.7 / 2.7 & 3.3 / 2.0 & 6.0 / 1.3 & 2.0 / 4.7 \\ \hline
**aya-23-sb** & 64.7 / 34.0 & 52.7 / 19.3 & 22.0 / 2.0 & 22.7 / 6.7 & 24.0 / 12.0 & 13.3 / 12.7 & 20.1 / 23.5 \\
**aya-23-35b** & 64.0 / 39.3 & 75.3 / 54.7 & 56.7 / 43.3 & 41.3 / 34.7 & 39.3 / 41.3 & 43.3 / 36.0 & 48.3 / 30.2 \\
**qwen-25.7h** & 44.7 / 41.3 & 64.7 / 46.7 & 39.3 / 34.0 & 45.3 / 26.0 & 36.0 / 32.7 & 40.0 / 28.7 & 40.3 / 28.2 \\
**quenn-2.5-32b** & 65.3 / 32.7 & 61.1 / 40.9 & 71.3 / 49.3 & 65.3 / 52.0 & 72.0 / 51.3 & 66.7 / 46.0 & 62.4 / 47.7 \\
**gemini-1.5-flash** & 79.3 / 20.0 & 72.7 / 26.0 & 58.0 / 18.7 & 45.3 / 16.0 & 62.0 / 18.7 & 56.7 / 18.0 & 71.1 / 17.4 \\
**gpt-4** & 84.7 / 40.0 & 72.7 / 45.3 & 84.0 / 65.3 & 84.0 / 70.0 & 74.7 / 61.3 & 63.3 / 52.7 & 74.5 / 56.4 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Morphological systematicity 3-shot ID / OOD coherence results for **Turkish** in **English** template for all examined models.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{8}{c}{**Number of morphemees (excl. root)**} \\
**Models** & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**majority** & 33.3 / 33.3 & 33.1 / 33.1 & 44.3 / 44.3 & 44.4 / 44.4 & 44.4 / 44.4 & 44.4 / 44.4 \\
**random** & 40.0 / 45.3 & 40.2 / 43.1 & 44.4 / 41.5 & 40.2 / 43.4 & 43.9 / 40.3 & 43.9 / 41.8 & 39.8 / 41.0 \\ \hline
**aya-23-sb** & 75.3 / 51.8 & 68.0 / 43.3 & 64.5 / 34.2 & 60.3 / 44.2 & 64.3 / 52.7 & 55.6 / 55.3 & 64.0 / 63.3 \\
**aya-23-35b** & 74.9 / 57.3 & 83.6 / 68.2 & 86.2 / 78.1 & 79.5 / 75.3 & 77.3 / 77.3 & 78.8 / 72.8 & 80.3 / 68.2 \\
**qwen-2.5-7h** & 60.7 / 56.9 & 75.3 / 62.9 & 76.9 / 72.2 & 78.4 / 67.9 & 74.1 / 72.3 & 74.9 / 67.3 & 74.9 / 68.4 \\
**qwen-2.5-32b** & 76.4 / 53.8 & 74.0 / 60.2 & 87.3 / 75.6 & 88.6 / 78.0 & 91.0 / 76.6 & 89.0 / 76.2 & 86.5 / 78.7 \\
**gemini-1.5-flash** & 86.0 / 45.3 & 81.6 / 50.4 & 79.1 / 55.0 & 71.2 / 53.9 & 81.2 / 55.7 & 79.1 / 55.1 & 87 / 64.1 \\
**gpt-4** & 89.6 / 59.1 & 81.8 / 62.9 & 92.5 / 84.9 & 94.9 / 85.7 & 92.2 / 84.7 & 88.0 / 81.8 & 92.4 / 84.7 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Morphological systematicity 3-shot ID / OOD macro-F1 results for **Turkish** in **English** template for all examined models.





### Templates in Turkish

**Productivity task prompt [ID root]**

Size Turkce bir kok ve biir ek listesi (virgulle aynlms) verilecek v sizden bu koken verilen tim ekleri kullanarak dibligisel olarak dogru blik leime uretmeniz isteleck. Sadece verilen ekkeri kullanabilirsiniz ve her bir ek sadece bir kez kullanlabiliar. Sadece uretilen kelimeyi cktli olarak verin.

Ornek 1: Kok: kucuk Ekler: umuz, lug, den Cevap: ktculkluğumizden Ornek 2: Kok: sevgi Ekler: in, li, m Cevap:

\begin{table}
\begin{tabular}{p{45.3pt} p{76.8pt} p{76.8pt} p{76.8pt} p{76.8pt} p{76.8pt}} \hline \hline
**Models** & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**majority random** & 0.0 / 0.0 / 0.0 & 0.0 / 0.0 / 0.0 & 0.0 / 0.0 / 0.0 & 0.0 / 0.0 / 0.0 & 0.0 / 0.0 / 0.0 \\
**aya-23-3b** & 60.0 / 64.0 & 26.7 / 19.3 & 3.3 / 2.7 & 0.7 / 0.0 & 1.3 / 0.0 & 0.7 / 0.0 & 0.7 / 0.0 \\
**aya-23-35b** & 76.0 / 87.3 & 46.0 / 34.7 & 14.7 / 10.0 & 5.3 / 3.3 & 1.3 / 0.0 & 2.0 / 0.0 & 1.3 / 0.0 \\
**qwen-2.5-7b** & 70.7 / 65.3 & 27.3 / 16.7 & 8.0 / 6.7 & 3.3 / 0.7 & 0.7 / 0.7 & 0.0 / 0.0 & 0.7 / 0.0 \\
**qvn-2.5-32b** & 83.3 / 86.7 & 48.3 / 39.6 & 20.7 / 14.7 & 12.0 / 10.0 & 2.7 / 2.0 & 0.0 / 0.0 & 2.0 / 0.0 \\
**gemini-1.5-flash gpmi-1.5-flash gptt-d** & 91.3 / 79.3 & 56.7 / 51.3 & 39.3 / 29.3 & 12.0 / 8.7 & 10.0 / 6.0 & 2.7 / 1.3 & 2.7 / 0.0 \\
**human** & 96.0 / 96.7 & 85.3 / 72.0 & 66.0 / 55.5 & 43.7 / 37.3 & 40.0 / 23.9 & 28.0 / 11.6 & 20.6 / 10.1 \\ \hline \multicolumn{1}{p{45.3pt}}{**human**} & 100.0 / 100.0 & 100.0 / 95.0 & 100.0 / 95.0 & 100.0 / 100.0 & 100.0 / 95.0 & 90.0 / 100.0 & 90.0 / 80.0 \\ \hline \hline \end{tabular}
\end{table}
Table 16: Morphological productivity 5-shot **ID / OOD macroy** results for **Turkish** in **English** template for all examined models. 1-shot and 3-shot results can be found in Tables 10, 13 respectively.

\begin{table}
\begin{tabular}{p{45.3pt} p{76.8pt} p{76.8pt} p{76.8pt} p{76.8pt} p{76.8pt} p{76.8pt}} \hline \hline
**Models** & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**majority random** & 33.3 / 33.3 & 33.1 / 33.1 & 44.3 / 44.3 & 44.4 / 44.4 & 44.4 / 44.4 & 44.4 / 44.4 & 44.4 / 44.4 \\
**ava-23-3b** & 74.0 / 49.6 & 71.3 / 49.8 & 67.7 / 44.3 & 65.6 / 52.8 & 69.7 / 56.7 & 61.3 / 48.0 & 63.3 / 59.4 \\
**aya-23-35b** & 77.3 / 63.6 & 87.1 / 68.2 & 85.7 / 79.4 & 82.2 / 75.2 & 82.1 / 76.7 & 80.5 / 70.5 & 77.8 / 71.2 \\
**qvn-2.5-7b** & 66.0 / 58.0 & 76.2 / 62.7 & 76.2 / 71.1 & 78.6 / 70.4 & 74.4 / 68.6 & 76.6 / 67.1 & 74.5 / 63.8 \\
**qvn-2.5-32b** & 80.4 / 58.4 & 78.1 / 68.9 & 89.5 / 79.1 & 90.3 / 80.8 & 89.7 / 80.8 & 88.1 / 81.0 & 85.5 / 77.9 \\
**gemini-1.5-flash gemii-1.5-flash gptt-d** & 86.9 / 53.3 & 82.9 / 55.8 & 83.4 / 66.3 & 85.5 / 60.0 & 84.8 / 60.5 & 85.5 / 66.3 & 89.2 / 72.3 \\
**bpf-d** & 92.0 / 52.7 & 90.9 / 82.9 & 94.4 / 86.1 & 93.9 / 84.7 & 90.8 / 80.6 & 89.1 / 82.9 & 90.2 / 81.4 \\ \hline
**human** & 100.0 / 100.0 / 96.7 & 100.0 / 100.0 & 97.2 / 100.0 & 100.0 / 100.0 & 98.8 / 97.6 & 95.2 / 100.0 \\ \hline \hline \end{tabular}
\end{table}
Table 17: Morphological systematicty 5-shot **ID / OOD macroy** results for **Turkish** in **English** template for all examined models. 1-shot and 3-shot results can be found in Tables 10, 13 respectively.





\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**Number of morphemes (excl. root)**} \\
**Models** & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**majority** & 0.0/ 0.0 & 0.0/ 0.0 & 0.0/ 0.0 & 0.0/ 0.0 & 0.0/ 0.0 & 0.0/ 0.0 & 0.0/ 0.0 \\
**random** & 24.7/ 30.0 & 26.0/ 20.0 & 2.0/ 1.3 & 4.7/ 1.3 & 2.0/ 4.7 & 4.0/ 4.0 & 2.7/ 2.0 \\ \hline
**aya-23-8b** & 62.7/ 32.0 & 58.7/ 30.7 & 24.7/ 7.3 & 30.0/ 14.7 & 34.0/ 15.3 & 18.0/ 8.0 & 24.2/ 20.8 \\
**aya-23-35h** & 68.0/ 50.7 & 80.7/ 53.5 & 56.0/ 49.3 & 47.3/ 36.0 & 49.3/ 38.7 & 46.0/ 30.0 & 43.0/ 34.9 \\
**qven-2.5-7b** & 51.3/ 40.7 & 66.7/ 46.0 & 34.0/ 30.7 & 43.3/ 30.7 & 37.3/ 31.3 & 39.3/ 27.3 & 43.6/ 24.8 \\
**qwen-2.5-32b** & 71.3/ 40.7 & 67.1/ 54.2 & 72.0/ 48.0 & 72.0/ 50.0 & 67.3/ 53.3 & 62.0/ 49.3 & 55.7/ 42.3 \\
**gemini-1.5-flash** & 80.7/ 31.3 & 75.3/ 34.7 & 65.3/ 34.7 & 69.3/ 26.7 & 64.7/ 27.3 & 68.0/ 34.7 & 71.8/ 43.6 \\
**gpt-4** & 88.0/ 31.3 & 86.7/ 74.7 & 86.0/ 57.3 & 78.0/ 56.7 & 68.7/ 46.0 & 64.7/ 50.7 & 64.4/ 43.0 \\ \hline
**human** & 100.0/ 100.0 & 100.0/ 95.0 & 100.0/ 100.0 & 95.0/ 100.0 & 100.0/ 100.0 & 95.0/ 90.0 & 80.0/ 100.0 \\ \hline \hline \end{tabular}
\end{table}
Table 18: Morphological predicativity 5-shot ID / OOD coherence results for Turkish in English template for all examined models. 1-shot and 3-shot results can be found in Tables 12, 15 respectively.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{6}{c}{**Number of morphemes (excl. root)**} \\
**Models** & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline
**majority** & 0.0/ 0.0 & 0.0/ 0.0 & 0.0/ 0.0 & 0.0/ 0.0 & 0.0/ 0.0 & 0.0/ 0.0 \\
**random** & 100.0/ 100.0 & 43.8/ 67.5 & 18.8/ 10.0 & 7.5/ 10.0 & 5.0/ 5.0 & 1.2/ 3.8 \\ \hline
**qwen-2.5-7b** & 55.0/ 45.0 & 22.5/ 15.0 & 2.5/ 1.2 & 1.2/ 0.0 & 0.0/ 0.0 & 0.0/ 0.0 \\
**qwen-2.5-32b** & 72.5/ 71.2 & 41.2/ 32.5 & 18.8/ 8.8 & 2.5/ 2.5 & 0.0/ 0.0 & 0.0/ 0.0 \\
**gemini-1.5-flash** & 75.0/ 73.8 & 38.8/ 36.2 & 12.5/ 8.8 & 6.2/ 1.2 & 2.5/ 3.8 & 0.0/ 0.0 \\
**gpt-4** & 81.2/ 86.2 & 76.2/ 58.8 & 26.2/ 21.2 & 15.0/ 6.2 & 18.8/ 8.8 & 8.8/ 7.5 \\ \hline \hline \end{tabular}
\end{table}
Table 19: Morphological productivity 1-shot ID / OOD accuracy results for Finnish in English template for all examined models.





\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{6}{c}{**Number of morphemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline
**majority** & 33.3 / 33.3 & 33.3 / 33.3 & 44.4 / 44.4 & 44.4 / 44.4 & 44.4 / 44.4 & 44.4 / 44.4 \\
**random** & 41.7 / 37.1 & 42.1 / 46.7 & 42.3 / 42.7 & 40.7 / 41.6 & 42.5 / 42.2 & 45.0 / 44.4 \\ \hline
**qwen-2.5-7b** & 65.8 / 57.1 & 77.5 / 71.2 & 61.5 / 57.2 & 53.3 / 47.7 & 53.5 / 47.2 & 56.0 / 47.2 \\
**qwen-2.5-32b** & 39.2 / 34.2 & 58.8 / 41.7 & 56.6 / 49.3 & 55.4 / 46.5 & 52.2 / 45.1 & 49.7 / 45.1 \\
**gemini-1.5-flash** & 50.8 / 33.3 & 45.0 / 33.3 & 54.7 / 44.4 & 49.7 / 44.4 & 46.5 / 44.4 & 49.7 / 44.4 \\
**gpt-4** & 79.2 / 34.2 & 73.3 / 36.7 & 73.9 / 47.9 & 64.4 / 45.8 & 71.2 / 44.4 & 58.2 / 44.4 \\ \hline \hline \end{tabular}
\end{table}
Table 20: Morphological systematicity 1-shot **ID / OOD macro-F1** results for **Finnish** in **English** template for all examined models.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Models** & \multicolumn{6}{c}{**Number of morphemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline
**majority** & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 \\
**random** & 23.8 / 21.2 & 25.0 / 30.0 & 3.8 / 1.2 & 0.0 / 5.0 & 6.2 / 0.0 & 2.5 / 7.5 \\ \hline
**qwen-2.5-7b** & 50.0 / 41.2 & 67.5 / 57.5 & 26.2 / 17.5 & 12.5 / 5.0 & 15.0 / 5.0 & 16.2 / 5.0 \\
**qwen-2.5-32b** & 11.2 / 3.8 & 38.8 / 12.5 & 22.5 / 8.8 & 18.8 / 3.8 & 13.8 / 1.2 & 8.8 / 1.2 \\
**gemini-1.5-flash** & 28.7 / 0.0 & 17.5 / 0.0 & 16.2 / 0.0 & 10.0 / 0.0 & 3.8 / 0.0 & 8.8 / 0.0 \\
**gpt-4** & 68.8 / 2.5 & 60.0 / 5.0 & 52.5 / 6.2 & 33.8 / 2.5 & 45.0 / 0.0 & 25.0 / 0.0 \\ \hline \hline \end{tabular}
\end{table}
Table 21: Morphological systematicity 1-shot **ID / OOD coherence** results for **Finnish** in **English** template for all examined models.





### Templates in Finnish

**Productivity task prompt [IOD root]**

Sinulle annetaan sanan sananvartalo ja luettelo pilkulla erotettuja paatteita kiellla suomi. Tehtavasi on luoda tista juresta kieliophilisesti oikea sana kayttamalla kaikkia annettuja paatteita. Voit kaytta vain annettuja paatteita ja ktakin paatetta vain kerran. Vastaa vain luodulla sanalla.

Esimerkki 1: Sananvartalo: markiise Paatteteet: j, a

Vastaus: markiseja Esimerkki 2: Sananvartalo: kasvatamis Paatteteet: si, ta

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{3}{*}{**Models**} & \multicolumn{8}{c}{**Number of morphemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline
**majority** & 33.3 / 33.3 & 33.3 / 33.3 & 44.4 / 44.4 & 44.4 / 44.4 & 44.4 / 44.4 / 44.4 \\
**random** & 40.0 / 43.8 & 43.8 / 37.9 & 45.9 / 48.5 & 46.1 / 42.9 & 44.7 / 45.3 & 39.3 / 44.0 \\ \hline
**qwen-2.5-7b** & 62.9 / 56.7 & 81.2 / 77.1 & 67.2 / 55.4 & 62.8 / 52.5 & 57.7 / 52.0 & 60.4 / 50.4 \\
**quven-2.5-32b** & 52.9 / 37.5 & 70.0 / 57.5 & 71.6 / 59.4 & 66.3 / 54.7 & 65.7 / 56.1 & 68.7 / 62.8 \\
**gemini-1.5-flash** & 84.2 / 50.8 & 70.4 / 52.1 & 72.1 / 55.1 & 64.9 / 48.8 & 66.8 / 46.4 & 69.0 / 48.6 \\
**gpt-4** & 81.7 / 52.5 & 87.9 / 65.8 & 84.4 / 67.7 & 83.0 / 68.9 & 78.4 / 69.7 & 83.1 / 69.2 \\ \hline \end{tabular}
\end{table}
Table 23: Morphological systematicity 3-shot **ID / OOD macro-F1** results for **Finnish** in **English** template for all examined models.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{3}{*}{**Models**} & \multicolumn{8}{c}{**Number of morphemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline
**majority** & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 \\
**random** & 100.0 / 100.0 & 56.2 / 46.2 & 12.5 / 17.5 & 7.5 / 12.5 & 13.8 / 3.8 & 3.8 / 1.2 \\ \hline
**qwen-2.5-7b** & 58.8 / 55.0 & 15.0 / 11.2 & 5.0 / 2.5 & 1.2 / 0.0 & 1.2 / 1.2 & 0.0 / 0.0 \\
**qwan-2.5-32b** & 70.0 / 67.5 & 41.2 / 36.2 & 12.5 / 10.0 & 3.8 / 2.5 & 1.2 / 2.5 & 2.5 / 0.0 \\
**gemini-1.5-flash** & 80.0 / 76.2 & 51.2 / 42.5 & 13.8 / 7.5 & 5.0 / 1.2 & 6.2 / 6.2 & 5.0 / 3.8 \\
**gptl-4** & 81.2 / 90.0 & 73.8 / 62.5 & 32.5 / 20.0 & 20.0 / 13.8 & 26.2 / 11.2 & 10.0 / 12.5 \\ \hline \end{tabular}
\end{table}
Table 22: Morphological productivity 3-shot **ID / OOD accuracy** results for **Finnish** in **English** template for all examined models.





\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{6}{c}{**Number of morphemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline
**majority** & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 \\
**random** & 26.2 / 26.2 & 27.5 / 18.8 & 5.0 / 8.8 & 5.0 / 2.5 & 3.8 / 6.2 & 2.5 / 2.5 \\ \hline
**qwen-2.5-7b** & 45.0 / 41.2 & 72.5 / 66.2 & 30.0 / 15.0 & 27.5 / 11.2 & 22.5 / 12.5 & 17.5 / 8.8 \\
**qwen-2.5-32b** & 31.2 / 8.8 & 56.2 / 36.2 & 45.0 / 25.0 & 35.0 / 16.2 & 31.2 / 17.5 & 40.0 / 28.7 \\
**gemini-1.5-flash** & 76.2 / 28.7 & 56.2 / 28.7 & 43.8 / 16.2 & 36.2 / 7.5 & 37.5 / 3.8 & 38.8 / 7.5 \\
**gpt-4** & 72.5 / 32.5 & 82.5 / 48.8 & 63.7 / 38.8 & 61.3 / 38.8 & 53.8 / 41.2 & 58.8 / 38.8 \\ \hline \hline \end{tabular}
\end{table}
Table 24: Morphological systematicy 3-shot ID / OOD coherence results for **Finnish** in **English** template for all examined models.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{6}{c}{**Number of morphemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline
**majority** & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 \\
**random** & 100.0 / 100.0 & 50.0 / 46.2 & 13.8 / 17.5 & 7.5 / 11.2 & 6.2 / 2.5 & 1.2 / 3.8 \\ \hline
**qwen-2.5-7b** & 62.5 / 60.0 & 26.2 / 17.5 & 3.8 / 6.2 & 1.2 / 1.2 & 1.2 / 1.2 & 1.2 / 0.0 \\
**qwen-2.5-32b** & 72.5 / 77.5 & 43.8 / 33.8 & 10.0 / 11.2 & 3.8 / 2.5 & 1.2 / 1.2 & 2.5 / 1.2 \\
**gemini-1.5-flash** & 81.2 / 73.8 & 51.2 / 45.0 & 17.5 / 10.0 & 5.0 / 3.8 & 8.8 / 7.5 & 5.0 / 3.8 \\
**gpt-4** & 83.8 / 90.0 & 73.8 / 66.2 & 40.0 / 23.8 & 22.5 / 5.0 & 30.0 / 12.5 & 15.0 / 8.8 \\ \hline
**human** & 90.0 / 95.0 & 90.0 / 90.0 & 100.0 / 95.0 & 80.0 / 80.0 & 60.0 / 80.0 & 90.0 / 60.0 \\ \hline \hline \end{tabular}
\end{table}
Table 25: Morphological productivity 5-shot ID / OOD accuracy results for **Finnish** in **English** template for all examined models. 1-shot and 3-shot results can be found in Tables 19, 22 respectively.





**Productivity task prompt [ID root] (with context)**

Allaolevassa lauseessa (kirjointettu kielella suomi) on tyhja kohta (__) joka tulee tayttaa kieliopillisesti oikealla sanalla. Alla on myos sananvartalo seka pilkulla eroteltu luertelo paatteista. Tehtavasi on kayttaa vartaloa seka pgautteita ja johtaa nistakieliopillisesti oikein taivvetatu sama joka spoji tyjahan kothaalausessa asiaythteys/kontektsti huomioonottaan. Kayta jokaisata piatettu vankern. Vastana vain generoidulla sanalla, ala sano mitaan mmuta. Esimerkkki 1: Sananvartalo: markiise Paattetet: a, j Laause: --- saatavana yksivirarisina, raidallisiana ja voit myos valia halautoko markissisi vevivi-vai sinkokkayttoisena. Vasatus: markiiseja Esimerkki 2: Sananvartalo: suitheut Pazattete: na, tu Lause: -- vakilukuuun, suomessa on enemman metsaau kinin missaan muussa euroopen maassa. Vastaus:

\begin{table}
\begin{tabular}{p{85.4pt} c c c c c} \hline \hline \multirow{3}{*}{**Models**} & \multicolumn{6}{c}{**Number of morplemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline
**majority** & 33.3 / 33.3 & 33.3 / 33.3 & 44.4 / 44.4 & 44.4 / 44.4 & 44.4 / 44.4 & 44.4 / 44.4 \\
**random** & 43.8 / 42.5 & 40.4 / 42.9 & 43.7 / 39.8 & 41.6 / 47.3 & 39.4 / 42.4 & 45.7 / 40.0 \\ \hline
**qwen-2.5-7b** & 65.8 / 55.4 & 86.7 / 75.4 & 65.0 / 58.1 & 66.2 / 54.2 & 62.5 / 57.7 & 63.5 / 55.9 \\
**qwen-2.5-32b** & 57.1 / 40.0 & 72.9 / 66.7 & 71.7 / 69.8 & 72.8 / 63.6 & 70.7 / 71.1 & 68.6 / 62.3 \\
**gemini-1.5-flash** & 84.2 / 49.6 & 80.8 / 65.4 & 79.6 / 62.0 & 75.4 / 52.8 & 74.9 / 53.5 & 71.4 / 57.7 \\
**gpt-4** & 84.2 / 61.3 & 89.2 / 78.8 & 84.2 / 83.8 & 88.0 / 76.7 & 83.0 / 76.5 & 82.7 / 71.6 \\ \hline
**human** & 73.3 / 86.7 & 93.3 / 100.0 & 93.2 / 97.6 & 95.5 / 90.9 & 91.2 / 90.5 & 89.7 / 84.4 \\ \hline \hline \end{tabular}
\end{table}
Table 26: Morphological systematicty 5-shot **ID / OOD macro-F1** results for **Finnish** in **English** template for all examined models. 1-shot and 3-shot results can be found in Tables 20, 23 respectively.

\begin{table}
\begin{tabular}{p{85.4pt} c c c c c c} \hline \hline \multirow{3}{*}{**Models**} & \multicolumn{6}{c}{**Number of morplemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline
**majority** & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 & 0.0 / 0.0 \\
**random** & 26.2 / 26.2 & 25.0 / 23.8 & 3.8 / 0.0 & 1.2 / 5.0 & 2.5 / 1.2 & 6.2 / 1.2 \\ \hline
**qwen-2.5-7b** & 48.8 / 36.2 & 81.2 / 65.0 & 28.7 / 16.2 & 31.2 / 15.0 & 22.5 / 18.8 & 22.5 / 15.0 \\
**qwen-2.5-32b** & 37.5 / 13.8 & 61.3 / 51.2 & 41.2 / 38.8 & 46.2 / 28.7 & 33.8 / 41.2 & 35.0 / 25.0 \\
**gemini-1.5-flash** & 76.2 / 28.7 & 71.2 / 48.8 & 47.5 / 26.2 & 43.8 / 12.5 & 42.5 / 12.5 & 32.5 / 21.2 \\
**grpt-4** & 76.2 / 45.0 & 83.8 / 68.8 & 56.2 / 61.3 & 68.8 / 48.8 & 57.5 / 40.0 & 55.0 / 40.0 \\ \hline
**human** & 60.0 / 80.0 & 90.0 / 100.0 & 75.0 / 90.0 & 85.0 / 75.0 & 70.0 / 60.0 & 70.0 / 50.0 \\ \hline \hline \end{tabular}
\end{table}
Table 27: Morphological systematicty 5-shot **ID / OOD coherence** results for **Finnish** in **English** template for all examined models. 1-shot and 3-shot results can be found in Tables 21, 24 respectively.





\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{1}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{1}{c}{**Morph. Systematicticity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**aya-23-8b** & 13.7 / 7.0 & 11.5 / 9.6 & 64.6 / 69.0 & 49.3 / 46.4 & 31.4 / 38.7 & 15.7 / 8.0 \\
**aya-23-35b** & 19.8 / 17.1 & 17.7 / 16.9 & 80.1 / 81.6 & 71.0 / 57.1 & 52.6 / 59.6 & 39.9 / 24.7 \\
**qwen-2.5-7b** & 14.9 / 11.2 & 12.9 / 11.4 & 73.6 / 73.7 & 66.8 / 61.4 & 44.3 / 44.8 & 33.9 / 28.2 \\
**qwen-2.5-32b** & 23.7 / 20.7 & 21.8 / 19.9 & 84.7 / 86.1 & 71.3 / 65.2 & 66.3 / 70.5 & 45.7 / 36.8 \\
**gemini-1.5-flash** & 30.5 / 23.0 & 25.7 / 20.7 & 80.8 / 56.9 & 52.8 / 41.2 & 63.6 / 25.3 & 19.3 / 0.0 \\
**gpt-4** & 52.1 / 51.5 & 40.5 / 40.6 & 90.2 / 88.6 & 77.7 / 66.3 & 76.8 / 76.3 & 55.9 / 39.9 \\ \hline \hline \end{tabular}
\end{table}
Table 29: 3-shot English / Turkish template results for Turkish for all examined models across tasks.

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**aya-23-8b** & 12.8 / 8.4 & 8.8 / 5.3 & 62.0 / 62.4 & 53.9 / 44.3 & 27.9 / 31.9 & 19.1 / 4.5 \\
**aya-23-35b** & 17.4 / 14.6 & 14.6 / 11.7 & 69.9 / 79.6 & 64.6 / 48.9 & 36.8 / 59.2 & 29.2 / 12.2 \\
**spani-2.5-7b** & 15.0 / 9.3 & 13.2 / 10.0 & 71.1 / 69.9 & 65.7 / 59.1 & 40.5 / 38.1 & 33.5 / 23.6 \\
**qwen-2.5-32b** & 22.6 / 22.9 & 21.7 / 20.4 & 77.3 / 78.5 & 53.1 / 44.5 & 56.7 / 60.0 & 18.5 / 5.7 \\
**gemini-1.5-flash** & 28.8 / 21.8 & 24.9 / 19.3 & 60.8 / 45.1 & 41.4 / 41.2 & 32.2 / 5.9 & 0.4 / 0.0 \\
**gpt-4** & 49.0 / 48.5 & 36.7 / 38.1 & 85.5 / 76.3 & 61.9 / 50.4 & 71.4 / 58.6 & 33.5 / 15.7 \\ \hline \hline \end{tabular}
\end{table}
Table 28: 1-shot English / Turkish template results for Turkfish for all examined models across tasks.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**aya-23-8b** & 13.3 / 8.6 & 12.3 / 8.1 & 67.5 / 62.0 & 51.5 / 47.9 & 36.0 / 30.6 & 18.4 / 15.8 \\
**aya-23-35b** & 21.0 / 17.1 & 19.3 / 18.0 & 81.8 / 81.7 & 72.1 / 64.3 & 55.8 / 57.7 & 41.8 / 30.4 \\
**quenn-2.5-7b** & 15.8 / 13.0 & 12.9 / 12.3 & 74.6 / 73.5 & 66.0 / 65.2 & 45.1 / 42.4 & 33.1 / 32.3 \\
**gpt-4** & 24.1 / 23.3 & 21.8 / 21.9 & 85.9 / 86.9 & 75.3 / 70.7 & 66.8 / 70.5 & 48.3 / 44.2 \\
**gemini-1.5-flash** & 30.7 / 25.8 & 25.1 / 22.1 & 85.4 / 61.6 & 62.1 / 41.6 & 70.7 / 32.9 & 33.3 / 0.7 \\
**gpt-4** & 54.2 / 53.1 & 43.9 / 40.7 & 91.6 / 92.7 & 78.8 / 76.3 & 76.6 / 82.1 & 51.4 / 51.9 \\ \hline \hline \end{tabular}
\end{table}
Table 30: 5-shot English / Turkish template results for Turkish from all examined models across tasks. Results for 1-shot and 3-shot can be found in Tables 28 and 29.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**aya-23-8b** & 13.3 / 8.6 & 12.3 / 8.1 & 67.5 / 62.0 & 51.5 / 47.9 & 36.0 / 30.6 & 18.4 / 15.8 \\
**ajwa-23-35b** & 21.0 / 17.1 & 19.3 / 18.0 & 81.8 / 81.7 & 72.1 / 64.3 & 55.8 / 57.7 & 41.8 / 30.4 \\
**ugwen-2.5-7b** & 15.8 / 13.0 & 12.9 / 12.3 & 74.6 / 73.5 & 66.0 / 65.2 & 45.1 / 42.4 & 33.1 / 32.3 \\
**gpt-4** & 24.1 / 23.3 & 21.8 / 21.9 & 85.9 / 86.9 & 75.3 / 70.7 & 66.8 / 70.5 & 48.3 / 44.2 \\
**gemini-1.5-flash** & 30.7 / 25.8 & 25.1 / 22.1 & 85.4 / 61.6 & 62.1 / 41.6 & 70.7 / 32.9 & 33.3 / 0.7 \\
**gpt-4** & 54.2 / 53.1 & 43.9 / 40.7 & 91.6 / 92.7 & 78.8 / 76.3 & 76.6 / 82.1 & 51.4 / 51.9 \\ \hline \hline \end{tabular}
\end{table}
Table 31: 1-shot English / Finnish template results for Finnish for all examined models across tasks.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**qwen-2.5-7b** & 13.5 / 10.4 & 11.7 / 10.2 & 65.4 / 65.0 & 57.3 / 55.7 & 35.8 / 32.5 & 25.8 / 22.5 \\
**qwen-2.5-32b** & 21.9 / 21.0 & 19.8 / 17.3 & 65.9 / 60.9 & 54.7 / 56.4 & 39.8 / 31.7 & 22.1 / 24.8 \\
**gemini-1.5-flash** & 26.9 / 24.6 & 22.9 / 19.2 & 71.2 / 74.7 & 50.3 / 48.0 & 48.1 / 51.7 & 15.4



\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{1}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{1}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**aya-23-Sb** & 13.7 / 13.6 & 11.5 / 9.6 & 64.6 / 50.3 & 49.3 / 42.5 & 31.4 / 16.8 & 15.7 / 9.6 \\
**aya-23-35b** & 19.8 / 24.3 & 17.7 / 17.8 & 80.1 / 66.3 & 71.0 / 51.0 & 52.6 / 34.9 & 39.9 / 16.1 \\
**qven-2.5-7h** & 14.9 / 13.5 & 12.9 / 9.8 & 73.6 / 63.4 & 66.8 / 59.3 & 44.3 / 30.3 & 33.9 / 25.2 \\
**qvren-2.5-32b** & 23.7 / 24.2 & 21.8 / 19.3 & 84.7 / 61.0 & 71.3 / 47.3 & 63.6 / 33.2 & 45.7 / 10.2 \\
**gemini-1.5-flash** & 30.5 / 38.4 & 25.7 / 26.5 & 80.8 / 75.6 & 52.8 / 60.1 & 63.6 / 49.7 & 19.3 / 27.2 \\
**gpt-4** & 52.1 / 59.8 & 40.5 / 45.6 & 90.2 / 85.2 & 77.7 / 67.1 & 76.8 / 71.9 & 55.9 / 41.5 \\ \hline \hline \end{tabular}
\end{table}
Table 35: 3-shot **No context / With context results for Turkish in English template for all examined models across tasks.**

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{1}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{1}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**avn-2.5-7h** & 13.5 / 11.0 & 11.7 / 10.8 & 65.4 / 63.5 & 57.3 / 55.3 & 35.8 / 34.0 & 25.8 / 21.5 \\
**qvren-2.5-32b** & 21.9 / 22.5 & 19.8 / 16.0 & 65.9 / 62.0 & 54.7 / 47.3 & 39.8 / 33.8 & 22.1 / 10.6 \\
**gemini-1.5-flash** & 26.9 / 32.3 & 22.9 / 23.3 & 71.2 / 72.3 & 50.3 / 57.3 & 48.1 / 43.3 & 15.4 / 23.3 \\
**gpt-4** & 40.6 / 52.3 & 35.0 / 33.5 & 83.1 / 83.8 & 65.6 / 65.2 & 65.4 / 67.9 & 39.8 / 39.2 \\ \hline \hline \end{tabular}
\end{table}
Table 38: 3-shot **No context / With context results for Finnish in English template for all examined models across tasks.**

\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{1}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{1}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**avn-2.5-7h** & 13.5 / 11.0 & 11.7 / 10.8 & 65.4 / 63.5 & 57.3 / 55.3 & 35.8 / 34.0 & 25.8 / 21.5 \\
**qvren-2.5-32b** & 21.9 / 22.5 & 19.8 / 16.0 & 65.9 / 62.0 & 54.7 / 47.3 & 39.8 / 33.8 & 22.1 / 10.6 \\
**gemini-1.5-flash** & 26.9 / 32.3 & 22.9 / 23.3 & 71.2 / 72.3 & 50.3 / 57.3 & 48.1 / 43.3 & 15.4 / 23.3 \\
**gpt-4** & 40.6 / 52.3 & 35.0 / 33.5 & 83.1 / 83.8 & 65.6 / 65.2 & 65.4 / 67.9 & 39.8 / 39.2 \\ \hline \hline \end{tabular}
\end{table}
Table 37: 1-shot **No context / With context results for Finnish in English template for all examined models across tasks.**

\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{1}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{1}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**avn-2.5-7h** & 13.5 / 11.0 & 11.7 / 10.8 & 65.4 / 63.5 & 57.3 / 55.3 & 35.8 / 34.0 & 25.8 / 21.5 \\
**qvren-2.5-32b** & 21.9 / 22.5 & 19.8 / 16.0 & 65.9 / 62.0 & 54.7 / 47.3 & 39.8 / 33.8 & 22.1 / 10.6 \\
**gemini-1.5-flash** & 26.9 / 32.3 & 22.9 / 23.3 & 71.2 / 72.3 & 50.3 / 57.3 & 48.1 / 43.3 & 15.4 / 23.3 \\
**gpt-4** & 40.6 / 52.3 & 35.0 / 33.5 & 83.1 / 83.8 & 65.6 / 65.2 & 65.4 / 67.9 & 39.8 / 39.2 \\ \hline \hline \end{tabular}
\end{table}
Table 39: 5-shot **No context / With context results for Finnish in English template for all examined models across tasks.**



\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{3}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicticity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**aya-23-5b** & 13.7 / 14.9 & 11.5 / 12.5 & 64.6 / 63.8 & 49.3 / 49.1 & 31.4 / 30.7 & 15.7 / 16.4 \\
**aya-23-35b** & 19.8 / 25.7 & 17.7 / 23.7 & 80.1 / 80.2 & 71.0 / 73.0 & 52.6 / 52.2 & 39.9 / 41.9 \\
**qven-2.5-7h** & 14.9 / 17.8 & 12.9 / 15.5 & 73.6 / 76.1 & 66.8 / 69.4 & 44.3 / 47.9 & 33.9 / 38.0 \\
**quren-2.5-32b** & 23.7 / 30.7 & 21.8 / 32.1 & 84.7 / 86.9 & 71.3 / 75.7 & 66.3 / 70.3 & 45.7 / 53.0 \\
**gemini-1.5-flash** & 30.5 / 39.6 & 25.7 / 33.4 & 80.8 / 85.4 & 52.8 / 58.0 & 63.6 / 71.6 & 19.3 / 27.4 \\
**gpt-4** & 52.1 / 70.3 & 40.5 / 63.5 & 90.2 / 92.9 & 77.7 / 81.0 & 76.8 / 82.0 & 55.9 / 59.6 \\ \hline \hline \end{tabular}
\end{table}
Table 45: 3-shot **Shuffled / Correct** morpheme order results for **Turkish in English** template for all examined models across tasks.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{5}{c}{**Number of morphemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**gpt-4** & 94.7 / 84.7 & 81.3 / 72.5 & 64.0 / 58.0 & 49.3 / 48.2 & 30.7 / 39.5 & 25.3 / 36.8 & 19.3 / 23.8 \\ \hline \hline \end{tabular}
\end{table}
Table 41: GPT-4 morphological productivity 3-shot **morphologically aligned / tokenizer aligned accuracy** results on the ID test set for **Turkish** in **English** template.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{5}{c}{**Number of morphemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**gpt-4** & 96.0 / 88.0 & 85.3 / 69.8 & 66.0 / 64.0 & 43.7 / 44.8 & 40.0 / 42.2 & 28.0 / 32.4 & 20.6 / 25.7 \\ \hline \hline \end{tabular}
\end{table}
Table 42: GPT-4 morphological productivity 5-shot **morphologically aligned / tokenizer aligned accuracy** results on the ID test set for **Turkish** in **English** template.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{5}{c}{**Number of morphemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**gpt-4** & 95.3 / 84.0 & 80.7 / 67.8 & 62.7 / 52.7 & 43.8 / 42.1 & 27.3 / 32.0 & 19.3 / 26.5 & 13.8 / 13.9 \\ \hline \hline \end{tabular}
\end{table}
Table 40: GPT-4 morphological productivity 1-shot **morphologically aligned / tokenizer aligned accuracy** results on the ID test set for **Turkish** in **English** template.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{5}{c}{**Number of morphemes (excl. root)**} \\  & 1 & 2 & 3 & 4 & 5 & 6 & 7 \\ \hline
**gpt-4** & 94.7 / 84.7 & 81.3 / 72.5 & 64.0 / 58.0 & 49.3 / 48.2 & 30.7 / 39.5 & 25.3 / 36.8 & 19.3 / 23.8 \\ \hline \hline \end{tabular}
\end{table}
Table 41: GPT-4 morphological productivity 3-shot **morphologically aligned / tokenizer aligned accuracy** results on the ID test set for **Turkish** in **English** template.





\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**qven-2.5-7b** & 13.5 / 16.5 & 10.2 / 12.3 & 61.3 / 62.9 & 54.6 / 55.2 & 31.2 / 34.2 & 21.9 / 22.5 \\
**qwen-2.5-32b** & 22.5 / 24.8 & 19.2 / 23.1 & 52.0 / 53.1 & 43.6 / 44.0 & 19.0 / 20.4 & 5.2 / 5.8 \\
**gemini-1.5-flash** & 22.5 / 26.7 & 20.6 / 25.6 & 49.4 / 51.2 & 40.7 / 40.9 & 14.2 / 17.5 & 0.0 / 0.2 \\
**gpt-4** & 37.7 / 46.0 & 31.5 / 39.6 & 70.0 / 70.7 & 42.2 / 43.7 & 47.5 / 48.3 & 2.7 / 5.0 \\ \hline \hline \end{tabular}
\end{table}
Table 47: 1-shot **Shuffled / Correct** morpheme order results for **Finnish in English** template for all examined models across tasks.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**qven-2.5-7b** & 16.0 / 18.8 & 14.4 / 17.1 & 68.3 / 68.7 & 59.4 / 61.6 & 39.2 / 40.2 & 27.7 / 29.4 \\
**qwen-2.5-32b** & 22.3 / 27.9 & 21.3 / 31.7 & 69.0 / 71.3 & 62.2 / 64.7 & 42.5 / 46.7 & 33.1 / 37.3 \\
**gemini-1.5-flash** & 28.1 / 35.0 & 24.0 / 30.6 & 77.7 / 79.9 & 56.8 / 57.7 & 52.3 / 58.5 & 25.0 / 26.9 \\
**gpt-4** & 44.2 / 59.6 & 34.4 / 52.1 & 85.2 / 86.9 & 74.8 / 78.4 & 66.2 / 70.0 & 50.6 / 57.1 \\ \hline \hline \end{tabular}
\end{table}
Table 48: 3-shot **Shuffled / Correct** morpheme order results for **Finnish** in **English** template for all examined models across tasks.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD \\ \hline
**qven-2.5-7b** & 16.0 / 18.8 & 14.4 / 17.1 & 68.3 / 68.7 & 59.4 / 61.6 & 39.2 / 40.2 & 27.7 / 29.4 \\
**qwen-2.5-32b** & 22.3 / 27.9 & 21.3 / 31.7 & 69.0 / 71.3 & 62.2 / 64.7 & 42.5 / 46.7 & 33.1 / 37.3 \\
**gemini-1.5-flash** & 28.1 / 35.0 & 24.0 / 30.6 & 77.7 / 79.9 & 56.8 / 57.7 & 52.3 / 58.5 & 25.0 / 26.9 \\
**gpt-4** & 44.2 / 59.6 & 34.4 / 52.1 & 85.2 / 86.9 & 74.8 / 78.4 & 66.2 / 70.0 & 50.6 / 57.1 \\ \hline \hline \end{tabular}
\end{table}
Table 49: 5-shot **Shuffled / Correct** morpheme order results for **Finnish** in **English** template for all examined models across tasks.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD \\ \hline
**aya-23-8b** & 74.4 / 64.6 / 61.9 & 53.0 / 49.3 / 47.1 & 45.6 / 31.4 / 28.1 & 21.5 / 15.7 / 13.4 \\
**aya-23-35b** & 88.2 / 80.1 / 78.8 & 80.4 / 71.0 / 71.0 & 72.9 / 52.6 / 48.9 & 60.5 / 39.9 / 38.1 \\
**qwen-2.5-7b** & 81.2 / 73.6 / 71.6 & 75.3 / 66.8 / 65.6 & 51.0 / 36.3 & 54.7 / 38.8 & 53.5 / 33.9 / 32.2 \\
**qwen-2.5-32b** & 88.3 / 84.7 / 83.4 & 74.5 / 71.3 / 69.8 & 78.3 / 66.3 / 63.6 & 55.4 / 45.7 / 42.4 \\
**gemini-1.5-flash** & 80.2 / 80.8 / 79.9 & 51.7 / 52.8 / 51.5 & 65.3 / 63.6 / 60.8 & 71.8 / 19.3 / 17.3 \\
**gpt-4** & 93.7 / 90.2 / 89.1 & 82.4 / 77.7 / 74.1 & 88.1 / 76.8 / 72.7 & 66.8 / 55.9 / 45.8 \\ \hline \hline \end{tabular}
\end{table}
Table 50: 1-shot **Random / Language-agnostic / Language-specific** negative sample selection results for **Turkish** in **English** template for all examined models across tasks.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD \\ \hline
**aya-23-8b** & 77.4 / 67.5 / 66.6 & 58.1 / 51.5 / 50.6 & 51.0 / 36.0 / 33.5 & 25.5 / 18.4 / 17.3 \\
**aya-23-35b** & 89.6 / 81.8 / 80.5 & 80.7 / 72.1 / 70.9 & 75.8 / 55.8 / 51.8 & 60.5 / 41.8 / 38.8 \\
**quen-2.5-7b** & 83.1 / 74.6 / 72.1 & 76.0 / 67.0 / 65.0 & 64.3 / 45.1 / 39.7 & 52.9 / 33.1 / 30.3 \\
**qwen-2.5-32b** & 90.1 / 85.9 / 83.9 & 81.3 / 75.3 / 73.6 & 80.3 / 66.8 / 61.8 & 64.7 / 48.3 / 45.8 \\
**gemini-1.5-flash** & 87.8 / 85.4 / 85.3 & 61.6 / 62.1 / 59.4 & 78.3 / 70.7 / 76.8 & 34.5 / 33.3 / 29.1 \\
**gpt-4** & 95.4 / 91.6 / 89.4 & 83.7 / 78.8 / 72.1 & 89.2 / 76.6 / 70.8 & 64.7 / 51.4 / 38.7 \\ \hline \hline \end{tabular}
\end{table}
Table 52: 5-shot **Random / Language-agnostic / Language-specific** negative sample selection results for **Turkish** in **English** template for all examined models across tasks.





\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**gpt-4 (top_p=0)*** & 44.0 & 34.0 & 85.0 & 75.0 & 66.0 & 51.0 \\
**gpt-4 (temp=0.3)** & 45.0 & 36.0 & 85.0 & 74.0 & 64.0 & 48.0 \\
**gpt-4 (temp=0.5)** & 45.0 & 34.0 & 85.0 & 73.0 & 64.0 & 45.0 \\
**gpt-4 (temp=0.7)** & 44.0 & 36.0 & 86.0 & 73.0 & 65.0 & 46.0 \\
**gpt-4 (temp=0.9)** & 44.0 & 33.0 & 84.0 & 71.0 & 63.0 & 42.0 \\ \hline \hline \end{tabular}
\end{table}
Table 55: 5-shot results for **Finnish** in **English** template for GPT-4 across tasks and different temperature values. *Corresponds to default decoding setting for main results.

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**gpt-4 (top_p=0)*** & 54.0 & 44.0 & 92.0 & 79.0 & 77.0 & 51.0 \\
**gpt-4 (tep=0.3)** & 53.0 & 43.0 & 92.0 & 79.0 & 80.0 & 53.0 \\
**gpt-4 (temp=0.5)** & 55.0 & 43.0 & 92.0 & 80.0 & 80.0 & 53.0 \\
**gpt-4 (temp=0.7)** & 53.0 & 43.0 & 92.0 & 80.0 & 80.0 & 53.0 \\
**gpt-4 (temp=0.9)** & 53.0 & 42.0 & 92.0 & 79.0 & 80.0 & 51.0 \\ \hline \hline \end{tabular}
\end{table}
Table 53: 5-shot results for **Turkish** in **English** template for GPT-4 across tasks and different temperature values. *Corresponds to default decoding setting for main results.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**gpt-4 (top_p=0)*** & 44.0 & 34.0 & 85.0 & 75.0 & 66.0 & 51.0 \\
**gpt-4 (temp=0.3)** & 45.0 & 34.0 & 85.0 & 73.0 & 64.0 & 45.0 \\
**gpt-4 (temp=0.7)** & 44.0 & 36.0 & 86.0 & 73.0 & 65.0 & 46.0 \\
**gpt-4 (temp=0.9)** & 44.0 & 33.0 & 84.0 & 71.0 & 63.0 & 42.0 \\ \hline \hline \end{tabular}
\end{table}
Table 55: 5-shot results for **Finnish** in **English** template for GPT-4 across tasks and different top_p values. *Corresponds to default decoding setting for main results.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**gpt-4 (top_p=0)*** & 44.0 & 34.0 & 85.0 & 75.0 & 66.0 & 51.0 \\
**gpt-4 (temp=0.3)** & 45.0 & 34.0 & 85.0 & 73.0 & 64.0 & 45.0 \\
**gpt-4 (temp=0.7)** & 44.0 & 36.0 & 86.0 & 73.0 & 65.0 & 46.0 \\
**gpt-4 (temp=0.9)** & 44.0 & 33.0 & 84.0 & 71.0 & 63.0 & 42.0 \\ \hline \hline \end{tabular}
\end{table}
Table 55: 5-shot results for **Finnish** in **English** template for GPT-4 across tasks and different temperature values. *Corresponds to default decoding setting for main results.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Models** & \multicolumn{2}{c}{**Morph. Productivity (accuracy)**} & \multicolumn{2}{c}{**Morph. Systematicity (macro-F1)**} & \multicolumn{2}{c}{**Morph. Systematicity (coherence)**} \\  & ID & OOD & ID & OOD & ID & OOD \\ \hline
**gpt-4 (top_p=0)*** & 44.0 & 34.0 & 85.0 & 75.0 & 66.0 & 51.0 \\
**gpt-4 (top_p=0.95)** & 43.0 & 34.0 & 86.0 & 73.0 & 65.0 & 46.0 \\
**gpt-4 (top_p=0.9)** & 43.0 & 34.0 & 85.0 & 79.0 & 64.0 & 44.0 \\ \hline \hline \end{tabular}
\end{table}
Table 56: 5-shot results for **Finnish** in **English** template for GPT-4 across tasks and different top_p values. *Corresponds to default decoding setting for main results.