Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model
Authors: Leonie Weissweiler
Valentin Hofmann
Anjali Kantharuban
Anna Cai
Ritam Dutt
Amey Hengle
Anubha Kabra
Atharva Kulkarni
Abhishek Vijayakumar
Haofei Yu
Hinrich Schuetze
Kemal Oflazer
David R Mortensen
Venue: EMNLP 2023 Main (2023)
Keywords: wug-testing
morphology
chatGPT
morphological generalization

Abstract: Large language models (LLMs) have recently reached an impressive level of linguistic capability, prompting comparisons with human language skills. However, there have been relatively few systematic inquiries into the linguistic capabilities of the latest generation of LLMs, and those studies that do exist (i) ignore the remarkable ability of humans to generalize, (ii) focus only on English, and (iii) investigate syntax or semantics and overlook other capabilities that lie at the heart of human language, like morphology. Here, we close these gaps by conducting the first rigorous analysis of the morphological capabilities of ChatGPT in four typologically varied languages (specifically, English, German, Tamil, and Turkish). We apply a version of Berko's (1958) wug test to ChatGPT, using novel, uncontaminated datasets for the four examined languages. We find that ChatGPT massively underperforms purpose-built systems, particularly in English. Overall, our results---through the lens of morphology---cast a new light on the linguistic capabilities of ChatGPT, suggesting that claims of human-like language skills are premature and misleading.

Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model
Leonie Weissweiler 2,4 Valentin Hofmann" *2-5 Anjali Kantharuban', Anna Cai'l, Ritam Dutt t1,
Amey Hengleio, Anubha Kabra*1, Atharva Kulkarmi") Abhishek Vijayakumar") Haofei Yut!, Hinrich Schütze?*, Kemal Oflazer', David R. Mortensen'
CCarnegie Mellon University LLM Munich SUniversity of Oxford
MMnich Center for Machine Learning 'Allen Institute for AI 'IIT Delhi weissweiler@cis 1mu. de
Abstract
Large language models (LLMs) have recently reached an impressive level of linguistic capa- bility, prompting comparisons with human lan- guage skills. However, there have been rel- atively few systematic inquiries into the lin- guistic capabilities of the latest generation of LLMs, and those studies that do exist (i) ig- nore the remarkable ability of humans gen- eralize, (ii) focus only on English, and (iii) investigate syntax or semantics and overlook other capabilities that lie at the heart of hu- man language, like morphology. Here, we close these gaps by conducting the first rigor- analysis of the morphological capabilities of ChatGPT in four typologically varied lan- guages (specifically, English, German, Tamil, and Turkish). We apply version of Berko's (1958) wug test to ChatGPT, using novel, un- contaminated datasets for the four examined languages. We find that ChatGPT massively underperforms purpose-built systems, particu- larly in English. Overall, our results thsults- the lens of morphology cast a new light on the linguistic capabilities of ChatGPT, suggest- ing that claims of human-like language skills are premature and misleading.
LLM
zuluyemedii
gel + gelemedik ye ylyemedik
Evaluation
zuiu
Humans
zuluyamadik
Figure I: Experimental paradigm for this study (illus- trated with Turkish). Human annotators and an LLM are given examples and nonce word to be inflected. The generated inflected forms are compared.
clauses, and identifying the grammatical number of subjects and objects (Jin al., 2022).
Despite these encouraging results, the existing body of work has so far examined a relatively lim- ited part of the full spectrum of phenomena that are known characterize human language, with a heavy focus on syntax and semantics. One area that has been neglected in particular is morphol- ogy, i.c., the capacity to create words according to systematic patterns of covariation in form and meaning (Haspelmath and Sims, 2010). This gap in the LLM literature is noteworthy given that mor- phology has been a hallmark of research on com- putational approaches to language since the very beginnings of neural language processing in the 1980s (Rumelhart and McClelland, 1986b; Plun- kett and Juola, 1999; Albright and Hayes, 2002, 2003; Goldberg. 2019).
1 Introduction
Do large language models (LLMs) possess human- like linguistic capabilitios? With the advent of the latest generation of LLMs such as GPT-4 (OpenAl, 2023b), LLaMA (Touvron et al., 2023), and PaL.M (Chowdhery et al., 2022), there appears to be grow- ing evidence for answering this question with yes (Bubeck al., 2023): LLMs capable of gen- crating text that crowdworkers cannot distinguish from human-generated text (Clark et al., 2021) and excel at linguistic probing tasks such as predicting grammaticality, detecting the subject and tense of
"qqual contribution.
In this study, we present the first systematic anal- ysis of the morphological capabilities of LLMs, fo-
AAtthrrs sorted alphabetically.

---

cusing on ChatGPT (OpenAI, 2023a) the most prominent and most widely-used LLM. Specifi- cally, we investigate ChatGPT's morphological ca- pabilities using the wug test (Berko, 1958), an experimental paradigm in which a participant is asked to provide an inflected or derived form of a nonce word. An example for our evalua- tion setup is given in Figure 1. Our experiments cover broad range of morphological construc- tions and four typologically diverse languages: En- glish, German, Tamil, and Turkish. We find that ChatGPT falls short not only of human perfor- mance but also of various supervised baselines. In sum, our contributions are as follows:
(with soft attention hard monotonic attention) various transformer models. A special subtype of this task is morphological reinflection, where the input can be a form that is itself inflected (Cot- terell et al., 201 Kann and Schütze, 2016; Kann et al., 2017; Silfverberg et al., 2017; Pimentel et al., 2021). Other typical tasks in computational re- search on inflection are morphological segmenta- tion (Cotterell et al., 2015, 201 6b,c; Kann et al., 20, unsupervised morphology induction (Ham- marström and Borin, 2011; Soricut and Och, 2015; Xu et al., 2018; Weissweiler et al., 2022), and mor- phological paradigm completion (Erdmann et al., b020a,b; Jin et al., 2020). There has also been some interest in the modeling of derivation (Cot- terell et al., 2017b; Vylomova et al., 2017; Deutsch et al., 2018; Hofmann et al., 2020b,c).
We conduct the first systematic analysis into the morphological capabilities of LLMs.
More recently, there have been few studies examining the morphological capabilities of lan- guage models (Edmiston, 2020; Hofmann et al., 2020a), but they focus on smaller language models such as BERT (Devlin et al., 2019). By contrast, we examine ChatGPT, a model whose parameter count is three orders of magnitude larger, and we analyze its zero, one-, and few-shot capabilities, an approach fully neglected by prior work.
Our study covers a diverse set of morphological sesestuuttions/langgagge and introduces datasets for future research in the area.'
We show that ChatGPT has not achieved human parity-or even state-of-the-art performance- nonce-word inflection/rrinfleettoon tasks but performs about well some older super- vised models. We furthermore find evidence for the existence of real word bias in ChatGPT that is the more pronounced the more data ChatGPT has seen for a given language.
2.2 Multilingual Capabilities of LLMs
Recent studies have extensively examined the eval- uation of LLMs in multilingual settings. Some of these studies have specifically investigated the ex- tent to which LLMs can be used for traditional multilingual NLP tasks such as machine transla- tion (Bawden et al., 2022: Hendy et al., 2023; Jiao et al., 2023; Wang et al., 2023). Brown et al. (2023) demonstrate that LLMs perform well across mul- tiple languages even with minimal task-specific training, highlighting their transferability and gen- eralization multilingual understanding.
2 Related Work
2.1 Computational Morphology
Linguists divide morphology into inflection and derivation (Haspelmath and Sims, 2010). While inflection accounts for the differrent word forms of a lexeme, e.g., listen, listens, and listened, deriva- tion accounts for the different lexemes of a word family, e.g., listen, listener, and listenable. Both inflection and derivation have been addressed in computational linguistics and natural language pro- cessing (NLP), albeit with heavy focus on in- flection. One line of work, which is conceptu- ally similar to wug testing, has sought to gener- ate inflected forms, given a stem and a morpho- logical tag (Cotterell et al., 2017a, 2018; Vylo- mova et al., 2020; Goldman et al., 2022), using systems ranging from weighted finite state trans- ducers and GRU/LSTM encoder-decoder models
2.3 LLM Performance Unseen Data
The fact that LLMs have been pretrained on mas- sive amounts of data means that they seen seen and potentially memorized a substantial amount of the items of data used in typical evaluation setups (Ma- gar and Schwartz, 2022). There have been few attempts in NLP to specifically control for previ- ous exposure (Haley, 2020; Hofmann et al., 2020a; Maudslay and Cotterell, 2021). We follow this idea by generating datasets of novel and uncontam- inated nonce words, thus ensuring that the words have not been seen by ChatGPT before.
'We relcase our dataset along with our code at https:// gi thub. com/dmortae 1imort274 tetortt7/cchaaaa chh/gtts wugs, carefully following the guidelines laid out by Jacovi et al. (2023).

---

3 Data and Morphological Constructions
Lang.
Train
Dev
Test
Wug test
English
10,000
1,000
1,000
50
In this paper, we examine ChatGPT's morpho- logical behavior a typologically diverse set of languages: Lnglish, German, Tamil, and Turkish. While English and German belong to the same lan- guage family, German has more fusional mor- phological system than English. Turkish is cho- sen since it is non-Indo-Luropean language with a fully agglutinative morphology. Tamil is cho- sen since it is a Dravidian language exhibiting an agglutinative morphology with fusional elements. Thus, in terms of the classical triangle of fusional, isolating, and agglutinative morphologies (Dixon, 1994), the languages cover four different points: almost fully isolating (Englishh, intermediate be- tween isolating and fusional (German), interme- diate between fusional and agglutinative (Tamil), and fully agglutinative (Turkish). Furthermore, the chosen languages also cover different points in the spectrum from low-resource to high-resource, en- abling us form hypotheses about the impact of the amount of language-specifer speciffc training data on the morphological capabilities of an LLM. Statis- tics for the amount of data in train, dev, and test for the baselines, as well as the number of wug test words, are given in Table 1. We report the accuracy of one annotator at time against the judgments of all other annotators in Table 2.
German
10,000
1,000
1,000
174
Tamil 1,541
368
123
Turkish
8,579
851
846
40
Table Data statistics. Please see Appendix A.1 for the distribution of morphological tags across the differ- ent splits for the four languages. There was not enough data available for Tamil to form test set.
Accuracy (%)
Lang-
@
@
English
67.14 + 17.76
85.29 + 13.06
87.64 + 12.13
German
63.05 + 12.62
83.80 + 10,57
87.88
+ 10.34
Tamil 37.09 + 26.39
43.85 + 26.95
43.85 + 26.95
Table 2: Accuracy of one annotator at time against the judgments of the other annotators on our collected wug dataset, for different values of For Turkish, since the morphology is deterministic, there is no variation.
(nonce_word word) all the time. In fact, they just yesterday." This yielded mappings between a lemma and a ranked list of inflected verbs, e.g., veed + [veeded, ved, vode]. The modal annotation was always a regularly inflected form (-ed with ap- propriate allomorphic variation), but other inflec- tional classes were attested.
3.2 German
3.1 English
The German plural of nouns is morphological phenomenon intensely studied linguistics and the cognitive sciences due to the general complex- ity of the alternation between the eight different op- erations that can be used to express it. German plu- ralization is particularly notable due to the fact that none of the possible operations express it in ma- jority of cases (McCurdy et al., 2020). In fact, the most frequent German plural noun suffix -en has been argued not to be the default (i.e., the suffix that applies to novel nouns) an honor that goes to - (Marcus et al., 1995).
The English past tense has a long and storied history in computational studies of morphology (Rumelhart and McClelland, 1986a; Pinker and Prince, 1988; Ullman et al., 1997; Plunkett and Juola, 1999; Albright and Hayes, 2002, 2003; Kirov and Cotterell, 2018; Ma and Gao, 2022). En- glish displays a handful of conjugation classes as well as frequent morphographemic alternations consonant doubling and c-deletion, for example- affecting past forms of verbs.
To create the English data, 50 two- to five-letter irregular verbs (defined as verbs that do not form the past tense simply by adding -ed) were sam- pled from the UniMorph 4.0 dataset (Batsuren et al., 2022). These items were each perturbed by two letters (substituting phonetically simi- lar sounds) producing word not included in Uni- Morph. These verbs were then annotated by 28 volunteer annotators. Participants were asked to provide the past tense of the nonce word and given an example (wug- + wugged) and the frame "They
To create the dataset of novel German nonce nouns, we drew upon Unipseudo. We generated 200 nonce words with length between four and seven characters (50 nonce words per character length), using German nouns as input to the algo- rithm. We then had one German native speaker un- related to the study (i) generate articles (der, die, or das) for cach of the nonce words, and (ii) generate a plural based on the nonce words and the previ-
Phttp://wwe lexique, waeeee.rg/sseee wrt/shiig/gu/nnm httpiiin//niisuuoo//

---

ously selected articles. We manually filtered out words whose plural blocked by existing German lexemes, resulting in final set of 174 nonce nouns. These nouns were then annotated by 21 volunteer annotators. Participants were asked to provide the plural of the nonce word and were given an exam- ple (Wig + Wugs) and the frame "Hier ist ein/e (nonce_ shonce_word].... Jetzt sind es zwei "" Simi- larly to English, this yielded mappings between lemma and ranked list of inflected nouns.
Features
Example
First person singular agreement and past tense
zöbür-ür-üm - zöbür-dü-mm
Second person plural agreement,
zöbür-ür-sünüz -> zöbür-me-mij-sine
reported/in fereential past tense, and negative polarity
Dative case, first person possessive
zürp-ten zürb-lim-c
Accusative singular
börüt
börüdd-üi
3.3 Tamil
Table 3: Turkish tasks. Forms with colored suffixes are actually used in the long prompt a contextually mean- ingful short sentence. Hyphens represent morpheme boundaries. The last row is for simple inflection. The predicted forms (to be predicted, the right) have the following morphosemantics: "I [verb]-ed", "(I heard that) you have not [verb]-ed", "to my [noun]", "the [noun] (as definite object)".
Tamil is Dravidian language primarily spoken in regions of South India and Sri Lanka. It is an agglutinative languange in which verbs are conju- gated for tense, transitivity, person, number, and (in some cases) gender. For the most part, af- fixes display allomorphy only due to phonologi- cal conditioning and are otherwise invariant across verbs, as is the case with the person person/numberrgender (PNG) affix (Arden, 1891, 71). This is not the case, however, for tense markers. Among linguists working on Tamil, it is not completely agreed upon how many verb classes there are the language, with some proposing up to and others as few as three (Lisker, 1951; Agesthialingom, 1971). In the spoken form of Tamil, there are points where verbs are part of completely different classes than their literary counterpart, so in this study we focus ex- clusively the written form (Schiffiman and Ren- ganathan, 2009).
prompts, 86 common Tamil verbs were sampled and conjugated with every possible combination of tense and PNG suffixes. These conjugations were generated automatically and then validated by two native speakers for accuracy. Unlike in the nonce word case, there 100% agreement between speakers. The nonce words were gener- ated by combining syllables from real verb roots and checking against Tamil dictionary to assure the words created were not real. Nonce verbs were created to be between two and six letters long to best match the distribution of real Tamil verbs. In order to get the "correct" past tense for these verbs, five native Tamil speakers were asked to pro- vide past tense forms (e.g., G nitu Gág nitugaan,, |LL nitamn فف ninaaa))).. The mode of these responses was taken be the gold form, with the level of agreement amongst speakers recorded for later analysis. The compar- atively lower inter-annotator agreement can be ex- plained by the lack of historical and linguistic con- text given to the annotators, since large part of classification is historical.
To simplify the analysis, we utilize a modifi- cation of Graul's classification seen in The En- glish Dictionary of the Tamil Verb, where there are seven primary classes (Schiffman and Ren- ganathan, 2009). The tense most impacted by these verb classes is the past tense, with each class having unique form, while the present and future only demonstrate three forms across the classes. As such, we focus on the past tense and desig- nate the same transitivity (intransitive) and PNG (third person singular masculine) affix across all experiments. examining this, we gain infor- mation about the ways LLMs handle morpholog- ically complex languages with inflectional classes defined in both phonological and morphological terms. This contrasts with English, where inflec- tion is not agglutinative, and Turkish, where mor- phology is agglutinative but where there are no in- flectional classes.
3.4 Turkish
Turkish is an agglutinative language where words consist of multiple morphemes attached root. Surface realizations of morphemes are influenced by deterministic morphophonological processes like vowel harmony, consonant assimilation, and clision. Unlike many other languages, Turkish has complex word form morphotactics, particu-
To create a dataset for training the baseline models and generating samples for the few-shot

---

larly when multiple derivations are present.
lyzer/generator (Oflazer, 1994), because the afore- mentioned SIGMORPHON 2023 datasct did not have sufficient number of examples for most of the feature combinations. The morphological gen- erator was set up to generate only Turkish word forms that corresponded to the selected inflectional morpheme combinations we selected, for all ap- plicable roots. For testing, we expected the base- line systems generate the word forms with the selected inflectional feature combinations, for 10 nonce roots. The nonce roots were chosen so that they would force the inflected forms to orthog- onally adhere to surface morphographemic con- straints and rules such as various types of vowel harmony, consonant elision, or assimilation at mor- pheme boundaries.
To simplify the task and reduce the number of feature combinations, we utilized four datasets with different levels of complexity and a limited number of inflectional features. In most cases, the context provides an inflected form with one set of features, and the model must predict the form with the requested set of features. The first three tasks are reinflection tasks, demanding proficiency in both morphotacties and morphographemics. The fourth task is a straightforward inflection task (see Table 3). Each task consists of up to shot exam- ples for real roots and 10 test examples with nonee roots. Stimuli and gold annotations were produced by our (single) Turkish annotator.
4 Methodology
Similarly, for Tamil, we split the data into train and dev sets. Since we have a limited amount of Tamil data, we kept the split ratio at around 4:1 between train and dev sets.
We compare the outputs of ChatGPT under a vari- ety of prompting regimens and a substantial set of supervised baselines (both neural and non-neural) to human annotations of the data described in Ap- pendix 3. Results are evaluated using accuracy at k (aco@k), i.e., model's response is regarded correct if it is in line with any of the top lt human responses. This evaluation method takes into account inter-speaker morphological variabil- ity, which more wide-spread than previously thought (Dammel and Schallert, 2019).
We report the results of all baselines in Table 4. Baselines generally perform as expected, validat- ing our usage of them. It should be noted that Min- Gen and AED are evaluated in IPA/feature space and may therefore be at a disadvantage compared to basclines operating directly in orthography. The training data was converted from orthography into IPA using Epirran (Mortensen et al., 2018).
4.1.2 Affix Rule Learner (ARL)
4.1 Baselines
As baseline for the 2020 and 2021 SIGMOR- PHON shared tasks, a simple non-neural system (Liu and Mao, 2016) was implemented that uses edit distance to "discover prefix and suffix rules in training data."* 14 At test time, the system modifies a lemma by applying the longest matching suffix rule and most frequently applied prefix rule for a given morphosyntactic description.
We investigate the efficacy of several baselines for the task of morphological inflection. The cho- sen baselines encompass both statistical and neu- ral architectures that have shown impressive per- formance on the morphological generalization task in recent years. We evaluate their performance on the SIGMORPHON 2023 task as well as constructed wug test set. The baselines have com- plementary strengths (see Section 5).
4.1.3 Minimal Generalization Learner (MinGen)
4.1.1 Training Data
Wilson and Li (2021) proposed a minimal gener- alization model based on a simplified form of AI- bright and Hayes (2002) to learn morphological rules. First, base rules that describe the changes needed to convert lemma to an inflected form are generated from training data. The rules are further generalized by comparing phonological features of the rule contexts. The rules are then scored by a confidence metric based on their accuracy and
We used the train/dev/test splits of the SIGMOR- PHON 2023 Inflection Shared Task3 for English and German. The choice of the train/dev/test splits was motivated by the fact that there was no over- lap of lemmata between the individual splits, thus mimicking a wukelike setting.
The Turkish training data for baselines was gen- erated directly using Turkish morphological ana-
//ttts com/s gmorphon/ 202tinT
/ thub. com/si 1aarppon//221/asss00 main/baselines

---

English
German
Turkish
Tamil
Model
Dev
Test
Dev
Test
Dev
Test
Dev
ARL
95.40
96.60
77.40
79.80
94.36
93.50
85.60
MinGen
81,40
78.70
72.70
70.70
93.65
93.03
87.23
FIT
96.22 0.19
94.93 + 0.49
79.01
1.16
81.04
1.39
97.00
0.22
96.25
0.26
64.24 +: 3.11
94.74 1 0.90
73.57
5.37
78.26
4.66
96.61 = 0.60
96.56
10.66
76.76.1 2.10
AED
71.06
70.16 + 5.79
64.44 1.85
67.44
2.02
95.54 0.77
95.19 11.4
50.70:t 2.84
Table 4: Results (acc@k) of the baselines on our development and test data. See Section 4.1.1 for full details.
scope. At test time, the rule with the highest score among the applicable rules is used.
4.2 Prompting
We employ three distinct prompting styles, namely zero-, one-, and few-shot, to interact with the lan- guage model. We start with a simple instruction each language, for example:
4.1.4 Feature Invariant Transformer (FIT)
Wu et al. (2021) proposed simple technique em- ploying a character-level transformer for feature- guided transduction that was used as baseline for the 2021 SIGMORPHON shared task. This is a generative model capable of performing character- level decoding to generate target inflections. In comparison to vanilla transformer model, posi- tional counts are used only for characters and not for features. The model also incorporates unique tokens to mark whether a given token is feature.
"Fill in the blank with the correct past tense of the word 'wug'. Give your re- sponse in one word. They wug all the time. In fact, they just yesterday."
For Tamil, the instruction portion of the prompt is omitted because of ChatGPT's unreliable perfor- mance when given instructions in that language. We select one example with jor inflection class of the phenomenon in question. We then perform multiple runs: 10 for the zero- shot scenario, one for every shot for the one-shot scenario, and 10 for the few-shot scenario, with new random permutation of all examples each time. We query first word of the response, and filter by removing non-word characters. We evaluate by computing the accuracy for each of the runs, averaged over all queried nonce words, and compute the mean and standard deviation across all runs. We employ acc@k as our evaluation metric, setting & for our main evaluation. We provide results for k and k = 3 in Appendix A.4. The k gold forms are the responses most frequently generated by hu- mans. Since only one Turkish response is possible (the morphology is deterministic). À is always for this language. We then perform an additional ex- periment for comparison in which we remove the context around the nonce word and only give the instructions well as the last line. We call this the short prompt and the original described above the long prompt. We provide instances of long and short prompt in Appendix A.5.
4.1.5 Principle Parts for Inflection (PPI)
We apply the approach of Liu and Hulden (2020), which recasts the task of morphological inflec- tion as "paradigm cell filling problem." This leverages a lexeme's principal parts-the the mini- mum subset of paradigm slots needed to gener- ate the other slots in its paradigm. Specifically, for low-resource scenarios, the principal parts of paradigm identify additional slots that crucial in generating the target-inflected lemma.
with real words for each ma-
-6113, select the
4.1.6 Analogical Encoder-Decoder (AED)
Following up on Albright and Hayes (2003) and Kirov and Cotterell (2018), Calderone et al. (2021) proposed a recurrent neural network encoder-decoder architecture augmented with pre- compiled analogical patterns for generating mor- phological inflections of nonce words. This model leverages the UniMorph Tags and fine alternation pattern (FAP) associated with each lemma in rela- tion its inflection form. FAPs analyze the posi- tioning of word forms within the system iden- tify recurrent patterns representing conventional linguistic elements.
'tttps://giie htw..//gttttubcoo wwt.uub..oomssiiiiigg http://sgmorphoo//// 220r Task0/ tree/ mainbaselines

---

5 Results
5.1 Overall Performance
For aco(s5 the performance of ChatGPT never ex- ceeded that of the strongest baselines (ARL, AED, and PPT) regardless of the prompting regime, as shown in Table 5. However, it beats certain older baselines such as MinGen (the minimum general- ization learner). ChatGPT performed best when it explicitly prompted to complete analogy with single example (i.e., short 1-shot), can be seen in Figure 2. We observe that similar trends hold for acc@@1 and acc@) (see Appendix A.4), but the gap between the strongest baselines and ChatGPT decreases with k.
Accumryy
Devoe
Demow
king
Method
Figure 2: Results for the different prompt scenarios, for- mats, languages, and values of k.
against the judgments of the other annotators, the few-shot accuracy is quite reasonable.
English ChatGPT's perfommacee on English was uniformly worse than both the average annota- tor (87.64%) and the strongest baselines. acc@@ falls below 60% in the O-shot condition but is markedly better when shots are supplied. Short prompts, which require the model to complete simple analogy, resulted in better performance than long prompts. all conditions, authentic Ln- glish words that did not occur in the reference anno- tations appeared as outputs when the nonce word and the authentic word were orthographically sim- ilar (see the discussion in Section 6.4).
Turkish The prompting performance for the Turkish inflection task worse than for English and German, especially in the long prompt case. For this task, the morphotactics is trivial but the selection of the allomorph depends on stem vow- els, stem-final consonants, whether there is a con- sonant cluster ending the stem, and whether the stem is monosyllabie or not. ChatGPT gets better results with the short prompt through an analogi- cal example. For the three reinflection tasks, Chat- GPT gets mixed results that are overall worse than for the inflection task (see Table 6).
German The best German result was 88.94% (short I-shot), which beat all of the baselines ex- cept for ARL and FIT. The other results are simi- larly strong in contrast to the other languages. The impact of & is not noticeable here. This, in com- bination with the fact that the human performance acc@s was 88%, indicates that the task is per- feetly performed by ChatGPT. has reached the upper bound given by the inherent subjectivity of the task (reflected the human variability) and the impact of & is, therefore, not measurable. This is further solidified by the very small impact of the long vs. short prompts.
Analysis
6.1 The Nature of the Task
The inherent complexity of the inflection tasks for the various languages (and the reinflection task for Turkish) varies greatly. English and Turkish are the simplest: the top-ranked form can always be obtained by adding a single suffix and applying a few morphographemic alternations. German anno- tations show no dominant pattern and assign nonce words morphological classes according com- plex criteria. However, German performance is clearly better, suggesting that factors other than in- herent complexity play role in ChatGPT's ability to generalize morphological patterns.
Tamil Tamil performance of ChatGPT was sig- nificantly worse than the provided baselines, even in the few-shot conditions. For the few-shot case, there was marginally better performance when us- ing short prompts, but this did not apply to the 0- I-shot case (in which no accurate outputs were generated). Across the board, the performance on Tamil was markedly worse than performance on English and German. However, considering that the average annotator had only 43.85% accuracy
6.2 Impact Tokenization
There is mounting evidence that the morpho- logically suboptimal nature of many tokenizers may limit the morphological capabilities of L.LMS (Bostrom and Durrett, 2020; Hofimann al., 2021). ChatGPT's tokenization, i.e., byte-pair encoding

---

Method
English
German
Tamil
Turkish
ARL
100.00
94.25
61,48
60.00
MinGien
62.00
64.37
49.18
40.00
FIT
98,00 1.26
92.87 = 0.74
63.28 4: 3.36
67. 67.00 + 4.58
PPI
94.60 2.54
85,98 +: 5.91
55.33 4: 1.84
68.00 4.00
AED
57.60 6.62
48.51 +: 5,45
58.69 5.46
56.00 4.90
long O-shot
58,40 5,28
86.49 = 1.07
0,00
28.00 = 14,00
long I-shot
73.60 6.97
85.42 a
14.52 7,48
20.00 1: 14.14
long few-shot
76,40 1:400=:: 4,45
47.3622.37
42.70 3.96
54.00 = 10.20
short 0-shot 75,40 1: 5.87
88.62 1: 1.64
0,00
3.00 L: 4.58
short 1-shot 82,80 5.60
88.94 + 2.35
3,28 + 3.99
58.00 7,48
short few-shot 78.60 2.84
88.33 + 1.15
43.36
59.00
Table S: Results (acol(k) for all languages except for Turkish where k cf. Section 4.2).
Type
0-shot
1-shot
few-shot
strict linguistic sense (see Section 2.1). Crucially, the stem of the generated form is always a real word (i.e., a word that exists in the respective lan- guage). Examples of this phenomenon include, for English: did the past tense of dedo, blushed the past tense of blus, fried the past tense of fride; and for German: Ozeane ('oceans") as the plural of Ozeak, Institute ('institutes") as the plu- ral of Instite, Sklaven ('slaves") as the plural of Schlave. is important to notice that in all these cases, (i) the generated form has the correct mor- phological properties e.g., the English forms did, the
long 3.00 + 1.80
20.67
33.33 4.94
short
7.00 + 4.33
18.67 + 6.18
31.00 + 4.23
Table 6: Results for Turkish averaged over the three reinflection tasks (k 1).
(Sennrich et al., 2016), has been shown to be par- ticularly problematic (Bostrom and Durrett, 2020; Hofimann al., 2022).
To examine the impact of tokenization, we mea- sured the number of tokens into which the nonce words are split for the individual languages and computed the accuracy ber of tokens. Our hypothesis was that longer to- ken sequenees are less optimal, potentially leading to worse performance. However, using two-sided ttessss, we did not find a significant difference be- tween nonce words with different token lengths. We interpret this as indicating that tokenization plays less pronounced role for ChatGPT.
blushed, fried are indced past tense stem is real word rather than the nonce word, and (ii) the stem that is generated in lieu of the nonce word a frequently occurring word in the respec- tive language and has certain (sometimes strong) orthographic similarity to the nonce word. We de- note this tendency real word bias.
function of the num-
The concept of real word bias allows us to make a hypothesis about the way in which ChatGPT ad- dresses morphological tasks. We think ChatGPT is not applying morphological rules to stem, which would be in line with item-and-process accounts of morphology (Hockett, 1954). Rather, it seems to linguistically decode the point in its representa- tional space defined by the semantic constraints in the prompt. In cases where this point (and its im- mediate neighborhood) is unoccupied, it generates a form based on the nonce word, but in cases where there is form of a real word close to the point (e.g., because of superficial orthographic similar- ity), it generates this form instcad. The fact that the real word bias is strongest for German and English (the two high-resource languages) suggests that the representational space is more dense for these two languages, increasing the probability that there a real word close to the point that the model is trying
6.3 Impact of k
We observe that the gap between the baselines and our results increases with & (see Table Ap- pendix A.4), suggesting that ChatGPT tends to generate either a top-ranked form or an implausi- ble inflection while the baselines tend to produce plausible inflections which are less frequent in the human annotations. ChatGPT's penchant for im- plausible inflections may be a result it real word bias (see Section 6.4 below).
6.4 Real Word Bias
Linlishh and German-and Turkish nurkish-many of the forms generated by Chat- GPT belong to a different lexeme than the nonce word and thus do not constitute inflections any
lesser extent in

---

as result of pretraining; (ii) it does not identically mirror the behavior of humans, but rather ampli- fies the productivity of certain morphemes. The finding that the most productive morphemes (for humans) are becoming more productive for Chat- GPT while the least productive morphemes (for hu- mans) are becoming less productive for ChatGPT bears some theoretical resemblance to discussions about bias amplification (Ahn al., 2022).
No change
317
10
0
0
0
47
262
77
79
0
13
5
22
19
3
20
64
18
4
0
0
11
25
Ro.chenge
wow change
viwe crange or
Bewe Word
Vper stenng
Future Directions
Prokical
Morphological patterns are only one kind of gen- cralization that can be investigated through a wug- like experimental paradigm. The form-meaning re- lationships encoded in language and multimodal models, including constructional and iconic pair- ings, can be investigated through prompting with nonce stimuli, leading to new insights regarding the generalizations they capture.
Figure 3: Confusion matrix for competing German plu- morphemes for the few-shot setting.
to decode based on the prompt.
6.5 Morphological Productivity
The productivity of morpheme is traditionally de- fined as its propensity to be used in novel combi- nations (Plag, 1999; Bauer, 2001; Haspelmath and Sims, 2010). Crucially, morphemes with the same meaning can differ in their productivity-fot for ex- ample, for English deadjectival nominalizing suf- fixes, -ness (c.g., robustness) generally more pro- ductive than -ity (e.g, equality), which in turn is more productive than the fully non-productive -th (c.g., warmth). We are interested to see whether there is any difference in the productivity of mor- phological patterns exhibited by ChatGPT com- pared to the human sample. We focus on German it has the most complex pattern of competing morphemes, and we examine the few-shot results they show the best performance overall.
Limitations
Our research conducted with single model (gpt-3.5-turbo 5-ttrbbo-06611, is not certain that our results will generalize other versions of GPT-3 or to GPT-4, let alone other LLMs. Although we went to great lengths to develop prompts that would maximize ChatGPT's performance on the tasks, it is not possible to state definitively that another strategy would not produce better perfor- mance. While the languages were typologically varied, it is not clear whether the results observed in the current study are generally robust or co- incidental properties of the small set of languages and datasets under investigation. Furthermore, comparing the languages to one another is prob- lematic because it was not possible to control other variables while varying the language. For example, the English and Tamil tasks involve verbal inflec- tion while the German and Turkish tasks involve nominal inflection. Finally, the number of annota- tors for Tamil was very small and inter-annotator agreement was very low, meaning that the results of the Tamil experiments must be approached with special caution (but see our discussion about mor- phological variation in Section 3).
We start by comparing the distribution over al- ternative plural morphemes generated by ChatGPT with the human responses. As shown in Figure 3, there several morphemes that are used by Chat- GPT similarly humans (c.g., g., the null morpheme). Cases of overgencralization, where ChatGPT sys- tematically generalizes the usage of a particular suffix to contexts where the suffix is not used by humans, are mainly limited to two plural mor- phemes: -en (77 generations for gold morpheme -e) and -s (79 generations for gold morpheme -e). Interestingly, these two plural morphemes most productive plural morphemes in Ger- man (Köpcke, 1988). This indicates two important points: (i) ChatGPT sensitive to the productiv- ity of morphemes, i.c., it has acquired the ability to model how productive certain morphemes are
the
Ethics
LLMs are already impacting the world's people in significant ways, for good and ill. Understanding their limitations, particularly with regard to non- hegemonic language communities, is an ethical im-

---

perative. This study highlights one specific way in which an LLM should not be treated as a sur- rogate human, thus motivating additional research on language modeling for structurally diverse and low-resource languages.
Maudslay, Josef Valvoda, Ran Zmigrod, Paula Czarnowska, Irene Nikkarinen, Aclita Salchak, Brijesh Bhatt, Christopher Straughn, Zoey Liu, Jonathan North Washington, Yuval Pinter, Duygu Ataman, Marcin Wolinski, Totok Suhardijanto, Anna Yablonskaya, Niklas Stoehr, Hossep Dolatian, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Aryaman Arora, Richard J. Hatcher, Ritesh Kumar, Jeremiah Young, Daria Rodionova, Anastasia Yemelina, Taras An- drushko, Igor Marchenko, Polina Mashkovtseva, Alexandra Serova, Fimily Prud"hommnn hommmeux,,, Maria Nepomniashchaya, Fausto Giunchiglia, Eleanor Chodroff, Mans Hulden, Miikka Silfverberg, Arya McCarthy, David Yarowsky, Ryan Cotterell, Reut Tsarfaty, and Ekaterina Vylomova. 2022. UniMorph 4.0: Universal Morphology. In Proceedings of the Thirteenth Language Resources and Evaluation Con- ference, pages 840-855, Marseille, France. Euro- pean Language Resources Association.
Acknowledgemmnts
This work was funded by the European Research Council (#740516). The second author was also supported by the German Academic Scholarship Foundation. We thank the reviewers for their ex- tremely helpful comments.
References
S Agesthialingom. 1971. A note on Tamil verbs. An- thropological Linguisticss pages 121-125.
Jaimeen Ahn, Hwaran Lee, Jinhwa Kim, and Alice Oh. 2022. Why knowledge distillation amplifies gen- der bias and how to mitigate from the perspective of DistilBERT. In Proceedings of the 4th Work- shop on Gender Bias in Natural Language Process- ing (GeBNLP), pages 266-272, Seattle, Washington. Association for Computational Linguistics.
Laurie Bauer. 2001. Morphological productivity. Cam- bridge University Press, Cambridge (UK).
Rachel Bawden, Jonathan Poinhos, Eleni Kogkitsidou, Philippe Gambette, Benoît Sagot, and Simon Gabay. 2022. Automatic normalisation of early Modern French. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3354- 3366, Marseille, France. European Language Re- sources Association.
Adam Albright and Bruce Hayes. 2002. Modeling en- glish past tense intuitions with minimal generaliza- tion. In Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning- - Volume 6, 1PLL '02, page 58- 69, USA. Association for Com- putational Linguistics.
Jean Berko. 1958. The child's learning of English mor- phology. Word, 1402-3)1100-177
Kaj Bostrom and Greg Durrett. 2020. Byte pair encod- ing is suboptimal for language model pretraining. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 4617 4624, Online. Association for Computational Linguisties.
Adam Albright and Bruce Hayes. 2003. Rules
vs. analogy in English past tenses: A computa- tional/experimental study. Cognition, 90(2):119-
161.
Albert Henry Arden. 1891. A progressive grammar of common Tamil. Society for Promoting Christian Knowledge.
Romina Brown, Santiago Paez, Gonzalo Herrera, Luis Chiruzzo, and Aiala Rosá. 2023. Experiments on au- tomatic error detection and correction for uruguayan learners of English. In Proceedings of the 12th Workshop on NLP for Computer Assisted Language Learning, pages 45-52, Torshavn, Faroe Islands. LiU Electronic Press.
Khuyagbaatar Batsuren, Omer Goldman, Salam Khal- ifa, Nizar Habash, Witold Kieraš, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yusti- nus Ghanggo Ate, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel El-Ksaissi, Tiago Pi- mentel, Michael Gasser William Abbott Lane, Mohit Raj. Matt Coler, Jaime Rafael Montoya Samame. Delio Siticonatzi Camaiteri, Esaú Zu- macta Rojas, Didier López Francis, Arturo Once- vay, Juan López Bautista, Gema Celeste Silva Vil- legas, Lucas Torroba Hennigen, Adam I.k, David Guriel, Peter Dirix, Jean-Philippe Bernardy, An- drey Scherbakov, Aziyana Bayyr-ool, Antonios Anastasopoulos, Roberto Zariquiey, Karina Sheifer, Sofya Ganieva, Hilaria Cruz, Ritván Karahóga, Stella Markantonatou, George Pavlidis, Matvey Plu- garyov, Elena Klyachko, Ali Salchi, Candy An- gulo. Jatayu Baxi, Andrew Krizhanovsky, Na- talia Krizhanovskaya, Elizabeth Salesky, Clara Va- nia, Sardana Ivanova, Jennifer White, Rowan Hall
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Fric Horvitz, Eee Kamar, Pe- ter Lec, Yin Tat Lec, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Zhang. 2023. Sparks of Artificial General In- telligence: Early experiments with GPT-4
Basilio Calderone, Nabil Hathout, and Olivier Bonami. 2021. Not quite there yet: Combining analogical patterns and decoder-decoder networks for cogni- tively plausible inflection. In Proceedings of the 18th SIGMORPHION Workshop on Computational Research in Phonetics, Phonology, and Morphology. pages 274 282, Online. Association for Computa- tional Linguistics.

---

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Scbastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Fmiil Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Mor- eira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Ieellstern, Douglas Eek, Dean, Slav Petrov, and Noah Fiedel. 2022. PaIM: Scal- ing Language Modeling with Pathways. 2204.02311.
Ryan Cotterell, Arun Kumar, and Hinrich Schüizze 2016b. Morphological segmentation inside-out. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2325 2330, Austin, Texas. Association for Compu- tational Linguistics.
Ryan Cotterell, Thomas Müller, Alexander F Fraser, and Hinrich Schütze. 2015. Labeled morphological seg- mentation with semi-Markov models. In Proceed- ings the Nineteenth Conference on Computational Natural Language Learning, pages 164-174, Bei- jing, China. Association for Computational Linguis- tics.
Ryan Cotterell, Tim Vieira, and Hinrich Schüitze. 2016c. A joint model of orthography and morphological seg- mentation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 664-669, San Diego, California. As- sociation for Computational Linguistics.
Arxiv,
Ryan Cotterell, Ekaterina Vylomova, Huda Khayral- lah, Christo Kirov, and David Yarowsky. 2017b. Paradigm completion for derivational morphology. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 714-720, Copenhagen, Denmark, Association for Computational Linguistics.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 7282-7296, Online. Association for Computational Linguistics.
Antje Dammel and Oliver Schallert. 2019. Morpholog- ical variation: Theoretical and empirical perspec- tives. John Benjamins, Amsterdam.
Daniel Deutsch, John Hewitt, and Dan Roth, 2018. A distributional and orthographic aggregation model for English derivational morphology. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages 1938 1947, Melbourne, Australia. As- sociation for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Arya D. McCarthy, Katharina Kann, Sabrina J. Mielke, Gar- rett Nicolai, Miikka Silfverberg, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. The CoNLL- SIGMORPHON 2018 shared task: Universal mor- phological reinflection. In Proceedingy of the CONIL-SIGMORPHOO 2018 Shared Task: Univer- sal Morphological Reinflection, pages 1 27, Brus- sels. Association for Computational Linguisties.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Associ- ation for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John Sylsk-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler,, David Yarowsky, Jason Eisner, and Mans Hulden. 2017a. CoNLL- SIGMORPHON 2017 shared task: Universal mor- phological reinflection in 52 languages. In Pro- ceedings the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 1-30, Vancouver. Association for Computational Linguistics.
Robert M. Dixon. 1994, Ergativity. Cambridge Uni- versity Press, Cambridge, UK.
Daniel Edmiston. 2020. A systematic analysis of mor- phological content in BERT models for multiple lan- guages. Arxiv, abs/2004.03032.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016a. The SIGMORPHON 2016 shared Task- Morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. pages 10 22, Berlin, Germany, Association for Com- putational Linguistics.
Alexander Erdmann, Micha Elsner, Shijic Wu, Ryan Cotterell, and Nizar Habash. 2020a. The paradigm discovery problem. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7778 7790, Online. Association for Computational Linguistics.

---

Alexander Erdmann, Tom Kenter, Markus Becker, and Christian Schallhart. 2020b. Frugal paradigm com- pletion. In Proceedings of the 58th Annual Meet- ing the Association for Computational Linguistics. pages 8248 8273, Online. Association for Computa- tional Linguistics.
Valentin Hofmann, Hinrich Schuetze, and Janet Pierre- humbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained lan- guage model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pages 385-393, Dublin, Ireland. Association for Computa- tional Linguistics.
Yoav Goldberg, 2019. Assessing BERT's 's syntactic abil- ities. Arxiv, 1900 .05287.
Valentin Hofmaan, Hinrich Schützz,, and Janet Pierre- humbert. 2020c. A graph auto-encoder model of derivational morphology. In Proceedings of the 58th Anmual Meeting of the Association for Compu- tational Linguistics, pages 1127-1138, Online. Asso- ciation for Computational Linguistics.
Omer Goldman, David Guriel, and Reut Tsarfaty. 2022, (Un)solving morphological inflection: Lemma over- lap artificially inflates models' performance. In Pro- ceedings of the 60th Anmual Meeting of the Associa- tion for Computational Linguistics (Volume 2: Short Papers). pages 864 870, Dublin, Ireland. Associa- tion for Computational Linguistics.
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contam- ination by evaluation benchmarks. arXiv preprint 2rXiv:230..1010
Coleman Haley. 2020. This is a BERT. now there are several of them. can they generalize to novel words? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. pages 333-341, Online. Association for Com- putational Linguistics.
Wenxiang Jiao, Wenxuan Wang, JT Huang, Xing Wang, and ZP Tu. 2023. Is ChatGPT a good translator? Yes with GPT-4 as the engine. arXiv, 2301.08745.
Harald Hammarström and Lars Borin. 2011. Unsuper- vised learning of morphology. Computational Lin- guistics, 37(2):3093355 350.
Huiming Jin, Liwei Cai, Yihui Peng, Chen Xia, Arya McCarthy, and Katharina Kann. 2020. Unsuper- vised morphological paradigm completion. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 6696- 6707, Online. Association for Computational Lin- guistics.
Martin Haspelmath and Andrea Sims. 2010. Under- standing Morphology. Routledge, Oxford (UK).
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Has- san Awadalla. 2023. How good are GPT models at machine translation? comprehensive evaluation. Arxiv, 2302.09210.
Zijia Jin, Xingyu Zhang, Mo Yu, and Lifu Huang. 2022. Probing script knowledge from pre-trained models. In Proceedings of the Workshop on Unimodal and Multimodal Induction of Linguistic Structures (UM- IoS), pages 87-93, Abu Dhabi, United Arab Emi- rates (Hybrid). Association for Computational Lin- guistics.
Charles F. Hockett. 1954. Two Models of Grammatical Description. WORD, 10(2-3):210-23.
Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2020a. DagoBERT: Generating deriva- tional morphology with a pretrained language model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3848-3861, Online, Association for Computa- tional L Linguistics.
Katharina Kann, Ryan Cotterell, and Hinrich Schütze. 2016. Neural morphological analysis: Encoding- decoding canonical segments. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 961-967, Austin, Texas. Association for Computational Linguistics.
Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2020b. Predicting the growth of morpho- logical families from social and linguistic factors. In Proceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 7273- 7283, Online. Association for Computational Lin- guistics.
Katharina Kann, Ryan Cotterell, and Hinrich Schütze. 2017. Neural multi-source morphological reinflec- tion. In Proceedings of the 15th Conference of the European Chapter of the Association for Computa- tional Linguistics: Volume 1, Long Papers, pages 514-524, Valencia, Spain. Association for Compu- tational Linguistics.
Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Deriva- tional morphology improves BERT's interpretation of complex words. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence Natural Language Processing (Volume I: Long Papers), pages 3594 3608, Online. Associa- tion for Computational Linguistics.
Katharina Kann and Hinrich Schütze. 2016. MED: The LMU system for the SIGMORPHON 2016 shared task on morphological reinflection. In Proceedings of the 14th SIGMORPHOON Workshop on Computa- tional Research in Phoneties, Phonology, and Mor- phology, pages 62 70, Berlin, Germany, Associa- tion for Computational Linguistics.

---

Christo Kirov and Ryan Cotterell, 2018. Recurrent neu- ral networks in linguistic theory: Revisiting pinker and prince ( (1988) and the past tense debate. Transac- tions of the Association for Computational Linguis- tics, 6:651 665.
David R. Mortensen, Siddharth Dalmia, and Patrick Lit- tell. 2018. Epitran: Precision G2P for many lan- guages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Re- sources Association (ELRA).
Klaus-Michael Köpcke. 1988. Schemas in German plu- ral formation. Lingua, 74(4):303 335.
Two-level description of Turk- Kemal Oflazer. 1994.
ish morphology. Literary and Linguistic Computing, 9():137-144.
Leigh Lisker. 1951. Tamil verb classification, Journal of the American Oriental Society, 11(2)111-114.
OpenAL. 2023a. ChatGPT. Large language model.
Ling Liu and Mans Hulden. 2020. Leveraging princi- pal parts for morphological inflection. In Proceed- ings of the 17th SIGMORPHON Workshop on Com- putational Research in Phonetics, Phonology, and Morphology, pages 153-161, Online. Association for Computational Linguistics.
OpenAL. 2023b. GPT-4 Technical Report.
Tiago Pimentel, Maria Ryskina, Sabrina J. Mielke, Shijie Wu, Eleanor Chodroff, Brian Leonard, Gar- rett Nicolai, Yustinus Ghanggo Ate, Salam Khal- ifa, Nizar Habash, Charbel El-Khaissi, Omer Gold- man, Michael Gasser, William Lane, Matt Coler, Arturo Oncevay, Jaime Rafael Montoya Samame, Gema Celeste Silva Villegas, Adam Ek, Jean- Philippe Bernardy, Andrey Shcherbakov, Aziyana Bayyr-ool, Karina Sheifer, Sofya Ganieva, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Andrew Krizhanovsky, Natalia Krizhanovsky, Clara Va- nia, Sardana Ivanova, Aclita Salchak, Christo- pher Straughn, Zoey Liu, Jonathan North Wash- ington, Duygu Ataman, Witold Kieras, Marcin Wolinski, Totok Suhardijanto, Niklas Stoehr, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Richard Hatcher, Emily Prud' Prud'hommeaux, Ritesh Kumar, Mans Hulden, Botond Barta, Dorina Lakatos, Gábor Szolnok, Ju- dit Acs, Mohit Raj, David Yarowsky, Ryan Cotterell, Ben Ambridge, and Ekaterina Vylomova. 2021, SIG- MORPHON 2021 shared task on morphological re- inflection: Generalization across languages. In Pro- ceedings of the 18th SIGMORPHOON Workshop on Computational Research in Phonetics, Phonology; and Morphology, pages 229 259, Online. Associa- tion for Computational Linguistics.
Ling Liu and Lingshuang Jack Mao. 2016. Morpho- logical reinflection with conditional random fields and unsupervised features. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 36-40, Berlin, Germany, Association for Com- putational Linguisties.
Xiaomeng Ma and Lingyu Gao. 2022. How do we get there? Evaluating transformer neural networks as cognitive models for English past tense inflec- tion. In Proceedings of the 2nd Conference of the Asia-Pacifiic Chapter of the Association for Compu- tational Linguistics and the 12th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 1101-1114, Online only. Association for Computational Linguistics.
Inbal Magar and Roy Schwartz. 2022. Data contami- nation: From memorization to exploitation. In Pro- ceedings of the 60th Annual Meeting of the Associa- for Computational Linguistics (Volume 2: Short Papers), pages 165,,,61111666 Dublin, Ireland. Associa- tion for Computational Linguistics.
Steven Pinker and Alan Prince. 1988, On language and connectionism: Analysis of parallel distributed pro- cessing model of language acquisition. Cognition, 18)1-)):33
Gary F Marcus, Ursula Brinkmann, Harald Clahsen, Richard Wiese, and Steven Pinker. 1995. German inflection: The exception that proves the rule. Cog- nitive psychology, 29(3):189-256
Ingo Plag. 1999. Morphological productivity: Struc- tural constraints in English derivation. De Gruyter, Berlin.
Rowan Hall Maudslay and Ryan Cotterell. 2021. Do syntactic probes probe syntax? experiments with jab- berwocky probing. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguisties: Human Lan- guage Technologies, pages 124-131, Online. Associ- ation for Computational Linguistics.
Kim Plunkett and Patrick Juola. 1999. A connectionist model of English past tense and plural morphology. Cognitive Science, 23(4):463 4900
David E Rumelhart and James I. McClelland. 1986a. On learning the past tenses of English verbs. In Par- allel Distributed Processing: Explorations in the Mi- crostructure of Cognition, volume 2, pages 216-271. MIT Press, Cambridge, MA.
Kate McCurdy, Sharon Goldwater, and Adam Lopez. 2020. Inflecting when there's majority: Limi- tations of encoder-decoder neural networks as cog- nitive models for German plurals. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1745 1756, On- line. Association for Computational Linguistics.
David E. Rumelhart and James L. McClelland. 1986b. Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations. The
MIT Press.

---

Harold F Schiffmaa and Vasu Renganathan. 2009. An English dictionary of the Tamil verb. Linguistic Data Consortium,
Research in Phonetics, Phonology, and Morphology. pages 1 1-39, Online. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725, Berlin, Germany. Association for Computa- tional Linguisties.
Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu, 2023. Document-level machine translation with large language models. arXiv, 2304.02210.
Leonie Weissweiler, Valentin Hofmann, Masoud Jalili Sabet, and Hinrich Schuetze. 2022. CaMEL: Case Marker Extraction without Labels. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 5506-5516, 5516, Dublin, Ircland. Associa- tion for Computational Linguistics.
Miikka Silfverberg, Adam Wiemerslage, Ling Liu, and Lingshuang Jack Mao. 2017. Data augmentation for morphological reinflection. In Proceedings of the CoNLI SIGMORPHHON 2017 Shared Task: Univer- sal Morphological Reinflection, pages 90-99, Van- couver. Association for Computational Linguistics.
Colin Wilson and Jane S.Y. Li. 2021. Were we there already? Applying minimal generalization to the SIGMORPHON-UUiMorro
Radu Soricut and Franz Och. 2015. Unsupervised mor- phology induction using word embeddings. In Pro- ceedings of the 2015 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1627 1637, Denver, Colorado. Association for Com- putational Linguistics.
shared task on cogni- tively plausible morphological inflection. In Pro- ceedings of the 18th SIGMORPHON Workshop Compulational Research in Phonetics, Phonology, and Morphology, pages 283-291, Online. Associa- tion for Computational Linguistics.
Shijie Wu, Ryan Cotterell, and Mans Hulden. 2021. Ap- plying the transformer to character-level transduc- tion. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguisties: Main Volume, pages 1901-1907, Online, Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models.
Hongzhi Xu, Mitchell Marcus, Charles Yang, and Lyle Ungar. 2018 Unsupervised morphology learning with statistical paradigms. In Proceedings of the 27th International Conference Computational Linguistics, pages 44-54, Santa Fc, New Mexico. USA. Association for Computational Linguistics.
Michael T Ullman, Suzanne Corkin, Marie Coppola, Gregory Hickok, John H Growdon, Walter J Ko- roshetz, and Steven Pinker. 1997. A neural dissoci- ation within language: Evidence that the mental dic- tionary is part of declarative memory, and that gram- matical rules are processed by the procedural system. Journal of Cognitive Neuroscience, 9(2):666-270
A Appendices
A.1 Morphological Tags
Ekaterina Vylomova, Ryan Cotterell, Timothy Baldwin, and Trevor Cohn. 2017. Context-aware prediction of derivational word-forms. In Proceedings of the
In Table 7, we provide details about the morpholog- ical tags that are comprised by the train, dev, test, and wug test sets for the four languages. The tags for English (eng), German (deu), and Tamil (tam) are defined in accordance with the description in UniMorph 4.0 dataset. For Turkish (tur),the tags are defined in Section 3.
15th Conference of the European Chapter of the As- sociation for Computational Linguistics: Volume 2. Short Papers, pages 118-124, Valencia, Spain. As- sociation for Computational Linguistics.
Ekaterina Vylomova, Jennifer White, Eliza- beth Salesky, Sabrina J. Mielke, Shijie Wu, Edoardo Maria Ponti, Rowan Hall Maudslay, Ran Zmigrod, Josef Valvoda, Svetlana Toldova, Francis Tyers, Elena Klyachko, Ilya Yegorov, Natalia Krizhanovsky, Paula Czarnowska, Irene Nikkarinen, Andrew Krizhanovsky, Tiago Pimentel, Lucas Torroba Hennigen, Christo Kirov, Garrett Nicolai, Adina Williams, Antonios Anastasopoulos, Hilaria Cruz, Eleanor Chodroff, Ryan Cotterell, Miikka Silfverberg, and Mans Hulden. 2020. SIGMOR- PHON 2020 shared task 0: Typologically diverse morphological inflection. In Proceedings of the 1 17th SIGMORPHON Workshop on Computational
A.2 Hyperparameter Tuning
For all baselines, we follow the hyperparameter settings from the publicly available code reposito- rics. The only exception is AED, where the num- ber of epochs was increased from 40 to 200.
A.3 Qualtries Details
Our study leveraged Qualtrics, a robust and com- prehensive survey software tool that facilitates the

---

Lang
Tags
Train
Dev
Test
Wug
253
15
9
eng
VINFIN
2015
206
202
(.P.S.NOM(3.SG)
1987
190
213
0
V.PST
1981
198
185
50
V.V.PTCP:RRS
2018
201
200
0
cag V.N.TCPPPST
1999
205
200
0
2
deu V.PTCP:PRS
246
19
19
0
dcu V.IMP.NOM(2PPL)
246
19
16
VoAN change
0
1
23
0
dcu V.MP.NOM(2.SG)
241
22
17
0
Unisows
10
23
deu VIND,PRS:NOMII (PL))
246
20
deu V.IND:PRS/NOMIC (SSG)
227
21
Vieel crange
dcu .INDD.P).NOM((.L))
250
29
Vp coerge
Se
deu HIND.PRS/NOMM2..0
258
25
BIND-PRS:NOMM(2SSO
HIND.PST.NOM(LPL.
235
26
deu VINNDPST:NNMUL.SS
236
17
deu VENND-PSTENOMC,.P
257
20
Figure 4: Confusion matrix for competing German plu- ral morphemes for the one-shot setting.
deu VINDDPPTTNNMMG.PP..
243
15
deu VJNDDPST.NOMM33SG
247
22
27
deu VINFIN
248
21
13
deu V.RII:PRS:NOM(IPP (PP..
234
25
18
No change
45
10
deu VSBIV.PSTTNOMM(LPPM
243
20
20
deu VSSHIVPPTTENMM(2.SC
229
27
45
251
80
99
34
deu VSSIY.PSTENOM(3..)
247
22
--
15
63
2
21
16
N:ACC(PL)
368
44
N:ACC(SG)
18
11
2
116
0
9
N:DAT(PL)
N:DAT(SG)
382
Voael change
deu deu N:GEN(SG)
N,GEN(PL)
364
44
385
47
N.NOM(P)) deu
370
44
49
174
Vowd cron
N,NOM(SG) deu deu V.PTCP;PST
391
47
53
Suod
242
23
23
deu VIIND,RRS/NOMS.HLL
232
25
19
dzu VIND.PSTINOM2..G))
215
25
18
Pood cod
deu NSMM22)
248
20
deu 3L
247
26
Figure 5: Confusion matrix for competing German plu- ral morphemes for the zero-shot setting.
dcu V.SBIV.PRS.NOM/. SG)
238
26
VSSRIVPSTTNNMECS.O
V.SHI..PRS.NOMILLSO))
246
V.SBIVPSTTIOMMIL.O
238
239
17
18
deu (SBM)
222
18
Y.POS:PAST.AISE
2005
201
V.NEG.NARR:AZPL
2005
201
202
tur
N.SSSAPPSSG.DTT
2170
214
214
10
tur
N:ASSG:POONAAC
2172
214
214
10
V.PRS-ISG
67
16
V.FUT-ISG
67
16
tam
V,PST-2SG
67
16
lam
V.PRS-2SG
67
16
Ho.
Vaedi
Werd
tam
V.FUT-2SG
67
16
tam
V.PST-3SG.M
67
16
123
Figure 6: Confusion matrix for competing English past tense morphemes for the one-shot setting.
lam
V.PRS-3SG.M
67
16
tam
V.FUT-3G..M
67
16
tam
V.PST-3SG.F
16
V.PRS-3SG.F
V.FUT-3SGFF
16
26
12
V:PST-JSG.HON
67
16
V:PRS-3SG.HON
67
16
V.FUT-3SG.HON
67
16
V.PST-IPL.
67
16
tam
IPI
67
16
lam
V.FUT-IPL
67
16
tam
V.PST-2PL
67
16
20
lam
V.PRS-2PI.
67
16
tam
V.FUT-2PL.
67
16
Word
tam
V.PST-3PL
67
16
3.PRS-3PI.
Figure 7: Confusion matrix for competing English past tense morphemes for the zero-shot setting.
V.FUT-3PI.
Table Distribution of tags over the different splits for the four languages.

---

design of intricate online surveys.
German:
We initiated the survey by presenting an intro- duction that detailed the concept of a wug test and the associated information for the survey. This in- troductory passage served to inform participants of the nature and intent of the research study, and it also provided examples to further facilitate their understanding of our task requirements.
Long: "Fülle die Lücke mit dem korrek- ten Plural des Nomens X aus. Antworte mit einem Wort. Hier ist ein X. Jetzt sind es zwei _!_
Short: "Bilde den korrekten Plural des Nomens X. Antworte mit einem Wort. X
Our data collection phase consisted of two parts: the English wug test and the German wug test. Upon consenting to participate, respondents were guided through series of thoughtfully designed prompts related to the wug test. These prompts en- couraged them to provide suitable responses based on their understanding of the task.
Tamil:
Long: Tbūg eieurii "B X" σा Gyea.. 3005 a C5L G 2ιe GUTT
Short: "X:"
For the English wug test, employed the fol- lowing exemplary prompt: "Fill in the blank with the correct past tense of the word 'wug'. There is no predetermined correct answer. We encourage you to rely on your linguistic intuition. If you be- lieve there are multiple possible responses, simply note the form that seems most accurate to you. For instance, 'They wug all the time. In fact, they just yesterday!" Such prompts stimulated the par- ticipants to produce responses that were entirely their own, drawing on the provided information. For the German wug test, we translated the task in- structions and prompts into German, ensuring easy comprehension for native German speakers.
Turkish:
Long: "Boşluklar X ile verilen eylemin birinci tekil şahıs geçmiş zaman formlar ile doldurun. Ben her zaman X. Ama dün
Short: "Tck bir sözcüik ilc farazi X eyleminin birinci tekil yahıs geçmiş za- man hali nasil olur? x
In total, the English wug test incorporated 50 unique words for participants to respond to, while the German version consisted of 74 unique words. We received 28 responses for the English wug test and 21 responses for the German wug test.
A.4 Other Values of k
Table 8 presents results for k = and = 3. Re- sults for k = are given in Section 5.
A.5 Prompts
We leveraged the following prompts for the indi- vidual languages:
English:
Long: "Fill in the blank with the correct past tense of the verb X. Answer with one word. They X all the time. In fact, they just yesterday!
Short: "Form the correct past tense of the verb X. Answer with one word. X :

---

English
German
Tamil
Method
A
k == 3
A =
==
k: == 1
k =
ARI.
66.00
98.00
71.84
91.95
49.18
61.48
MinGen
56.00
60.00
39.66
60.92
39.34
49.18
FIT
84.00 2.97
96.20
70,06 1.67
90.69 = 0,99
44.75
59.84 3.07
PPI
72.60 +: 6.00
90.80 4.49
60.17 6.80
82.59 = 6.56
2.57
51.07 =: 1,56
AED
44.20 1: 7.18
56,20 = 6.54
27.82 3.94
42.87 = 4.65
46.15 4.118
$7.87 a: 5.46
long 0-shot
42.60 4.90
$5.60
62.18 2.45
81.55 = 1.77
0.00
long 1-shot
58,40
6.50
4.01
81.61 3.09
=.28
12.65 6.19
long few-shot
$7.60
74.60
3.03
82.59 1.96
15.25
38.03
short 0-shot
55.40 6.07
72.20
66.38 2.48
0.00
short 1-shot
61.20
81,60
67.31 3.92
1.99
3.04
short few-shot
61.40 3.69
77.20 + 3.12
68.97 2.02
84.43 = 1.07
38.77 2.86
Table 8: Results for other values of k.


Reviews:
[{"review_id": "93HseCBo8u", "reviewer": "Reviewer_oAs1", "summary": "", "questions": "(a) The final section refers to productivity, I wonder if the authors have considered a tangential thread: rare inflections of a paradigm? Presumably, some inflectional features will be more prevalent than others. For example, the future perfect continuous in the passive voice in English vs. Past tense. \n\n(b) During the production stage of nonce words, the authors describe the process as :\n[For English]: \"These items were each perturbed by one or two letters (substituting phonetically similar sounds) producing a word not included\" \n[For Tamil]:  \"combining syllables from real verb roots and checking against a Tamil dictionary\"\n[For German]: \"We generated 200 nonce words with a length between four and seven characters\"...using Unipseudo\n[For Turkish]: The process isn't transparent/explained. \nIs it possible that certain sound changes, or sequences can identify the word as a loan-word and therefore guide it's inflection pattern? How might letter frequency in each respective language affect your results (if at all)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### “I think claiming \"typological diversity\" is vague.”\n\nWe thank the reviewer for encouraging us to provide more details about the specific ways in which the considered languages are typologically diverse. German and English were chosen since they exhibit morphological phenomena that have been investigated in great depth in the literature, thus providing a good opportunity for benchmarking LLMs. While English and German belong to the same language family (as rightly mentioned by the reviewer), the morphological system of German is considerably more complex and (in terms of typology) more fusional than English. Turkish was chosen since it is a non-Indo-European language with a fully agglutinative morphology, thus occupying a different point in the typological spectrum. Tamil was chosen since it is a Dravidian language exhibiting an agglutinative morphology with fusional elements. Thus, in terms of the classical triangle of fusional, isolating, and agglutinative morphologies, our chosen languages cover four different points: almost fully isolating (English), intermediate between isolating and fusional (German), intermediate between fusional and agglutinative (Tamil) and fully agglutinative (Turkish). Furthermore, the chosen languages also cover different points in the spectrum from low-resource to high-resource, enabling us to form hypotheses about the impact of the amount of language-specific training data on the morphological capabilities of LLMs.\n\nWe agree that we should have discussed these points in greater detail in the paper and will do so in the final version.\n\n### “I would've welcomed further discussion on the inter-annotator agreement.”\n\nWe thank the reviewer for pointing out that our discussion of the low inter-annotator agreement (IAA) for Tamil is insufficient. We would like to draw the reviewer’s attention to two crucial points:\n\nRecent research in linguistics has shown that inter-speaker variability is more wide-spread in morphological systems than previously thought (see, for instance, Dammel and Schallert, 2019). Relatively low IAA for Tamil indicates that the examined morphological construction exhibits a particularly pronounced degree of variation — this is not a problem of our analysis, but an inherent property of morphology. For Tamil, the verb classes are not conditioned on phonemic or morphomeic features, “the correspondence between phonemic shape and class membership is not one that enables us to predict absolutely the latter from the former… in fact, pairs that are phonemically identical are found in different classes” (Lisker 1951). Because variation is grounded in historical factors, new verbs may belong to one of several classes based on how the speaker chooses to extend their knowledge of Tamil’s grammar, leading to low IAA. In other words, it is to be expected that IAA for Tamil annotators will be fairly low. We hope that the Tamil data (which we will release) might be an interesting starting point for future research into morphological variation in non-Indo-European languages.\n\nWhile not specifically focussing on morphological variation, our analysis is specifically set up such that it takes variation between humans into account: we evaluate using accuracy@k, i.e., if there is a high degree of variation between speakers, the model’s response is evaluated as correct if it is in line with any of the speakers (assuming there are not more than k answers in total, which is never the case for Tamil). This reflects our belief that when there is no one right answer for humans, we should not require this of the model either, but rather expect it to represent the full spectrum of variation in human judgement. Please note that this is exactly in line with a recent call in NLP to pay greater attention to annotator variation (e.g., Röttger et al., 2022).\n\nThat being said, we now see that we should have devoted more attention to these points, and we will add a discussion to the final version of the paper.\n\n### For example, \"factors other than inherent complexity play a role in ChatGPT’s ability to generalize morphological patterns\". This feels like a bit of a throw-away line.\n\nWe agree that this was expressed clumsily. Our intent was to account for the high performance of ChatGPT on the German, relative to the English, wug test. One hypothesis might be that the English task was harder. However, in actual fact, the model could achieve a perfect score on the English dataset simply by guessing an -ed form for every verb, whereas the situation with German nouns was *much* more complicated (there was more variation and a richer set of conditions drove it). We speculated that ChatGPT does relatively poorly in English, then, not because the task is harder but because it has seen *too much* English and is suffering due to “real word bias.” The idea that we were trying to express was actually quite important and we would welcome the opportunity to express it better.\n\n### “I wonder if the authors have considered a tangential thread: rare inflections of a paradigm?”\n\nWe did consider this, but felt that tackling it was too much for this study. All of the inflectional categories explored here (English present versus past tense, Tamil present versus past tense, German singular versus plural, Turkish nominative versus accusative) are exceptionally common. Since high frequency inflectional categories tend to show greater variability (more “irregularity”) we focused our efforts there.\n\n### “[For Turkish]: The process isn't transparent/explained.”\n\nWe agree that the process of producing and annotating the Turkish data got short shrift. In brief, the data were generated by one author (an expert in the subject matter). They manually produced nonce roots consistent with the phonotactic rules of Turkish and checked to ensure that they did not correspond to dictionary words. We originally intended to recruit a team of Turkish annotators. However, this proved unnecessary since we discovered that the morphotactics and morphophonology of Turkish are so regular that the inflected forms followed deterministically from the form of the roots. For this reason, the annotations were produced by a single individual (unlike the other languages). Given an additional page, we will explain this matter (as well as the Tamil dataset)  in more detail.\n\n### \n“Is it possible that certain sound changes, or sequences can identify the word as a loan-word and therefore guide its inflection pattern? How might letter frequency in each respective language affect your results (if at all)?”\n\nThis is an excellent question (and one with which morphologists have wrestled for some time): are language users relying on formal factors (like sequences of sounds that make a new word sound “foreign”) when they decide what inflectional class to assign it to? Are they relying on meaning? Are they relying on explicit knowledge of the etymological origin of a word? Our study shows that German and Tamil speakers, at least, rely to a considerable extent on form. What we have not yet studied is whether the frequencies of graphemes in the stem (or, for that matter, the entropy of stems) predicts inflectional class (and whether LLMs like ChatGPT show the same behaviour). We think that this is an excellent avenue for future research.\n\n### Typos Grammar Style And Presentation Improvements\n\nThanks so much for drawing our attention to these mistakes — we will correct them in the final version of the paper!\n\nAntje Dammel and Oliver Schallert. 2019. Morphological Variation: Theoretical and empirical perspectives. Amsterdam: John Benjamins.\n\nPaul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190, Seattle, United States. Association for Computational Linguistics.\n\nLisker, Leigh. 1951. Tamil Verb Classification. Journal of the American Oriental Society 71.2, pages 111-114.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It's great to see linguistically motivated probing into LLMs. Given the recent hype, this kind of work is important. The authors present a set of experiments to test the morphological generalising capacities of LLMs - specifically gpt-3.5-turbo-0613.", "reasons_to_reject": "I think claiming \"typological diversity\" is vague. My first thought in reading this description followed by English and German is that these languages are in the same language family! The authors should be specific, the research question narrows down the considered design space to morphology. So what morphological properties make these languages of interest? \n\nI would've welcomed further discussion on the inter-annotator agreement. It was briefly mentioned in the Tamil results section. Why was inter-annotator agreement so low? \n\nIt feels as though the authors had a lot of results to share, which came at a cost of the narrative and explanation. There were a few conclusions that felt like a leap. For example, \"factors other than inherent complexity play a role in ChatGPT’s ability to generalize morphological patterns\". This feels like a bit of a throw-away line. Again, being more specific will help. Do the authors mean complexity in assigning morphological classes?", "questions_to_authors": "(a) The final section refers to productivity, I wonder if the authors have considered a tangential thread: rare inflections of a paradigm? Presumably, some inflectional features will be more prevalent than others. For example, the future perfect continuous in the passive voice in English vs. Past tense. \n\n(b) During the production stage of nonce words, the authors describe the process as :\n[For English]: \"These items were each perturbed by one or two letters (substituting phonetically similar sounds) producing a word not included\" \n[For Tamil]:  \"combining syllables from real verb roots and checking against a Tamil dictionary\"\n[For German]: \"We generated 200 nonce words with a length between four and seven characters\"...using Unipseudo\n[For Turkish]: The process isn't transparent/explained. \nIs it possible that certain sound changes, or sequences can identify the word as a loan-word and therefore guide it's inflection pattern? How might letter frequency in each respective language affect your results (if at all)?", "paper_topic_and_main_contributions": "This paper examines the morphological 'understanding' of LLMs by prompting the production of nonce words. The authors focus on specific parts of inflection for English, German, Tamil and Turkish. The study promises a contribution of the datasets used. The results show that ChatGPT has yet to achieve human-level performance on the wug test.  Interestingly, the authors report 'real word biases' where a similar stem with the same grammatical features is produced (e.g., blushed for the past tense of blus).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "-line193 what \"it\" refers to is unclear\n-line596 I'm assuming the reference should be to figure 3 not 8. \n-There were a few figures and tables which weren't referenced in text, so it took extra effort to see how they tied into the research presented."}}, {"review_id": "yOG7U1gtTo", "reviewer": "Reviewer_hyZR", "summary": "", "questions": "Question A: Did you use the openai provided version or the one on Microsoft Azure?\nQuestion B: Beyond the investigation of the real-word bias in 6.4., did you find any other similarly groupable types of errors?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Generalizability: Most importantly, the generalizability of the study beyond the combination of the particular GPT 3.5 version with the particular nonce word datasets is not clear.\n\nWe would like to point out that at the time of writing, there were no other large models available that were comparable in performance with GPT3.5 on morphological generalisation, or indeed reached the level at which an evaluation such as ours can be expected to return reasonable results. For instance, we found in initial experiments with Llama and Alpaca that there was no substantial morphological generalisation from nonce words. We had planned to include results from these models, but changed our plans when we discovered that they were not even remotely competitive with the GPT models, no matter how we constructed the prompts. This situation has since changed, with the release of open models such as Llama 2 (which actually does quite well at this task), and we heartily agree with the reviewer that a broader investigation is desirable for future work. \n\n### Tamil data: Futhermore, the inter-annotator agreement on the Tamil dataset is low, which might indicate a problem with the approach to its construction, also, the dataset is small.\n\nWe thank the reviewer for pointing out that our discussion of the low inter-annotator agreement (IAA) for Tamil is insufficient. We would like to draw the reviewer’s attention to two crucial points:\n\nRecent research in linguistics has shown that inter-speaker variability is more wide-spread in morphological systems than previously thought (see, for instance, Dammel and Schallert, 2019). Relatively low IAA for Tamil indicates that the examined morphological construction exhibits a particularly pronounced degree of variation — this is not a problem of our analysis, but an inherent property of morphology. For Tamil, the verb classes are not conditioned on phonemic or morphomeic features, “the correspondence between phonemic shape and class membership is not one that enables us to predict absolutely the latter from the former… in fact, pairs that are phonemically identical are found in different classes” (Lisker 1951). Because variation is grounded in historical factors, new verbs may belong to one of several classes based on how the speaker chooses to extend their knowledge of Tamil’s grammar, leading to low IAA. In other words, it is to be expected that IAA for Tamil annotators will be fairly low. We hope that the Tamil data (which we will release) might be an interesting starting point for future research into morphological variation in non-Indo-European languages.\nWhile not specifically focussing on morphological variation, our analysis is specifically set up such that it takes variation between humans into account: we evaluate using accuracy@k, i.e., if there is a high degree of variation between speakers, the model’s response is evaluated as correct if it is in line with any of the speakers (assuming there are not more than k answers in total, which is never the case for Tamil). This reflects our belief that when there is no one right answer for humans, we should not require this of the model either, but rather expect it to represent the full spectrum of variation in human judgement. Please note that this is exactly in line with a recent call in NLP to pay greater attention to annotator variation (e.g., Röttger et al., 2022).\n\nThat being said, we now see that we should have devoted more attention to these points, and we will add a discussion to the final version of the paper.\n\n### References\n\nAntje Dammel and Oliver Schallert. 2019. Morphological Variation: Theoretical and empirical perspectives. Amsterdam: John Benjamins.\n\nPaul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190, Seattle, United States. Association for Computational Linguistics.\n\nLisker, Leigh. 1951. Tamil Verb Classification. Journal of the American Oriental Society 71.2, pages 111-114.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Regarding large language models, \"understanding their limitations, [...], is an ethical imperative\" (l 654). I agree with that. The presented analysis of morphological capabilities of ChatGPT (first one, also to my knowledge) is a decent contribution to this respect.\n* The study is broad, taking into account different morphological phenomena across different languages any many baseline models.\n* Interesting findings regarding ChatGPT and morphology (e.g., l 555, l 605). \n* The datasets are potentially valuable to further research.", "reasons_to_reject": "The limitations section addresses the most important points already:\n* Generalizability: Most importantly, the generalizability of the study beyond the combination of the particular GPT 3.5 version with the particular nonce word datasets is not clear.\n* Tamil data: Futhermore, the inter-annotator agreement on the Tamil dataset is low, which might indicate a problem with the approach to its construction, also, the dataset is small.\n\n", "questions_to_authors": "Question A: Did you use the openai provided version or the one on Microsoft Azure?\nQuestion B: Beyond the investigation of the real-word bias in 6.4., did you find any other similarly groupable types of errors?", "paper_topic_and_main_contributions": "The paper presents a study of the morphological capabilities of ChatGPT on the basis of the \"Wug Test\": The LLM is queried to provide an inflected or a derived form of a nonce word. The performance of the LLM and several baselines models is evaluated against human annotations. The performance of the baseline models is additionally evaluated on the SIGMORPHON 23 task. The \"Wug\" datasets are specially constructed for this study for four languages, German, English, Tamil and Turkish and will be published.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* l 73, l 249: \"LLMs\" -> \"ChatGPT\".\n* l 415: For the avoidance of any doubts regarding the evaluation, you could mention what \"acc@k\" stands for and why it makes sense as an evaluation metric.\n* l 304: SIGMOPHON -> SIGMORPHON"}}, {"review_id": "YK6Rowqwpf", "reviewer": "Reviewer_hkP1", "summary": "", "questions": "A> I am wondering of real world use cases of non-word morphology like this, I think if llm's could be harnessed as a tool to help e.g. linguists in lexicography and other work of e.g. lesser resourced languages, are there other directions to go from here besides probing potential capabilities or of llms?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their helpful comments. Question A raises an important point. LLMs in general (and especially open source LLMs, as they improve in performance) encode a tremendous richness of information about human languages that can be tapped for analysis and can be used for multiple purposes:\n1. To construct sets of stimuli for conducting quantitative studies and psycholinguistic experiments.\n2. To perform experiments regarding grammatical patterns (including morphological patterns) that are latent within unlabelled language data (which is directly relevant to the current study).\n3. To develop varied and challenging language teaching materials, especially for low-resource languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Article presents some evidence on linguistic capabilities of the internal representations of large language models, which is both very relevant to the venue and surprisingly rare topic", "reasons_to_reject": "* there is a limited scientific value in merely using one off-the-shelf large language model via web-based api; i.e. reproducibility, issues etc.", "questions_to_authors": "A> I am wondering of real world use cases of non-word morphology like this, I think if llm's could be harnessed as a tool to help e.g. linguists in lexicography and other work of e.g. lesser resourced languages, are there other directions to go from here besides probing potential capabilities or of llms?", "paper_topic_and_main_contributions": "The article probes chatgpt on morphological skills by getting it to inflect non-words in English, German and Turkish amd Tamil. Based on the findings the chatgpt's internal representation of morphology seems to be lacking in what is generally being advertised. The article gives good linguistic backgrounds and hypotheses of morphological inflection relevant to the task and an experimental setup. The selection of languages is limtied and all belong in relatively well resourced languages with large speaker base and ample written material in the publically available data sets.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigated ChatGPT's morphological capabilities by prompting the generation of inflected or derived forms of nonce words. The findings indicate that ChatGPT has not yet reached human-level performance. Additionally, the creation of the dataset covering four languages is a contribution to the research community.

Testing only a single closed model through a web API is undeniably disappointing. However, as the authors aptly pointed out in their rebuttal, there was understandably no alternative available. I believe this paper deserves publication in some form.
Decision: Accept-Main