SlowGuess/ACL_2020_OCR huggingface dataset:
Kate McCurdy Sharon Goldwater Adam Lopez
kate.mccurdy@ed.ac.uk, {sgwater, alopez}@inf.ed.ac.uk
Can artificial neural networks learn to represent inflectional morphology and generalize to new words as human speakers do? Kirov and Cotterell (2018) argue that the answer is yes: modern Encoder-Decoder (ED) architectures learn human-like behavior when inflecting English verbs, such as extending the regular past tense form  $/-(\mathrm{e})\mathrm{d}/$  to novel words. However, their work does not address the criticism raised by Marcus et al. (1995): that neural models may learn to extend not the regular, but the most frequent class — and thus fail on tasks like German number inflection, where infrequent suffixes like  $/-\mathrm{s}/$  can still be productively generalized. To investigate this question, we first collect a new dataset from German speakers (production and ratings of plural forms for novel nouns) that is designed to avoid sources of information unavailable to the ED model. The speaker data show high variability, and two suffixes evince 'regular' behavior, appearing more often with phonologically atypical inputs. Encoder-decoder models do generalize the most frequently produced plural class, but do not show human-like variability or 'regular' extension of these other plural markers. We conclude that modern neural models may still struggle with minority-class generalization.
Morphology has historically been the site of vigorous debate on the capacity of neural models to capture human speaker behavior, and hence ground claims about speaker cognition. In 1986, Rumelhart and McClelland described a neural network model which learned to map English present tense verbs to their past tense forms. Importantly, the network handled both regular verbs, whose past tense is formed systematically by adding the suffix /-(e)d/ (e.g. jumped), and irregular verbs where the present and past tenses bear no systematic relationship (e.g. ran). The authors suggested their model provided "an alternative [...] to the implicit knowledge of rules" (1986, 218), a claim
which sparked considerable controversy. Pinker and Prince (1988) highlighted many empirical inadequacies of the Rumelhart and McClelland model, and argued that these failures stemmed from “central features of connectionist ideology” and would persist in any neural network model lacking a symbolic processing component.
Recently, however, Kirov and Cotterell (2018, henceforth K&C) revisited the English past tense debate and showed that modern recurrent neural networks with encoder-decoder (ED) architectures overcome many of the empirical limitations of earlier neural models. Their ED model successfully learns to generalize the regular past tense suffix  $/-(\mathrm{e})\mathrm{d}/$ , achieving near-ceiling accuracy on held-out test data. Moreover, its errors result from overapplication of the regular past tense (e.g. throw-throwed)—a type of error observed in human language learners as well—as opposed to the unattested forms produced by Rumelhart and McClelland's model. K&C conclude that modern neural networks can learn human-like behavior for English past tense without recourse to explicit symbolic structure, and invite researchers to move beyond the 'rules' debate, asking instead whether the learner correctly generalizes to a range of novel inputs, and whether its errors (and other behavior) are human-like.
This challenge was first taken up by Corkery et al. (2019), who showed that, on novel English-like words designed to elicit some irregular generalizations from humans, the ED model's predictions do not closely match the human data. While these results suggest possible problems with the ED model, English may not be the best test case to fully understand these, since the sole regular inflectional class is also by far the most frequent. In contrast, many languages have multiple inflectional classes which can act 'regular' under various conditions (Seidenberg and Plaut, 2014; Clahsen, 2016).
In this paper, we examine German number inflection, which has been identified as a crucial test case
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1745-1756 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
for connectionist modeling (Köpcke, 1988; Bybee, 1995; Marcus et al., 1995; Clahsen, 1999b). The German plural system features eight plural markers (c.f. Table 1), none of which hold a numerical majority in type or token frequency. Different linguistic environments favor different plural markers (e.g. Köpcke, 1988; Wiese, 1996; Yang, 2016), and even the famously rare suffix /-s/ is nonetheless productive, in the sense that speakers readily extend it to new words. $^{1}$  In their analysis of the German plural system, Marcus et al. (1995, henceforth M95) argue that neural networks generalize the most frequent patterns to unfamiliar inputs, and thus struggle to represent productive but rare classes such as /-s/. We investigate that claim using the novel German-like nouns M95 developed.
Because the design and results of previous human studies have been somewhat inconsistent, and because we want to compare to fine-grained results from individuals (not just published averages), we first collect a new dataset of plural productions and ratings from German speakers. Our speaker data show high variability: no class holds a majority overall, and two less frequent suffixes show a relative preference for phonologically atypical inputs ("Non-Rhymes"). We then compare our human data with the predictions of the encoder-decoder (ED) model proposed by K&C. While our human data paint a more complex picture of the German plural system than M95 claimed, nevertheless M95's central idea is borne out: when given Non-Rhymes, the ED model prefers the most frequent plural class, but speakers behave differently. This finding reveals that while modern neural models are far more powerful than earlier ones, they still have limitations as models of cognition in contexts like German number inflection, where no class holds a majority. The model may correctly identify the most frequent class, but fails to learn the conditions under which minority classes are productive for speakers.
2 Study 1: Speaker plural inflection
To evaluate whether neural models generalize correctly, we need to compare their behavior with that of humans on the same task. Unfortunately, no existing datasets were suitable, so our first study asks how German speakers inflect novel nouns.
Suffix	Singular	Plural	Type	Token
/-(e)n/	Strasse	Strassen	48%	45%
/-e/	Hund Kuh	Hunde Kühe	27%	21%
/-∅/	Daumen Mutter	Daumen Mütter	17%	29%
/-er/	Kind Wald	Kinder Wälder	4%	3%
/-s/	Auto	Autos	4%	2%
Table 1: German plural system with examples, ordered by CELEX type frequency (Sonnenstuhl and Huth, 2002).
Wug testing and productivity If an English speaker needs to produce the plural form of an unknown word such as wug, that speaker must decide whether wug belongs to the same inflectional class as dog and cat (yielding plural wugs) or the same class as sheep and deer (yielding wug). Speakers' overwhelming preference for wugs in this scenario indicates that the /-s/ plural class is productive in English: a productive morphological process can be generalized to new inputs. This task of inflecting novel (nonce) words is known as the wug test (Berko, 1958), and is the standard method to determine productivity in psycholinguistic research. While the concept of morphological 'regularity' is not well-defined (Herce, 2019), productivity is nonetheless an essential component: an inflectional class that is not productive cannot be regular.
Productivity in German plurals The German plural system comprises five suffixes: /-e/, /-er/,  $-\emptyset /^2$  , /-(e)n/, and /-s/. The first three can optionally combine with an umlaut over the root vowel.3 Umlaut varies semi-independently of plural class (Wiese, 1996), and is not fully predictable; for simplicity, this study will focus only on the five main suffix classes for analysis. Examples in all forms are shown in Table 1. Each plural suffix is also shown with its type frequency (counting each word type only once, how many types in the lexicon take this plural?) and token frequency (how often do words with this plural suffix appear in the corpus overall?). German nouns can have one of
For example, the Institut für Deutsche Sprache (https://www.owid.de/service/stichwortlisten/neo_neuste) officially added multiple /-s/-inflecting nouns to the German language in 2019, including Verhütungsapp, Morphsuit and Onesie.
$^{2} / - \emptyset /$  refers to the so-called "zero plural", and is indicated as "zero" on all figures in this paper.
${}^{3}$  Umlaut is a process which fronts a back vowel,so only roots with back vowels can take an umlaut (e.g. Dach  $\rightarrow$  Dächer,Fuss  $\rightarrow$  Füsse).
three grammatical genders — masculine, feminine, or neuter — and this lexical feature is highly associated with plural class: most feminine nouns take  $/-(\mathrm{e})\mathrm{n}/$ , while  $/-\mathrm{e}/$  and  $/-\emptyset/$  nouns are often masculine or neuter. The phonological shape of a noun also influences its plural class; for example, most nouns ending with schwa take  $/-(\mathrm{e})\mathrm{n}/$  (Elsen, 2002). Although there are statistical tendencies, there are no absolute rules, and no suffix holds a majority overall. Researchers continue to debate which plural markers are productive, and in which circumstances.
The dispute has historically centered on the infrequent class /-s/, which, despite its rarity, occurs across a wide range of linguistic environments. Examples include proper names (e.g. der Bader  $\rightarrow$  die Bader 'the barber  $\rightarrow$  the barbers' but meinie Freunden, die Baders 'my friends, the Barbers'), acronyms, and truncated and quoted nouns (e.g. der Asi  $\rightarrow$  die Asis, short for Asozialer 'antisocial person'). In addition, /-s/ tends to be the plural class for recent borrowings from other languages, and children reportedly extend /-s/ to novel nouns (Clahsen et al., 1992). For these reasons, M95 argue that /-s/ is the default plural: it applies in a range of heterogeneous elsewhere conditions which do not define a cohesive similarity space, serving as the "emergency" plural form when other markers do not seem to fit. They further assert that, as the default form, /-s/ is also the only regular plural form, in the sense that it "applies not to particular sets of stored items or to their frequent patterns, but to any item whatsoever" (1995, 192). Under this minority-default analysis, other German plural classes may be productive, but in a limited sense — they can only extend to novel inputs which are similar in some respect to existing class members, while infrequent /-s/ can apply to any noun regardless of its form (Clahsen, 1999b). M95 claim that this behavior should be particularly difficult for connectionist, i.e. neural, models to learn: /-s/ cannot be generalized based on its frequency, as it is rare, and it cannot be generalized based on similar inputs, as it applies to heterogeneous, unfamiliar inputs.
Other researchers have challenged the minority-default account with evidence of regular, productive behavior from the two more common suffixes /-e/ and /-(e)n/. /-(e)n/ is argued to be the default class for feminine nouns and nouns ending with the weak vowel schwa (Wiese, 1996; Dressler, 1999), and children have also been found to overgeneral
ize /-(e)n/ (Köpcke, 1998). Indefrey (1999, 1025) argues that /-(e)n/ and /-e/ are “regular and productive allomorphs with gender-dependent application domains”, noting that /-e/ and /-(e)n/ are extended in elsewhere conditions where /-s/ is blocked for phonological reasons, such as letters (die “X”e) and acronyms (die MAZen, Magnetaufzeichnungen, ‘magnetic recordings’). Bybee (1995) argues that, while /-s/ does act as the default plural, it is still less productive than other plural classes due to its low type frequency.
Wug testing for German plurals To assess whether German speakers treat /-s/ as a productive default for novel words, M95 developed a list of 24 monosyllabic nonce nouns for wug testing. The stimuli represented two phonological classes: 'familiar' or Rhyme words, which rhymed with one or more existing words in German (e.g. Bral, rhyming with Fall; Spert, rhyming with Wert), and 'unfamiliar' or Non-Rhyme words (e.g. Plaupf, Fnohk), which were constructed using rare but phonotactically valid phone sequences. They hypothesized that Non-Rhymes, as phonologically atypical words, should be more likely to take the /-s/ plural. M95 conducted a rating study in which stimuli were presented across three different sentence contexts. If the word Bral was presented in the "root" condition, subjects would rate a set of sentences where the nonce word referred to some object: Die grünen BRAL sind billiger ("The green brals are cheaper"), Die grünen BRALE ..., Die grünen BRÄLE ..., etc.; whereas in the "name" condition, the nonce word would refer to people: Die BRAL sind ein bβchen komisch ("The Brals [family name] are a bit weird"), Die BRALEN ..., Die BRALS ..., etc. With data from 48 participants, /-s/ was the top-rated plural form for 2 out of 12 rhyme words, and 7 out of 12 non-rhyme words; while /-e/ was rated highest overall, /-s/ was the only marker favored more for non-rhymes. Clahsen (1999a) cites this asymmetry as crucial evidence for /-s/ as the only default plural form, at least with respect to these stimuli.
These results, however, have been called into question. Zaretsky and Lange (2016, henceforth Z&L) conducted a large-scale follow-up study with 585 participants, using the same nonce words but a different task: instead of rating the plural forms within a sentence context, subjects were presented with the noun in isolation (e.g. Der Bral) and asked
to produce its plural form.4 They found a much lower preference for /-s/ than expected based on M95's results, and a significant effect for feminine (die) versus non-feminine (der; das) grammatical gender, where M95 reported no effect of gender. The authors conclude from their data that /-(e)n/, /-e/, and /-s/ are all productive in German, and also speculate that task differences (production versus rating) could account for the discrepancy between the two studies.
2.2 Data collection
Motivation Although M95 published average rating data for each word in the appendix to their paper, we felt it necessary to collect our own data. Z&L's findings suggest that the M95 /-s/ effect might reflect task artefacts: speaker behavior could differ for production and rating tasks, and with and without sentential context for the nonce words. We seek to evaluate K&C's performance claims for ED models, which were based on speaker production probabilities rather than ratings. To do so, we need speaker data which closely parallels the model task: given a noun in isolation, produce its plural inflected form. We collect production data, and also ratings, to see whether speaker behavior is consistent across tasks.
Another issue raised by Z&L's findings is the role of grammatical gender. Although Z&L reported significant gender effects, M95 did not: their reported rating averages combine all gender presentations (e.g. Der Bral, Die Bral, Das Bral). Previous experiments have found neural models of German plurals to be sensitive to grammatical gender (Goebel and Indefrey, 2000); therefore, the stimuli presented to speakers should be consistent with model inputs to enable valid comparison. For simplicity, we opted to select one grammatical gender for presentation: neuter, or Das. Based on similar experimentation by Kopcke (1988), speakers do not have a strong majority class preference for neuter monosyllabic nouns, hence this environment may be the most challenging for a neural model to learn. For this reason, we present all stimuli as neuter to study participants.
Method The current study uses the same Rhyme and Non-Rhyme stimuli from M95's original experiment. We collected both production and rating data on plural inflection for the 24 M95 nonce nouns through an online survey with 150 native
Plural	Prod %	N	Rating (SE)
/-e/	R	45.3	815	3.53 (.021)
NR	44.7	805	3.51 (.024)
/-(e)n/	R	25.0	450	3.73 (.026)
NR	34.7	624	3.84 (.025)
/-er/	R	17.4	314	3.08 (.022)
NR	6.7	120	3.06 (.024)
/-s/	R	4.2	75	2.39 (.027)
NR	6.4	116	2.52 (.028)
/-∅/	R	2.7	48	2.24 (.020)
NR	2.7	48	2.38 (.024)
other	R	5.4	98
NR	4.8	87
overall	R	1800	2.99 (.011)
NR	1800	3.04 (.012)
Table 2: Survey results. Production reported as percentages out of all Rhymes (R) and Non-Rhymes (NR); ratings are averages over a 1 (worst) - 5 (best) scale, with standard errors in parentheses. Highest numbers in each category are bolded.
German-speaking participants. Survey respondents were first prompted to produce a plural-inflected form for each noun (i.e. filling in the blank: "Das Bral, Die ----"). After producing plural forms for all nouns, they were prompted to rate the acceptability of each potential plural form for each noun on a 1-5 Likert scale, where 5 means most acceptable. For example, a participant would see Das Bral, and then give an acceptability rating for each of the following plural forms: Bral, Bräl, Brale, Bräle, Bralen, Braler, Bräler, Brals. For details of the survey design, please see Appendix A.
Our study results are shown in Table 2. The production data collected in our survey appears broadly consistent with the distribution observed by Z&L and Köpcke: /-e/ is favored in production, followed by /-(e)n/. The rhyme vs non-rhyme comparison is also consistent with Z&L's results. /-s/ is produced more for Non-Rhymes than for Rhymes, as emphasized by Clahsen (1999a); however, /-(e)n/ also shows the same directional preference, and at a much higher frequency.
Our rating results diverge from production results in some ways — for example,  $/-(\mathrm{e})\mathrm{n}$  is fa
5The article das indicates singular number, neuter gender; as all nouns were presented in neuter gender (see preceding discussion), all nouns were preceded by das. Die here indicates plural number, so the following noun will be pluralized.
${}^{4}\mathrm{Z}$  & L's data is unfortunately not freely available.
vored instead of  $/-\mathrm{e}/$  — and are consistent in others: both  $/-\mathrm{s}/$  and  $/-(\mathrm{e})\mathrm{n}/$  are rated higher for Non-Rhymes compared to Rhymes. The low ratings for  $/-\mathrm{s}/$  conflict with M95's findings, and suggest that presentation in sentence context is an important methodological difference from presentation in isolation. For example, family surnames obligatorily take  $/-\mathrm{s}/$  in German, so it's possible that exposure to surnames in the "name" context primed subjects in the M95 rating study to find  $/-\mathrm{s}/$  more acceptable generally, across conditions.[6] In any case, our results demonstrate task effects: although  $/-\mathrm{e}/$  is the most produced plural form,  $/-(\mathrm{e})\mathrm{n}/$  obtains the highest ratings from the same speakers.[7] We compare these results with the modeling study in Section 4, focusing on production data.
3 Study 2: Encoder-Decoder inflection
Our second study trains an encoder-decoder (ED) model on the task of German plural inflection, following the method of Kirov and Cotterell (K&C). We then compare its predictions on the M95 stimuli to the behavior of participants in Study 1.
Wug testing and computational models Wug tests have also been used to evaluate how computational models generalize, although the appropriate method of comparison to speakers is still under debate. Albright and Hayes (2003) collected spoken productions and acceptability ratings of past tense inflections for English nonce verbs, comparing the prevalence of regular inflection (e.g.  $rife \rightarrow rifed$ ) to one or two pre-selected irregular forms for each nonce verb (e.g.  $rife \rightarrow rofe$ ,  $riff$ ). They then evaluated two different computational models on their wug data, focusing on correlation between model scores and participant ratings to select a rule-based learner as the best-performing model. K&C also tested their ED model on Albright and Hayes' nonce words and evaluated performance using correlation with model scores; however, instead of the rating data, they focused on production probabilities: the percentage of speakers who produced each pre-selected irregular form. Corkery et al. (2019)
call this methodology into question, noting that different random initializations of the ED model lead to highly variable rankings of the output forms, and thus to unstable correlation metrics. Instead, they correlate the speaker production probabilities to the aggregated predictions of models with different random seeds, treating each model instance as simulating a unique "speaker". Our study follows the latter approach: we aggregate production probabilities over several model initializations and compare these results to the speaker production data.
Modeling German plurals The same M95 stimuli used in our Study 1 have also been applied to wug test computational models. To date, no computational studies have reproduced the high /-s/ preference reported for participants in the original rating study. Hahn and Nakisa (2000) framed the problem as a classification task, mapping noun inputs to their plural classes. They trained a "single-route" exemplar-based categorization model (Nosofsky, 1988) alongside a "dual-route" version of the same model, which had an additional symbolic rule component to handle the /-s/ class. Hahn and Nakisa also collected their own speaker productions of the M95 wug stimuli, and found that the single-route model showed a higher overall correlation to speaker production probabilities, relative to the dual-route model. They did not explicitly compare model and speaker behavior on Rhymes versus Non-Rhymes, so we don't know whether the model learned speaker-like generalizations for phonologically atypical stimuli, or whether the model could achieve similar performance on the more challenging task of sequence prediction.
Goebel and Indefrey (2000) used a simple recurrent network (Elman, 1990) for sequence prediction on the M95 wug stimuli. The model did produce  $/-\mathrm{s}/$  more often for Non-Rhymes than Rhymes, but as the overall production of  $/-\mathrm{s}/$  was relatively low, the authors did not consider this evidence of default behavior. Instead, they find that the model learns to condition regular plural inflection on grammatical gender. For both Rhymes and Non-Rhymes, the model predicted  $/-(\mathrm{e})\mathrm{n}/$  when the input was preceded by the feminine article die, and  $/-\mathrm{e}/$  when the input began with masculine der; neuter das was not tested. Goebel and Indefrey reanalyze the original M95 rating data and argue that its results are hypothetically consistent with the model's behavior; they conclude that  $/-\mathrm{s}/, /-(\mathrm{e})\mathrm{n}/,$  and  $/-\mathrm{e}/$  are all reg-
8"Hypothetically" because M95 did not report results split by grammatical gender.
$^{6}$ Hahn and Nakisa (2000) reanalyze the M95 ratings and find that  $/-\mathrm{s}/$  is rated much higher for family surnames than other kinds of names within the "name" condition (e.g. first names), reflecting the strong link between this category and the  $/-\mathrm{s}/$  plural class.
Further analysis indicates that individual survey participants rated a plural form they did not produce as better than the form they did produce in fully one-third of cases.
Plural	% All	Neut	M95 R	1 Syll
/-(e)n/	37.3	3.2	13.9	14.0
/-e/	34.4	51.9	72.6	66.5
/-∅/	19.2	21.5	0.5	1.4
/-er/	2.9	10.6	7.3	4.7
/-s/	4.0	7.7	3.1	12.5
other	2.1	5.1	2.6	.9
N	11,243	2,606	642	570
ular plural classes in German, with the latter two conditioned on grammatical gender. These findings show the importance of controlling for grammatical gender in comparing speaker and model results.