SlowGuess/ACL_2020_OCR huggingface dataset:
Overview We model German number inflection using the sequence-to-sequence Encoder-Decoder architecture (Sutskever et al., 2014). This comprises a recurrent neural network (RNN) which reads in an input sequence and encodes it into a fixed-length vector representation, and another RNN which incrementally decodes that representation into an output sequence. Following Kann and Schütze (2016), our decoder uses neural attention (Bahdanau et al., 2015).
For our task of morphological transduction, the ED model takes character-level representations of German nouns in their singular form as inputs (e.g.  $\langle m\rangle$  HUND  $\langle eos\rangle$ ), and learns to produce the noun's inflected plural form (e.g. HUNDE  $\langle eos\rangle$ ). Each character sequence starts with  $\langle m\rangle$ ,  $\langle f\rangle$ , or  $\langle n\rangle$ , to indicate grammatical gender. Unlike English, the phonological-orthographic mapping is straightforward in German, so we can use a written corpus for model training. We keep a held-out dev set for hyperparameter selection, and a held-out test set to assess the model's accuracy in generalizing to unseen German nouns. In addition, the 24 M95 nouns were used for comparison with speaker behavior. They were presented to the model as neuter gender, consistent with Study 1.
Corpus We trained all models on the UniMorph German data set $^{9}$  (Kirov et al., 2016; Sylak-Glassman et al., 2015), which provides the singular and plural forms of 11,243 nouns. Only nominative case forms were used. Grammatical gender was
Table 3: Distribution (percentages) of plural class for 1) nouns overall, 2) only neuter nouns, 3) nouns rhyming with M95 stimuli, 4) one-syllable nouns from Unimorph German dataset (Kirov et al., 2016).
Train	Dev	Test
99.9% (8694)	92.1% (1229)	88.8% (1320)
Table 4: Model accuracy (N) by UniMorph corpus split, averaged over 25 random initializations.
obtained by merging the Unimorph dataset with a more recent Wiktionary scrape containing this feature. $^{10}$  Table 3 gives the distribution of plural suffixes for the UniMorph corpus overall, and for three relevant subsets: nouns with neuter gender, monosyllabic nouns (like the M95 stimuli), and nouns which were phonologically similar to the M95 stimuli, i.e. shared a rhyme. The number of items in the train, dev, and test splits is shown (in parentheses) in Table 4.
Implementation Following K&C and Corkery et al. (2019), our model is implemented using OpenNMT (Klein et al., 2018) with their reported hyperparameters (after Kann and Schütze, 2016): 2 LSTM encoder layers and 2 LSTM decoder layers, 300-dimensional character embeddings in the encoder, and 100-dimensional hidden layers in both encoder and decoder; Adadelta optimization for training with a batch size of 20 and inter-layer dropout rate of 0.3; and a beam size of 12 for decoding during evaluation.
Since Corkery et al. (2019) found the ED model to be highly sensitive to initialization, we trained multiple simulations with the same architecture, varying only the random seed. Reported results combine predictions from 25 separate random initializations. The one hyperparameter we tuned was early stopping. Best performance on the validation set was achieved at 10 epochs, which was sufficient to memorize the training data.
Results The model achieves  $88.8\%$  accuracy on the held-out test set (Table 4). It performs best on  $/-(\mathrm{e})\mathrm{n}/$ , the most frequent class (Table 5). Unsurprisingly, the worst performance appears on the 'other' category, which comprises the long tail of idiosyncratic forms which must be memorized (e.g. Latinate plurals Abstraktum  $\rightarrow$  Abstrakta or other borrowings Zaddik  $\rightarrow$  Zaddikim). In keeping with the findings of Hahn and Nakisa (2000),  $/-\mathrm{s}/$  is the plural suffix with the worst generalization perfor
10https://github.com/gambolputty/ German-nouns/ To ensure our results were not limited by the small size of the UniMorph dataset, we also trained the model on this larger dataset, including about 65,000 nouns. As the outcome was consistent with our findings here, we report results from the smaller model.
<sup>9</sup>https://github.com/unimorph/deu
Figure 1: Plural class productions by item.
Test	M95
Prec.	Rec.	F1	%R	%NR	ρ
/-(e)n/	.95	.95	.95	6.3	3.3	.28
/-e/	.86	.89	.87	68.3	91.7	.13
/-∅/	.96	.91	.92	0	0
/-er/	.83	.85	.84	21.7	2.7	.05
/-s/	.64	.56	.60	3.7	2.3	.33
other	.37	.48	.42	0	0
Table 5: Model results by plural suffix for: (left) test set performance (averaged over plural seed); (right) production percentages for rhyme (R) and non-rhyme (NR) M95 stimuli, and correlation (Spearman's  $\rho$ ) to speaker productions.
mance; this cannot be attributed to low frequency alone (c.f. Table 3), as the model does much better on the similarly rare suffix /-er/.
We use the M95 stimuli to compare model predictions to speaker data from Study 1. The model shows an overwhelming preference for  $/ - \mathrm{e} /$  on these words (Table 5); roughly  $80\%$  of its productions are  $/ - \mathrm{e} /$  relative to  $45\%$  of speaker productions (Figure 1). In contrast, the model rarely predicts  $/ - (\mathrm{e})\mathrm{n} /$  , which speakers use  $30\%$  of the time. The model's treatment of Rhymes and Non-Rhymes is even farther off the mark: where speakers use  $/ - (\mathrm{e})\mathrm{n} /$  and  $/ - \mathrm{s} /$  more for Non-Rhymes relative to Rhymes, the ED model uses them less, producing  $/ - \mathrm{e} /$  for over  $90\%$  of Non-Rhymes. Following K&C and Corkery et al. (2019), we calculate the Spearman rank correlation coefficient (Spearman's  $\rho$
between model and speaker production probabilities within inflectional categories rather than across categories. $^{11}$  This means that, for each potential plural suffix, we compare speaker and model productions for that suffix on each individual M95 word. Table 5 reports the correlation for each suffix. None show a statistically significant difference from the null hypothesis of no correlation.
Figure 2 shows the distribution of plural classes in the top 5 most likely forms predicted by the model for each M95 word. While all of the model's top-ranked predictions are well-formed outputs in the sense that they conform to one of the main German plural classes, the lower-ranked predictions are rapidly dominated by "other" forms which do not cohere to standard plural production. An example from one model instance: the Rhyme input Spert had as its top five predictions Sperte, Spelte, Spente, Sperten, and Fspern; the Non-Rhyme input Bneik had Bneiken, Bneiks, Bneikke, Bneikz, and Bneikme. Corkery et al. (2019) observed instability in the ranking of irregular forms in ED models trained on the English past tense; however, English irregular forms are very diverse, which makes it difficult to draw broad conclusions about the plausibility of lower-ranked forms in the model's output. In contrast, the five main plural suffixes for German cover  $98\%$  of the nouns in the UniMorph dataset,
<sup>11</sup>For the English analyses in the prior works, this means calculating separate correlations for regular and irregular forms.
Figure 2: Distribution of plural classes by rank in ED model output.
and  $95\%$  of speaker productions on M95 stimuli in Study 1. The predominance of ill-formed plurals in lower-ranked predictions<sup>12</sup> suggests ED model scores may not be cognitively plausible analogues to speaker behavior; if they were, we would expect forms with standard plural inflections to receive consistently high rankings.
The current study asks whether modern Encoder-Decoder neural models learn the full set of correct generalizations — that is, human-like behavior — with respect to German number inflection, which requires the learner to generalize non-majority inflectional classes. The short answer is no: our model learns part of that set. In particular, it correctly identifies  $/ - \mathrm{e}/$  as the 'best' plural class for this context.  $/ - \mathrm{e}/$  is the most frequent class in the training data for similar inputs (neuter gender, monosyllabic, phonologically close to M95; c.f. Table 3), and it is also the plural suffix most frequently produced by speakers (Table 2). Like all plural classes,  $/ - \mathrm{e}/$  does not characterize a majority of German nouns overall (Table 1), so the model has technically learned to generalize a minority class in its appropriate context. Nonetheless, it does not reproduce the behavior of survey participants in response to the same stimuli, which shows a more variable distribution over plural classes and different generalization patterns for Non-Rhymes relative to Rhymes.
This outcome is not surprising when one considers that the model is trained to produce one correct form rather than a distribution over plausible forms; however, this is exactly the task faced by human language learners as well. All the models of morphology discussed here assume that exposure to correct forms alone should suffice for learning speaker-like behavior. Corkery et al. (2019, 3872, fn. 4) note that training on single target forms produces highly skewed ED model scores, with a great deal of probability mass on the top-ranked form and instability in lower rankings, but that training on a distribution would not be a cognitively plausible alternative. However, it could be the case that German speakers do regularly encounter variable realizations of plural forms. Kopcke observes that German plural inflection shows regional variation, for example northern speakers using /-s/ (die Mädels 'girls') where southern dialects prefer /-(e)n/ (die Mädeln). Incorporating dialect-informed variability into training might be one way to encourage neural models toward speaker-like generalization.[13]
Parallel issues arise for model evaluation: how should we evaluate models of production when the target output is a distribution? On simplified versions of the task, such as classification (Hahn and Nakisa, 2000), the output distribution is constrained within a space of plausible forms, but sequence-to-sequence models deal with the open-ended domain of all possible strings. For
<sup>13</sup>Like previous studies on these stimuli, our Study 1 did not collect data on speakers' dialect background; we are addressing this issue in follow-up research. We note that Study 1 began with an onboarding task prompting speakers to inflect existing nouns in Modern High German, which hopefully primed use of the standard variety for the following tasks.
12 Interestingly, while less frequent classes such as  $/ - \mathrm{s}/$  and  $/ - \emptyset /$  appear more often in the model's lower-ranked outputs, the class  $/ - (\mathrm{e})\mathrm{n}/$  is almost never predicted — despite being the second most frequent class in speaker data productions.
encoder-decoders, the likelihood scores produced during beam-search decoding offer an intuitive option, and K&C use these scores to evaluate their model with respect to Albright and Hayes' wug data; however, Corkery et al. (2019) demonstrate that these model scores are not a suitable metric for that comparison. Other recent research has highlighted the limitations of both beam search and model scores globally in neural sequence-to-sequence models (Stahlberg and Byrne, 2019). Our results provide further evidence that lower-ranked ED predictions do not reflect cognitively plausible distributions: they contain many ill-formed outputs, and omit inflectional classes such as  $/-(\mathrm{e})\mathrm{n}/$  which is prevalent in speaker productions. An alternative to model scores is to treat each randomly initialized instance of a model as an individual, and compare aggregate productions with speaker data (Goebel and Indefrey, 2000; Corkery et al., 2019). For our experiments, this did not produce the distribution observed in the speaker data. The discrepancy between speaker production and rating preferences poses another challenge, as it's not clear how the ED model might represent these different task modalities.
Beside variability, the other key discrepancy between speaker and ED behavior is the treatment of Non-Rhyme words. If German has a default plural class, it should be realized more often on these phonologically atypical stimuli than the more familiar Rhyme words. Speakers in Study 1 use  $/-\mathrm{s}/$  and  $/-(\mathrm{e})\mathrm{n}/$  more for Non-Rhymes than for Rhymes. These results are consistent with earlier studies: M95 found that  $/-\mathrm{s}/$  was the only plural form to receive higher average ratings for Non-Rhymes compared to Rhymes, and Z&L found that speakers produced both  $/-(\mathrm{e})\mathrm{n}/$  and  $/-\mathrm{s}/$  more often for Non-Rhymes. In contrast, the ED model appears to treat  $/-\mathrm{e}/$  as a default, producing  $/-\mathrm{e}/$  inflections for under  $70\%$  of Rhymes but over  $90\%$  of Non-Rhyme inputs. This asymmetry suggests that the model has not induced the full set of correct generalizations for German plural inflection — it has not recognized which plural classes are more productive for phonologically atypical nouns. In fact, the model's preference for  $/-\mathrm{e}/$ , the most frequent (if non-majority) suffix, is the behavior anticipated by M95: "frequency in the input to a pattern associator causes a greater tendency to generalize" (1995, 215). It seems that the productivity of less frequent inflectional classes continues to challenge neural models and limit their cognitive application.
German number inflection has been claimed to have distributional properties which make it difficult for neural networks to model. Our experimental speaker data does not necessarily support all of these claims; in particular,  $/-\mathrm{s}/$  does not appear to be the only plural suffix which speakers treat as a 'default' for phonologically unfamiliar words, as the more frequent marker  $/-(\mathrm{e})\mathrm{n}/$  shows similar trends. Nonetheless, the German plural system continues to challenge ED architectures. Our neural model struggles to accurately predict the distribution of  $/-\mathrm{s}/$  for existing German nouns. On novel nouns, it generalizes the contextually most frequent plural marker  $/-\mathrm{e}/$ ; its predictions are less variable than speaker productions, and show different patterns of response to words which are phonologically typical (Rhymes) as opposed to atypical (Non-Rhymes). Regardless of the minority-default question, it seems that ED models do not necessarily function as good cognitive approximations for inflectional systems like German number, in which no class holds the majority.
The authors thank Yevgen Matushevych, Maria Corkery, Timothy O'Donnell, the Agora reading group at Edinburgh, and the ACL reviewers for helpful feedback. This work was supported in part by the EPSRC Centre for Doctoral Training in Data Science, funded by the UK Engineering and Physical Sciences Research Council (grant EP/L016427/1) and the University of Edinburgh. This work was also supported by a James S McDonnell Foundation Scholar Award (#220020374) to the second author.
Adam Albright and Bruce Hayes. 2003. Rules vs. analogy in English past tenses: A computational/experimental study. Cognition, 90(2):119-161.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR.
Jean Berko. 1958. The Child's Learning of English Morphology. WORD, 14(2-3):150-177.
Joan Bybee. 1995. Regular morphology and the lexicon. Language and Cognitive Processes, 10(5):425-455.
Harald Clahsen. 1999a. The dual nature of the language faculty. Behavioral and Brain Sciences, 22(6):1046-1055.
Harald Clahsen. 1999b. Lexical entries and rules of language: A multidisciplinary study of German inflection. Behavioral and Brain Sciences, 22(6):991-1013.
Harald Clahsen. 2016. Contributions of linguistic typology to psycholinguistics. Linguistic Typology, 20(3).
Harald Clahsen, Monika Rothweiler, Andreas Woest, and Gary F. Marcus. 1992. Regular and irregular inflection in the acquisition of German noun plurals. Cognition, 45(3):225-255.
Maria Corkery, Yevgen Matushevych, and Sharon Goldwater. 2019. Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 3868-3877, Florence, Italy. Association for Computational Linguistics.
Wolfgang U Dressler. 1999. Why collapse morphological concepts? Behavioral and Brain Sciences, 22(6):1021-1021.
Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179-211.
Hilke Elsen. 2002. The acquisition of German plurals. In Morphology 2000: Selected Papers from the 9th Morphology Meeting, Vienna, 24-28 February 2000, number v. 218 in Amsterdam Studies in the Theory and History of Linguistic Science, pages 117-127. J. Benjamins, Amsterdam; Philadelphia.
Rainer Goebel and Peter Indefrey. 2000. A recurrent network with short-term memory capacity learning the German-s plural. Models of language acquisition: Inductive and deductive approaches, pages 177-200.
Ulrike Hahn and Ramin Charles Nakisa. 2000. German Inflection: Single Route or Dual Route? Cognitive Psychology, 41(4):313-360.
Borja Herce. 2019. Deconstructing (ir)regularity. Studies in Language, 43(1):44-91.
Peter Indefrey. 1999. Some problems with the lexical status of nondefault inflection. Behavioral and Brain Sciences, 22(6):1025-1025.
Katharina Kann and Hinrich Schütze. 2016. Single-Model Encoder-Decoder with Explicit Morphological Representation for Reinflation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 555–560, Berlin, Germany. Association for Computational Linguistics.
Christo Kirov and Ryan Cotterell. 2018. Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate. Transactions of the Association for Computational Linguistics, 6:651-665.
Christo Kirov, John Sylak-Glassman, Roger Que, and David Yarowsky. 2016. Very-large Scale Parsing and Normalization of Wiktionary Morphological Paradigms. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3121-3126, Portorož, Slovenia. European Language Resources Association (ELRA).
Guillaume Klein, Yoon Kim, Yuntian Deng, Vincent Nguyen, Jean Senellart, and Alexander M. Rush. 2018. OpenNMT: Neural Machine Translation Toolkit. arXiv:1805.11462 [cs].
Klaus-Michael Köpcke. 1988. Schemas in German plural formation. *Lingua*, 74(4):303-335.
Klaus-Michael Koppke. 1998. The acquisition of plural marking in English and German revisited: Schemata versus rules. Journal of Child Language, 25(2):293-319.
Gary F Marcus, Ursula Brinkmann, Harald Clahsen, Richard Wiese, and Steven Pinker. 1995. German inflection: The exception that proves the rule. Cognitive psychology, 29(3):189-256.
Robert M. Nosofsky. 1988. Exemplar-based accounts of relations between classification, recognition, and typicality. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14(4):700-708.
Steven Pinker and Alan Prince. 1988. On language and connectionism: Analysis of a parallel distributed processing model of language acquisition. Cognition, 28(1-2):73-193.
DE Rumelhart and J McClelland. 1986. On Learning the Past Tenses of English Verbs. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, pages 216-271. MIT Press, Cambridge, MA.
Mark S. Seidenberg and David C. Plaut. 2014. Quasiregularity and Its Discontents: The Legacy of the Past Tense Debate. Cognitive Science, 38(6):1190-1228.
Ingrid Sonnenstuhl and Axel Huth. 2002. Processing and Representation of German -n Plurals: A Dual Mechanism Approach. *Brain and Language*, 81(1-3):276-290.
Felix Stahlberg and Bill Byrne. 2019. On NMT Search Errors and Model Errors: Cat Got Your Tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3354-3360, Hong Kong, China. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems, pages 3104-3112.
John Sylak-Glassman, Christo Kirov, David Yarowsky, and Roger Que. 2015. A Language-Independent Feature Schema for Inflectional Morphology. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 674-680, Beijing, China. Association for Computational Linguistics.
Richard Wiese. 1996. The Phonology of German. Oxford University Press on Demand.
Charles D. Yang. 2016. The Price of Linguistic Productivity: How Children Learn to Break the Rules of Language. The MIT Press, Cambridge, Massachusetts.
Eugen Zaretsky and Benjamin P Lange. 2016. No matter how hard we try: Still no default plural marker in nonce nouns in Modern High German. In A Blend of MaLT: Selected Contributions from the Methods and Linguistic Theories Symposium 2015, number Band 15 in Bamberger Beiträge Zur Linguistik, pages 153-178. University of Bamberg Press, Bamberg.
A Study design
A.1 Stimuli
Table 6 provides the complete list of nouns used in the experiment.
Rhymes	Non-rhymes
Bral	Bnaupf
Kach	Bneik
Klot	Bnöhk
Mur	Fnahf
Nuhl	Fneik
Pind	Fnohk
Pisch	Plauff
Pund	Pleik
Raun	Pläk
Spand	Pnähf
Spert	Pröng
Vag	Snauk
Table 6: Experimental stimuli (Marcus et al., 1995)
A.2 Procedure
We designed an online survey comprising three sections, in order of presentation: 1) an introductory production task with existing German words, 2) a nonce-word production task, and 3) a nonce-word rating task. For the introductory production task, eight existing German nouns were used, one from each of the eight plural classes under consideration. The goal of this section was to familiarize participants with the task of producing the plural, and avoid biasing them toward any particular plural marker by showing all eight options. We also hoped that inflecting nouns in Modern High German would encourage participants to approach the following tasks with the standard variety primed, thus reducing the possible effects of dialectal variation. For the second and third sections, the production and rating tasks, the twenty-four M95 nonce words were presented. All stimuli were presented with neuter grammatical gender in the nominative case. In all tasks, each noun was preceded by the article  $Das$ , indicating neuter gender and singular number, and each prompt for participant responses was preceded by  $Die$ ..., to indicate plural number. The eight existing nouns presented in the introductory production task were selected for neuter gender, so they followed this pattern as well.
We recruited 192 participants through the online survey platform Prolific $^{14}$ , using the site's demo
graphic filters to target native German speakers. Participants were additionally asked about their age and exposure to languages other than German within the survey. Participants were shown the three tasks, introduction, production, and rating, in order, meaning that participants had to produce a plural form for all 24 nonce words before performing the rating task. For the production task, participants saw the noun on its own, preceded by Das, e.g. Das Bral. Above the response box, the text Die... appeared, to indicate that a plural form of the noun should be typed into the response box below the text. For the rating task, participants were prompted to rate each potential plural on a Likert scale of Sehr gut ('very good'; 5) to Sehr schlecht ('very bad'; 1). After filtering out 42 respondents who failed a preliminary attention check, data from 150 participants was available for analysis. The cleaned, anonymized survey data will be published online along with this paper.
14http://www.prolific.com