How “deep” is learning word inflection?
Authors: Franco Alberto Cardillo, Marcello Ferro, Claudia Marzi, Vito Pirrelli (CNR-ILC, 2017)
Topic: Lexical and Semantic Resources and Analysis

Abstract
Machine learning offers two basic strategies for morphology induction: lexical segmentation and surface word relation. The first one assumes that words can be segmented into morphemes. Inducing a novel inflected form requires identification of morphemic constituents and a strategy for their recombination. The second approach dispenses with segmentation: lexical representations form part of a network of associatively related inflected forms. Production of a novel form consists in filling in one empty node in the network. Here, we present the results of a recurrent LSTM network that learns to fill in paradigm cells of incomplete verb paradigms. Although the process is not based on morpheme segmentation, the model shows sensitivity to stem selection and stem-ending boundaries.

Introduction
Morphological induction can be defined as the task of singling out morphological formatives from fully inflected word forms. These formatives are understood to be part of the morphological lexicon, where they are accessed and retrieved, to be recombined and spelled out in word production. The view requires that a word form be segmented into meaningful morphemes, each contributing a separable piece of morpho-lexical content. Typically, this holds for regularly inflected forms, as with Italian cred-ut-o 'believed' (past participle, from \textsc{credere}), where cred- conveys the lexical meaning, and -ut-o is associated with morpho-syntactic features. A further assumption is that there always exists an underlying base form upon which all other forms are spelled out. In an irregular verb form like Italian appes-o 'hung' (from \textsc{appendere}), however, it soon becomes difficult to separate morpholexical information (the verb stem) from morpho-syntactic information. A different formulation of the same task assumes that the lexicon consists of fully-inflected word forms and that morphology induction is the result of finding out implicative relations between them. Unknown forms are generated by redundant analogy-based patterns between known forms, along the lines of an analogical proportion such as: rendere `make' :: reso `made' = appendere `hang' :: appeso `hung'. Support to this view comes from developmental psychology, where words are understood as the foundational elements of language acquisition, from which early grammar rules emerge epiphenomally AUTHOR. After all, children% are exposed to fully inflected forms in acquisition, and have no privileged access to underlying base forms. Besides, they {} appear to be extremely sensitive to subregularities holding between inflectionally-related forms AUTHOR. Further support is lent by neurobiologically inspired computer models of language, blurring the traditional dichotomy between processing and storage AUTHOR. In particular we will consider here the consequences of this view on issues of word inflection by recurrent Long Short Term Memory (LSTM) networks AUTHOR.

Conclusion
To meaningfully assess the relative computational difficulty of the cell-filling task, we calculated a simple baseline performance, with 695 forms of our original datasets selected for training, and 55 for testing. For this purpose, we used the baseline system for Task 1 of the CoNLL- SIGMORPHON-2017 Universal Morphological Reinflection shared task.5 The model changes the infinitive into its inflected forms through rewrite rules of increasing specificity: e.g. two Italian forms such as badare ‘to look after’ and bado ‘I look after’ stand in a BASE :: PRES _ IND _3 S relation. The most general rule changing the former into the latter is -are -> -o, but more specific rewrite rules can be extracted from the same pair: -dare -> -do, -adare -> -ado, -badare -> -bado. The algorithm then generates the PRES _ IND _3 S of - say - diradare ’thin out’, by using the rewrite rule with the longest left-hand side matching diradare (namely -adare > -ado). If there is no matching rule, the base is used as a default output. The algorithm proves to be effective for regu- lar forms in both languages (Table 2). However, per-word accuracy drops dramatically on German irregulars (0.23), and Italian irregulars (0.5). The same table shows accuracy scores on test data obtained by running 128, 256 and 512 LSTM blocks. Each model instance was run 10 times, and overall per-word scores are averaged across repetitions. The CoNLL baseline is reminiscent of AUTHOR Minimal Generalization Learner, inferring Italian infinitives from first singular present indicative forms AUTHOR. In the present case, however, the inference goes from the infinitive (base) to other paradigm cells. The inference is much weaker in German, where stem allomorphy is more consistently distributed within each paradigm. In Appendix, Table 3 contains a list of all German forms wrongly produced by the CoNLL baseline, together with per-word accuracy of our models. Most wrong forms are inflected forms requiring ablaut, which turn out to be over-regularised by the CoNLL baseline (e.g. *stehtet for standet, *beginntet for begannt). It appears that, in German, a purely syntagmatic approach to word production, deriving all inflected forms from an underlying base, has a strong bias towards over-regularisation. Simply put, the orthotactic/phonotactic structure of the German stem is less criterial for stem allomorphy than the Italian one. LSTMs are considerably more robust in this respect. Memory resources allowing, they can keep track of local syntagmatic constraints as well as more global, paradigmatic constraints, whereby all paradigmatically-related forms contribute to fill in gaps in the same paradigm. For example, knowledge that a paradigm contains a few stem allomorphs is good reason for an LSTM to produce a stem allomorph in other (empty) cells. The more systematic the distribution of stem alternants is across the paradigm, the easier for the learner to fill in empty cells. German conjugation proves to be paradigmatically well-behaved. An LSTM recurrent network has no information about the morphological structure of input forms. Due to the predictive nature of the production task and the LSTM re-entrant layer, however, the network develops a left-to-right sensitivity to upcoming symbols, with per-symbol accuracy being a function of the network confidence about the next output symbol. To assess the correlation between per-symbol accuracy and “perception” of the morphological structure, we used a Linear Mixed Effects (LME) model of how well structural features of German and Italian verb forms interpolate the “average” network accuracy in producing an upcoming symbol (1 for a hit, 0 for a miss) in both training and test. The marginal plots of Figure 2 show that there is a clear structural effect of the distance to the stem-ending boundary of the symbol currently being produced, over and above the length of the input string. Besides, stems and suffixes of regulars exhibit different accuracy slopes compared with stems and suffixes of irregulars. Intuitively, production of an inflected form by a LSTM network is fairly easy at the beginning of the stem, but it soon gets more difficult when approaching the morpheme boundary, particularly with irregulars. Accuracy reaches the minimum value on the first symbol of the inflectional ending, which marks a point of structural discontinuity in an inflected verb form. From that position, accuracy starts increasing again, showing a characteristically V-shaped trend. Clearly, this trend is more apparent with test words (Figure 2, bottom), where stems and endings are recombined in novel ways. The same results hold for German. On the other hand, no evidence of structure sensitivity was found in a LME model of the baseline output for both German and Italian. The cell-filling problem is an ecological, developmentally motivated task, based on evidence of fully inflected forms. Although other (simpler) models have been proposed to account for form- meaning mapping in Morphology (AUTHOR; AUTHOR, among oth- ers), we do not know of any other artificial neural networks that can simulate word inflection as a cell-filling task. Unlike more traditional connectionist architectures AUTHOR, recurrent LSTMs do not presuppose the existence of underlying base forms, but they learn possibly alternating stems upon exposure to full forms. Admittedly, the use of orthogonal one-hot vectors for lemmas, unigram temporal series for inflected forms, and abstract morphosyntactic features as a proxy of context-sensitive functional agreement effects, are crude representational short-hands. Nonetheless, in tackling the task, LSTMs prove to be able to orchestrate “deep” knowledge about word structure, well beyond pure surface word relations: namely stem- affix boundaries, paradigm organisation and degrees of regularity in stem formation. Acquisition of different inflectional systems may require a different balance of all these pieces of knowledge.