SlowGuess/NAACL_2021_OCR huggingface dataset:
Miikka Silfverberg $^{3}$  Francis Tyers $^{1}$  Garrett Nicolai $^{3}$  Mans Hulden $^{\chi}$
$^{3}$ University of British Columbia Indiana University
$\chi_{\text{University of Colorado}}$
first.last@ubc.ca ftyers@iu.edu first.last@colorado.edu
Sequence-to-sequence models have delivered impressive results in word formation tasks such as morphological inflection, often learning to model subtle morphophonological details with limited training data. Despite the performance, the opacity of neural models makes it difficult to determine whether complex generalizations are learned, or whether a kind of separate rote memorization of each morphophonological process takes place. To investigate whether complex alternations are simply memorized or whether there is some level of generalization across related sound changes in a sequence-to-sequence model, we perform several experiments on Finnish consonant gradation—a complex set of sound changes triggered in some words by certain suffixes. We find that our models often—though not always—encode 17 different consonant gradation processes in a handful of dimensions in the RNN. We also show that by scaling the activations in these dimensions we can control whether consonant gradation occurs and the direction of the gradation.
Recent work on computational morphology demonstrates that neural networks can very effectively learn to inflect words, given adequate amounts of training data (Cotterell et al., 2016, 2017). However, in computational morphology and in NLP at large, the interpretability of neural models remains a serious concern (Doshi-Velez and Kim, 2017)—it is unclear how networks trained to inflect words actually accomplish their task. It is also unclear to which extent networks are able to learn linguistic generalizations from their input data instead of simply memorizing training examples and exhibiting a kind of nearest-neighbor behavior.
In this paper, we shed light on what kind of linguistic generalizations neural networks are capable of learning from data. We report on an in
Figure 1: Scatter plot of the activation for two encoder hidden state dimensions which activate strongly during gradation. The letters  $\mathbf{p}$ ,  $\mathbf{t}$  and  $\mathbf{k}$  refer to examples which undergo gradation for the respective consonants. Lower-case characters mark direct (weakening) gradation and upper-case characters inverse (strengthening) gradation (see Section 3 for further details). Black dots mark examples which do not undergo gradation.
vestigation into how consonant gradation, a particular morphophonological alternation which is common in Finnish and other Uralic languages, is encoded in the hidden states of an LSTM encoder-decoder model trained to perform word inflection. Specifically, we train character-based sequence-to-sequence models for inflection of Finnish nouns into the genitive case, an inflection type which commonly triggers consonant gradation.
Consonant gradation is a morphophonological alternation where voiceless stops  $\mathbf{p}$ ,  $\mathbf{t}$  and  $\mathbf{k}$  are lenited in certain positions (see Section 3 for further details). We first demonstrate that inflection networks tend to learn an abstract representation for consonant gradation, where the alternation is triggered by the same dimensions in encoder hidden states regardless of which stop  $\mathbf{p}$ ,  $\mathbf{t}$  or  $\mathbf{k}$  undergoes gradation. This echoes the treatment of gradation in linguistic literature (Hakulinen et al., 2004, §41) Nevertheless, we also find evidence that
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5501-5513 June 6-11, 2021. ©2021 Association for Computational Linguistics
this behavior is not universal and that networks can sometimes fail to generalize gradation and instead learn to represent gradation using distinct dimensions for each stop  $\mathbf{p}$ ,  $\mathbf{t}$  and  $\mathbf{k}$ .
Our second contribution is to show that networks can learn a general representation encompassing both so-called quantitative gradation and qualitative gradation (these are further described in Section 3). This presents further evidence that the phonological representations learned by encoder-decoder models can learn to group linguistic generalizations that target different sounds.
As our third contribution, we show evidence of a remarkable property whereby directionality of gradation is encoded as positive or negative hidden state activations: Consonant gradation is called direct when the base form of a noun displays the strong grade (such as  $\mathbf{kk}$ ) and the genitive form displays the weak grade of a stop (such as  $\mathbf{k}$ ). In inverse (or 'strengthening') gradation, the opposite alternation occurs. We find hidden state dimensions which encode for the direction of gradation by a positive or negative activation. This behavior is demonstrated in Figure 1 where a negative activation of dimension 487 in the encoder hidden state marks inverse gradation of a stop, and positive activation instead marks direct gradation (see Section 6 for further discussion of this phenomenon).
Interpretation of neural representations in recurrent neural models has been an active area of research over a long period of time starting with Elman (1990). However, representations in models of phonology have received less attention than many other subfields of NLP. Rodd (1997) investigates learning of Turkish vowel harmony by a character-based RNN language model trained on word forms. The paper investigates hidden state activations of RNN models while varying the hidden state dimensionality between 1 and 4. It presents evidence that RNN hidden states can capture Turkish vowel harmony patterns when a sufficient number of hidden dimensions are available. In a similar vein, Silfverberg et al. (2018) investigate phoneme representations for Finnish, Spanish and Turkish finding correlations between embedding representations and phonological distinctive features. Kolachina and Magyar (2019) present an investigation of phone embeddings learned using word2vec (Mikolov et al., 2013) for simulated data showing
that phone embeddings capture phonemic and allophonic relationships. They also show that phone embeddings capture co-occurrence restrictions for vowels well, while largely failing to do this for consonants. Our encoder representations, in contrast, are able to capture these co-occurrence restrictions.
Begus (2020b) investigates representations learned by a generative adversarial network or GAN (Goodfellow et al., 2014) trained on audio recordings of speech, showing that some of the latent variables of the GAN correspond to phonological features of the speech signal: specifically the presence or absence of the fricative [s] in the output of the network and the amplitude of frication. They show that manipulation of the variables changes these features in a predictable manner. Similarly to our work, Begus (2020b) also scales state activations and observes the effect on the output of the network. In a related investigation of reduplication, Begus (2020a) train GAN models on speech and identify variables which trigger reduplication in the speech signal.
Extensive work exists on linguistic probing experiments for neural representations (Conneau et al., 2018a,b; Clark et al., 2019). A recent probing paper by Torroba Hennigen et al. (2020) is more directly related to our work. They present a decomposable probe for finding small sets of hidden states which encode for linguistically relevant information, particularly morphosyntactic information. Our work shares the aim of not only identifying if information is present in a neural system, but also examining how it is represented. However, we additionally perform experiments on manipulating network activations and examine how such manipulations influence the outputs of the network.
Our approach was inspired by the now-classic paper on visualization and interpretation of recurrent networks by Karpathy et al. (2015) in that we also seek individual interpretable dimensions. The work by Dalvi et al. (2019) on analyzing individual neurons in networks trained for linguistic tasks (POS tagging as well as semantic and morphological tagging) is more closely related to the present work. They present a general methodology for uncovering neurons which encode linguistic information by training a classifier to predict linguistic features of the input based on the representations generated by the network. They also show that it is possible to manipulate specific neurons to force the
Nominative	Genitive	Gloss
Quantitative	pappi	papin	'priest'
kenttä	kentän	'field'
kiukku	kiukun	'anger'
ripe	rippeen	'remain(s)'
laite	laitteen	'device'
liike	liikkeen	'motion'
Qualitative	sopu	soyun	'agreement'
johto	johdon	'lead'
aika	ajan	'time'
kyky	kyvyn	'skill'
olento	olennon	'creature'
kenkä	kengän	'shoe'
silta	sillan	'bridge'
rumpu	rummun	'drum'
ranne	ranteen	'wrist'
salko	salon	'pole'
Table 1: Examples of various kinds of consonant gradation in Finnish. Adding the genitive suffix -n closes the final syllable triggering gradation.
network to generate a particular linguistic feature. Meyes et al. (2019) investigate the effect of scaling network activations, something they call ablation studies. They train classifiers and look at how classification performance varies when zeroing out the activations for particular states or groups of states.
3 Consonant Gradation
Consonant Gradation (CG), common in many Uralic languages, is a set of assimilation and lenition processes, usually targeting the final syllable in a word stem. Historically the trigger for the alternation has been purely phonological, but in Finnish, the alternation is no longer entirely predictable from the phonological structure (Karlsson, 2017). The trigger for gradation is usually an affix that closes the final syllable, such as the genitive -n, e.g. katto  $\sim$  katon ('roof' sg. nom.  $\sim$  sg.2 gen.). The overall process is divided into quantitative gradation where, for example, geminate pp, tt, kk alternate with their non-geminate counterparts, p, t, k, and qualitative gradation where a large variety of lenition and assimilation processes are found. For example, strong grade k can alternate with the weakened j, v, g, etc. See Table 1 for a summary of these types of gradation processes found in our data set. The lenited or elided forms are commonly called the weak grade (e.g. katon) and the alter
nant the strong grade (e.g. katto). Sometimes the weak and strong grades appear in the inverse position, i.e. the weak grade appears with open syllables as in rike  $\sim$  rikkeen ('offense' sg. nom.  $\sim$  sg. gen.).
While quantitative gradation remains productive in the language, many stems from more recent loanwords in particular, do not tend to alternate qualitatively; for example auto  $\sim$  auton, *audon ('car' sg. nom.  $\sim$  sg. gen.). Speakers must therefore know the lexical status of each stem to inflect it correctly. Our data set includes both grading and non-gradating lexemes.
The advantages of studying Finnish consonant gradation in this context is that the set of sound changes is very diverse, but that the trigger for all of them is the same. Also, the Finnish writing system is very phonemic and surface-oriented and therefore no conversion to an IPA representation is necessary to reveal the sound changes that occur as a result of gradation.
Of particular interest to us is that there are many similar-looking alternations in Finnish that are not a result of consonant gradation, but paradigmatic variation. For example, varis ('crow' sg. nom.) is inflected variksen in the sg. gen. form. Note the similarity of this alternation to the actual CG case of liike ('motion' sg. nom.)  $\sim$  likkeen (sg. gen.) which also involves a  $\varnothing \sim \mathrm{k}$  alternation. It is therefore of some interest to observe whether neural inflection models encode the two cases differently in some respect.
In total we count 17 different types of lenition or fortition falling under the rubric of consonant gradation in our data set; an example of each type is shown in Table 1.
This section presents our nominative  $\rightarrow$  genitive inflection models and our approach to finding encoder hidden state dimensions which are associated with consonant gradation.
4.1 Inflection Models
As our inflection model, we use the well-known attentional BiLSTM encoder-decoder model which was presented by Bahdanau et al. (2014) and first applied to inflection by Kann and Schütze (2016). This neural model transduces a nominative input form which is represented as a sequence of characters  $x_{[1:T]}$  of length  $T$  into a genitive output form
Nevertheless, the alternation is fully determined by the phonological context in the nominative and genitive cases which are the focus of this paper. Therefore, we treat gradation as a phonological alternation.
2Underlining marks the position affected by gradation.
Quantitative gradation
Qualitative gradation
Figure 2: Activation of encoder hidden state dimension 487 in model 3 when (1) quantitative gradation, (2) qualitative gradation and (3) no gradation occurs in the input form. Red squares indicate positive and blue squares negative activation. Forms with strikethrough are inflected incorrectly by the decoder: *fonetikan (fonetiikan), *salkon (salon) and *bammun (bambun).
No gradation
$y_{[1:S]}$  of length  $S$
The encoder network in an attentional model generates one hidden state vector  $\mathbf{h}_t = \mathbf{f}_t \oplus \mathbf{b}_t \in \mathbb{R}^{2n}$  for every position in the input sequence. Due to the bidirectionality of the encoder, the hidden state vector is a concatenation of a forward state  $\mathbf{f}_t \in \mathbb{R}^n$  and a backward state  $\mathbf{b}_t \in \mathbb{R}^n$ . We refer to the vectors  $\mathbf{f}_t$  as hidden states and the elements in the vectors  $\mathbf{f}_t[d]$  as activations. Here  $d \in \{1, 2, 3, \dots, 2n\}$  is called a dimension.
4.2 Finding Dimensions Associated with Gradation
Our aim is to investigate encoder hidden state dimensions  $d$  which are associated with gradation. To this end, we extract the encoder hidden state activations  $\mathbf{h}_1[d]$ , ...,  $\mathbf{h}_T[d]$  for each example  $(x_{[1:T]},y_{[1:S]})$  in our development set and dimension  $d\in \{1,2,3,\dots,2n\}$ . In order to find dimensions which activate strongly at positions where gradation occurs, we compare the mean activation of each dimension for forms which undergo gradation and forms which do not.
Let  $a_{X}(d)$ , as defined by Equation (1) below, be the mean activation for dimension  $d$  in a set of encoder hidden states  $X$ . For each dimension  $d$ , we extract the mean activation  $a_{G}(d)$ , where  $G$  is the set of encoder hidden states at positions where gradation occurs. As explained in Section 3, gradation applies to the final stop in word forms which undergo gradation. Usually, this would refer to position  $T - 1$  in a string of length  $T$  as in tupa 'cottage sg. nom.', where  $\mathbf{p}$  undergoes gradation, but can also happen at position  $T - 2$  as in the form ratas 'wheel sg. nom.', where  $\mathbf{t}$  undergoes gradation.
$$
a _ {X} (d) = \frac {\sum_ {h \in X} \mathbf {h} [ d ]}{| X |} \tag {1}
$$
The mean activation  $a_{G}(d)$  is compared to the activation  $a_{N}(d)$  of dimension  $d$  at the penultimate position  $T - 1$  in base forms of length  $T$  which do not undergo gradation. In order to specifically capture dimensions which encode for gradation as opposed to simply encoding for consonants, we limit this examination to base forms like kana 'chicken sg. nom.' and auto 'car sg. nom.', where the penultimate character is a consonant. We retrieve the top-  $N$  dimensions  $d$  where the difference in mean activation  $|a_{N}(d) - a_{G}(d)|$  is maximized and consider these candidate dimensions for gradation.
Our dataset was produced by taking the most frequent 5,000 lexemes tagged as singular nominative nouns from the Turku Dependency Treebank (Haverinen et al., 2014) and generating the singular genitive forms using the OmorFi finite-state morphological transducer (Pirinen, 2015). We excluded compound nouns (e.g. ammatikorkeakoulutukinnoista 'from the professional high-school examinations') and words marked as nouns which contained punctuation or numerals (e.g. G8-neuvottelut 'G8 negotiations', 2000-luvulla 'in the 2000s',  $^\circ \mathrm{C}$  ssa 'in  $^\circ \mathrm{C}$  etc.). Loan words were included, both unadapted such as workshop and bungalow and partially or fully adapted such as brosyyri 'brochure' and sampanja 'champagne'. This gave a total of 4,797 nominative-genitive pairs. We randomly ordered them and then split these into disjoint sets:  $90\%$  for
Ungrad.	Grad.
Qual.	Quant.	Total
p	35	14	40	54
t	159	24	30	54
k	128	4	50	54
Total:	322	42	120	162
Table 2: Composition of the manually annotated validation set. The Ungrad. column refers to forms that do not exhibit gradation in the genitive, Grad. those forms that exhibit gradation and Qual. is qualitative and Quant. is quantitative gradation.
training (4,317 pairs) and  $10\%$  for validation.
We then took the validation set (479 pairs) and annotated them for: gradation (yes, no), type of gradation (qualitative, quantitative), consonant  $(\mathbf{p},$  t, k) and direction (direct, inverse). This gave a total of 84 examples of nouns exhibiting consonant gradation.
This set was heavily skewed towards t gradation (54 out of 84 examples). So we randomly sampled another 84 words from the frequency list, which were not found in the training data or in the existing validation set and which contained  $\mathbf{p}$  and  $\mathbf{k}$ , and annotated them and added them to the validation set. Statistics on the composition of the hand-annotated dataset can be found in Table 2 and the full data is freely available on GitHub.
We investigate representation of consonant gradation in encoder hidden states in the following way: As explained in Section 4.2, we identify individual dimensions in encoder hidden states which activate strongly during gradation regardless of the identity of the consonant undergoing gradation. We then investigate the association of these states using two experiments: we (1) perform significance tests on a held-out dataset to determine if the states activate significantly more strongly when gradation occurs, and (2) scale the state activations and observe the effect on the output of the network.
6.1 Training Details
We train ten encoder-decoder models with different random initializations for inflection using the OpenNMT toolkit (Klein et al., 2018). We use a 2-layer BiLSTM encoder with hidden dimension
Figure 3: Percent inflection accuracy on the development set for steps 500-3,000 during the training of our ten inflection models. The figure demonstrates that changes in accuracy on the development set are modest after step 2,000.
250. Due to the bidirectionality of the encoder this results in 500-dimensional hidden states (consisting of a forward and backward hidden state). Our model uses 500-dimensional character embeddings both in the encoder and decoder and we use an attentional decoder with 250-dimensional hidden states. The model is trained for a total of 3,000 steps using stochastic gradient descent and a batch size of 64. See Figure 3 for a plot of the development accuracy during the training process. As can be seen, changes in development accuracy are modes after training step 2,000.
We report inflection accuracy for our ten inflection models measured on held-out data in Table 3. The accuracy is reported separately for forms undergoing gradation and forms not undergoing gradation. In addition, we report an overall accuracy for all forms. We can see that the mean performance is close  $95\%$  for all forms and performance tends to be higher on forms undergoing gradation than other forms.
6.2 Investigation of State Activations
We randomly split our development set into two disjoint parts of equal size. The first part of the development set we use to discover the top-5 encoder hidden state dimensions which are strongly associated with gradation (as described in Section 4.2). The rest of the development set is used for significance testing. We perform a two-sided t-test to check if the mean activations of our top-5 dimensions differ significantly (at the  $99.5\%$  significance level) between positions which undergo gradation
3This follows character-level frequency patterns in Finnish, e.g. in the treebank t appears 122,821 times, k appears 64,513 times and p appears 23,130 times.
<sup>4</sup>https://github.com/mpsilfve/gradation
Model	# States	+Grad.	-Grad.	Overall
1	2	94.4	94.0	94.1
2	2	95.1	94.0	94.3
3	1	94.4	94.3	94.3
4	3	96.9	95.0	95.6
5	3	95.1	95.0	95.0
6	3	97.5	95.3	95.9
7	1	95.7	94.5	94.9
8	0	95.7	94.3	94.7
9	0	96.9	95.8	96.1
10	2	96.9	93.0	94.1
Table 3: Percent inflection accuracy for 10 NOM to GEN models trained using different random seeds. The column # States refers to the number of states found in Table 4 that have significant activations for all gradation types.
and positions which do not undergo gradation. As explained in Section 4.2, we limit this examination to nominative forms where the penultimate character is a consonant to better zone in on gradation.
Table 4 shows the results separately for  $\mathbf{p}$ ,  $\mathbf{t}$  and  $\mathbf{k}$  gradation. The table also shows results for qualitative and quantitative gradation. We can see that eight of the ten models contain at least one dimension where activation is significantly stronger for all stops  $\mathbf{p}$ ,  $\mathbf{t}$  and  $\mathbf{k}$  undergoing gradation than other stem-final consonants indicating that these states are associated with gradation in general rather than gradation of one of the individual consonants  $\mathbf{p}$ ,  $\mathbf{t}$ , or  $\mathbf{k}$ . We note that these dimensions also typically activate both for qualitative and quantitative gradation indicating that the network has learned an abstraction for both types of gradation.
6.3 Scaling State Activations
As a direct test of the effect of hidden state dimensions on gradation, we scale the activations of dimensions which are strongly associated with gradation. Our hypothesis is that negatively scaling these dimensions will prevent forms from undergoing gradation.
We experiment on a dataset consisting of all development examples which undergo gradation. For each nominative input form such as luukku, we identify the correct gold standard genitive form luukun (where  $\mathbf{kk} \to \mathbf{k}$  alternation has applied) and an alternate output form *luukkun which is correct apart from the fact that the form has not undergone gradation. We then compute (1) the number of gold standard forms, (2) the number of alternate forms, and (3) the number of nonce forms
generated by our models. Nonce forms here refer to erroneous outputs like *luukukuukkun which do not belong in category (2).
We scale the hidden state activations at positions where gradation occurs, that is at the final stop in the nominative form, before feeding the encoder hidden states into the decoder. For each input form, we scale the top- $N$  encoder hidden states which are associated with gradation according to the mapping  $a \mapsto x \cdot a$  where  $x$  varies between 1 and -25. The number of states which are scaled (that is  $N$ ) is tuned for maximal effect on the number of alternate forms which are generated.
Figure 4 shows the results for the scaling experiment when tuning  $N$ . The first graph shows that for most models the number of alternate forms first increases when the scaling factor  $x$  approaches -25, and then gradually decreases. As the number of alternate forms increases, the number of gold standard forms undergoing gradation naturally decreases as demonstrated by the second graph. We also see an increase in the number of nonce forms which do not belong to either category. This is to be expected as scaling represents a deviation from learned model weights which disturbs the network.