SlowGuess/NAACL_2021_OCR huggingface dataset:
The effect of scaling varies between models: When scaling activations for Model 9, over half of the output forms do not undergo gradation. In contrast, for Model 7, the best scaling factor only produces around  $7\%$  of non-gradating output forms. Crucially, however, we do see an effect for nearly all models (apart from model 8). Contrast this with Figure 5 which shows results when scaling a set of five random states instead of states which are associated with gradation, showing that scaling of randomly sampled states has very small if any effect on the number of alternate forms produced by the models. Based on the graphs in Figure 4, scaling has very limited effect on Model 8. Even when scaling by  $a = -25$ , there is only a small decrease in the number of gold standard forms and a corresponding small increase in nonce forms. This might be evidence of a more redundant representation of information in Model 8, whereby scaling a few states will not strongly perturb the network.
Figure 2 shows the activation for a hidden state dimension which is strongly associated with grada
For completeness, all results for scaling the top-  $N$  states where  $N$  varies between 1 and 5 are shown in Appendix A.
MODEL 1
Gradation	Dimension
458	252	395	489	396
K	0.247	0.162	0.048	0.156	0.649
P	0.291	0.350	0.390	0.283	0.056
T	0.175	0.062	0.022	0.043	0.193
Qual.	0.259	0.176	0.200	0.136	0.069
Quant.	0.252	0.218	0.199	0.194	0.245
MODEL 3
Gradation	Dimension
487	207	203	484	221
K	0.205	0.412	0.393	0.263	0.276
P	0.717	0.471	0.248	0.622	0.366
T	0.567	0.080	0.324	0.475	0.127
Qual.	0.684	0.254	0.268	0.118	0.249
Quant.	0.440	0.423	0.323	0.358	0.289
MODEL 5
Gradation	Dimension
345	109	408	390	191
K	0.272	0.206	0.775	0.041	0.225
P	0.466	0.381	0.105	0.498	0.183
T	0.498	0.186	0.057	0.161	0.257
Qual.	0.397	0.241	0.068	0.206	0.162
Quant.	0.395	0.296	0.298	0.224	0.222
MODEL 7
Gradation	Dimension
142	108	230	207	405
K	0.212	0.223	0.138	0.068	0.309
P	0.319	0.452	0.421	0.438	0.169
T	0.172	0.227	0.032	0.028	0.122
Qual.	0.180	0.099	0.148	0.170	0.166
Quant.	0.268	0.282	0.278	0.232	0.226
MODEL 9
Gradation	Dimension
446	459	438	380	351
K	0.184	0.375	0.487	0.157	0.109
P	0.416	0.293	0.142	0.283	0.379
T	0.208	0.030	0.075	0.124	0.135
Qual.	0.258	0.137	0.044	0.242	0.117
Quant.	0.293	0.314	0.282	0.196	0.220
Table 4: Mean differences in activation strength for dimension  $d$  where we first find the top-5 states associated with gradation using  $50\%$  of the development data and then perform significance tests using the remaining  $50\%$  of the development data. We present results for 10 different random initializations of model parameters. We compare activation when  $\mathbf{k}$ ,  $\mathbf{p}$  or  $\mathbf{t}$  gradation occurs to activation at -CV word endings where gradation does not occur. We also report results for qualitative and quantitative gradation irrespective of the consonant undergoing gradation. Statistically significant differences in activation strength at the  $99.5\%$  significance level are shown in bold face. Dimensions with significantly stronger association for all stops as well as qualitative and quantitative gradation are marked using a gray box.
MODEL 2
Gradation	Dimension
255	127	453	399	469
K	0.080	0.285	0.395	0.319	0.281
P	0.557	0.354	0.371	0.260	0.228
T	0.168	0.158	0.122	0.233	0.277
Qual.	0.330	0.047	0.132	0.242	0.224
Quant.	0.314	0.368	0.335	0.277	0.252
MODEL 4
Gradation	Dimension
349	163	124	416	254
K	0.229	0.498	0.540	0.078	0.364
P	0.260	0.081	0.115	0.414	0.099
T	0.386	0.193	0.082	0.070	0.139
Qual.	0.222	0.122	0.006	0.294	0.068
Quant.	0.270	0.290	0.297	0.214	0.273
MODEL 6
Gradation	Dimension
480	22	418	48	320
K	0.303	0.240	0.257	0.191	0.651
P	0.394	0.353	0.346	0.297	0.045
T	0.593	0.073	0.012	0.198	0.055
Qual.	0.320	0.140	0.180	0.162	0.018
Quant.	0.388	0.294	0.275	0.262	0.267
MODEL 8
Gradation	Dimension
441	234	238	283	125
K	0.033	0.272	0.191	0.075	0.013
P	0.463	0.243	0.240	0.352	0.260
T	0.038	0.085	0.025	0.087	0.241
Qual.	0.238	0.080	0.154	0.149	0.192
Quant.	0.238	0.227	0.193	0.185	0.164
MODEL 10
Gradation	Dimension
56	270	284	367	306
K	0.563	0.172	0.108	0.357	0.438
P	0.153	0.326	0.429	0.145	0.132
T	0.194	0.177	0.064	0.076	0.075
Qual.	0.173	0.240	0.215	0.120	0.158
Quant.	0.348	0.232	0.218	0.235	0.305
Figure 4: The amount (in  $\%$  ) of gold standard outputs, alternate outputs not displaying gradation and nonce forms which are generated when scaling top encoder hidden dimensions associated with gradation.
tion: dimension 487 in model 3. This dimension displays positive activation for consonants undergoing direct gradation as in laukku 'bag sg. nom.'  $\sim$  laukun 'bag sg. gen'. Remarkably, the state displays negative activation for consonants undergoing inverse gradation as in the example lauseke 'phrase' where k is strengthened into a geminate kk resulting in the genitive form lausekkeen 'phraseGEN'. This effect can be seen both in forms where quantitative and qualitative gradation occurs. However, as the example basilika 'basil' in the third heat map demonstrates, dimension 487 can also activate strongly when no gradation occurs.6 This
Figure 5: The amount (in  $\%$  ) of alternate outputs not displaying gradation when five randomly sampled encoder hidden dimensions are scaled.
prompted us to investigate hidden state activations more directly using the scaling experiments described in Section 6.3.
Figure 1 shows a scatter plot of two encoder hidden state dimensions (487 and 484 in model 3) which activate strongly during gradation. Each point in the plot corresponds to one example in our development dataset. Clearly, examples which do not undergo gradation cluster around  $(0,0)$ . In contrast, gradation for  $\mathbf{k}$  and  $\mathbf{p}$  lead to a positive activation for state 484, whereas t-gradation gives a negative activation. Moreover, direct gradation results in a positive activation for state 487 and inverse gradation gives a negative activation. Examples which do not undergo gradation can also have high values for 484 ( $>0.4$ ). Many of these examples end in -jV, -vV or -mV which could actually be examples where inverse gradation occurs but it happens not to be the case for these particular ones. Examples where the activation for 484 is low ( $< -0.5$ ) span a small number of forms ending -tV, -bV, and -gV. There is also a substantial number of non-gradating forms where the activation for 484 is  $>0.5$ . Most of these fall into the linnoitus 'fortress' / linnoituksen 'fortress sg. gen.' patterns where a  $\mathbf{k}$  is inserted in the penultimate syllable. This alternation bears great resemblance to gradation as mentioned in Section 3. There are also a few examples of the type tase 'balance sheet' / taseen 'balance sheet sg. gen.' where the stem-final vowel is doubled displaying large activation for 484. This is perhaps somewhat harder to explain. However, note that this vowel doubling fre
state 487, our model still correctly inflects basilika into basilikan instead of applying gradation, which would give a form like *basilijan or *basilian.
The single  $\mathbf{t}$  at  $(0,0)$  represents the pair  $\mathbf{olut} \sim \mathbf{oluen}$ , where  $\mathbf{t} \rightarrow \emptyset$ . This is an extremely infrequent gradation type.
<sup>6</sup>The form basilika is a loan word and would probably undergo gradation if it were a native Finnish word. It is noteworthy, however, that regardless of the strong activation of
sequently co-occurs with gradation as in tarvike ‘accessory’, tarvikkeen ‘accessory sg. gen’.
In our experiments we found that the system would sometimes output a gradated form even when the exact type of gradation was not present in the training data, for example bambu  $\sim$  bammun ('bamboo' sg. nom.  $\sim$  sg. gen.). Since Finnish natively lacks b and g, examples of gradation with these consonants are rare. However, it is indeed the case that loanwords that include such voiced stops do undergo gradation, e.g. dubata  $\sim$  dubbaan ('to dub' inf.  $\sim$  1p sg. pres. sg.) (Voutilainen, 2008). Since native Finnish speakers seem to extend gradation from voiceless stops to their voiced counterparts in loanwords, the question whether neural models can exhibit such generalizing behavior as well is an interesting one. Our initial investigations into whether the similarity of the learned embeddings for p and b could trigger such generalizations across similar sounds failed to identify a clear reason for the behavior, and we leave a detailed study of this to future work.
We have presented an investigation of encoder representations of phonological alternations, specifically consonant gradation in Finnish. We found evidence of a generalized representation of gradation covering all stops which undergo gradation and different types of gradation. We also found that scaling hidden states can "switch off" gradation, prompting the model to generate alternate forms which do not display gradation. Moreover, the direction of gradation can be encoded as positive vs. negative hidden dimension activation.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Gasper Begus. 2020a. Identity-based patterns in deep convolutional networks: Generative adversarial phonology and reduplication. arXiv preprint arXiv:2009.06110.
Gasper Begus. 2020b. Modeling unsupervised phonetic and phonological learning in generative adversarial phonology. Proceedings of the Society for Computation in Linguistics, 3(1):138-148.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? An analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for
NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018a. What you can cram into a single  $\$ \& !\#$ * vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Australia. Association for Computational Linguistics.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018b. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 1-30, Vancouver. Association for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The SIGMORPHON 2016 shared task—morphological reinfection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10-22.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. 2019. What is one grain of sand in the desert? Analyzing individual neurons in deep NLP models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6309-6317.
Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179-211.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks.
Auli Hakulinen, Maria Vilkuna, Riitta Korhonen, Vesa Koivisto, Tarja-Riitta Heinonen, and Irja Alho. 2004. Iso Suomen Kielioppi [Comprehensive grammar of Finnish]. Suomalaisen Kirjallisuuden Seura [Finnish Literature Society].
Katri Haverinen, Jenna Nyblom, Timo Viljanen, Veronika Laippala, Samuel Kohonen, Anna Missila, Stina Ojala, Tapio Salakoski, and Filip Ginter. 2014. Building the essential resources for Finnish: the Turku Dependency Treebank. Language Resources and Evaluation, 48(3):493-531.
Katharina Kann and Hinrich Schütze. 2016. MED: The LMU system for the SIGMORPHON 2016 shared task on morphological reinfection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 62-70.
Fred Karlsson. 2017. Finnish: A comprehensive grammar. Routledge.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078.
Guillaume Klein, Yoon Kim, Yuntian Deng, Vincent Nguyen, Jean Senellart, and Alexander Rush. 2018. OpenNMT: Neural machine translation toolkit. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 177-184, Boston, MA. Association for Machine Translation in the Americas.
Sudheer Kolachina and Lilla Magyar. 2019. What do phone embeddings learn about phonology? In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 160-169.
R. Meyes, M. Lu, C. Waubert de Puiseau, and T. Meisen. 2019. Ablation studies to uncover structure of learned representations in artificial neural networks. In Proceedings on the International Conference on Artificial Intelligence (ICAI), pages 185-191. The Steering Committee of The World Congress in Computer Science.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Tommi A. Pirinen. 2015. Development and use of computational morphology of Finnish in the open source and open science era: Notes on experiences with Omorfi development. _SKY Journal of Linguistics_, 28:381-393.
Jennifer Rodd. 1997. Recurrent neural-network learning of phonological regularities in Turkish. In CoNLL97: Computational Natural Language Learning.
Miikka P. Silfverberg, Lingshuang Mao, and Mans Hulden. 2018. Sound analogies with phoneme embeddings. In Proceedings of the Society for Computation in Linguistics (SCiL) 2018, pages 136-144.
Lucas Torroba Hennigen, Adina Williams, and Ryan Cotterell. 2020. Intrinsic probing through dimension selection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 197-216, Online. Association for Computational Linguistics.
Eero Voutilainen. 2008. Bloggaaminen laajentaa astveihtelua [Blogging extends consonant gradation]. https://www.kotus.fi/nyt/kolumnit_ artikkelit_ja_esitelmat/kieli-ikkuna_ (1996_2010)/bloggaaminen_laajentaa_ astevahtielua.
A Appendix: Scaling experiments
This appendix contains all results for the scaling experiment presented in Section 6. Figure 6 presents the amount of alternate forms produced by each model when 1 - 5 top gradation encoding hidden state dimensions associated with gradation are scaled. Figure 7 presents results for the gold standard forms undergoing gradation. For each model, we also present results for scaling a set of five randomly selected encoder hidden state dimensions.
As Figures 6 and 7 show, the effect of scaling dimensions associated with gradation has a clear positive effect on the number of output forms which do not undergo gradation. In contrast, scaling randomly selected encode hidden state dimensions has small effect overall on the number of these output forms although it does tend to reduce the number of gold standard outputs undergoing gradation. This means that the number of nonce output forms still increases when the scaling factor approaches  $-25$  as might be expected because we are deviating from the learned models parameters.
Figure 6: Results when scaling the activations for the top 1-5 (T1 - T5) encoder dimensions associated with gradation for each of out ten models M1 - M10. These graphs show the amount of outputs which do not undergo gradation which are produced when encoder dimensions are scaled. As comparison, the green TR graph shows the effect of scaling 5 randomly selected encoder dimensions.
Figure 7: Results when scaling the activations for the top 1-5 (T1 - T5) encoder dimensions associated with gradation for each of out ten models M1 - M10. These graphs show the amount of gold standard outputs undergoing gradation which are produced when encoder dimensions are scaled. As comparison, the green TR graph shows the effect of scaling 5 randomly selected encoder dimensions.