SlowGuess/NAACL_2021_OCR huggingface dataset:
Morphological changes. Latin had a very developed morphology, with several classes of special conjugations and irregular forms. The network struggles to reconstruct correctly irregular forms, as these forms were mostly lost in the daughter languages. An instance of such irregular verbs is <praeferre>, reconstructed as <praefere> by the network. Moreover, other special morphological classes, such as Latin neuters, tend to be reconstructed as more usual forms. Another interesting class of errors is change of morphological category: some nouns have suffixes reminiscent of those of verbs, and hence are wrongly reconstructed as such. A separated case is that of Greek words: Latin contained several Greek loanwords that conserved their original morphology, different from the Latin one. Since these peculiarities were, for most part, not retained in the daughter languages, the network reconstructs them with normal Latin suffixes. For example, the greek [syntaksis] was reconstructed as [syntaksεm], with the normal Latin suffix.
Other vowel changes. Latin contained several diphthongs, among them the diphthongs [ai] and [oi]. These sounds did not survive in any of the daughter languages (although in some rare cases they may be represented in the orthography), and both changed into [e] in the different Romance languages. This left to reconstruction errors such as reconstructing  $\langle \text{eigrum} \rangle$  instead of Latin  $\langle \text{aegrum} \rangle$ . Some changes also occurred with the vowel [a], which was reconstructed as a different vowel.
Greek orthography. some Latin words from Greek origin retained some orthographic conventions alien to Latin, such as the use of  $<\mathrm{y}>$ ,  $<\mathrm{ph}>$ ,  $<\mathrm{th}>$ ,  $<\mathrm{rh}>$  etc. These conventions were only partially retained in the daughter languages, which creates some inconsistencies in their reconstruction by the network.
Tense-Lax alternation. this is the largest cate
tense-lax errors
Figure 2: Phonological mistakes resulting from alternations between vowels, on the phonetic dataset. The numbers signify the number of errors, excluding singleton errors.
backness errors
gory found in the networks errors on the phonetic dataset – up to  $26\%$  of all errors. As said previously, the tense-lax contrast reflects vowel length in Latin, which is not entirely predictable based on the daughter languages. The network tends to confuse between the lax and the tense vowels.
Figure 2 shows clearly that the network's errors are internally consistent and not random: all the vowel errors fit neatly in one of the aforementioned categories, while other possible errors do not occur.
Orthographic vs. Phonetic Importantly, the phonetic and orthographic tasks differ in their error distributions: while the performance of the network on the orthographic task displays many syllable changes – changes that alter the structure of the syllable (mostly changes in consonant clusters and deletion of segments) – on the phonetic tasks the model tends to retain syllable structure, but perform more segment-related errors (i.e., changing a specific vowel or consonant for another one). The IPA performance contained more idiosyncratic errors that could not be categorized in one of the main categories. Such errors tended to occur when the network had only one or two cognates from the daughter languages. Even though the orthographic performance also exhibited poorer reconstructions in these cases, it seems that the IPA performance was even more affected by the singular words, leading to more erratic reconstructions.
6.2 Learnt generalizations
This section will focus on the phonetic dataset. A closer inspection of the errors made by the model, and of those that do not occur in the data, can shed light on the processes of phonological change learnt by the model. We will first focus on the vowels. The Latin vowel [a] is quite resilient to
changes, and most of the daughter languages retain it without change (only in French and Romanian some phonological changes occur, in certain phonological environments). Indeed, the network has almost no mistakes in recovering it, apart from some isolated cases that derives from insufficient cognates in the daughter languages. The network also makes virtually no errors regarding the reconstruction of vowel backness – here also the only few cases are caused by the paucity of cognates and by assimilation processes in the daughter languages that make the Latin source opaque (metaphony processes). All in all, the network learns correctly the phonological changes that occurred in Latin vowels, and the main errors are a result of changes that cannot be fully reverted from the daughter languages.
The model learnt well the mapping of consonants between Latin and its daughter languages, and vowel reconstruction errors are considerably more prevalent. Focusing on one type of errors, palatalization, shows that the network failed to reconstruct the original consonant in opaque contexts, that is, when phonological cues crucial for the right reconstruction were lacking. Specifically, the network confused between the consonants [t] and [k] in the Latin reconstruction, since they palatalize to the same segments in Spanish and French. Without the other daughter languages, it is impossible to reconstruct correctly the original sound in Latin.
Finally, the network correctly generalized the occurrence of nasals in Latin clusters. Latin nasal tended to assimilate to the place of articulation of the adjacent consonant, deriving clusters such [ŋk], [mp] and [nt]. When the network deleted a consonant in a cluster containing a nasal, or changed a consonant adjacent to a nasal, the nasal consonant always changed to match the place of articulation of the following consonant. Hence, by deleting [k] in the cluster [ŋkt], the network reconstructs [nt]. Similarly, by changing [p] to [t] in the cluster [mp], the nasal consonant accordingly: [nt].
6.3 Evaluating Rules of Phonetic Change
To what extent did the model learn known rules of phonetic change?
The evolution of the Romance languages is well studied and linguists documented the set of phonological transformations that underwent between Latin and its daughter languages. We
collected 33 of these phonological change rules, and used them to create a "synthetic" test set, containing syllable examples each focusing on a different phonological change. An example of a row in this dataset, corresponding to the rule of change of Latin [j] at word initial, is:
$$
\begin{array}{l} x = 3 a ^ {R M}, 3 a ^ {F R}, d 3 a ^ {I T}, x a ^ {S P}, 3 a ^ {P T} \\ y = \mathrm {j a} \\ \end{array}
$$
Since the model was trained on complete words, isolated syllables tended to be unnatural for the network, and the output often contained additional consonants (usually morphological endings). When evaluating the model output we focus on the specific phonemes involved in the phonological change, and we ignore additional phonological material.
Results The complete list of synthetic examples and predictions is available at Table 3. The network correctly predicted 22 out of the 33 phonological rules (66.67% of the changes). The results are compatible with the results of the main reconstruction experiment: In both experiments, the network correctly reconstructed phonemes retained with little or no changes in all languages (e.g. [a] in different -phonological environments). Another class of phonemes correctly reconstructed in both cases are those which changed in a predictable way in each one of the daughter languages. Thus, [w] was correctly reconstructed since it predictably changed to [v] in all the daughter languages (apart from Spanish, which merged it with [b]). Phonemes that tended to change differently, but consistently, were also faithfully recovered: even though Latin [k] tended to change differently depending on the daughter language ([s] in French and Portuguese, [θ] in Spanish and [tʃ] in Italian and Romanian), it was reconstructed correctly because of the consistence of the change in each daughter language. The phonemes wrongly reconstructed tended to be those whose phonological change was "opaque". The "opaqueness" of their change can be ascribed to the fact that they were neutralized in the daughter languages, making it impossible to recover them without additional information. Relevant to this case are mostly vowels and diphthongs, as Latin [e] and [ɪ], which both became [e] in all the different daughter languages (with variants influenced by the phonological environment).
Latin phoneme	Romanian	French	Italian	Spanish	Portuguese	Latin	Latin - reconstruction	Correct
/e/ blocked syllable	pep	pep	pep	pep	pep	pep	pip	no
/o/ blocked syllable	pop	pup	pop	pop	pop	pop	pup	no
/ε/ blocked syllable	pjep	pεp	pεp	pjep	pεp	pεp	pep	no
/kt/ medially, before nasals	-	anta	anta	anta	anta	ankta	antam	no
/aɪ/	pe	pe	pe	pe	pe	paɪ	pɛm	no
/ɔɪ/	pe	pe	pe	pe	pe	pɔɪ	pɛm	no
/b/ intervocalic	aa	ava	ava	aβa	ava	aba	awam	no
/e/ free syllable	pe	pwa	pe	pe	pe	pe	pɛm	no
/o/ free syllable	po	pø	po	po	po	po	pʊm	no
/ɪ/ free syllable	pe	pwa	pe	pe	pe	pi	pɛm	no
/n/ before front vowels	ji	ni	ni	ni	ni	ni	ŋidɛm	no
/a/ before nasal	pin	pan	pan	pan	pan	pan	pan	yes
/a/ blocked syllable	pap	pap	pap	pap	pap	pap	pap	yes
/i/	pi	pi	pi	pi	pi	pi	pi	yes
/u/	pu	py	pu	pu	pu	pu	pu	yes
/ɪ/ blocked syllable	pep	pep	pep	pep	pep	piŋ	piŋ	yes
/ʊ/ blocked syllable	pup	pup	pop	pop	pop	pup	pup	yes
/ɔ/ blocked syllable	pop	pɔp	pɔp	pwep	pɔp	pɔp	pɔp	yes
/k/ before front vowels	tfi	si	tfi	θi	si	ki	ki	yes
/sk/ before front vowels	fti	si	fi	θi	fi	ski	ski	yes
/kt/ medially, elsewhere	apta	ata	atta	atfa	ata	akta	aktam	yes
/au/	pau	pɔ	pɔ	po	po	paʊ	paʊm	yes
/pl/ word initial	pla	pla	pja	ʌa	fɑ	pla	plam	yes
/a/ free syllable	pa	pa	pa	pa	pa	pa	pam	yes
/ε/ free syllable	pje	pje	pje	pje	pε	pε	pɛm	yes
/w/	va	va	va	ba	va	wa	wam	yes
/b/ word initial	ba	ba	ba	ba	ba	ba	bam	yes
/j/ word initial	3a	3a	dʒa	xa	3a	ja	jam	yes
/f/ word initial	fa	fa	fa	a	fa	fa	fam	yes
/f/ elsewhere	afa	afa	afa	afa	afa	afa	affam	yes
/ʊ/ free syllable	pu	pɒ	pɒ	pɒ	pɒ	pu	pʊpɒm	yes
/ɔ/ free syllable	po	pɒ	pwɔ	pwe	pɔ	pɔ	pɒdɛm	yes
/ɪ/ before front vowels	ji	ji	ʌi	xi	ʌi	li	gɪlʊm	yes
Table 3: the set of test phonemes used to evaluate the model's generalizations. Each row represents a distinct rule of phonetic change, which focuses on a single phoneme. The phoneme in question is bolded, and other consonants / vowels are added to simulate the phonological environment of the rule. The added consonants / vowels were chosen because they did not affect the evolution of the examined phonemes from Latin to the Romance languages. "correct" signifies whether the network's prediction were correct.
Figure 3: Hierarchical clustering of French phoneme embeddings
6.4 Learnt phoneme representations
Does training on proto-word reconstructions implicitly encourage the model to acquire phonologically-meaningful representations? We visualize the representation learned by network on the phonetic task by performing hierarchical clustering on the characters embedding vectors using the sklearn (Pedregosa et al., 2011) implementation of Ward variance minimization
algorithm (Ward Jr, 1963).
Here we will briefly discuss the learned French phoneme representations (Figure 3). For all other languages, see appendix §5. As can be seen, the primary division that the network performs is between vowels and consonants, displayed on two different branches of the tree. On a lower level other phonologically motivated groupings are found: the network tends to place under the same node pairs of voiced and unvoiced consonants (as [ʃ] and [ʒ], [d] and [t]), allophones ([æ] and [ø]) or phonemes of the same category (as the glides [j] and [w]). To conclude, the results demonstrate the learning of a phonologically meaningful taxonomy of phonemes, without explicit supervision.
Orthographic
Phonetic
Figure 4: position in output vs. most attended language (left) and output letter vs. most attended language (right) for the orthographic (upper) and phonetic (lower) tasks.
6.5 Attention analysis
Since different languages can diverge to a varying extent from their proto-language, we hypothesize that the 5 daughter languages we use in this work would be of different importance for the model. To test this hypothesis, we inspect the learned attention weights. We focus on the most attended input character at each time step (the character having the largest attention weight) and count the number of times each of the 5 input languages is the most attended language, as a function of the location in the output and of the identity of the Latin character produced in that time step. We normalize the count with respect to time step, letter frequency and language frequency in the corpus.
Results The results for the phonetic and orthographic tasks are presented in Figure 4. In both cases, Italian is the most attended language. There are some differences between the settings, however. For the orthographic task, the network focuses noticeably more on French than in the phonetic task. This tendency can be attributed to the very conservative orthography of French, that masks the phonological innovations that occurred in the language. Indeed, the network focuses exclusively on French for the reconstruction of the characters  $\langle \mathrm{h} \rangle$  and  $\langle \mathrm{y} \rangle$ , which are consistently represented only in French orthography, disappearing from the written form of the other Romance languages. The comparison to the atten
tion of the phonetic dataset shows that the network tends to actually ignore French, favoring other sources instead. Similarly, in the orthographic dataset, French is favored in the initial positions, a tendency that disappears in the phonetic dataset. Finally, an interesting trend in the phonetic dataset is a tendency to attend to Romanian at the initial positions and to Portuguese at later ones.
In this work, we introduce a new dataset for the task of proto-word reconstruction in the Romance language family, and used it to evaluate the ability of neural networks to capture the regularities of historic language change. We have shown that neural methods outperform previously suggested models for this task. Analysis of the linguistic generalizations the model acquires during training demonstrated that the mistakes are related to the complexity of the phonetic change. A controlled experiment on a set of rules for phonetic alternations between Latin and its daughter languages demonstrated the model internalizes some of the systematic processes that Latin had undergone during the evolution of the Romance languages. Visualizing the learned phoneme-embedding vectors has revealed a hierarchical division of phonemes that reflects phonological realities, and inspection of attention patterns demonstrated the model attributes different importance to different languages, in a position-dependent manner.
While the task examined in this paper is commonly called "proto-word reconstruction", in practice the task the model faces is considerably less challenging than the work of historical linguists, as the model is trained in a supervised setting. A future line of work we suggest is applying neural models for the end task of proto word reconstruction, without relying on cognates lists, in a way that would more naturally model the historical linguistic methodology.
We thank Arya McCarthy for pointing out to relevant references. This project received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme, grant agreement No. 802774 (iEXTRACT).
Ti Alkire and Carol Rosen. 2010. *Romance languages: A historical introduction*. Cambridge University Press.
W Sidney Allen and William Sidney Allen. 1989. Vox Latina. Cambridge University Press.
Raimo Anttila. 1989. Historical and comparative linguistics, volume 6. John Benjamins Publishing.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Alexandre Bouchard-Côté, Thomas L Griffiths, and Dan Klein. 2009. Improved reconstruction of protolanguage word forms. In Proceedings of human language technologies: The 2009 annual conference of the north american chapter of the association for computational linguistics, pages 65-73. Association for Computational Linguistics.
Alexandre Bouchard-Côte, David Hall, Thomas L Griffiths, and Dan Klein. 2013. Automated reconstruction of ancient languages using probabilistic models of sound change. Proceedings of the National Academy of Sciences, 110(11):4224-4229.
Alexandre Bouchard-Côté, Percy Liang, Thomas L. Griffiths, and Dan Klein. 2007. A probabilistic approach to diachronic phonology. In EMNLP-CoNLL 2007, Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, June 28-30, 2007, Prague, Czech Republic, pages 887-896.
Peter Boyd-Bowman. 1980. From Latin to Romance in sound charts. Georgetown University Press.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bah-danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST@EMNLP 2014, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, Doha, Qatar, 25 October 2014, pages 103-111.
Alina Maria Ciobanu and Liviu P Dinu. 2014a. Automatic detection of cognates using orthographic alignment. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 99-105.
Alina Maria Ciobanu and Liviu P Dinu. 2014b. Building a dataset of multilingual cognates for the Romanian lexicon. In Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC, pages 1038-1043.
Alina Maria Ciobanu and Liviu P Dinu. 2018. Ab initio: Automatic latin proto-word reconstruction. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1604-1614.
J Halvor Clegg and Willis C Fails. 2017. Manual de fonética y fonología españolas. Routledge.
Josette Rey Deabove and Alain Rey. 2000. Le Nouveau Petit Robert: Dictionnaire alphabetique et analogique de la langue française. Dictionnaires Le Robert.
Friedrich Christian DIEZ and TC Donkin. 1864. An Etymological Dictionary of the Romance Languages; chiefly from the German of F. Diez. By TC Donkin. Williams and Norgate.
Gerhard Ernst. 2003. Romanische Sprachgeschichte: Ein Internationales Handbuch Zur Geschichte Der Romanischen Sprachen Und ihrer Erforschung/Manuel International Sur L'Histoire Et L'Etude Linguistique DES Langues Romanes. Mouton de Gruyter.
Félix Gaffiot and Pierre Flobert. 1934. Dictionnaire latin-français. Hachette Paris.
Robert A Hall. 1944. Italian phonemes and orthography. Italica, 21(2):72-82.
Martin Harris and Nigel Vincent. 2003. The romance languages. Routledge.
Günter Holtus, Michael Metzeltin, and Christian Schmitt. 1989. LRL, volume 3. M. Niemeyer.
Diana Inkpen, Oana Frunza, and Grzegorz Kondrak. 2005. Automatic identification of cognates and false friends in french and english. In Proceedings of the International Conference Recent Advances in Natural Language Processing, volume 9, pages 251-257.
Grzegorz Kondrak. 2001. Identifying cognates by phonetic and semantic similarity. In Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, pages 1-8. Association for Computational Linguistics.
Adam Ledgeway and Martin Maiden. 2016. The Oxford guide to the Romance languages, volume 1. Oxford University Press.
Charlton T. Lewis and Charles Short. 1879. A latin dictionary. Perseus Digital Library.
Dylan Lewis, Winston Wu, Arya D. McCarthy, and David Yarowsky. 2020. Neural transduction for multilingual lexical translation. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4373-4384. International Committee on Computational Linguistics.
Johann-Mattis List, Philippe Lopez, and Eric Baptiste. 2016. Using sequence similarity networks to identify partial cognates in multilingual wordlists. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 599-605.
Gideon S. Mann and David Yarowsky. 2001. Multipath translation lexicon induction via bridge languages. In Language Technologies 2001: The Second Meeting of the North American Chapter of the Association for Computational Linguistics, NAACL 2001, Pittsburgh, PA, USA, June 2-7, 2001.
Maria Helena Mateus and Ernesto d'Andrade. 2000. The phonology of Portuguese. OUP Oxford.
Robert McColl Millar. 2013. Trask's historical linguistics. Routledge.
Andrea Mulloni and Viktor Pekar. 2006. Automatic detection of orthographic cues for cognate recognition. In LREC, pages 2387-2390.
Yuta Nishimura, Katsuhito Sudoh, Graham Neubig, and Satoshi Nakamura. 2020. Multi-source neural machine translation with missing data. IEEE ACM Trans. Audio Speech Lang. Process., 28:569-580.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825-2830.
Taraka Rama, Johann-Mattis List, Johannes Wahle, and Gerhard Jäger. 2018. Are automatic methods for cognate detection good enough for phylogenetic reconstruction in historical linguistics? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 393-400.
Mika Sarlin. 2014. *Romanian grammar*. BoD-Books on Demand.
Joe H Ward Jr. 1963. Hierarchical grouping to optimize an objective function. Journal of the American statistical association, 58(301):236-244.
Winston Wu, Garrett Nicolai, and David Yarowsky. 2020. Multilingual dictionary based construction of core vocabulary. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 4211-4217. European Language Resources Association.
Winston Wu and David Yarowsky. 2018. Creating large-scale multilingual cognate tables. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA).
In order to perform the reconstruction task, we required a large dataset of cognates and their protowords, in both orthographic and phonetic (IPA) forms.
Despite growing interest in recent years, high-quality digital resources for the tasks of proto-word reconstruction and cognates detection are scarce. Our departure point is the dataset provided by Ciobanu and Dinu (2014b), which, to the best of our knowledge, is the most extensive dataset for proto-word reconstruction of a well-attested proto-language. The dataset contains 3,218 complete cognate sets in five Romance languages (Spanish, Italian, Portuguese, French, Romanian) together with their Latin etymological ancestor. Although being a valuable resource, this dataset was constructed via automatic method of cognate extraction, and a comparison with references on the development of Romance languages (Boyd-Bowman, 1980; Alkire and Rosen, 2010) reveals some problems, such as false cognates, truncated forms, non-existent words and mismatch between the part of speech of the cognates and the ancestor. Another salient problem of the dataset regarded the grammatical case of Latin nouns: Romance languages derived their words from the accusative Latin case (Harris and Vincent, 2003), while in the dataset Latin words were displayed in the nominative case, an inconsistency making the reconstruction inherently more challenging.