SlowGuess/NAACL_2021_OCR huggingface dataset:
Carlo Meloni*1 Shauli Ravfogel*2,3 Yoav Goldberg*2,3
$^{1}$ Department of Comparative Language Science, University of Zurich
$^{2}$ Computer Science Department, Bar Ilan University
carlomeloni@mail.tau.ac.il, {shauli.ravfogel, yoav.goldberg}@gmail.com
Historical linguists have identified regularities in the process of historic sound change. The comparative method utilizes those regularities to reconstruct proto-words based on observed forms in daughter languages. Can this process be efficiently automated? We address the task of proto-word reconstruction, in which the model is exposed to cognates in contemporary daughter languages, and has to predict the proto word in the ancestor language. We provide a novel dataset for this task, encompassing over 8,000 comparative entries, and show that neural sequence models outperform conventional methods applied to this task so far. Error analysis reveals a variability in the ability of neural model to capture different phonological changes, correlating with the complexity of the changes. Analysis of learned embeddings reveals the models learn phonologically meaningful generalizations, corresponding to well-attested phonological shifts documented by historical linguistics.
Historical linguists seek to identify and explain the various ways in which languages change through time. Research in historical linguistics has revealed that groups of languages (language families) can often be traced into a common, ancestral language, a "proto-language". Large-scale lexical comparison of words across different languages enables linguists to identify cognates: words sharing a common proto-word. Comparing cognates makes it possible to identify rules of phonetic historic change, and by back-tracing those rules one can identify the form of the proto-word, which is often not documented. That methodology is called the comparative method (Anttila, 1989), and is the main tool used to reconstruct the lexicon and phonology of extinct languages. Inferring the form of proto-words from existing cog
nates in daughter languages is possible since historical sound changes within a language family are not random. Rather, the phonological change is characterized by regularities that are the result of constraints imposed by the human articulatory and cognitive faculties (Millar, 2013). For example, we can find such regular change—commonly called "systematic correspondence"—by looking at the evolution of the first phoneme of Latin's word for "sky":<sup>1</sup>
Figure 1: the evolution of Latin word for "sky" is several Romance languages.
The Spanish word's first sound is  $[\theta]$ , while the Italian word begins with  $[tʃ]$ , the French word with [s], Romansh with [ts] and Sardinian with [k]. This pattern is systematic, and will be found throughout the languages. Working this way, historical linguists reconstruct words in the proto-language from existing cognates in the daughter languages, and determine how words in the proto-language may have sounded.
To what extent can a machine-learning model learn to reconstruct proto-words from examples in this way? And what generalizations of phonetic change will it learn? We focus on the task of proto-word reconstruction: the model is trained on sets of cognates and their known proto-word, and is then tasked with predicting the proto-word for an unseen set of cognates. Our study concentrate on the romance language family<sup>2</sup> and the model is trained to reconstruct the Latin origin. We show
<sup>1</sup>The words as transcribed with International Phonetic Alphabet (IPA) characters.
2 All the languages that derived from Latin.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4460-4473 June 6-11, 2021. ©2021 Association for Computational Linguistics
that a recurrent neural-network model can learn to perform this task well (outperforming previous attempts).<sup>3</sup>
More interesting than the raw performance numbers are the learned generalizations and error patterns. The Romance languages are widely studied (Ernst (2003); Ledgeway and Maiden (2016); Holtus et al. (1989) among others), and their phonological evolution from Latin is well mapped. The existence of this comprehensive knowledge allows exploring to what extent neural models internalize and capture the documented rules of language change, and where do they deviate from it. We provide an extensive error analysis, relating errors patterns to knowledge in historical linguistics. This is often not possible in common NLP tasks, such as parsing or semantic inference, in which the rules governing linguistic phenomena—or even the suitable framework to describe them—are still in dispute among linguists.
Contributions Inspection of existing datasets of cognates in Romance languages has revealed inherent problems. We thus have collected a new comprehensive dataset for performing the reconstruction task (§4). Besides the dataset, our main contribution is the extensive analysis of what is being captured by the models, both on orthographic and phonetic versions of the dataset (§6). We find that the error patterns are not random, and they correlate with the relative opacity of the historic change. These patterns were divided in different categories, each one motivated by a sound phonological explanation. Moreover, in order to further evaluate the learning of rules of phonetic change, we evaluated models on a synthetic dataset (§6.3), showing that the model is able to correctly capture several phonological change rules. Finally, we analyze the learned inner representations of the model, and show it learns phonologically meaningful properties of phonemes (§6.4) and attributes different importance to different daughter languages (§6.5).
The related task of cognates detection has been extensively studied. In this task, a set of cognates should be extracted from word lists in different languages. Most effort in Machine learn
ing approaches to this task has been focused on distance-based methods, which quantify the distance (according to some metric), or the similarity, between a given candidate of cognates. The similarity can be either static (e.g. Levenshtein distance) or learned. Once the metric is established, a classification can be performed either based on hard-decision (words below a certain threshold are considered cognates) or by learning a classifier over the distance measures and other features (Kondrak, 2001; Mann and Yarowsky, 2001; Inkpen et al., 2005; Ciobanu and Dinu, 2014a; List et al., 2016); Mulloni and Pekar (2006) have evaluated an alternative approach, in which explicit rules of transformation are derived based on edit operations. See Rama et al. (2018) for a recent evaluation of the performance of several cognates detection algorithms.
Several studies have gone beyond the stage of cognates extraction, and used resulted list of cognates to reconstruct the lexicon of protolanguages. Most studies in this direction borrowed techniques from computational phylogeny, drawing a parallel between the hypothesized branching of (latent) proto words into their (observed) current forms and the gradual change of genes during evolution. Bouchard-Côte et al. (2007) has applied such a model to the development of the Romance languages, based on a dataset composed of aligned-translations. Bouchard-Côte et al. (2009, 2013) used an extensive dataset of Austronesian languages and their reconstructed proto-languages, and built a parameterized graphical model which models the probability of a phonetic change between a word and its ancestral form; the probability is branch-dependent, allowing for the learning of different trends of change across lineages. While achieving impressive performance, even without necessitating a cognates lists as an input, their model is based on a given phylogeny tree that accurately represents the development of the languages in question.
Wu and Yarowsky (2018) have automatically constructed cognate datasets for several languages, including Romance languages, and used a character-level NMT system to complete missing entries (not necessarily the proto-form). Several works studied the induction of multilingual dictionaries from partial data in related languages. Wu et al. (2020) reconstruct cognates in Austronesian languages (where the proto-language is not attested). Lewis et al. (2020) employ a mixture
<sup>3</sup>We note that the role of the ML model is easier than that of the historical linguist, as it is trained on sets of words that it took the historical linguistics discipline a considerable effort to acquire.
of-experts approach for lexical translation induction, combining neural and probabilistic methods, and Nishimura et al. (2020) translate from a multi-source input that contains partial translations to different languages, concatenated. Finally, Ciobanu and Dinu (2018) have applied a CRF model with alignment to a dataset of Romance cognates, created from automatic alignment of translations (Ciobanu and Dinu, 2014b). The researchers also applied RNNs on the same dataset, but reported negative results.
3 Proto-word Reconstruction
Our proto-word reconstruction is as follows: the training set is composed of pairs  $(x_{i},y_{i})$ , where each  $x_{i} = c_{i}^{\ell_{1}},\ldots ,c_{i}^{\ell_{n}}$  is a set of cognate words, each tagged with a language  $\ell_{j}$ , and  $y_{i}$  is the protoword (Latin word) of that set. We consider an orthographic task, where the cognates and protowords are spelled out as written. As the orthography is often arbitrary and more conservative than spoken language, we consider also a phonetic task, in which the cognates and proto-words are represented as their phonetic transcriptions into IPA.
An example of a training instance  $(x,y)$  for the orthographic task is:
$$
x = \text {l a p t e} ^ {\mathrm {R M}}, \text {l a i t} ^ {\mathrm {F R}}, \text {l a t t e} ^ {\mathrm {I T}}, \text {l e c h e} ^ {\mathrm {S P}}, \text {l e i t e} ^ {\mathrm {P T}}
$$
$$
y = \text {l a c t e m}
$$
$$
a n d \text {f o r}
$$
$$
x = \operatorname {l a p t e} ^ {\mathrm {R M}}, \mathrm {l} \varepsilon^ {\mathrm {F R}}, \operatorname {l a t t e} ^ {\mathrm {I T}}, \operatorname {l e f t} \mathrm {f e} ^ {\mathrm {S P}}, \operatorname {l e j t i} ^ {\mathrm {P T}}
$$
$$
y = \mathrm {l a k t} \varepsilon \mathrm {m}
$$
A cognate in one of the languages may be missing, in which case we represent it by a dash. Here, we are missing the Italian and Romanian cognates:
$$
x = - ^ {\mathrm {R M}}, \operatorname {t r a v a j} ^ {\mathrm {F R}}, - ^ {\mathrm {I T}}, \operatorname {t r a b a x o} ^ {\mathrm {S P}}, \operatorname {t r e} \beta \mathrm {a} \Lambda^ {\mathrm {P T}}
$$
$$
y = \operatorname {t r i p a l} \varepsilon \mathrm {m}
$$
At test time, we are given a set of cognates and are asked to predict their proto-word.
4 Comprehensive Romance Dataset
The different experiments described in the paper were performed on a large dataset of our creation, which contained cognates and their proto-words in both orthographic and phonetic (IPA) forms. The dataset's departure point is Ciobanu and Dinu (2014b), which consists of 3,218 complete cognate sets in six different languages: French, Italian, Spanish, Portuguese, Romanian and Latin  ${}^{4}$  . We augmented the dataset’s items with a
freely available resource, Wiktionary, whose data were manually checked against DIEZ and Donkin (1864) to ensure their etymological relatedness with the Latin source. The entries were transcribed into IPA using the transcription module of the eSpeak library<sup>5</sup>, which offers transcriptions for all languages in our dataset, including Latin. The final dataset contains 8,799 cognate sets (not all of them complete), which were randomly split into train, evaluation and test sets: 7,038 cognate sets  $(80\%)$  were used for training, 703  $(8\%)$  for evaluation and 1,055  $(12\%)$  for testing. Overall, the dataset contains 41,563 distinct words for a total of 83,126 words counting both the orthographic and the phonetic datasets. Vowel lengths were found to be difficult to recover (see Table 1), hence we created the following variations of the dataset: with and without vowel length (for both the orthographic and phonetic datasets), and without a contrast (for the phonetic dataset); see section §6 for further discussion.
A detailed description of the dataset collection process is available at the appendix §A.1. We make our additions to the dataset of Ciobanu and Dinu (2014b) publicly available<sup>6</sup>.
5.1 NMT-based Neural Model
Our proto-word reconstruction setup follows an encoder-decoder with attention architecture, similar to contemporary neural machine translation (NMT) systems (Bahdanau et al., 2015; Cho et al., 2014).
We use a standard character-based encoder-decoder architecture with attention (Bahdanau et al., 2015). Both encoder and decoder are GRU networks with 150 cells. The encoder reads the forms of the words in the daughter languages, and output a contextualized representation of each character. At each decoding step, the decoder attends to the encoder's representations via a dot-product attention. The output of the attention is then fed into a MLP with 200 hidden units, which outputs the next Latin character to generate.
Input representation Each character (a letter in the orthographic case, and a phoneme in the phonetic case) is represented by an embedding vector
$^{5}$ https://github.com/espeak-ng/espeak-ng
<https://github.com/shauli-ravfogel/
Latin-Reconstruction-NAACL. The entries that appeared in the original dataset are not publicly available.
4We thank Ciobanu and Dinu for sharing their data with us.
Edit Distance
0	≤1	≤2	≤3	≤4	Average	Avg, norm
Ortographic	64.1%	84.0%	92.7%	96.8%	98.5%	0.65	0.064
IPA	50.0%	73.9%	85.9%	93.3%	97.0%	1.022	0.100
Ortographic, added vowel lengths	42.5%	72.0%	86.3%	92.5%	96.7%	1.13	0.102
IPA, added vowel lengths	47.9%	62.0%	80.2%	87.1%	93.8%	1.331	0.119
IPA, no contrast	65.1%	80.0%	87.5%	93.6%	96.7%	0.797	0.077
Table 1: Distribution of edit distances between the reconstructed and original Latin form, on the orthographic and transcribed detaets. Edit distance of 0 corresponds to perfect reconstruction. "Average" refers to average edit distance, and "Avg, norm" to normalized average edit distance.
of size 100. While all Romance languages are orthographically similar, the same letters represent different sounds, and thus convey different kinds of information for the task of Latin reconstruction. A possible approach would encode each language's characters using a unique embedding table. We instead share the character embedding table across all languages (including Latin), but concatenate to each character vector also a language-embedding vector. The final representation of a character  $c$  in language  $\ell$  is then  $WE[c] + UE[\ell]$  where  $E$  is a shared embedding matrix,  $c$  is a character id,  $\ell$  is a language id, and  $W$  and  $U$  are a linear projection layers.
5.2 Evaluation Metric
Our main quantitative metric for evaluation is the edit distance between the reconstructed word and the gold Latin word. We use the standard edit distance with equal weight of 1 for deletion, insertion and substitution. We report test set average edit distance and average normalized edit distance (divided by word length), as well as the percentage of instances with less than  $k$  edit operations between the reconstruction and the gold, for  $k = 0$  to 4.
Table 1 summarizes our main quantitative results. "Orthographic, added vowel lengths" and "IPA, added vowel lengths" refer to variations of the datasets that include explicit marking of vowel length in Latin words, marked by  $<:>$  after long vowels. The models performance on the orthographic dataset demonstrates a substantial improvement over previously reported results. Our method has achieved average edit distance of 0.65, average normalized edit distance of 0.064, and  $64.1\%$  complete reconstruction rate (edit distance of 0). These numbers compare favorably with the edit distance of 1.07, normalized edit distance of 0.13 and  $50\%$  complete reconstruction reported
by Ciobanu and Dinu (2018). We note, however, that as our method is different both in the training corpus and in the type of model we employ, it is not clear whether this improvement should be attributed to the quality of the data, to the model, or to both of them.
The performances on the phonetic dataset were lower than those derived from the orthographic one: in the phonetic dataset the average edit distance was of 1.022, and the average normalized edit distance of 0.1, with  $50.0\%$  complete reconstruction rate.
This disparity can be explained at least partially by a peculiarity of the phonetic dataset: it implicitly encodes vowel length, which was neutralized in the orthographic dataset. The reason for this difference is that length contrast in Latin co-occurred with quality differences: short vowels tended to be more open than their long counterparts, a contrast also called "tense-lax" (Allen and Allen, 1989). This contrast is not present in Latin orthography, but it appears in its phonetic transcription. This results in a noticeable gap between the results of the orthographic dataset with vowel lengths and without vowel lengths (0.064 average normalized edit distance vs. 0.119), while the differences between the phonetic IPA dataset with vowel lengths and without vowels lengths are much smaller. When the contrast "tense-lax" is manually neutralized<sup>8</sup>, the performances achieved are similar to the ones on the orthographic dataset (as it is possible to see from the performances on "IPA, no contrast", whose Latin entries do not contain a "tense-lax"
7When we train a smaller version of our model (75-dimensional GRU) on the original dataset of Ciobanu and Dinu (2014b) we achieve average edit distance of 0.881, average normalized edit distance of 0.103, and complete reconstruction rate of  $59.1\%$ . Training a similar model on their dataset after cleaning resulted in average edit distance of 0.612, average normalized edit distance of 0.062 and complete reconstruction rate of  $68.8\%$ .
8We achieved that by respectively changing the characters  $<  \mathrm{o}>$ $<  \mathrm{o}>$ $<  \mathrm{i}>$ $<  \varepsilon >$  to  $<  \mathrm{u}>$ $<  \mathrm{o}>$ $<  \mathrm{i}>$ $<  \mathrm{e}>$  in the Latin words
Error type	Orthographic	Phonetic
High-mid	18%	8%
Deletion	14%	6%
Consonant	13%	15%
Cluster	12%	3%
Morphology	11%	10%
Vowel	7%	8%
Length	—	26%
Orthography	5%	—
Other	20%	24%
Table 2: Error type distribution based on 650 orthographic and 650 phonological errors.
contrast).
6.1 Error Patterns
The following subsections focus on the model performances on the orthographic and phonetic datasets without explicit vowel length marking. A thorough analysis of both datasets reveals that the model's errors are not arbitrary, but rather tend to correspond to one of a few well-defined linguistic phenomena characterizing the evolution of Latin to its daughter languages. From an analysis of about 1300 errors, equally divided between the orthographic and the phonetic datasets, we find that  $80\%$  of the errors of the model on the orthographic dataset, and  $75\%$  on the phonetic one can be grouped into one of the following groups: high-mid vowel alternations, segment deletion, segment changes, cluster changes, morphological changes and other vowel changes. Additionally, one error category is unique to the phonetic dataset, tenselax errors, and one is unique to the orthographic dataset, orthography errors. Table 2 summarizes the results, and Figure 2 visualizes the vowels error patterns on the phonetic dataset.
We briefly discuss each of these groups.
High-mid alternation. The largest number of errors on the orthographic dataset,  $18\%$ , can be attributed to confusion between high and mid-high vowels (correspondingly  $<\mathrm{i}>$ ,  $<\mathrm{u}>$  and  $<\mathrm{e}>$ ,  $<\mathrm{o}>$ ), as shown by the reconstruction  $<\text{pescarium}>$  instead of the Latin  $<\text{piscarium}>$  (alternation between  $<\mathrm{e}>$  and  $<\mathrm{i}>$ ). That error is much rarer in the phonetic dataset, accounting only for  $8\%$  of all the errors. The reason of this error can be attributed to the origin of the mid-vowels in the daughter languages: while Latin
long vowels [ɪː], [eː], [ɔː] and [uː], always evolved into [i], [e], [o] and [u] in the daughter languages (with minor changes related to syllable structure), Latin short vowels—[ɪ], [ε], [ɔ] and [ʊ]—are not deterministically mapped into corresponding vowels in the daughter languages: Latin [ɪ] and [ε] both usually became Romance [e] (with alternations related to syllable structure, as diphthongization to [je]), while Latin [ɔ] and [ʊ] have different reflexes in the daughter languages as [u], [o], [ɔ], [ɒ] or as diphthongs. Because of this complex evolution, which merges different Latin phonemes into the same one in the daughter languages, the model is unable of unequivocally predicting the Latin vowel. Nonetheless, it seems that the tenselax contrast present in the phonetic dataset eases the task of distinguishing the different phonemes, and enables the network to reconstruct their origin more often.
Segment deletion. examples of these errors are the reconstruction of  $\langle \text{aspargum} \rangle$  instead of Latin  $\langle \text{asparagus} \rangle$ , and the reconstruction of [abitatem] instead of Latin [habilitatem]. During the evolution from Latin to Romance languages, unstressed syllables tended to be dropped. This phenomenon was not systematic, and occurred in different ways among and within the languages. Such process could affect either whole syllables (consonant + vowel) or only the vowel, creating new consonant clusters. Because of the erratic nature of this process, it seems that the network struggles with the exact reconstruction of segments eliminated in the daughter languages. A special kind of deletion is that of the consonant [h]. This consonant did not survive in any Romance languages (although it may be represented orthographically), and hence many times the network does not reconstruct it.
Segment changes. this category encompasses errors in the reconstruction of consonants—such as voicing changes (reconstructing  $<\text{faculdadem}>$  vs. Latin  $<\text{facultatem}>$ ), assimilation ([wessare] vs. [weksare]) and gemination ([agregationem] vs. [aggregationem]). All these errors reflect processes that took place in all of the daughter languages, that obscures the original form of the proto-word.
Cluster changes. These are changes that occur with two contiguous consonants. Consider, for example, the reconstruction of [rətɪnɪm] instead of Latin [rəktɪnɪm], and of <sennorem> instead of Latin <seniorem>. The former is an
The orthographic characters will be displayed between two angle brackets, while phonetic characters between two square brackets.
instance of cluster simplification, while the latter is an instance of cluster palatalization. In many of the daughter languages clusters of two different sounds underwent simplification, either by the dropping of one of the sound or the assimilation of one of them. Palatalization is the process by which certain sounds tend to be pronounced more closely to the palate, usually because of an adjacent front vowel. This change occurred in all Romance languages, even though its orthographic representation may vary among them.