Cognate Transformer for Automated Phonological Reconstruction and Cognate Reflex Prediction
Authors: V.S.D.S.Mahesh Akavarapu
Arnab Bhattacharya
Venue: EMNLP 2023 Main (2023)
Keywords: Computational historical linguistics
Phonological reconstruction
Cognate reflex prediction
Transformer

Abstract: Phonological reconstruction is one of the central problems in historical linguistics where a proto-word of an ancestral language is determined from the observed cognate words of daughter languages. Computational approaches to historical linguistics attempt to automate the task by learning models on available linguistic data. Several ideas and techniques drawn from computational biology have been successfully applied in this area of computational historical linguistics. Following these lines, we adapt MSA Transformer, a protein language model, to the problem of automated phonological reconstruction. MSA Transformer trains on multiple sequence alignments as input and is, thus, apt for application on aligned cognate words. We, hence, name our model as Cognate Transformer. We also apply the model on another associated task, namely, cognate reflex prediction where a reflex word in a daughter language is predicted based on cognate words from other daughter languages. We show that our model outperforms the existing models on both the tasks, especially when it is pre-trained on masked word prediction task.

Cognate Transformer for Automated Phonological Reconstruction and Cognate Reflex Prediction
V.S.D.S.Mahesh Akavarapu and Arnab Bhattacharya Dept. of Computer Science and Engineering Indian Institute of Technology Kanpur maheshakēcse. iitk. ac. in, arnabbêcse. litk.ac. in
Abstract
respondences, and finally reconstructing the proto- phonemes. This procedure known as the 'com- parative method' (Ringe and Eska, 2013), which is traditionally carried out mamually.
Phonological reconstruction is one of the cen- tral problems in historical linguistics where a proto-word of an ancestral language is de- termined from the observed cognate words of daughter languages. Computational ap- proaches to historical linguisties attempt to au- tomate the task by learning models on avail- able linguistic data. Several ideas and tech- niques drawn from computational biology have been successfully applied in the of putational historical linguistics. Following these lines, we adapt MSA Transformer, a pro- tein language model, to the problem of au- tomated phonological reconstruction. MSA Transformer trains on multiple sequence align- ments input and is, thus, apt for application on aligned cognate words. We, hence, name our model Cognate Transformer. We also apply the model on another associated task, namely, cognate reflex prediction, where a reflex word in daughter language is predicted based cognate words from other daughter languages. We show that our model outperforms the exist- ing models on both tasks, especially when it is pre-trained on masked word prediction task.
Several automated phonological reconstruction algorithmss emerged in the last decade. Some of these are drawn or inspired from computational bi- ology, for example, Bouchard-Côtt et al. (2013). general, computational historical linguistics draws techniques such as sequence alignment and phylo- genetic inference from computational biology, in addition to the techniques known from historical linguistics and computational linguistics or natu- ral language processing (Jäger, 2019). On similar lines, we adapt the MSA transformer, introduced in Rao et al. (2021) for modeling multiple sequence alignment (MSA) protein sequences, for the prob- lem of phonological reconstruction which takes as input a cognate word set in the form of MSAs. Henceforth, name the model introduced here Cognate Transformer.
We also apply our model on cognate reflex predi- cion task, where unknown, i.e., a masked reflex in daughter language is to be predicted based the attested reflexes from other daughter languages (List et al., 2022b). For instance, in the previous example, if we mask French enfant, the task would involve arriving at the word form correctly based on Spanish infantes and Italian infanti. One can notice that this task can serve as a pre-training ob- jective for the proto-language reconstruction task described previously. Hence, we also pre-train the Cognate Transformer the cognate reflex predic- tion task.
1 Introduction
Phonological reconstruction of word in an an- cestral proto-language from the observed cognate words, i.c., words of supposed common origin, in the descendant languages is one of the central prob- lems in historical linguistics, a discipline that stud- ies diachronic evolution of languages (Camphell, 2013). For example, the cognate words French en- fant, Spanish infantes and Italian infanti all trace to the proto-form infantes in Latin meaning "chil- dren', which is attested language in this case. In most cases, the lroto-language is attested and has be rather reconstructed. The process of arriv- ing at such phonological reconstruction usually in- volves multiple steps including gathering potential cognate words, identifying systematic sound cor-
Further, most of the existing models are fitted on per language family basis, i.c., time consisting of single language family. Thus, the utility of either transfer learning or simultane- ous fitting across several language families has not yet been demonstrated. This is desirable even from the linguistic perspective since it well known
dataset

---

that the sound changes are phonologically system- atic and, thus, often similar sound changes oper- ate across different language families (Campbell, 2013). For instance, the sound change involving palatalization of a velar consonant say /k/ > /t// can be observed in the case of Latin caelum /kalom/ to Italian cielo /tfe:l// as well as in the supposed cognate pairs cold versus chill, which is reminis- cence of historical palatalization in Old English. Hence, owing the presence of commonalities across language families terms of sound change phenomena, training models simultancously across multiple language families should be expected to yield better results than when training on a sin- gle language family data at time. This is well reflected in our present work.
applied problems, namely, proto-language reconstruction and cognate reflex prediction, where performs comparable to the existing methods. We have further demonstrated the use of pretrain- ing in proto-language reconstruction, where the pre-trained Cognate Transformer outperforms all the existing methods.
The rest of the paper is organized as follows. Existing methodologies are outlined in $2. The workflow of Cognate Transformer is elaborated in $3. Details of experimentation including dataset information, model hyperparameters, evaluation metrics, etc. are mentioned in $4. Results along with discussions and error analysis are stated in $5.
2 Related Work
1.1 Problem Statements
Several methods to date exist for proto-language re- construction, as mentioned previously. We mention a notable few. Bouchard- et al. (2013) employs a probabilistic model of sound change given the language family's phylogeny, which is even able to perform unsupervised reconstruction on Austrone- sian dataset. Ciobanu and Dinu (2018) performed proto proto-word reconstruction Romance dataset ing conditional random fields (CRF) followed by an ensemble of classifiers. Meloni et al. (2021) employ GRU-attention based neural machine trans- lation model (NMT) on Romance dataset. List et al. (2022a) presents datasets of several families and employs SVM on trimmed alignments.
There are two tasks at hand as mentioned be- fore, namely, cognate reflex prediction and proto- language reconstruction.
An input instance of the cognate reflex predic- tion task consists of a bunch cognate words from one or more related languages with one language marked as unknown; the expected output would be the cognate reflex in that particular language which is marked unknown. An example from the Romance languages is:
Inpul: [French] 3 n V.R, [Portuguese] ?.
[Ttalian] d3 derro
Output: [Portuguese] 3 unipiro
The input for the proto-language reconstruction task consists of cognate words in the daughter languages and the expected output is the corre- sponding word the ancestral (proto-) language. We model this as a special case of cognate reflex prediction problem where the proto-language is always marked as unknown. For instance, in the above example, Latin would be marked as unknown:
The problem of cognate reflex prediction was part of SIGTYP 2022 shared task (List et al.. 2022b), where the winning team (Kirov et al., 2022) models it as an image inpainting problem and em- ploys a convolutional neural network (CNN). Other high performing models include a transformer model by the same team, support vector machine (SVM) based baseline, and a Bayesian phyloge- netic inference based model by Jäger (2022). Other previous approaches include sequence-to-sequee LSTM with attention, i.e., standard NMT based (Lewis al., 2020) and mixture of NMT experts based approach (Nishimura et al., 2020).
Input:
[Latin] ?, [French] 3 zonjeve V B, [Por-
tuguese] 5 unipprr
Output: [Latin] u munipe niperom
1.2 Contributions
The architecture of MSA transformer is part of Evoformer used in AlphaFold2 (Jumper et al., 2021), protein structure predictor. Pre-training of MSA transformer was demonstrated by Rao et al. (2021). Handling MSAs as input by using 2D con-
Our contributions are summarized as follows. We have designed new architecture, Cognate Trans- former, and have demonstrated its efficiency when
For International Phonetic Alphabet (IPA) notation, Phonetic
volutions or GRUs was demonstrated by Mirabello and Wallner (2019) and Kandathil et al. (2022).
see htts://en, Alphabet.

---

[French]
3
n
j
V
B
[French]
3
n
j.E
V
If
[[talian]
d3
P
[Italian]
d3
e
P
0
[Spanish]
[Spanish]
P
[Latin]
3
I
[Latin]
?
?
?
?
[Latin]
U.m
Table 1: Aligned phoneme sequences
Table 2: Trimmed input and output alignments
3
Methodology
In this section, the overall workflow is described. The input phoneme sequences first aligned ($3.1), the resulting alignments are trimmed ($3.2), and then finally passed into the MSA transformer with token classification head ($3.3). In the training phase, the output sequence is also aligned while in the testing phase, trimming is not performed. The first two steps are the same as described in List et al. (2022a) and are briefly described next.
Row Attn.
Col. Attn.
Fully Conn.
Figure 1: A single layer of MSA Transformer.
to the following phoneme with a separator and that specific site is removed. For the last site, the lone phoneme is appended to the penultimate site. Fol- lowing (List et al., 2022a), trimming is skipped for testing as it has been observed to cause de- crease in performance. The reason for this is men- tioned in (Blum and List, 2023). Briefly stating it, gaps in daughter languages can point to a poten- tial phoneme in the proto-language. While training however, they are redundant and can be trimmed since proto-language is part of alignment.
3.1 Multiple Sequence Alignment
The phenomenon of sound change in spoken lan- guages and genetic mutations similar. As result, multiple sequence alignment and the meth- ods surrounding are naturally relevant here as much as they are biology. The phonemes of each language in a single cognate set are aligned based on the sound classes to which they belong. An example of an alignment is given in Table 1.
We use the implementation imported from the library lingpy (List and Forkel, 2021) which uses the sound-class based phonetic alignment de- scribed in List (2012). In this algorithm, the weights pairwise alignments following Needle- man and Wunsch (1970) are defined based on the sound classes into which the phonemes fall. Multi- ple sequences are aligned progressively following the tree determined by UPGMA (Sokal and Mich- ener, 1975).
3.3 MSA Transformer
The MSA Transformer, proposed in (Rao al., 2021), handles two-dimensional inputs with sep- arate row and column attentions (each with mul- tiple heads) in contrast with the usual attention heads found in standard transformer architectures (Vaswani et al., 2017). It uses learned positional embeddings only across rows since group of rows does not make up any sequential data. The out- puts of row attentions and column attentions summed up before passing into fully connected linear layer (see Figure 1). MSA Transformer, de- spite its name, not an encoder-decoder trans- former but rather only an encoder like BERT (De- vlin et al., 2018), 8), except with the ability to handle 2D input (see Figure 2).
3.2 Trimming Alignments
In the example given in Table one can observe that during testing, the final gap (hyphen) in the input languages (i.e., excluding Latin) will not be present. Since the task essentially a token classi- fication, the model will not predict the final token 'm' of Latin. To avoid this, alignments are trimmed illustrated in Table 2 for the same example.
3,4 Workflow
This problem is discussed in detail in (List et al., 2022a) and the solution presented there has been adopted here. In particular, given the sequences to be trimmed, site all tokens are gaps except in one language, then that phoneme is prepended
The aligned input sequences thus trimmed are passed into MSA Transformer as tokens. A single input instance an MSA Transformer is 2D array of tokens. The overall architecture of the Cognate Transformer is illustrated in Figure 2. Due to trim-

---

[CLS]
[La.]
u.m
[SEP]
Classifier
MSA Transformer
[CLS]
[Fr.]
3
18
V
[SEP]
[PAD]
[PAD]
[CLS]
[SEP]
[PAD]
[PAD]
[CLS]
[Sp.]
X
P
[SEP]
[PAD]
[PAD]
[CLS]
[La.]
[MASK]
[MASK]
[MASK]
[MASK]
[MASK]
[MASK]
[MASK]
[MASK]
[SEP]
[PAD]
[PAD]
[PAD]
[PAD]
[PAD]
[PAD]
[PAD]
[PAD]
[PAD]
[PAD]
[PAD]
[PAD]
[PAD]
[PAD]
[PAD]
Figure
Cognate Transformer architecture: an input instance is passed into an MSA transformer, where the
resultant emheddings summed and normalized along columns, which are then finally passed into a classifier.
ming, several phonemes can be joined together as one token. Hence, with trimming the total number of tokens the vocabulary size can be above 1000 even 2000 based on the training dataset, while without trimming the vocabulary size would essen- tially be close to the total number of phonemes possible which would be only a few hundreds.
The normal output of an MSA Transformer is a 2D array of embeddings per instance. To this, we add addition layer that sums over columns give a ID array of embeddings per instance as output. In other words, the overall dimensions of the MSA transformer output were (batch_size x num_languages x msa_length x hidden_size) then, for our case, the final di- mensions after summing up along columns are (batch_size x msa_length x hidden_size) To this, we add normalizer layer followed by clas- sifier, i.e., linear layer followed by cross-entropy loss. This illustrated in Figure
Meloni al. (2021) incorporate the informa- tion regarding the language of word through a language embedding concatenated to the charac- ter/token embedding. We instead treat language information as separate token attached to the beginning of the phoneme sequence. Use of lan- guage embeddings with transformer based models was initially present in the multi-language model XLM (Conneau and Lample, 2019). It was how- ever discontinued in the later versions(Conneal et al., 2020). We similarly have decided to remove the language embedding and instead use special token denoting language as it is less complex in implementation. Other special tokens used include the usual [CLS] to mark the beginning, [SEP] to mark the ending of word, [PAD] for padding, and [MASK] replace "?" in the unknown word (see Table 2) or the word to be predicted. Thus, the
3.5 Pre-training
The described model can support pre-training in form similar to masked language modeling where word from a cognate is entirely masked but the language token remains unmasked corresponding to the language that is to be predicted. other words, pre-training is the same as training for cog- nate prediction task. For proto-language reconstruc- tion, however, pre-training can be done. As result, we pre-train Cognate Transfomer on the data of the cognate reflex prediction task. It is further fine- tuned the proto-language reconstruction task.
input batch padded appropriately is passed on to the MSA Transformer.
We have used the publicly available implementa-

---

Family
Lngs.
Words
Cogs.
Family
Lngs.
Words
Cogs.
Training data
Bai
10
459
3866
Tshanglic
2063
403
Burmish
269
1711
Bai
9
5773
969
Karen
11
365
3231
Sino-Tibetan
7
1426
248
Talo (Yi)
1251
7815
Sui
16
10139
1048
Purus
199
693
Uto-Aztecan
771
118
Romance
4147
18806
Afro-Asiatic
19
2583
340
6690
36122
Dogon
16
4405
971
Table 4: Dataset for Proto-language reconstruction task
Japonic
10
1802
278
Sud--European
1320
512
Burmish
2501
576
into test proportion 0. by List et al. (2022a). We further test for proportions 0.5 and 0.8. For pre- training the Cognate Transformer for this task, we the entire training data of both the tasks with words from proto-languages removed.
32783
5463
Surprise data
Atlantic-Congo
10
1218
388
Hui
19
9750
518
Chapacuran
10
939
187
Western Kwa
8
5214
915
Berta
600
204
4.2
Model Hyperparameters
Palaung
16
1911
196
We have tested two variations of the proposed Cog- nate Transformer architecture, namely CogTran- tiny and Cog CogTran-small. CogTran-ting -iny has hid- den size 128, intermediate size 256, 2 attention heads, and 2 layers with overall million parame- ters. Cog Tran-small has hidden size 256, interme- diate size 512, 4 attention heads, and layers with overall 4.4 million parameters. Both models have a vocabulary size of about 2,300.
Burmish
9
2202
467
Indo-Europen
5
565
212
Karen
2363
379
Bai
10
4356
658
29118
4124
Table 3: Dataset for reflex prediction task
tion of MSA transformer by the authors?, on top of which we added the layers required for the Cognate Transformer architecture. We have used tokeniza- tion, training, and related modules from Hugging- Face library (Wolf et al., 2020). The entire code is made publicly available?.
For pre-training, only CogTran-small is used, since it consistently outperforms CogTran-tiny, The training is carried out with 48 epochs for pre- training, with 9 epochs for finetuning in the proto- language reconstruction task, 24 epochs for non- pre-trained in the same task, and epochs for cognate-reflex prediction task, using Adam opti- mizer with weight decay (Loshchilov and Hutter, 2017) as implemented in HuggingFace transform- ers library (Wolf et al., 2020) with learning rate le-3 and batch size of 64. For finetuning the pre- trained model, the batch size 48.
4 Experimental Setup
4.1 Datasets
We use the SIGTYP 2022 dataset (List et al., 2022b) for the cognate reflex prediction task. consists of two different subsets, namely, training and surprise, i.e., evaluation data from several lan- guage families. The statistics for this dataset is provided in Table 3. Surprise data divided into different test proportions of 0.1, 0.2, 0.3, 0.4, and 0.5 for evaluation. Among these, only report for the test proportions 0. 0.3, and 0.5.
4.3 Evaluation
We use the metrics average edit distance (ED). average normalized edit distance (NED), and B- Cubed FI score (BC) following List et al. (2022a) for evaluating the models. Edit distance is the well- known Levenshtein distance (Levenshtein, 1965), both with without normalization by the lengths of the source and target strings being compared. B- B-Cubed Fl score (Amigó et al., 2009) was ap- plied to phoneme sequences by List (2019), where similarity measured between aligned predicted and gold sequences. B-Cubed Fl score measures
For the proto- lanoula reconstruction task, the dataset provided by List et al. (2022a) is used. consists of data from language families, namely, Bai, Burmish, Karen, Lalo, Purus, and Romance whose statistics are listed in Table 4. This is divided

---

the similarity in the structures and, hence, in the presence of systematic errors, carries less penalty than edit distance. As (normalized) edit distance is a distance measure, the lower the distance, the better the model. On the other hand, for B-Cubed Fl it is opposite, i.c., the higher the score, the better the model. We import the metric functions from the LingRex package (List and Forkel, 2022).
tions of the corresponding scores obtained by the SVM-based model for the Romance dataset: ED 1.579 and NED 190, which we generate using the code made available by List et al. (2022a). Alterna- tively, the disparity in NED could also be attributed to differences in the sizes of the dataset used for training. However, it is unclear how agreement in ED score could have been then possible. Due to absence of both appropriate model and data, we sume that the NMT model we have built is a good reproduction of that built by Meloni al. (2021).
4.4 Methods for Comparison
The results of the cognate reflex prediction task are compared directly against those of the top per- forming model in the SIGTYP 2022 task Kirov al. (2022). Here, direct comparison between the models possible since the datasets including the test divisions are the same.
All models compared in the proto-language re- construction task are 10-fold cross-validated.
Results
In this section, we present and discuss in detail the results of our Cognate Transformer and other state-of-the-ar models on the two tasks.
However, for the proto-language reconstruction task, the previous state-of-the-ee trr---rttte-ttttt model (Meloni al., 2021) reports only on the Romance dataset with test proportion 0.12 and the baseline SVM model (List et al., 2022a) with additional features such as position, prosodic structure, etc., marked as SVM+PosStr is tested only with test proportion 0. However, the code is openly provided for the SVM- based model and, hence, results were generated for other test proportions 0.5 and 0.8 as well.
5.1 Cognate Reflex Prediction
The results of the cognate reflex prediction task are summarized in Table 5. The edit distance (ED), nor- malized edit distance (NED), and B-Cubed F1 (BC) scores are provided for Cognate Transformer across the test proportions 0.1, 0.3, and 0.5 along with the best performing model of the SIGTYP 2022 (List et al., 2022b) task, namely, the CNN inpainting (Kirov et al., 2022). CogTran- -mall consistently outperforms the previous best models across all test proportions. In particular, the difference scores between Cognate transformer and the CNN inpainting model becomes prominent with increas- ing test proportion. Hence, it can be concluded here that Cognate Transformer is more robust than other models. The language family wise results for the best performing model, CogTran-small are provided Appendix A.
To compare the results of proto-language recon- struction with the NMT model given by Meloni et al. (2021) for which the code is not publicly available, build best-effort appropriate model identical to the one described there with 128 units Bidirectional GRU encoder followed by same sized GRU decoder followed by attention and linear layer with dimension 256 followed by a classifier. The input is encoded as a 96-dimensional embedding for each token concatenated with 32-dimensional language embedding. The training parameters are the same previously stated in $4.2 except that the number of epochs trained is 32 and the batch size is 16. For the Romance data part, the results ob- tained are ED 1.287 and NED 0.157 whereas those reported by Meloni et al. (2021) for Romance data (IPA) with almost similar test proportion 12) are ED 1.331 and NED 091 Thus, the edit distances match whereas normalized ones do not. We specu- late that the NED reported by Meloni et al. (2021) could be erroneous due to possible inclusion of de- limiter while calculating the length of the strings, since by (mis)considering delimiters, we obtain a similar NED 0.121 for the model we train. This can be confirmed by observing the ED-to-NED propor-
5.2 Proto-Language Reconstruction
The results of the proto-language reconstruction task are summarized Table with the same eval- uation metrics along with comparisons with other previously high performing models, namely, SVM with extra features by List et al. (2022a) and NMT (GRU-attention) based by Meloni et al. (2021) for the test proportions 0.1, 0.5, and 0.8. Previously, there were no comparisons between SVM-based and NMT-based models. Here. we find that the SVM-based model performs consistently better than the NMT-based model. In other words, the GRU-Attention baaeed NMT model does not appear

---

Test proportion
Method
ED
NED
BC
CogTran-tiny
1.0901
0.2997
0.7521
0.1
CogTran-small
0.8966
0.2421
0.7823
Mockingbird Inpaint (Kirov et al., 2022)
0.9201
0.2431
0.7673
CogTran-tiny
1.3223
0.3497
0.6612
0.3
CogTran-small
1.1235
0.2919
0.6954
Mockingbird Inpaint (Kirov et al., 2022)
1.1762
0.2899
0.6717
CogTran-tiny
1.4521
0.3873
0.6257
0.5
CogTran-small
1.2786
0.3332
0.6477
Mockingbird Inpaint (Kirov et al., 2022)
1.4170
0.3518
0.6050
Table 5: Cogmate reflex prediction results.
Test proportion
Method
ED
NED
BC
CogTran-tiny
0,8081
0.1760
0.7946
CogTran-smais
0.7772
0.1683
0.7968
0.1
CogTran-smmll Pretrained
0.7459
0.1595
0.8081
SVM + PosStr (List et al., 2022a)
0.7612
0.1633
0,8080
NMT GRU + Attn. (Meloni et al., 2021)
1.0296
0.1909
0.7560
Craattnnin
0.9013
0.1966
0.7279
Cog Tran-small
0,8750
0.1899
0.7330
0.5
Cog Tran-smalll Pretrained
0.8177
0.1760
0.7534
SVM PosStr (List al., 2022a)
0.8455
0.1839
0.7425
NMT GRU Attn. (Meloni et al., 2021)
1.2585
0.2362
0.6733
CogTran-ting Tran-tiny
1.1043
0.2455
0.6781
Cog Tron-smmaal
1.0697
0.2359
0,6817
0.8
Cog Trannlsmmml Pretrained
0.9754
0.2142
0.7132
SVM + PosStr (List et al., 2022a)
1.0630
0.2391
0.6800
NMT GRU Attn. (Meloni et al., 2021)
1.8640
0.3546
0.5538
Table 6: Proto- lanoula reconstruction results.
to scale well in harder situations, i.e., for higher test proportions when compared with the other mod- els. While CogTran-small achieves results similar to the SVM-based models, pre-training makes difference. The pre-trained Cognate transformer outperforms all the other models in all test pro- portions. Although the increase in the proportion 0.1 much significant, paired I-test between best performing model and the next best model i.e. CogTran-smal Pretrained and SVM-based yield significance of p < 0.01 in low-resource propor- tions i.c. 0.5 and 0.8 The language family wise results and standard deviations for the best perform- ing model, Cog Tran-mmlll Pretrained are provided in Appendix B. Note that SVM-based model was also part of SIGTYP 2022 (List al., 2022b) where it lags well behind CNN inpainting model. Hence, cognate transformer generalizes well across tasks hence gains from architecture are obvious.
5.3 Error Analysis
To analyze errors, we consider the pre-trained and finetuned CogTran-sman on the proto-language re- construction task for the easiest and hardest test proportions 0.1 and 0.8 over fixed data (without cross-validation). Figure 3 shows the 30 most com- mon sound exchange errors by the models. An ex- ample of sound exchange error, u/a means either is predicted in place of 'u' or vice versa. To make this plot, we first gather the frequencies of sound exchanges for the various language families in data by comparing the aligned predicted and gold recon- structions. These frequencies are normalized for each proto-language language family and finally combined and normalized again. Normalization the language family level important since few language families show more tendencies for certain types of errors than others. Since data is not equally available from all families, a language family with

---

0.175
Proportion (Norma lized
Test Proportion
0.150
0.125
0.100
0.050
0.025
0.000
k/ts t/c
r/j
Ve
a///m//e/aa
Ud w/g
M 2:p1 bbpp ?/?
Sound exchange kk/g w/g s1/s a/a dz/d
w/n 2/21 l:ft
e:/e rvm
0/i
9s/ts 4/2
Proportion (Nomnalized)
0.36
Test Proportion
0.05
0.04
0.03
0.02
0.01
0.00
w/a
Iif
4/-
8//
u/o
V/a
t/c
4/2
3/
t/j
p/b
ts/:
We
p/n
e:/e a:/a +/- 4/
n/j
0t/3
6i
k/g
c/a
LE/0
N/h
cja
4/1 p\/p
Sound
exchange
Figure 3: Top-30 most common sound exchange errors out of over 400 errors for pre-trained Cog sman on proto-language reconstruction task with test proportions 0.1 (top) and 0.8 (bottom).
more data influences the outcome. For example, among the datasets used for the task, the Romance dataset comprises half of them. We observe that Romance data shows more vowel-length-relatee errors as also observed by Meloni et al. (2021) and, thus, proportion of such errors is inflated. Hence, normalization is carried out the language family level prevent such biases. We normalize per fam- ily by dividing the frequency type in family by the total number of errors in that family. Normalized frequencies thus obtained per error type per family are combined by adding up across families and then normalized again.
lation like /n/ /n/, /s/ ~ /b/, etc. Tone exchange errors like / P\ also exist which contribute to about 10% in each of the cases. Affricatives ex- change error /tj/ /ts/ appears prominently in the case of test proportion 0.1. Overall, these are the most general kinds of eπrоrs; however, exact types of errors are observed to be dependent the lan- guage family. Hence, although most general ones are universally observed, significant differences can expected based on the particular datasets.
particular error
5.4 Zero-shot Attempt
Previously, discussed the results of proto- language reconstruction for various test pro- portions. Among these, the highest proportion considered, i.e., 0.8, can be thought of as shot learning case, since for some of the language fami- lies like Purus and Burmish, the number of training instances, i.c., cognate sets is less than 50. We next consider the pre-trained model for the same task without any finetuning: in other words, consider the zero-shot case. The scores achieved by such model are 2.6477 ED, 0.5758 NED, and 0.5499 BC, which means that more than 40% of word on average in generated reconstructions are correct. An example input instance and its correspond- ing output and gold data from the Romance dataset: Input: [Latin] ?, [French] & B i k y t [ u
The most frequent sound exchange errors are plotted in Figure which make up respectively, for test proportions 0.1 and 0.8, about 71% % and 60% of total such errors. One can observe from the plot that the most common vowel errors the exchange of short vowels /u/ and /i/ with neutral vowel /a/, vowel raising-lowering i.e., exchange fi/ /el, ful /ol. grgghthong-mooohhthhee changes /ai/ ~ fi/, tense-laxed exchanges, i.e., /e/ ~ /el and /5/ fol. Vowel length confusions, i.e., fi:/ /1/./e; ~ fe/, /a:/ /a/. /o:/ /s/, /uz/ ~ /u/ also make up significant portion. Overall, vowel/consonant length errors make up about 10% sound exchange errors each in both cases. Among consonant errors, one can observe voiced- unvoiced or glottalized-ungllttaaize
ex-
consonant ex- changes like /p/ ~ /b/, /2k/ / ~ /g/, aspiration errors, i.e., /p"/ ~ /p/. /11 ~ /t/ change of place of articu-
Output: ptrenchpperppike
Gold: [Latin] pattnpperspike ott....mm
ac.t...

---

Conclusions
In the above example, the output language token is incorrect. Since the proto-languages (in this case, Latin) have been excluded entirely in pre-training, this can be expected. One can also observe that the output word completely agrees with neither Italian nor French, although the inclination is more toward the former (with ED of 1). A similar observa- tion was made by Meloni et al. (2021) where the network attended most to Italian since is conser- vative than most other Romance languages.
In this paper, we adapted MSA transformer for two phonological reconstruction tasks, namely, cog- nate reflex prediction and proto-language recon- struction. Our novel architecture, called Cognate Transformer, performs either comparable to or bet- ter than the previous methods across various test- train proportions consistently. Specifically, the pre- trained model outperforms the previous methods by significant margin even at high test-train propor- tions, i.e., with very less trainable data reflecting a more realistic scenario.
5.5 Learned Sound Changes
To the best of our knowledge, this work demon- strates the utility of transfer learning when applied to historical linguistics for the first time. In this paper, the data is in IPA representation, but this is not necessary long words be aligned with properly defined sound classes in the respec- tive orthographic representations. Thus, relaxing the IPA input constraint can increase the amount of trainable data and pre-traue training with more data would most likely improve the performance of not only the problem of automated phonological recon- struction but can be demonstrated in the future for an important related task, namely automated cog- nate word detection. Further, more standard ways of pre-training such as masking only couple of tokens across all languages instead of a complete word single language can be adapted in future.
Here, we consider the finetumed pre-trained model on the proto-language reconstruction task to observe the learned sound changes by the network in the hardest scenario, i.e., with test proportion 0.8. The following example reveals an instance where palatalization appearing in Romance languages is correctly reconstructed
non-palatal consonant: Input: [Latin] ?, [French] is j [Spanish] e [Latin] k
Output:
We now consider metathesis, non-trivial com- plex sound change where positions of phonemes are interchanged. The following example is from the training set which the network learns cor- rectly and demonstrates the metathesis -bil- -ble-. Imput: [Latin]] ?, [French] p s peptt ptpii tibbi [Spanish]] m mpprrep: tißle
Output: [Latin] m www.repptbrllee..mm
Limitations
Following is an example from the test set where the model confuses a complex metathesis pattern occurring in Hispano-Romance, -bil- -Ib- Input: [Latin] ?, [Spanish] i a. [French] y b e, [Portuguese] s
In the task proto-language reconstruction, can be seen from the results (Table 6) that CogTran- small i.e. the plain Cognate Transformer model without pre-training slightly underperforms the SVM-based model at low test proportions. Only the pre-trained model performs well in this scenario.
Output:
[Latin] s b a: re
Gold:
[Latin] s i: b Ee
Further, has already been mentioned in $5.5 that metathesis sound change not being captured correctly by the network which requires further investigation. Overall, very few languages and lan- guage families are included in the data used. Thus. it is desirable to create such datasets for other lan- guages with at least cognacy information to im- prove the unsupervised training firstly, which be then employed in supervised training success- fully with fewer training examples.
Even the model finetuned on test proportion 0.1 does not get this example correct. Its output is Output: [Latin] y b re
Hence, metathesis can be seen as a hard sound change to be learned by this model. This is not sur- prising since metathesis or site exchange does not naturally fit into the sequence alignment approach which fundamentally only models insertions and deletions at any site. Thus, it is worthwhile to investigate more on this aspect by training the net- work on language families that exhibit systematic metathesis to understand its behavior.
Ethies Statement
The data and some modules of code used in this work are obtained from publicly available sources.

---

As stated in $4.4, the code for the model defined by Meloni et al. (2021) was not publicly available, hence we implemented it our own. Thereby the re- sults produced by our implementation may slightly differ from those that would be produced by the original model. Otherwise, there are no foreseen ethical concerns nor conflicts of interest.
Research in Computational Linguistic Typology and Multilingual NLP. pages 63-69, Seattle, Washington. Association for Computational Linguisties.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger. Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. 2021. Highly accurate pro- tein structure prediction with alphafold. Nature,
(96(7873):583-58)
References
Shaun M Kandathil, Joe G Greener, Andy M Lau, and David T Jones. 2022. Ultrafast end-to-end pro- tein structure prediction enables high tigh-throughput exploration of uncharacterized proteins. Pro- ceedings of the National Academy of Sciences, 119(4)e2213334819
Enrique Amigó, Julio Gonzalo, Javier Artiles, and Fe- lisa Verdejo. 2009. A comparison of extrinsic clus- tering evaluation metries hased formal constraints. Information retrieval, 12:461-486.
Frederic Blum and Johann-Mattis List. 2023. Trimming phonetic alignments improves the inference of sound correspondence patterss from multilingual wordlists. In Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP. pages 52-64, Dubrovnik, Croatia. Association for Computational Linguistics.
Christo Kirov, Richard Sproat, and Alexander Gutkin. 2022. Mockingbird at the SIGTYP 2022 shared task: Two types of models for the prediction of cognate reflexes. In Proceedings of the 4th Workshop on Research in Computational Linguistic Typology and Multillingual NLP, pages 70-79, Seattle, Washington. Association for Computational Linguistics.
Alexandre Bouchard-Cãte Coté, David Hall, Thomas L Grif- fiths, and Dan Klein. 2013. Automated reconstruc- tion of ancient languages using probabilistic mod- els of sound change. Proceedings of the National Academy of Sciences, 110(11):422442222
Vladimir Levenshtein. 1965. Dvoičnye kody is-
pravlemiem vypadenij, vstavok i zameščenij simvolov. Doklady Akademij Nauk SSSR, 163(():845-848.
Dylan Lewis, Winston Wu, Arya D. McCarthy, and David Yarowsky. 2020. Neural transduction for mul- tilingual lexical translation. In Proceedings of the 28th International Conference on Computational Lin- guistics, pages 4373-4384, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics.
Lyle Campbell. 2013. Historical linguistics. Edinburgh University Press.
Alina Maria Ciobanu and Liviu P. Dinu. 2018. Ab ini- tio: Automatic Latin proto-word reconstruction. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1604 1604-1614, Santa Fe. New Mexico. USA. Association for Computa- tional Linguisties.
JM List and R Forkel. 2022. Lingrex. Linguistic recon- struction with LingPy.
Johann-Mattis List. 2012. Sea: Phonetic alignment based on sound classes. In New Directions in Logic, Language and Computation: ESSLLI 2010 and ESS- LLI 2011 Student Sessions. Selected Pupers, pages 32-51. Springer.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Out, Luke 7ettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 8440- 8451, Online. Association for Computational Lin- guistics.
Johann-Matie List. 2019. Beyond edit distances: Com- paring linguistic reconstruction systems. Theoretical Linguistics, 45(3-4):247-258.
Johann-Mart Mattis List and Robert Forkel. 2021. Lingpy. a python library for historical linguistics. version 2.6.9.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. Advances in neural information processing systems, 32.
Johann-Mart List, Robert Forkel, and Nathan Hill. 2022a. A new framework for fast automated phono- logical reconstruction using trimmed alignments and sound correspondence patterns. In Proceedings of the 3rd Workshop Computational Approaches to Historical Language Change, pages 89-96, Dublin, Ireland, Association for Computational Linguistics.
Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04880
Gerhard Jäger. 2019, Computational historical linguis- tics. Theoretical Linguistics, 151--2)
Johann-Matte List, Ekaterina Vylomova, Robert Forkel, Nathan Hill, and Ryan Cotterell. 2022b. The SIG- TYP 2022 shared task on the prediction of cognate reflexes. In Proceedings of the 4th Workshop on
Gerhard Jäger. 2022. Bayesian phylogenetic cognate prediction. In Proceedings of the 4th Workshop on

---

Research Computational Linguistic Typology and Muhtilingual NLP, pages 52-62, Seattle, Washington. Association for Computational Linguistics.
Appendix
A Detailed results of CogTran-small on Reflex prediction task
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.05001.
Family VTst prop.
0.1
0.3
0.5
Atlantic-Congo
0.8192
0.7245
0.7027
Carlo Meloni, Shauli Ravfogel, and Yoav Goldberg. 2021. Ab antiquo: Neural proto-language recon- struction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4460-4473, Online, Association for Computational Linguistics.
Hui
0.7933
0.7418
0.7143
Chapacuran
0.6624
0.5847
0.5335
Western Kho-Bwa
0.8572
0.8065
0.7161
Berta
0.7681
0.6758
0.6197
Palaung
0.8815
0.7401
0.6863
Burmish
0.6531
0.5931
0.5261
Claudio Mirabello and Björn Wallner. 2019, rawmsa: End-to-end deep learning using raw multiple se- quence alignments. PloS one, 148))e20220182
Indo-European
0,5012
0.3915
0.3637
Karen
0.8869
0.7914
0.7504
Bai
0.8047
0.7029
0.6444
Saul Needleman and Christian D Wunsch, 1970. general method applicable the search for simi- larities in the amino acid sequence of two proteins. Journal of molecular biology, 48(3)4443-43.
Family wise B-Cubed F scores for model Cog Tran- small against test proportions
Yuta Nishimura, Katsuhito Sudoh, Graham Neubig. and Satoshi Nakamura. 2020. Multi-source neural machine translation with missing data. IEEE/ACM Transactions Audio, Speech, and Language Pro- cessing, 28:569-580.
B Detailed results of Satrrran-smalllpperrrriinin on Proto-language reconstruction task
Family VTest prop.
0.1
0.5
0.8
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexan- der Rives. 2021. Msa transformer. In International Conference on Machine Learning, pages 2844-8856. PMLR.
Romance
0.7765
0.7570
0.7353
Bai
0.7465
0.7108
0.6748
Burmish
0.8426
0.7250
0.6460
Karen
0.8666
0.7845
0.7373
Donald A Ringe and Joseph Eska. 2013. Historical linguistics: Toward a twenty-first century reintegra- fion. Cambridge University Press.
Lalo
0.7221
0.6769
0.6416
Purus
0.8941
0.8662
0.8440
Family wise B-Cubed scores for model Cog Tran- small Pretrained against test proportions
Robert R Sokal and Charles D Michener. 1975. A statis- tical method for evaluating systematic relationships. Multivariate statistical methods, among-grooe gronpp co- variation, page 269.
Test prop.
ED
NED
BCF
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser. and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
0.1
0.065
0.015
0.018
0.5
0.028
0.006
0.008
0.8
0.027
0.006
0.007
Standard Deviations for model CogTran-small Pre- trained
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of the-art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.


Reviews:
[{"review_id": "sjfvJBtZRc", "reviewer": "Reviewer_ZH7n", "summary": "", "questions": "- My impression is that most of the gains from this method come from pretraining on the cognate reflex prediction task rather than on the choice of model architecture. In particular, in Table 4, CogTran-small Pretrained seems to significantly outperform other methods, but CogTran-small and SVM + PosStr are roughly on par with each other. I’m not too familiar with the details of these other methods, but would it be similarly possible to apply the same type of pretraining approach to them? Is there a clear gain from the architecture used or is it mostly the pretraining that leads to performance improvements? \n- Lines 521-524: Do you mean to claim that more than 40% of the reconstructions exactly match their original forms? This seems highly unlikely to me given the zero-shot setting, and I’m not sure how you get that number from the edit distance metrics.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for giving an insightful review.\n\nAs pointed out correctly, the novelty lies in transferring of an idea from one field to the other, which in this case, is from computational biology to computational historical linguistics. Our work is still the first, to the best of our knowledge, to successfully apply a transformer-based architecture while obtaining SoTA results in this particular field.\n\nRegarding the comment that improvements in score (Tables 3-4) are small, this is true for small test-train proportions. However, the increase is indeed significant for higher test proportions. We also stated that high-test-proportion scenarios are realistic or real-life ones (lines 425-430) since these are the cases where linguists get benefited the most by using automated procedures. It is in this low-resource setting that the improvements become significant. Supporting this statement, we report p-values obtained from paired t-tests between the two best performing models on proto-language reconstruction, namely CogTran-small-pretrained and SVM + PosStr.\n\n\n| Prop | ED | NED | B3_F1 |\n|--------|------|--------|----------|\n| 0.1|0.60|0.60|0.90|\n|0.5|0.01|0.003|0.002|\n|0.8|2e-07|1e-08|3e-10|\n\nThe same test could not be carried out for the cognate reflex prediction task, as it was a SemEval task and cross-validation was not performed in order to directly compare the results from the competition. However, cross-validation can verily be incorporated for this task in future.\n\nRegarding the query of whether there is a clear gain from the architecture, the answer is yes but this is indeed unclear from Table 4 where SVM based approach performs similar or even slightly better than non-pre-trained CogTran-small. However, SVM based method was also part of cognate-reflex-prediction SemEval task where it lags significantly (see List 2022b), especially for high test proportions it is well behind the best models. Hence, there is clear gain from the architecture in terms of generalization. Regarding whether other methods, i.e., BiGRU + attention (Meloni et al. 2021) would benefit from pre-training, the answer is possibly so. However, as non-pretrained CogTran models are well ahead of non-pretrained BiGRU + attention (Table 4), it makes little sense to check if the pre-trained variant would bring any significant change in the overall ranking order amongst the models.\n\nRegarding the 40% accuracy mentioned in lines 521-524, the number is deduced from the normalized edit distance score, viz., 0.5758 ~ 60% (line 522). In other words, 40% of a word required no edits.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- As far as I can tell, this model does provide small concrete improvements over previous modeling approaches, at least for the protolanguage reconstruction task (Table 4).\n- Additionally, this paper brings in new methods from outside of NLP (in this case, computational biology). Doing so might expose readers to new techniques they aren’t yet familiar with", "reasons_to_reject": "- The primary contribution of this paper is the application of an existing method (the MSA transformer) to a new domain (phonological reconstruction). While there is some evidence that this slightly improves upon existing methods, the novelty of the work is limited.\n- Some of the improvements in edit distance (Tables 3-4) are reasonably small, and the paper would benefit either from significance testing on its results or from the inclusion of additional datasets. The paper briefly mentions cross-validation and significance testing for the proto-language reconstruction task but not for the cognate reflex prediction task (lines 448-450). The paper is also missing some details about how the significance testing works.\n\n**In response to the rebuttal**: I have adjusted my soundness score (2->3) based on the author response to my concerns about significance testing.", "questions_to_authors": "- My impression is that most of the gains from this method come from pretraining on the cognate reflex prediction task rather than on the choice of model architecture. In particular, in Table 4, CogTran-small Pretrained seems to significantly outperform other methods, but CogTran-small and SVM + PosStr are roughly on par with each other. I’m not too familiar with the details of these other methods, but would it be similarly possible to apply the same type of pretraining approach to them? Is there a clear gain from the architecture used or is it mostly the pretraining that leads to performance improvements? \n- Lines 521-524: Do you mean to claim that more than 40% of the reconstructions exactly match their original forms? This seems highly unlikely to me given the zero-shot setting, and I’m not sure how you get that number from the edit distance metrics.", "paper_topic_and_main_contributions": "This paper proposes a transformer-based model for the tasks of supervised phonological reconstruction and cognate reflex prediction. The approach works by (1) using an out-of-the-box method for aligning phonemes within a cognate set, (2) trimming these alignments and collapsing phonemes accordingly, and (3) feeding the trimmed alignments into an MSA transformer architecture, which is a sequence alignment model from computational biology.\n\nThe approach is evaluated using a SIGTYP 2022 dataset as well as a protolanguage reconstruction dataset from List, et al. (2022a) spanning six different language families. The proposed model shows small gains over existing approaches, at least when pretrained on the cognate reflex prediction task.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- When discussing related work: it would be useful to draw a clearer distinction between supervised and unsupervised approaches to phonological reconstruction. \n- Additional explanation of how the MSA transformer works would be appreciated, if space allows"}}, {"review_id": "XwWLpeDuKs", "reviewer": "Reviewer_8tWY", "summary": "", "questions": "Could please provide the t-test value of the experiments?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for giving an informative review with fine observations.\n\nRegarding the comment that the paper appears somewhat similar to (List 2022a), the pre-processing steps, namely, multiple sequence alignment, alignment trimming (Sec 3.1-2) and the dataset used are indeed same. For this reason and to save space, Sec 3.1 and 3.2 are not exclusively elaborated. Otherwise, sequence alignment and related procedures are quite common in this field and the reason behind can be intuitively understood by noting the similarity between the sound change phenomenon of historical linguistics and the phenomenon of genetic mutation. As pointed out and reiterated in the paper, the architecture is an adaptation from biology. Nevertheless, our work is the first to successfully apply a transformer-based architecture while obtaining SoTA results in this particular field.\n\nRegarding the lack of trimmings during testing, this is an empirical observation for which reason is not known.  Trimmings in the context of training applies mostly to the proto-language, i.e., gold output which often gets 'eroded' to give rise to the daughter languages. Since, proto-language form is unknown during testing phase, trimmings in this context would apply to daughter languages and as proto-languages are often not trimmed or 'eroded' forms of daughter languages, trimming is not only meaningless but also possibly brings down the performance.\n\nRegarding the deviations in the methodology from (Meloni et al. 2021), the transitions from it are motivated by the general trends in natural language processing. This is not only true with regard to the transition from BiGRU-Attention to attention only transformer architecture but also in discontinuing the usage of language embedding. The use of language embedding was found in the earliest multilingual large language model, XLM (Lample and Conneau 2019), but its usage discontinued in the later models. However, whether such step improves or decreases the performance is not verified in our case. Inputting language information as a token is convenient and less complex from an implementation perspective. \n\nRegarding the query to provide the t-test values, we report p-values obtained from paired t-test between the two best performing models on proto-language reconstruction, namely CogTran-small-pretrained and SVM + PosStr.\n\n| Prop | ED | NED | B3_F1 |\n|--------|------|--------|----------|\n| 0.1|0.60|0.60|0.90|\n|0.5|0.01|0.003|0.002|\n|0.8|2e-07|1e-08|3e-10|\n\nThe improvements in scores become significant at high test proportions or low resource settings. Since cognate reflex prediction task was a SemEval task, cross-validation was not performed in order to directly compare the results from the competition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of adapting the transformer model is good and innovative and would further. help in solving the tasks.\nThe paper provides a detailed error analysis report of the sound change error.\nThe experiment details are well documented and should be replicable.", "reasons_to_reject": "To start off with the most problematic: The paper is highly similar in terms of description of the models, experimentation and writing style (List et al., 2022a). The model architecture is also not new and is heavily adapted from the previous work; However, although the authors have cited the paper, they still failed to provide reasons for the adaptation. \n\nSome important facts that I would expect to be more clearly discussed were missing. In section 3.1, Multiple Sequence Alignment, the authors name the methodology and do not justify why they chose to do so. Citing the state-of-the-art does not justify why you are doing these steps in your experiment.\n- The authors also mention that they input language information as a separate token attached to be beginning of the phoneme sequence, which is dissimilar to (Meloni et al., 2021). However, they do not provide any motivation for the steps or the drawback of (Meloni et al., 2021) steps as it is certain that they try to compare both models.\n-Section 3.2 is poorly described. The authors should describe the reasons behind the decrease in performance of the trimming for testing.", "questions_to_authors": "Could please provide the t-test value of the experiments?", "paper_topic_and_main_contributions": "The Paper tries to adapt the MSA transformer model to the problem of automated phonological reconstruction and cognate reflex prediction. The authors also claim that the model can also be pre-trained on the cognate reflex prediction tasks for the proto-language reconstruction. The paper reports experimental results on both tasks and compares them with the state-of-the-art models. It also shows an extensive error analysis report of the models.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Though the paper has citied most of the recent works in the task they lacks citations while defining the terms such as 'Phonological reconstruction, cognate reflex predictions.'", "Typos_Grammar_Style_and_Presentation_Improvements": "There are two spaces after the full stop on some of the pages.  Please recheck the two spaces and make them as one space uniformly."}, "extra_text": {}}, {"review_id": "Ez1sGkHKhy", "reviewer": "Reviewer_LYee", "summary": "", "questions": "Could you report scores per language family as an appendix?\n\nCan you provide confidence intervals or statistical significance of numbers in tables?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We are grateful for your feedback.\n\nAnswering the queries, scores per language family can be verily made available in the Appendix. Standard deviations are provided here for the proto-language reconstruction task obtained from cross-validation for the best performing model CogTran-small-Pretrained.\n\n| Prop | ED | NED | B3_F1|\n|-------|------|--------|---------|\n| 0.1 | 0.06 | 0.02 | 0.02 |\n| 0.5 | 0.03 | 0.006 | 0.008 |\n| 0.8 | 0.03 | 0.006 | 0.007 |\n\nSince cognate-reflex-prediction was a SemEval task, cross-validation was not performed in order to directly compare with the results obtained in the competition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method seems effective, and importing novel ideas from biology is a good idea in this area.\n\nThis paper is meticulously documented and should be replicable.\n\nThe analysis section is useful in identifying further problems to solve in this area.\n\nBeyond standard experiments, some additional manipulation of the percent of data held out for testing is also done, as well as zero-shot experiments which look at reconstruction using a supervised model but without in-family data.\n\n---\n\nThanks for the additional data/analyses reported in the review responses.", "reasons_to_reject": "I don't see any.", "questions_to_authors": "Could you report scores per language family as an appendix?\n\nCan you provide confidence intervals or statistical significance of numbers in tables?", "paper_topic_and_main_contributions": "This paper offers results using a method from computational biology (a kind of transformer model with 2d input) on supervised cognate reconstruction and protolanguage reconstruction.\n\nInputs are aligned, then presented to the transformer encoder, and then results are decoded using a classification head at each position. Results beat a 2022 shared task winner (cognate reflexes) and Meloni 2021 (protolanguage) which are good baselines.\n\nThere is some nice phonetic/phonological analysis.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper applies a multiple sequence alignment (MSA) transformer designed to model protein sequences to the task of automatic phonological reconstruction. The authors find that their approach outperforms the best performing models from recent shared tasks on protoform reconstruction and cognate reflex prediction for a large number of language families. They extend their work with a qualitative error analysis, a discussion of zero-shot reconstruction, and a description of examples exploring how specific diachronic sound changes were learned by their models. 

The reviewers appreciated the novelty of the idea of adapting a method from computational biology to a linguistic task. They also noted the improvements in performance over prior work reported for two distinct tasks with three different accuracy metrics. One reviewer had a very positive evaluation of this paper, finding no reasons to reject. Two reviewers, however, questioned the significance and novelty of the contribution.  Both reviewers seem to have misunderstood parts of the paper, despite the paper's overall clarity, and the author rebuttal addressed their concerns very thoroughly. Despite the middling scores from these two reviewers, the paper is clear and offers a substantial contribution that is entirely appropriate for this track.
Decision: Accept-Main