JAMBU: A historical linguistic database for South Asian languages
Year: 2023

Abstract:
We introduce JAMBU, a cognate database of South Asian languages which unifies dozens of previous sources in a structured and accessible format. The database includes 287k lemmata from 602 lects, grouped together in 23k sets of cognates. We outline the data wrangling necessary to compile the dataset and train neural models for reflex prediction on the Indo-Aryan subset of the data. We hope that JAMBU is an invaluable resource for all historical linguists and Indologists, and look towards further improvement and expansion of the database.1

heading: 1 Introduction
text: A particular concern of historical linguists is studying relatedness and contact between languages. Two languages are related if they share words the arose from a common source, having undergone (potentially different) regular sound changes.2 For example, the German words schlafen and Schiff are cognate to the English words sleep and ship respectively, with the German words having undergone the sound change /p/ → /f/. Using evidence like this from all of the Germanic languages, historical linguists have reconstructed the historical words that gave rise to these terms: *slāpan and *skipą (Kroonen, 2013).
Computational historical linguistics is a growing field that seeks to apply modern computational methods to studying this kind of change (Jäger, 2019; List, 2023). Massive datasets of multilingual cognates are necessary for much of the current research in this area, e.g. on multilingual cognate detection and phoneme-level alignment (List et al., 2018) and automatic comparative reconstruction
1The entire dataset is available at https://github.com/ moli-mandala/data, and a web interface for browsing it is at https://neojambu.herokuapp.com/.
2Per the Neogrammarian hypothesis, sound changes are regular and exceptionless (Osthoff and Brugmann, 1878; Paul, 1880). The reality of sound change is sometimes less ideal.
of historical ancestors of languages (Ciobanu and Dinu, 2018).
South Asia3 as a region is home to a complex historical mesh of language contact and change, especially between the Indo-Aryan and Dravidian language families (Masica, 1976). Yet, South Asia is relatively understudied by linguists compared to the linguistic diversity of the region (Arora et al., 2022). There is no cross-family lexical dataset to facilitate computational study on South Asian historical and contact linguistics. In order to improve this unfortunate state of affairs, we introduce the JAMBU cognate database for South Asian languages. JAMBU includes all cognacy information from the major printed etymological dictionaries for the Indo-Aryan (Turner, 1962–1966) and Dravidian (Burrow and Emeneau, 1984) languages, as well as data from several more recent sources. In this paper, we introduce and analyse our database and train neural models on the reflex prediction task. We hope that this resource brings us closer to the ultimate goal of understanding how the lan-
3When using the term South Asia we refer to the Indian Subcontinent.
68
guages of South Asia have evolved and interacted over time.

heading: 2 Related work
text: CLDF format. CLDF was proposed by Forkel et al. (2018) as a standard, yet highly flexible, format for linguistic data (including cognate databases, etymological dictionaries with reconstructions, and even dictionaries). We use this format for the JAMBU database. Many etymological databases use CLDF to effectively encode complex relations (e.g. loaning) and metadata (e.g. references, phonetic forms, alignments). Some which informed our database design were Rankin et al. (2015); Greenhill et al. (2008).
Cognates. Batsuren et al. (2019) compiled a cognate database covering 338 languages from Wiktionary. They noted that the meaning of cognate varies between research communities—for our purposes as historical linguists, we prefer grouping terms with shared direct etymological sources, while much computational work (e.g. Kondrak et al., 2003) takes a broader definition which includes loanwords or even all semantic equivalents as cognates.
As shown in figure 1, computational historical linguistics has taken on tasks involving cognates such as automatic cognate identification from wordlists (Rama et al., 2018; List et al., 2018; Rama, 2016), cognate/reflex prediction, i.e. predicting the form of a cognate in another language based on concurrent or historical data (List et al., 2022; Bodt and List, 2022; Fourrier et al., 2021; Marr and Mortensen, 2020), and reconstruction of the ancestor form of a cognate set (Durham and Rogers, 1969; Bouchard et al., 2007; Ciobanu and Dinu, 2018; Meloni et al., 2021; He et al., 2022, inter alia).
Other South Asian cognate databases. Cathcart (2019a,b, 2020) and Cathcart and Rama (2020) also previously made use of data from Turner (1962– 1966) by scraping the version hosted online by Digitial Dictionaries of South Asia.
There was an effort to create a new digital South Asian etymological dictionary in the early 2000s, termed the SARVA (South Asian Residual Vocabulary Assemblage) project (Southworth, 2005a). This was unsuccessful however, and only a small portion of the possible cross-family entries were complete. Our database does not incorporate it.

heading: 3 Database
text: The JAMBU database incorporates data from three major language families of South Asia: Indo-European (the Indo-Aryan and Nuristani subbranches), Dravidian, and Austroasiatic (the Munda subbranch). This comes out to 287k lemmata from 602 lects across 23k cognate sets (table 1).
The data is stored in the CLDF structured data format. The overall database structure is described in the file Wordlist-metadata.json, which includes information about the type of data recorded in each column of each file. The file forms.csv includes all lemmata (word form) and associated etymological and linguistic information. The files parameters.csv and cognates.csv include all cognateset headwords and etymological notes for each. The file languages.csv lists all languages in the database and their geographical location. Finally, sources.bib lists all data references in BibTeX format.
For each lemma in forms.csv, we store the following information: a unique form ID; the language ID; the cognate set ID, linking it to other cognate lemmata; a normalised representation of the lemma itself, using our transcription scheme; a gloss in English; the spelling of the lemma in the native script; the phonemic IPA representation of the lemma; the unnormalised form of the lemma taken from the original source; a finer-grained cognate set ID; notes; and references.
For each cognate set, we store a headword, which is usually a common ancestor of the words in that cognateset or a reconstruction of that ancestor if possible. We also store desiderata such as definitions and etymological notes.
Finally, we take an expansive view of what constitutes a “language” in our database. If a word is
known to only be attested in a particular dialect, we list that dialect separately. For example, for the Shina language (northwestern Indo-Aryan), we list 32 geographical dialects. The distribution of languages by number of lemmata is depicted in figure 3.

heading: 3.1 Data sources and scraping
text: The two major data sources are CDIAL (Turner, 1962–1966) and DEDR (Burrow and Emeneau, 1984), which have been scraped in their entirety from web versions hosted by the University of Chicago’s Digital Dictionaries of South Asia project.4 Since the raw data is in HTML with limited structured markup, extracting CLDF-suitable data is a significant hurdle, including matching lemmata to the appropriate language and grouping associated metadata like grammatical gender and etymological notes under the correct form (figure 2). Further cleanup of data from these two sources will have to be done manually.
Since CDIAL and DEDR have not been updated in decades, we are also incorporating more recent sources that refer to them into our database, as well as etymologising newer fieldwork data manually.
4https://dsal.uchicago.edu/dictionaries/
The additional sources we added (some partially) are listed in appendix B.

heading: 3.2 Transcriptions
text: One serious issue has been reconciling differing transcription systems from different sources; transcription schemes vary across sources even for the same language, since there is no standard transcription for South Asian linguistics. An illustrative example of this issue is the variable transcription of the labiodental fricative as v or w.
Turner (1962–1966) normalises entries from various sources into a relatively mundane Indological transcription, i.e., IAST5 with many extensions for the varying phonologies of South Asian languages, but not always consistently. For example, the phoneme /e:/ is notated ⟨ē⟩ for Sanskrit entries, but ⟨e⟩ for Hindi (and in Burrow and Emeneau (1984), as ⟨é⟩ for Malto entries). Elsewhere, e.g., in Bengali and Punjabi, transcriptions adhere to the written form, which do not always adhere to any phonemic analysis of the languages in question. In the case of Kashmiri, Shina, and many other languages, there are now better analyses to base romanisation on than existed at the time of compilation of the sources of Turner (1962–1966). Meanwhile, (Burrow and Emeneau, 1984) does not attempt to conventionalise transcription at all, instead strictly copying the transcription from the original source; e.g. all Bengali entries strictly reflect spelling and do not indicate the differing surface realisations of the orthographic schwa (Johny and Jansche, 2018).
We created a new, more rigidly standardised transcription system based on Indological conventions to unify all our data. We did not want to use pure IPA because it obscures useful cross-lingual pat-
5https://en.wikipedia.org/wiki/International_
Alphabet_of_Sanskrit_Transliteration
terns6 and is not conventional in the Indological research community (especially considering that the database may be of use to non-linguist Indologists as well). For that reason, we use a modified IAST (for instance, using a superscript ⟨h⟩ to notate aspiration and breathy voice distinguishing these from genuine h-clusters) to suit cross-linguistic needs. Some contrasts are made more explicit while notational consistency is maintained across the board.
We used the segments Python library to create orthography normalisation profiles for each source’s transcription scheme (Moran and Cysouw, 2018); some examples of the changes are shown in table 2. So far, forms from all source have not yet been orthographically standardised to our system. However, we developed standardisation scripts covering 204k lemmata, of which 99.7% were automatically converted without errors.

heading: 3.3 Web interface
text: Finally, we developed a web interface for the JAMBU database; see figure 4. Originally, we used the pre-existing clld webapp toolkit for the pub-
6E.g. the Indological a (called a schwa) varies in pronunciation across South Asia, from [a] (Telugu) to [3] (Hindi) to [O~o] (Bengali) to [2] (Nepali).
lication of Cross-Linguistic Linked Data,7 but we later switched to a custom Flask web app designed from scratch in order to have finer control over the database structure and to execute searches on the backend more efficiently. This web interface supports search, filtering, and geographical visualisation. We hope this supersedes the unstructured search interfaces currently available for browsing older etymological dictionaries for these languages (Turner, 1962–1966; Burrow and Emeneau, 1984).

heading: 4 Experiment
text: As a demonstration of the usability of the dataset for computational historical linguistics, we replicate the reflex prediction task of Cathcart and Rama (2020). We train neural models on the task of reflex prediction in Indo-Aryan languages, i.e. predicting the descendant of an Old Indo-Aryan word in a given Indo-Aryan language. Rather than being restricted to data from Turner (1962–1966), we can draw on all the sources present in JAMBU.
We train on 80% of the data and test on the remaining 20%. We compare two models: a bidirectional GRU encoder-decoder with Bahdanau attention and a transformer encoder-decoder with learned positional embeddings. The optimised hyperparameters for the GRU are a learning rate of 2 ⋅10−3, 4 layers, and embedding and hidden size of 64. The transformer had a learning rate of 1 (using the parameter-based adjustment and warmup/decay schedule from Huang et al., 2022), 3 layers, 4 attention heads per layer, embedding size of 64, and FFN size of 128. Both models were trained for 50 epochs without early stopping with a batch size of 1024 on a single Quadro RTX 6000, with a run completing in about 15 minutes.
We evaluate BLEU and TER on the held-out set using the SacreBLEU implementation (Post, 2018), treating a single phoneme as a ‘word’. Even after comprehensive hyperparameter tuning we find that both models achieve similar performances, per the results in table 3. We leave analysis of these models for future work.
7https://github.com/clld/clld

heading: 5 Conclusion
text: In this paper, we introduced JAMBU, the largest and most up-to-date cognate database for South Asian languages. We are continuing to expand the database, incorporating all lexical data that has so far been unused in comparative linguistic work on the region. We believe that the open questions of South Asian historical linguistics cannot be resolved without examining all the information (both synchronic and diachronic) that linguists have collected about language of the region. The old etymological dictionaries are in desparate need of an update. However, much work remains. We briefly discuss some avenues of future work.
Many sources are yet to be incorporated, especially those recording loanwords from external languages (especially Persian, Arabic, English, and Portuguese) and from local literary languages (particularly Sanskrit). We have yet to distentangle cross-lectal interactions and mark lexical isoglosses, which seem necessary to understand the history of language interactions in the region; Kalyan et al. (2018)’s wave model of linguistic change has been thought by many scholars to be suited for South Asian languages, but it has not been operationalised yet due to a lack of comprehensive data (Toulmin, 2006; Kogan, 2017).
Another significant task ahead is extending our database structure to support indicating and analysing more complex cross-lingual interactions. For example, the database as it stands does not distinguish between inheritance from the parent language and loaning mediated by a sibling language.
We have also been working on a consistent orthography for tonemes in the languages where tones are contrastive, such as the northwestern IndoAryan languages (Baart, 2014). Older data from these languages either does not notate tone at all (for tonality was not yet recognized, as in Gawri and Torwali), or represents it indirectly through diachronically correct, but synchronically confusing, spelling systems (as in Punjabi and Kishtwari). So, our work will also involve analyzing and incorporating new data from tonal languages, both from existing sources and our own fieldwork.
Finally, we hope to manually improve data quality once the parsing of old sources is stable. This includes fixing known mistakes, reorganising entries to better indicate indirect derivations and crosslectal loans, and etymological notes that summarise
the extant literature.

heading: A Licensing
text: Data from Burrow and Emeneau (1984) and Turner (1962–1966) has been scraped using the approval of the SARVA project (of which one of the authors was previously involved in) for strictly academic purposes. Additional data added to the dataset has either been manually etymologised (and therefore is an original academic contribution) or obtained with permission of the respective authors.

heading: B Other data sources

heading: Future Work
text: We briefly discuss some avenues of future work. Many sources are yet to be incorporated, especially those recording loanwords from external languages (especially Persian, Arabic, English, and Portuguese) and from local literary languages (particularly Sanskrit). We have yet to distentangle cross-lectal interactions and mark lexical isoglosses, which seem necessary to understand the history of language interactions in the region; Kalyan et al. (2018)’s wave model of linguistic change has been thought by many scholars to be suited for South Asian languages, but it has not been operationalised yet due to a lack of comprehensive data (Toulmin, 2006; Kogan, 2017). Another significant task ahead is extending our database structure to support indicating and analysing more complex cross-lingual interactions. For example, the database as it stands does not distinguish between inheritance from the parent language and loaning mediated by a sibling language. We have also been working on a consistent orthography for tonemes in the languages where tones are contrastive, such as the northwestern IndoAryan languages (Baart, 2014). Older data from these languages either does not notate tone at all (for tonality was not yet recognized, as in Gawri and Torwali), or represents it indirectly through diachronically correct, but synchronically confusing, spelling systems (as in Punjabi and Kishtwari). So, our work will also involve analyzing and incorporating new data from tonal languages, both from existing sources and our own fieldwork. Finally, we hope to manually improve data quality once the parsing of old sources is stable. This includes fixing known mistakes, reorganising entries to better indicate indirect derivations and crosslectal loans, and etymological notes that summarise
the extant literature