Evaluating Cross Lingual Transfer for Morphological Analysis: a Case Study of Indian Languages
Year: 2023

Abstract:
Recent advances in pretrained multilingual models such as Multilingual T5 (mT5) have facilitated cross-lingual transfer by learning shared representations across languages. Leveraging pre-trained multilingual models for scaling morphology analyzers to low-resource languages is a unique opportunity that has been under-explored so far. We investigate this line of research in the context of Indian languages, focusing on two important morphological subtasks: root word extraction and tagging morphosyntactic descriptions (MSD), viz., gender, number, and person (GNP). We experiment with six Indian languages from two language families (Dravidian and Indo-Aryan) to train a multilingual morphology analyzers for the first time for Indian languages. We demonstrate the usability of multilingual models for fewshot cross-lingual transfer through an average 7% increase in GNP tagging in a cross-lingual setting as compared to a monolingual setting through controlled experiments. We provide an overview of the state of the datasets available related to our tasks and point-out a few modeling limitations due to datasets. Lastly, we analyze the cross-lingual transfer of morphological tags for verbs and nouns, which provides a proxy for the quality of representations of word markings learned by the model.

heading: 1 Introduction
text: Morphology analysis is the first step of processing in the classical NLP pipeline. Even in the transformer era, wherein the entire NLP pipeline is replaced with a transformer, the use of morphological segmentation for tokenization instead of statistical subword tokenization has been shown to produce better embeddings, especially for morphologically rich languages (Nzeyimana and Rubungo, 2022). The statistical subword tokenization used in tokenizers such as wordpiece cannot capture
*Work done while at IIT Bombay
morphological alternations (e.g. wordpiece doesn’t treat contextual allomorphs as related) and nonconcatenative morphology (Klein and Tsarfaty, 2020).
One of the tasks that we analyze in our work is root word extraction, which forms an integral component of morphologically informed segmentation. A morphology analyzer can also help speed up language documentation efforts for endangered languages, Moeller et al. (2020) leveraged interlinear glossed text to generate unseen forms of inflectional paradigm using a morphology analyzer. Availability of morphological information can also benefit various downstream tasks such as parsing (Seeker and Çetinoğlu, 2015), machine translation (Tamchyna et al., 2017), language modeling (Park et al., 2021), etc. Our scope of this work is inflectional and concatenative morphology. We also envision our work to be used in bias-aware machine translation, especially from morphologically poor languages to morphologically richer languages. For example, if we want to translate the sentence "My friend was a doctor" to Hindi, we would ideally prefer to have both masculine and feminine translations "Mera dost doctor tha" (masculine) and "Meri dost doctor thi"(feminine), as English sentence has no mention of gender and for Hindi, the gender markers are present on verbs (tha\thi) and pronouns (mera\meri).
Although high-quality morphology analyzers have been built for some Indian languages, they are either rule-based such as Agarwal et al. (2014), or are neural models trained on annotated data which is available in sufficient quantities only for high resource languages (Jha et al., 2018). Building morphology analyzers for low-resource languages remains a challenging task. For low-resource languages, morphological resources are sparse or virtually nonexistent. Multilingual models have shown promising results for cross-lingual transfer from high-resource to low-resource languages (Wu
14
and Dredze, 2019; Lauscher et al., 2020). The main goal of our work is to increase NLP inclusivity. The primary obstacle one encounters while expanding the coverage of NLP models is the lack of usable (annotated) data for most languages; collecting (annotated) data is a painstaking task, especially for endangered languages. When data is sparse, we turn to linguistics to help exploit universalities across languages.
In this work, we study the multilingual capability of mT5 (Xue et al., 2021) to carry out cross-lingual transfer of morphological features and extract the root words given the surface forms. We also test the multilinguality hypothesis that, in the presence of annotated examples of source languages, the required number of annotated examples of the target language to get identical results reduces. We carry out this analysis of cross-lingual transfer within language families and across (language) families and provide pointers to effective usage of multilingual data. The languages we carry out morphological analysis are of the Dravidian family (Tamil, Telugu, and Kannada) and the Indo-Aryan family (Bengali, Hindi, and Marathi). We also give a brief account of the state of datasets available for morphological analysis and their challenges. We finetune mT5 for gender, number, and person tagging for verbs and nouns in six Indian languages: Marathi, Hindi, Bengali, Tamil, Telugu, and Kannada. The features: gender, number, and person (GNP) are hereby referred to as morphosyntactic description (MSD) tags. The current state of the datasets and inconsistency of annotation across languages limits our analysis to GNP tags of verbs and nouns. Our contributions are as follows:
• We test the multilinguality hypothesis that the availability of annotated data of source languages reduces the number of examples of target language required to outperform the monolingual baseline.
• We study inter-family and intra-family transfer in the context of GNP tagging and root word extraction for languages from Dravidian and Indo-Aryan families.
• We analyze how multilingualism helps in the morphological analysis of verbs and nouns, root word extraction, and test the model’s ability to generalize to unseen suffixes.

heading: 2 Related Work
text: Morphological analysis: For morphological analysis, SIGMORPHON (Nicolai et al., 2021) has been one of the venues organizing shared tasks and workshops related to computational morphology and multilingual morphological analysis, especially in the low resource scenarios. Shared tasks such as Cotterell et al. (2016, 2017, 2018), etc. looked at morphological reinflection with an increasing number of languages each year. For morphological reinflection, the output is the surface form, and the inputs are: a root word (or any other form of the root word) and desired features in the surface form (the output). Task 2 in Cotterell et al. (2018) as well as McCarthy et al. (2019) explored morphological analysis and reinfection in context. Jin et al. (2020) and Wiemerslage et al. (2021) were aimed at unsupervised clustering of paradigms, wherein given a lemma list, the goal is to output all the possible forms of a lemma. Morphosyntactic lexicon generation is one task closely related to morphological analysis; Faruqui et al. (2016) used graph-based semi-supervised learning for label propagation. Hulden et al. (2014) used a semi-supervised approach for lexicon construction from concrete inflection tables by generalizing the inflection paradigms from the tables provided. For morphology resources, apart from UniMorph (Batsuren et al., 2022; McCarthy et al., 2020), the MorphyNet database (Batsuren et al., 2021) is a large dataset of methodologically annotated surface forms spanning 15 languages and is extracted from Wiktionary. There have also been efforts to create task-specific models for various components of cross-lingual morphological tagging (Cotterell and Heigold, 2017a; Malaviya et al., 2018) Indian language morphology: Regarding resources for Indian languages, Arora et al. (2022) points out resource scatteredness (rather than
scarcity) as the primary obstacle to developing South Asian language technology and proposes the study of language history and contact as one of the potential solutions. Workshops like DravidianLangTech (Chakravarthi et al., 2021) and WILDRE (Jha et al., 2020) are dedicated specifically to the development of technologies and resources for Indian languages. The UniMorph database (McCarthy et al., 2020) has been one of the recent efforts to extend the coverage of computational morphological resources. Cotterell and Heigold (2017b) trained bidirectional character-based LSTM-based models to demonstrate the effectiveness of the crosslingual transfer. They have trained bilingual models for languages from Romance, Slavic Germanic, and Uralic families. Gupta et al. (2020) trained various sequence labelling models for Sanskrit. Nguyen et al. (2021) trained transformer-based models for various NLP tasks such as PoS tagging, Morphological feature tagging, and dependency parsing for over 100 languages. Nair et al. (2021) carried out a comparative study of existing morphological analyzers for Indian languages to conclude that although morphological analyzers exist for Indian languages like Sanskrit and Malayalam, they are not accurate as compared to the high resource baselines. Elsner (2021) probed an analogical memory-based framework for oneshot morphological transfer to study the abstract representational concepts learned by the transfer networks.

heading: 3 Dataset Challenges
text: Creating a multilingual morphology analyzer would require a union of the sets of features across all the languages and all the parts of speech. The morphological features are modeled as categorical variables in fixed output space. The modeling difficulties arise primarily due to the following: (1) absence of feature annotations for Indian languages, (2) lack of data for all the parts of speech (PoS) except verbs and nouns and (3) variance of markings across PoS and languages. The dataset only contains data for verbs and nouns, which restricts our analysis to those PoS. For these PoS, the feature data is primarily available for Gender, number, and person compared to other features, so we carry out transfer analysis for only those features. We provide a summary of annotated data available in Table 1. Gender, number, and person also happen to be morphological features that are common to nouns and verbs. We provide detailed statistics of
the UniMorph dataset in appendix A. We have used various data sources to demonstrate the scalability of the morphology analyzer to 6 Indian languages. For languages Hindi, Telugu, Kannada, and Bengali, we have used the UniMorph 3.0 (McCarthy et al., 2020) dataset. The number of examples varies across languages. For Bengali, the number of examples available is 4443; for Kannada, it is around 6400; for Hindi, there are about 54K examples, while Telugu has about 1500 examples. All the examples in the UniMorph dataset are either verbs or nouns. For Tamil, morphologically annotated data from the Tamil dependency treebank (Ramasamy and Žabokrtský, 2014) was used. The number of annotated words (verbs+nouns) in the tree bank is 9521, all of which were used. For Marathi, we used the dataset from Bapat et al. (2010). The dataset consists of around 21k annotated words, out of which we used 15k words, nouns, or verbs, to have consistency with other datasets. Although there are other sources of data, such as Bhat et al. (2017), we stick to the UniMorph dataset wherever possible to ensure higher annotation accuracy. The scope of our work limits demonstrating the usefulness of cross-lingual transfer for morphological analysis, so dataset selection and optimizing the number of examples for creating the best morphology analyzer remains a challenge for future research.

heading: 4 Modelling Details

heading: 4.1 Morphological analysis as text to text problem
text: The Multilingual T5 (Xue et al., 2021) is a massively multilingual pre-trained text-to-text transformer model released by Google in 2020. It is pre-trained on the Common Crawl-based dataset and covers 101 languages. It is an encoder-decoder sequence generation model, unlike mBERT, which is an encoder-only multilingual model. Our task of root word extraction requires the generation of text sequences, so we use an encoder-decoder model to avoid training a decoder separately for the given languages. We use the mT5 base model with 580 million parameters for our experiments.
As mT5 is a text-to-text sequence generation model, the tags are generated as a sequence of text, one after the other. The input to the model is the surface form of the words, and the model generates the gender, number, and person tags as a text sequence. Not all the words in the dataset would be
morphologically marked for GNP; for example, in the case of person marking for nouns, the markings are only present on the pronouns (and the surface form changes according to the person). In contrast, the surface form remains the same for common and proper nouns, irrespective of the person. In such cases, where the marking is either trivial or marking is not present on the word or where the marking cannot be inferred from the surface form itself, the model’s expected output is the tag ‘unknown’. The datasets we use contain morphological tags without context; we, therefore, predict the tags solely based on the markings present on the words rather than the context and assign the tag ‘unknown’ to the words for which tags cannot be predicted without context. For all the experiments, unless and otherwise stated: we use the following evaluation strategy: We firstly remove the 20% data (randomly sampled) for each language (which is used for evaluation) and use the remaining 80% data for experiments. We ensure that the randomly sampled data contains unseen paradigms; no surface form of the lemma is present in the training dataset. Across the monolingual and multilingual experiments, the evaluation data remains the same. To avoid the error variation due to bias in sampling (wherein the test set contains all the paradigms available in the training set), we use k-fold cross-validation (with k=5) and report average numbers. The epochs used were 7-15 based on performance on validation data. As far as metrics for measuring model performance are concerned, we report per-tag accuracy for each of the GNP tags, and overall accuracy. The overall accuracy denotes the percentage of instances for which all three tags are predicted correctly by the model. For root word extraction, we consider exact-string match based accuracy.

heading: 4.2 Three modelling strategies
text: We consider three modelling strategies for MSD tagging and root word extraction.
• Joint model: We first use the mT5 as a sequence prediction model wherein the input is the surface form, and the outputs are the root words and MSD tags. The root word and MSD tags are generated as a sequence, with the root word being generated first, followed by MSD tags: gender, number, and person (in that order).
• Multitask model: In the second setting, we use mT5 as multitask model, with MSD tagging and root word extraction being treated as two separate tasks. We prepend a prefix (string) to the input to specify which task should be performed.
• Independent model: In the third setting, we train separate models for root word extraction and MSD tagging, with MSD tags being predicted as a sequence of letters and the input being the surface form.
The input to the model for the second task is the surface form, along with a prefix specifying the task. It should be noted that the choice of prefixes is arbitrary, as long as they are different for each task. While fine-tuning, we add explicit language flags with the respective surface words.
We compare the training strategies in Table 2. The joint sequential prediction leads to the least accuracy in both tasks. Although the multitask framework has higher accuracy than the joint prediction for MSD tagging, it has the lowest accuracy for root word prediction. The multitask framework is expected to have high accuracies because both tasks (MSD tagging and root word extraction) are closely related to each other in the following way: The suffix determines the MSD tags of the surface form, and thus identifying the suffix is an important part of MSD tagging while stripping away the suffix is one of the aspects of root word extraction. The joint multitask training leads to the mixing of outputs (the outputs of both the tasks are in different languages: The MSD tags are in English while
the root words are in the same language as the surface form), as observed during the performance on the test set. Training a separate model for both tasks yields the highest performance, and we use the strategy for all our subsequent experiments.

heading: 5 Low Resource Morphological Analysis Experiments

heading: 5.1 Multilinguality hypothesis
text: We test the multilinguality hypothesis by comparing monolingual models with multilingual models. As seen in Table 3, which shows per-tag accuracy for each gender, number, person tag, along with overall accuracy, multilingual models outperform the monolingual models for most of the languages except for Hindi and Telugu. One of the reasons for worse performance of multilingual model for Hindi is that the Hindi data containers phrases and post-positions with GNP markings, which are not present in other languages. Thus, adding multilingual data leads to drop in model performance due to confusion between word-based markers and post-position based markers. For Dravidian languages, the overall increase in the accuracy of the multilingual model is negligible in the case of Telugu (as compared to the monolingual baseline), the other two languages, Tamil and Kannada, show around 7.8% increase in overall accuracy. Multilingual models also show better scores in the case of per-tag accuracies for all the Dravidian languages, with gender tag having the highest average increase of 4.03% (averaged over languages).
As also seen in column 2 of Table 2, there is an increase in the accuracy of root word extraction and MSD tagging for Marathi. We show more evidence of the multilinguality hypothesis for MSD
tagging through controlled experiments on Bengali and Kannada. We chose these two languages to study the transfer because (1) Kannada shows the highest increase in overall accuracy among the Dravidian languages, and (2) the number of annotated examples of Bengali is the least among the IndoAryan languages. Choosing these two languages helps us clearly observe the effect of cross-lingual transfer and the low resource scenario. Tables 4 and 5 show that the multilingual models outperform the monolingual models irrespective of the source languages, with the increase in accuracy being the highest (around 54% for Bengali and 33% for Kannada) in sparse data scenario, where the number of examples of the target language is 1000. Tables 6 and 7 show evidence of the multilinguality hypothesis for root word extraction.

heading: 5.2 Inter-family and intra-family transfer
text: To study cross-family and intra-family transfer, we use Bengali and Kannada. Bengali has the least number of examples in the Indo-Aryan family and shows the highest increase in accuracy with the addition of multilingual data. Kannada shows the highest increase in overall accuracy when going from a monolingual to a multilingual setting. We do this by varying the number of examples of Bengali in the train set to simulate the low-resource scenario. We also add various sets of languages as a source to check inter-family and intra-family transfer. Note that the last row in all the tables named ‘All Languages’ implies that the data of all six languages were used for training. We study the effectiveness of (family-based) multilingual data by analyzing inter-family and intra-family transfer. In the case of Bengali, we observe that intra-family transfer from languages of the Indo-Aryan family,
viz., Marathi and Hindi, lead to, on average, 2.82% more accuracy as compared to transfer from the Dravidian family for MSD tagging (Table 4). For Kannada, the increase in accuracy from monolingual baselines is more from the languages of the Dravidian family as compared to the Indo-Aryan family when the number of examples of Kannada in the training data is 1000 (Table 5). In all other cases, the increase in accuracy with Dravidian languages is either less or similar to that with IndoAryan languages as a source. When languages from both families are used as source languages, we observe a sharp increase in accuracy for the root word extraction in Bengali and Kannada. For both the languages, Bengali and Kannada, there is a decrease in accuracy when all the languages are used as source languages, compared to the setting where languages from a particular family are used as source languages.

heading: 6 Analysis
text: In this section, we provide further analysis of the cross-lingual transfer of MSD tags for verbs and nouns and root word extraction.

heading: 6.1 GNP tagging for verbs and nouns
text: In Table 3, we note that the increase in overall accuracy in the case of the multilingual model is the highest for Kannada in the Dravidian family as compared to the monolingual model. Bengali has the least number of annotated examples and shows the highest increase in accuracy from monolingual baseline in the Indo-Aryan family. We dive further into the accuracies of Kannada and Bengali. To investigate the sources of multilingual signals, we conduct experiments separately for nouns and verbs. Nouns: For nouns, the person feature is trivially
third (except for pronouns), and the number feature can be inferred from the suffix, but the gender assignment is arbitrary, and we may require a dictionary to get the gender of the nouns. So, if the nouns (present in the test set) have not been seen during training by the model, one of the potential sources of signal regarding gender is the multilingual data. Another source of signals for gender is also the context that the model has seen during the pretraining (for example, the gender of the nouns is marked on verbs). It is hoped that the gender signals will be captured in the representations learned during the multilingual pretraining. The shared latent space, learned by the multilingual models, is assumed to cluster the words of the same meaning in different languages close to each other.
To test the hypothesis regarding the gender of nouns, we test the accuracy of Kannada and Bengali nouns with various training data from multiple languages. As the gender signal can be dictionarybased, we see that the accuracy increases irrespective of the source languages, as shown in Table 9 and Table 12 in appendix B. For both the languages, Bengali and Kannada, we note that the gender accuracy is higher when the source languages are Marathi and Hindi. The higher accuracy is because the number of training examples of Hindi and Marathi combined is around 70k, while the number of examples of all Dravidian languages combined is about 17K, so more the number of nouns in the training set, more would be the hope of getting dictionary signals. As additional evidence, we also carry out zero-shot transfer for nouns of each language. The training data consists of nouns from all the available languages, and the test data contains nouns from the target language, as shown in Table 13 in the appendix B. The zero-shot gender predic-
tion accuracy is non-trivially high for all languages except Tamil (as compared to the case where only verbs are used as source data, wherein we get trivial test accuracies). Tamil has less accuracy for gender as compared to other languages because the number of genders in the Tamil dataset is five, and in a zero-shot setting, the model has no way of knowing the presence of five genders. Verbs: In the case of verbs, all the features: gender, number, and person can be inferred from the suffix. Our hypothesis here is that increase in the accuracy of verbs in the multilingual setting depends on the source language data available for training. As seen in Table 8, the highest increase in the accuracy of Bengali verbs is seen when the source languages are from the same family. The gender accuracy is almost the same for all the languages as Bengali is a gender-less language, and there are no markings of gender on verbs. In the case of Kannada, as shown in Table 11, the highest increase is observed when the source language is Tamil and Marathi. A Significant increase in accuracy when source data from Marathi is used provides evidence of historical contact between these two languages, as has been discussed in Sengupta and Saha (2015).
The historical contact also shows the reason behind the highest increase in overall accuracy when the source languages are Marathi and Hindi (Table 9).
Generalization: We also test the model’s generalization ability to unseen patterns. For example, the suffix ‘raha hei’ in Hindi represents masculine, third person, and singular. We remove all instances of the suffix from the train set, add them to the test set, and check the accuracy of the model on it in multilingual and monolingual settings. In the case of monolingual and multilingual settings, the model’s overall accuracy is 50% for GNP tagging; the tags gender and number are correctly predicted for all the test instances, while the person tag is correctly predicted for 50% of all the instances. The number and gender can be inferred from the suffix itself; however, the person tag depends on the verb as well as the context, thus leading to confusion for the model (as we are not using the context currently.)

heading: 6.2 Root word extraction
text: To test cross-lingual transfer in the case of root word extraction, we test the copy bias learned by the model. The copy bias is an essential part of the learning process for root word extraction, as the output contains most of the characters present in the input except for a suffix. As can be seen in Tables 7 and 6, the root word extraction accuracy increases to a similar extent, irrespective of the source language. We test the copy bias by adding training examples from source languages such that the input and output are the same. The comparison of the effect of copy bias with our standard setup where the source inputs are surface form and source outputs are root words is shown in Table 10. The table highlights that copy bias plays a role in root word extraction and cross-lingual transfer of morphological knowledge (such as the similarity
between morphemes) across the shared embedding space is limited.

heading: 7 Conclusions
text: In this paper, we tested the multilinguality hypothesis for root word extraction and morphosyntactic descriptors (MSD) tagging. We trained multilingual models for MSD tagging and root word extraction using data of six Indian languages spanning two families of the Indian subcontinent. We demonstrated the effectiveness of data from languages of the same and different families and how it can be leveraged to train morphological analysis models for low resource languages. We also analyzed how cross-lingual transfer of morphological knowledge happens for nouns and verbs along with the copy bias, which forms a significant component of the root word extraction. Our framework can be extended to multiple tags as well as more low resources languages as annotated data becomes available. We see our work as an important step in the direction of bias-aware machine translation to morphologically rich languages.

heading: 8 Limitations
text: One of the limitations of our work is the unavailability of context data and unavailability of phrasebased annotations for all languages except Hindi. The unavailability of phrase-based annotations prevents the usage of universal tags because markings that are present on a single word in highly agglutinative languages like Marathi or Tamil get expressed on 2–3 words in isolating or fusional languages like Hindi or Bengali (where markings are present on post-positions). The benefits of using phrase level morphology over token level morphology have been discussed in Goldman and Tsarfaty
(2021). For example, the word ‘sochega’ in Hindi will have MSD tags: future tense and male gender, while in English, it would take two words, ‘he will think’ to express the same amount of morphological information. The presence of contextual data can also help to disambiguate MSD tags. The other limitation of our work is the mismatch between the languages for which pretrained models (especially encoder-decoder models) are available and the languages for which we have the annotated data. For example, UniMorph dataset contains annotated examples for Assamese and Sanskrit, but we do not have multilingual pretrained encoder-decoder models for these languages.

heading: 9 Ethics Considerations\Broader Impact
text: Our work is on morphological analysis of low resource languages. We aim to increase the coverage of NLP tools through our work. It is inline with making language technologies accessible for wider range of audiences who do-not have commonly researched high resource languages like English, French as their native language. Our work is also a step towards automating the process of documentation of endangered languages.

heading: Acknowledgements
text: We thank Simran Khanuja for participating in the early phases of this research. We also thank CFILT Lab, IIT Bombay for providing resources and some of the datasets. We thank Kuzman Ganchev and Srini Narayanan for comments on the draft. We would like to thank CMiNDS department, IIT Bombay along with Google India Research Lab for providing with the opportunity for conducting this research. Finally, we would like to thank Matthias Bauer for providing help with writing and structuring of the final draft.

heading: A Appendix: Satistics of UniMorph Dataset
text: The UnimOrph dataset’s statistics are shown in table 14. A total of 26 features are available in the meta-data of the UniMorph dataset. They include Aktionsart, Animacy, Argument marking, Aspect, Case, Comparison, Definiteness, Deixis, Evidentiality, Finiteness, Gender, Information Structure, Interrogativity, Language Specific features, Mood, Number, Other, Part of speech, Person, Polarity, Politeness, Possession, Switch reference, Tense, Valency, Voice. For most Indic languages, the annotations are present for not more than eight features
per language. The set of features for which annotations are current varies across languages. We give the proportion of words in the dataset for which feature annotations are present. We provide statistics for Gender, Number, Person, Tense, Aspect, and Modality, characteristic features of verbal morphology. We also provide statistics for case, number, number, and person for nouns. The ‘others’ section represents the features with the highest proportion of tags, from gender, number, person, tense, aspect, modality, and case. Also, one thing that must be noted is that the amount of data available for verbs is almost 5 times the data available for nouns for most of the languages, so the number in the ‘total’ row is dominated by statistics of verb. For Hindi, the nouns data is completely absent.

heading: B Appendix: Cross-Lingual transfer Nouns—Additional Tables

heading: Future Work
text: For example, the word ‘sochega’ in Hindi will have MSD tags: future tense and male gender, while in English, it would take two words, ‘he will think’ to express the same amount of morphological information. The presence of contextual data can also help to disambiguate MSD tags. The other limitation of our work is the mismatch between the languages for which pretrained models (especially encoder-decoder models) are available and the languages for which we have the annotated data. For example, UniMorph dataset contains annotated examples for Assamese and Sanskrit, but we do not have multilingual pretrained encoder-decoder models for these languages