Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages
Year: 2023

Abstract:
We present, Naamapadam, the largest publicly available Named Entity Recognition (NER) dataset for the 11 major Indian languages from two language families. The dataset contains more than 400k sentences annotated with a total of at least 100k entities from three standard entity categories (Person, Location, and, Organization) for 9 out of the 11 languages. The training dataset has been automatically created from the Samanantar parallel corpus by projecting automatically tagged entities from an English sentence to the corresponding Indian language translation. We also create manually annotated testsets for 9 languages. We demonstrate the utility of the obtained dataset on the Naamapadam-test dataset. We also release IndicNER, a multilingual IndicBERT model finetuned on Naamapadam training set. IndicNER achieves an F1 score of more than 80 for 7 out of 9 test languages. The dataset and models are available under open-source licences at https: //ai4bharat.iitm.ac.in/naamapadam.

heading: 1 Introduction
text: Named Entity Recognition (NER) is a fundamental task in natural language processing (NLP) and is an important component for many downstream tasks like information extraction, machine translation, entity linking, co-reference resolution, etc. The most common entities of interest are person, location, and, organization names, which are the focus of this work and most work in NLP. Given high-quality NER data, it is possible to train goodquality NER systems with existing technologies (Devlin et al., 2019). For many high-resource languages, publicly available annotated NER datasets (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003a; Pradhan et al., 2013; Benikova et al., 2014) as well as high-quality taggers (Wang et al., 2021; Li et al., 2020) are available.
∗Equal contribution †Project leads; correspondence to rmurthyv@in.ibm.com,
However, most Indic languages do not have sufficient labeled NER data to build good-quality NER models. All existing NER corpora for Indic languages have been manually curated (Lalitha Devi et al., 2014; Murthy et al., 2018; Pathak et al., 2022; Murthy et al., 2022; Malmasi et al., 2022; Litake et al., 2022). Given the number of languages, the expenses and the logistical challenges, these datasets are limited on various dimensions viz. corpus size, language coverage, and broad domain representation. In recent years, zero-shot crosslingual transfer from pre-trained models, fine-tuned on task-specific training data in English has been proposed as a way to support various language understanding tasks for low-resource languages (Hu et al., 2020). However, this approach is more suitable for semantic tasks and the cross-lingual transfer does not work as well for syntactic tasks like NER when transferring across distant languages like English and Indian languages (Wu and Dredze, 2019; Karthikeyan et al., 2020; Ruder et al., 2021). Hence, there is a need for in-language NER training
ankunchu@microsoft.com
10441
data for Indic languages. In recent years, the paradigm of mining datasets from publicly available data sources has been successfully applied to various NLP tasks for Indic languages like machine translation (Ramesh et al., 2022), machine transliteration (Madhani et al., 2022) and many natural language understanding and generation tasks (Kakwani et al., 2020; Kumar et al., 2022). These approaches have led to the creation of large-scale datasets and models with broad coverage of Indic languages in a short amount of time. Taking inspiration from these successes, we also explore the automatic creation of NER datasets by utilizing publicly available parallel corpora for Indian languages and high-quality English-named entity taggers. In this work, we undertake the task of building large-scale NER datasets and models for all major Indic languages.
The following are the contributions of our work: • We build Naamapadam1, the largest publicly available NER dataset for Indic languages for 11 languages from 2 language families. Naamapadam contains 5.7M sentences and 9.4M entities across these languages from three categories: PERSON, NAME, and LOCATION. This is significantly larger than other publicly available NER corpora for Indian languages in terms of the number of named entities and language coverage. Table 1 compares Naamapadam with other NER datasets. • We create the Naamapadam test set, containing human-annotated test sets for 9 languages on general domain corpora, that can help in benchmarking NER models for Indic languages. Existing testsets are limited to fewer languages or are domain-specific. • We also train a multilingual NER model, IndicNER, supporting 11 Indic languages. Our models achieve more than 80% F1 score on most languages
1Naamapadam means named entity in Sanskrit
on the Naamapadam test set. To the best of our knowledge, no publicly available NER models exist for Indian languages. • We create the NER training corpora by projecting annotations from English sentences to their Indic language translations in parallel corpora. We show that the projection approach is better than approaches based on zero-shot transfer or teacherstudent training. This allows for the inexpensive creation of data, at scale, while maintaining high quality. Hence, we recommend the use of a projection approach compared to these approaches when a reasonable amount of parallel corpora is available. This is a valid assumption for many mid-resource languages which today lack good NER models.

heading: 2 Related Work
text: We discuss the state of NER datasets for Indian languages and common methods used to improve NER for low-resource languages.

heading: 2.1 NER data for Indian languages
text: Very limited NER corpora are available for Indian languages. They are mostly small in size and do not cover all major Indian languages. The FIRE2014 dataset (Lalitha Devi et al., 2014) is available for 4 languages. It was created by collecting sentences/documents from Wikipedia, blogs, and, online discussion forums. The WikiAnn dataset (Pan et al., 2017) is available for around 16 Indian languages - but these are all Wikipedia article titles that are not representative of natural language sentences and the annotations are very noisy. Moreover, the examples are Wikipedia article titles and are not representative of natural language sentences. Murthy et al. (2022) contributed the largest human-annotated dataset for Hindi (CFILTHindi) in terms of volume and diversity with over 100k sentences, all annotated by a single expert individual over a span of several years. There are
a few small datasets for Indian languages: CFILTMarathi (Murthy et al., 2018), MahaNER (Litake et al., 2022), AsNER (Pathak et al., 2022) and MultiCoNER (Malmasi et al., 2022). In contrast, Naamapadam has greater language coverage and is much larger compared to other datasets. It is also representative of general domain text.

heading: 2.2 Annotation Projection
text: Named entity corpora can be created for lowresource languages by projecting named entity annotations from sentences in the source language (high-resource) to the corresponding words in the translated sentence in the target language (lowresource). Yarowsky et al. (2001) first demonstrated how annotations can be projected using word alignments given parallel corpora between two languages. In addition to word alignments, projection can also be based on matching tokens via translation and entity dictionaries as well as transliteration (Zhang et al., 2016; Jain et al., 2019). Agerri et al. (2018) extended this approach to multiple languages by utilizing multi-way parallel corpora to project named entity labels from multiple source languages to the target language. When parallel corpus is not available, but good quality MT systems are available, annotated corpora in one language can be translated to another language followed by annotation projection (Jain et al., 2019; Shah et al., 2010). Bilingual dictionaries or bilingual embeddings have been used as a cheap alternative for translation in low-resource scenarios (Mayhew et al., 2017; Xie et al., 2018). The WikiAnn project creates ‘silver standard’ NER corpora using a weakly supervised approach leveraging knowledge bases and cross-lingual entity links to project English entity tags to other languages (Pan et al., 2017). Given the availability of sufficient parallel corpora for major Indian languages (Ramesh et al., 2022), we use the annotation projection approach for building Indian language NER datasets.

heading: 2.3 Zero-shot Cross-lingual Transfer
text: This is a popular method for low-resource languages that relies on shared multilingual representations to help low-resource languages by transferring information from high-resource language NER models. Particularly, NER models finetuned on pre-trained language models like mBERT (Devlin et al., 2019), XLM-RoBERTa (Conneau et al., 2020) for high resource languages are used to tag low-resource language sentences (zero-shot NER).
Pires et al. (2019) demonstrate that multilingual models perform well for zero-shot NER transfer on related languages. However, zero-shot performance is limited for distant languages Wu and Dredze (2019), particularly when there are structural/word order differences between the two languages (Karthikeyan et al., 2020). Unlike many other NLP tasks, zero-shot cross-lingual NER has seen only limited benefit from recent advances in cross-lingual representation learning (Ruder et al., 2021). To overcome this limitation, a knowledge distillation approach has been proposed to create synthetic in-language training data (Wu et al., 2020). Here, the source language teacher NER model is used to create distillation data in the target language via zero-shot cross-lingual transfer, which is used to train a target language model. We find that the projection-based approaches outperform zero-shot transfer and knowledge distillation approaches.

heading: 3 Mining NER Corpora
text: Following Yarowsky and Ngai (2001a,b), our method for building NER corpora is based on projecting NER tags from the English side of an English-Indic language parallel corpora to the corresponding Indic language words. For our work, we use the Samanantar parallel corpus (Ramesh et al., 2022) which is the largest publicly available parallel corpora between English and 11 Indic languages. Figure 1 illustrates our workflow for extracting named entity annotated Indic sentences from an English-Indic parallel sentence pair. It involves the following stages: (a) tagging the English sentence with a high-accuracy English NER model (Sec 3.1), (b) aligning English and Indic language words in the parallel sentence pair (Sec 3.2), (c) projecting NER tags from the English sentence to Indic words using the word alignments (Sec 3.3). These stages are further described in this section.

heading: 3.1 Tagging the English side
text: We tag the named entities on the English side of the parallel corpus using a publicly available, highquality off-the-shelf English NER tagger. We evaluated various English taggers on the CoNLL dataset (see Table 9 for comparison). Since the parallel corpora contain a significant number of Indian named entities, we also performed a manual analysis to understand the taggers’ performance on these entities. Based on these comparisons, we used the BERT-
base-NER2 model for tagging the English portion of the Samanantar parallel corpus. We ignore the MISC tags predicted by the BERT-base-NER and focus on PERSON, LOCATION, and ORGANIZATION tags only. MISC is a very open-ended category and we found that it was not easy to reliably align MISC tagged entities from English to Indian languages.

heading: 3.2 Word Alignment
text: For every sentence pair in the parallel corpus, we align English words to the corresponding Indic language words. We explore two approaches for learning word alignments: (a) GIZA++ (Och and Ney, 2003) with default settings, (b) Awesomealign (Dou and Neubig, 2021) finetuned on parallel corpora with Translation Language Modeling and Self-training objectives. We use softmax to normalize the alignment scores in our experiments.

heading: 3.3 Projecting Named Entities
text: The next step is the projection of named entity labels from English to the Indic language side of the parallel corpus using English-Indic language word alignment information. We want the entity projection algorithm to ensure the following: (1) adjacent entities of the same type should not be merged into one single entity, and (2) small errors in word alignment should not cause drastic changes in the final NER projection. To ensure these, we project the entities as a whole (i.e., the entire English entity phrase and not word by word) by identifying the minimal span of Indic words that encompass all the aligned Indic words. Word alignment errors could lead to incorrect named entity projection as shown in Figure 2. In this case, alignments in one direction are erroneous leading to wrong projection. We rely on the intersection of alignments in both directions to reduce alignment errors and thus ensure improved projection as illustrated in Figure 3. We show some examples of projections from Awesome-align in Appendix C.
In Figure 2 and 3, we use black arrows to indicate the alignment from Hindi to English direction and blue arrows to indicate the alignment from English to Hindi. The alignment from Hindi to English is correct. On the contrary, the alignment in English to Hindi direction suffers due to the presence of additional Hindi words. The word ‘Soren’ gets aligned to additional Hindi words ’photo’ and ‘PTI’ which are not part of PERSON named entity
2https://huggingface.co/dslim/bert-base-NER
(Figure 2). In order to minimize such errors, we take advantage of bidirectional alignments. We take the intersection of alignments in both directions, which improves the precision of alignments and hence improves projection accuracy (Figure 3). We will include the description in the revised version. Figure C is described in detail in Appendix C.

heading: 3.4 Sentence Filtering
text: After NER projection, we apply the following filters to the tagged Indic sentences. Sentences without Named Entities. Many English sentences in the Samanantar corpus are not annotated with any entities. We retain only a small fraction of such sentences (≈ 1%) for training the NER model so the model is exposed to sentences without any NER tags as well. Sentences with low-quality alignments. We observe that most of the errors in the Indic-tagged corpus arise from word alignment errors. Hence, we compute a word alignment quality score for each sentence pair. This score is the product of the probability of each aligned word pair (as provided by the forward alignment model in the case of GIZA++ and the alignment model by awesome align) normalized by the number of words in the sentence. We retain the top 30-40% sentences to create the final NER-tagged data for Indic languages (See Table 11 for filtered data statistics).

heading: 3.5 Qualitative Analysis
text: To quantify the quality of the labeled data obtained, we select a small sample of 50 sentences3 and obtain manual annotation for the 9 languages namely Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and, Telugu. We also project the named entities on this small set of 50 sentences using the projection approach discussed earlier. Since the ground truths are known, the F1 scores can be calculated. Table 2 presents the F1 scores on the manually labeled set using various projection approaches. We observe that both GIZA++ and Awesome-align word alignment approaches obtain similar performance. On average, Awesome-align provides the best F1 scores, hence, moving forward, we consider the datasets from the Awesome-align approach unless specified otherwise.
3this set is later expanded by annotating more sentences to form the test set

heading: 3.6 Dataset Statistics
text: Table 3 shows the statistics of the final Naamapadam dataset. We create train, dev, and, test splits. Testsets are then manually annotated as described later in Section 4. Most languages have training datasets of more than 100K sentences and 500K entities each. Some languages like Hindi have more than 1M sentences in the training set. Compared to other datasets (See Table 1), the Naamapadam has a significantly higher number of entities. Even though the dataset is slightly noisy due to alignment errors, we hope that the large dataset size can compensate for the noise as has been seen in many NLP tasks (Bansal et al., 2022).
We have manually annotated testsets of around 500-1000 sentences for most languages. The Assamese and Oriya testsets are silver-standard (the named entity projections have not been verified yet). Work on the creation of larger, manually annotated testsets for these languages is in
progress.

heading: 4 Testset Creation
text: We have created Naamapadam-test: manually annotated test set for Indian language NER evaluation. The Naamapadam-test comprises 500-1000 annotated sentences per language for 9 languages namely Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and, Telugu. The annotators were provided sentences with named entity annotations obtained using the methodology described in Section 3. The annotators had to verify if the projected NER annotations were correct and rectify the annotations if incorrect. They were asked to follow the CoNLL 2003 annotation guidelines (Tjong Kim Sang and De Meulder, 2003b). The human annotations were contributed by volunteers who are native language speakers.

heading: 4.1 Inter-Annotator Agreement
text: We compute the inter-annotator agreement on a sample of two annotators for each language using Cohen’s kappa coefficient (Cohen, 1960). The scores are shown in Table 4. They are all above 69% signifying good-quality annotations.

heading: 5 Experimental Setup
text: We analyze the performance of models trained on the Naamapadam-train dataset with alternative approaches for low-resource NER and to models trained on publicly available datasets. To this end, we investigate the following research questions:
RQ1: Are models trained on data obtained from projection approach (Naamapadam-train) better than zero-shot and teacher-student models?
RQ2: How does the model trained on publiclyavailable dataset fare against the model trained on Naamapadam-train data? We evaluate it on the Naamapadam-test set.

heading: 5.1 Train Dataset
text: In order to demonstrate the usefulness of our Naamapadam-train dataset, we fine-tune the mBERT model (Devlin et al., 2019) on the Naamapadam-train data and test on Naamapadamtest set. We additionally fine-tune the mBERT model on the train split of publicly available datasets and test on Naamapadam-test set. We consider the following datasets in our experiments
• WikiANN: We use the train split of the data released by Rahimi et al. (2019). Due to the languages covered, this is the most widely used
dataset. However, we observe the tagged data to be highly erroneous and does not contain complete sentences, but just titles. Appendix A discusses the issues with the WikiNER dataset.
• FIRE-2014: The FIRE-2014 dataset (Lalitha Devi et al., 2014) contains named entity annotated dataset for Hindi, Bengali, Malayalam, Tamil, and, English languages. We train language-specific models on the train splits of these datasets and evaluate the performance on our test set.
• MultiCoNER: We use the Hindi and Bengali named entity annotated data from Malmasi et al. (2022).4
• CFILT: We use the CFILT-HiNER dataset created for Named Entity Recognition in Hindi language (Murthy et al., 2022). The dataset was from various government information web pages, and newspaper articles. The sentences were manually annotated. We also use the CFILT-Marathi dataset created for Named Entity Recognition in Marathi (Murthy et al., 2018).
• MahaNER: We use the Marathi named entity annotated data from Litake et al. (2022).
For a fair comparison with models trained on our dataset, we include only PERSON, LOCATION, and, ORGANIZATION entities. The rest of the named entities if present (FIRE 2014, CFILT Marathi, MultiCoNER) are considered non-named entities.

heading: 5.2 NER Fine-tuning
text: Recently, sequence labeling via fine-tuning of pretrained language models has become the norm (Devlin et al., 2019; Conneau et al., 2020; Kakwani et al., 2020). We fine-tune the pre-trained mBERT model (Devlin et al., 2019) and report the results in our experiments. The input to the model is a sequence of sub-word tokens that pass through the Transformer encoder layers. The output from the transformer is an encoder representation for each token in the sequence. We take the encoder representation of the first sub-word (in case the word gets split into multiple sub-words) and is passed through the output layer. The output layer is a linear layer followed by the softmax function. The
4https://multiconer.github.io/
model is trained using cross-entropy loss. We use the Dhamecha et al. (2021) toolkit for fine-tuning our models.

heading: 5.3 Baseline Comparison
text: Our proposed approach can be seen as a crosslingual approach since the training data is created by projection from English to Indic sentences. Hence, we compare the performance of our model with zero-shot learning (Pires et al., 2019) and teacher-student learning (Wu et al., 2020). We describe the baseline approaches in detail below:

heading: 5.3.1 Zero-shot NER
text: To perform Zero-shot transfer, we consider the mBERT model fine-tuned for NER task in English. We use the publicly available fine-tuned NER model5 which is trained for NER in 10 highresource languages (English, Arabic, Chinese, and some European languages). We directly test the performance of this model on Naamapadam largetest dataset (Bn, Gu, Hi, Kn, Ml, Mr, Pa, Ta, Te) and Naamapadam small-test datasets (As, Or) respectively.

heading: 5.3.2 Teacher-Student Learning
text: We use the publicly available fine-tuned NER model5 to create synthetic named entity annotated training data for the Indic languages. We annotate the Indic language portion of the Samanantar corpus using the above NER model. This synthetic labeled data is used to fine-tune for NER task in each of the languages respectively.
5Davlan/bert-base-multilingual-cased-ner-hrl
Wu et al. (2020) trained the student model to mimic the probability distribution of the entity labels by the teacher model. In our approach, we follow the Hard Labels approach where we round the probability distribution of the entity labels into a one-hot labeling vector to guide the learning of the student model.

heading: 5.4 Implementation Details
text: We use the Huggingface library6 (Wolf et al., 2020) to train our NER models. We use NVIDIA A100 Tensor Core GPU to run all the experiments. We use bert-base-multilingual-cased (169.05M) as the base pre-trained model in all our experiments. We tune hyper-parameters based on F1-Score on the validation set. We use the following range of values for selecting the best hyper-parameter.
• Batch Size: 8, 16, 32
• Learning Rate: 1e-3, 1e-4, 1e-5, 1e-6, 3e-3, 3e-4, 3e-5, 3e-6, 5e-3, 5e-4, 5e-5, 5e-6
Once we obtain the best hyper-parameter, we finetune the model for 2 epochs with 5 different random seeds. We report the mean and standard deviation of the 5 runs.

heading: 6 Results
text: We now present the results from our experiments.

heading: 6.1 RQ1
text: We now answer the question if the models trained using data from projection approach are better than
6https://github.com/huggingface/transformers/ tree/main/examples/pytorch/token-classification
cross-lingual zero-shot and teacher-student models?
Table 5 reports the results from our experiments. Apart from Hindi, Malayalam, and, Marathi we observe relatively poor results for other Indic languages in the Zero-Shot setting. Zero-shot techniques perform quite well in high-resource languages like Hindi, scoring a respectable 75.96%. However, for Assamese and Oriya languages the results are very poor. The Teacher-Student approach in comparison with the zero-shot approach gives very poor results.
We observe that the models trained using the Naamapadam-train dataset give the best F1 scores across languages. In general, we observe better performance from data obtained using Awesome-align (Dou and Neubig, 2021) compared to GIZA++ (Och and Ney, 2003). Moving forward, we choose the data obtained using Awesome-align (Dou and Neubig, 2021) in all our experiments.
IndicNER: Multilingual Fine-tuning
Multilingual fine-tuning on a downstream task has been shown to outperform language-specific fine-tuning in low-resource scenarios (Dhamecha et al., 2021). We also fine-tune a multilingual model on the combined data of all languages in Naamapadam-train. We refer to this model as IndicNER. Table 6 reports the results from our experiments. We observe that the multilingual model on average performs better than the monolingual models.
It can also be seen that for extremely lowresource languages like Assamese, the multilingual model performs a lot better than the others with a jump in F1 score from 45.37 to 60.19.

heading: 6.2 RQ2
text: In this section, we answer the question if the models trained on the Naamapadam-train data fare better against models trained on other publicly available labeled datasets when tested on Naamapadam-test set?
Table 8 reports the results of our experiments. We observe that the model fine-tuned on Naamapadam-train data outperforms all other models by a significant margin indicating the utility of our labeled data. Only the models trained using CFILT-HiNER (Murthy et al., 2022) and MahaNER (Litake et al., 2022) obtain reasonable F1 on Hindi and Marathi. This underlines the importance of large, high-quality data and shows that projection methods can help to create such data at scale.

heading: 6.3 Error Analysis
text: We observe that boundary error is our model’s most common error type. The model sometimes identifies named entities partially. For example, in the case of Organization entities, our model only tags A B C as an organization entity when the correct entity phrase is, say, A B C Limited. Similarly, for Location entities, our model only tags A B as location entity when the correct entity phrase is A B Hospital. This could be attributed to some of the boundary errors present in our training data due to alignment errors.

heading: 7 Conclusion
text: We take a major step towards creating publicly available, open datasets and open-source models for named entity recognition in Indic languages. We introduce Naamapadam, the largest entity recognition corpora for 11 Indic languages containing more than 100K training sentences per language, and covering 11 of the 22 languages listed in the Indian constitution. Naamapadam also includes manually labelled test set for 9 Indic languages. We also build IndicNER, an mBERT based multilingual named entity recognition model for 11 Indic languages. We also provide baseline results on our test set along with a qualitative analysis of the model performance. The datasets and models will be available publicly under open-source licenses. We hope the dataset will spur innovations in entity recognition and its downstream applications in the Indian NLP space.

heading: Acknowledgements
text: We would like to thank the Ministry of Electronics and Information Technology7 of the Government of India for their generous grant through the Digital India Bhashini project8. We also thank the Centre for Development of Advanced Computing9 for providing compute time on the Param Siddhi Supercomputer. We also thank Nilekani Philanthropies for their generous grant towards building datasets, models, tools and resources for Indic languages. We also thank Microsoft for their grant to support research on Indic languages. Most importantly, we would like to thank Archana Mhaske, Anil Mhaske, Sangeeta Rajagopal, Vindhya DS, Nitin Kedia, Yash Madhani, Kabir Ahuja, Shallu Rani, Armin Virk and Gowtham Ramesh for volunteering to annotate the testset.
Limitations
This work applies to languages that have a modest amount of data in parallel with English and are represented in pre-trained language models. These are typically high to mid-resource languages. Very low-resource languages might not have enough parallel corpora to extract sufficient NER training data. With limited parallel data and/or limited representation in pre-trained LMs, it will be difficult to get high-quality word alignments for projection. We use span-based annotation projection to alleviate word alignment errors to some extent.
Ethics Statement
The annotations are collected on a publicly available dataset and will be released publicly for future use. Some of these datasets originate from webcrawls and we do not make any explicit attempt to
7https://www.meity.gov.in/ 8https://www.bhashini.gov.in/ 9https://www.cdac.in/index.aspx?id=pune
identify any biases in these datasets and use them as-is. All the datasets used have been cited. All the datasets created as part of this work will be released under a CC-0 license10 and all the code and models will be release under an MIT license.11
The annotations in the testset were mostly contributed by volunteers interested in contributing to building a benchmark NER dataset. The volunteers were not made any payment and worked pro bono. Some annotators were paid for their services. These language experts were paid a competitive monthly salary to help with the task. The salary was determined based on the skill set and experience of the expert and adhered to the norms of the government of our country. The annotators were made aware that the annotations would be made publicly available. The annotations contains no personal information.

heading: B Comparison English NER taggers
text: We compared many English NER taggers. The results are shown in Table 9.

heading: C Examples of alignments generated by Awesome-align
text: We now present a few examples from our projection method. Figure 4 presents examples of correct alignments and hence correct projections of NER tags. As can be seen, the alignment is fairly sparse and the model aligns only those words in which it
is extremely confident. In this sentence, both words “Ravi" and “Shankar" had to be aligned to “Ravishankar" in Hindi, but only “Ravi" was aligned. But due to our range projection, the entire entity “Shri Ravi Shankar Prasad" was projected successfully with the tag PERSON.
Figure 5 shows an example of incorrect word alignment using the awesome align method for word alignment. In this sentence, “AAP" which is the abbreviated name of a political party is mapped only to “Aam" in Marathi instead of the entire phrase “Aam Aadmi pakshanche". This causes the projected entity to be only partially tagged with the entity type Organization.

heading: D Comparison with Other Pre-Trained Language Models
text: Table 10 reports the performance of various pretrained models fine-tuned on Naamapadam-train set in a multilingual fashion. We observe both MuRIL (Khanuja et al., 2021) and IndicBERT (Doddapaneni et al., 2022) outperform mBERT model.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: LimitationSection discusses the limitation of our work

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Section discusses the potential risks of our work.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 talks about creating the dataset

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Yes, Section 3 and Section 5.1 cites the creators of artifacts used

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Ethics Section describes the license

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Ethics Section
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Some of these datasets originate from web crawls and we do not make any explicit attempt to identify any biases in these datasets and use them as-is. The huge volume of data and lack of tools available to anonymize/ remove biases in the languages we are dealing with make it difficult to anonymize identities or remove offensive content

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? We have listed the languages used. It is discussed in section 3.6
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3.6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 5.4 provides details about the computation experiments performed

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5.4

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5.4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 6

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5.4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We follow conll 2003 annotation guidelines and have placed reference to the same in section

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Ethics Section talks about the same

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Ethics Section

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Ethics section describes the same

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: The annotators are native speakers from the Indian subcontinent. We mention the same in Ethics section

heading: Future Work