dataesr/scientific-paragraphs-categorization huggingface dataset:
Content analysis for building the indicator database The indicators identi ed from all these reviews were analyzed and combined into a database that provided a long list of 1556 indicators used or suggested in measuring the impact of health research. The indicators were then classi ed according to the ve CRIF dimensions: excellence, e cacy, economic, social and patient-reported outcomes. The payback model taxonomy  [29]  was chosen as a backbone for the indicator classi cation as it is a widely recognized framework for measuring health research impact  [52, 53]  and its categories were easily associated with the CRIF dimensions. Following an inductive approach, we ne-tuned the initial categorization and further group the indicators into 52 measurement aspects based on the topics they represented within each CRIF dimension.
A partnership between Google Arts & Culture (http://artsandculture. google. com) and the Musée des Instruments à Vent (Woodwinds Museum) has allowed the museum to make accessible online new documents from the archives, create stories and presentations, translate them into English, and share them worldwide. When the museum reopens, it will benefit from being georeferenced on the Google Arts & Culture app. The lack of public has allowed the museum to use the two galleries as working space. The staff, as well as an intern, have been working on cataloguing the objects with newly acquired database software and, in parallel, a website project has started, which will become the main access point to the collections, thanks to an integrated Online Public Access Catalogue (OPAC). During this pandemic period, regular social media activity has allowed the digital public to keep in touch with the museum and discover some behind the scenes aspects of the museum.
We take a step towards addressing the under representation of the African continent in NLP research by bringing together different stakeholders to create the first large, publicly available, highquality dataset for named en tity recognition (NER) in ten African lan guages. We detail the characteristics of these languages to help researchers and practition ers better understand the challenges they pose for NER tasks. We analyze our datasets and conduct an extensive empirical evalua tion of stateoftheart methods across both supervised and transfer learning settings. Fi nally, we release the data, code, and models to inspire future research on African NLP 1. 1 https://git. io/masakhane-ner.
Introduction Africa has over 2,000 spoken languages  (Eberhard et al. , 2020 ); however, these languages are scarcely represented in existing natural language process ing (NLP) datasets, research, and tools  (Martinus and Abbott, 2019). ∀ et al. (  2020 ) investigate the reasons for these disparities by examining how NLP for lowresource languages is constrained by several societal factors. One of these factors is the geographical and language diversity of NLP re searchers. For example, of the 2695 affiliations of authors whose works were published at the five ma jor NLP conferences in 2019, only five were from African institutions  (Caines, 2019). Conversely, many NLP tasks such as machine translation, text classification, partofspeech tagging, and named entity recognition would benefit from the knowl edge of native speakers who are involved in the development of datasets and models.
In this work, we focus on named entity recog nition (NER)-one of the most impactful tasks in NLP  (Sang and De Meulder, 2003; Lample et al. , 2016). NER is an important information extrac tion task and an essential component of numer ous products including spellcheckers, localization of voice and dialogue systems, and conversational agents. It also enables identifying African names, places and organizations for information retrieval. African languages are underrepresented in this crucial task due to lack of datasets, reproducible results, and researchers who understand the chal lenges that such languages present for NER. In this paper, we take an initial step towards im proving representation for African languages for the NER task, making the following contributions: (i) We bring together language speakers, dataset curators, NLP practitioners, and evaluation experts to address the challenges facing NER for African languages.
Based on the avail ability of online news corpora and language annotators, we develop NER datasets, mod els, and evaluation covering ten widely spo ken African languages. (ii) We curate NER datasets from local sources to ensure relevance of future research for native speakers of the respective languages. (iii) We train and evaluate multiple NER mod els for all ten languages. Our experiments provide insights into the transfer across lan guages, and highlight open challenges. (iv) We release the datasets, code, and models to facilitate future research on the specific chal lenges raised by NER for African languages.
Related Work African NER datasets NER is a wellstudied se quence labeling task  (Yadav and Bethard, 2018)  and has been the subject of many shared tasks in different languages  (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003; Sangal et al. , 2008; Shaalan, 2014; Benikova et al. , 2014). How ever, most of the available datasets are in high resource languages. Although there have been ef forts to create NER datasets for lowerresourced languages, such as the WikiAnn corpus  (Pan et al. , 2017)  covering 282 languages, such datasets con sist of "silverstandard" labels created by transfer ring annotations from English to other languages through crosslingual links in knowledge bases. Because the WikiAnn corpus data comes from Wikipedia, it includes some African languages; though most have fewer than 10k tokens.
Other NER datasets for African languages in clude SADiLaR  (Eiselen, 2016)  for ten South African languages based on government data, and small corpora of fewer than 2K sentences for Yorùbá  (Alabi et al. , 2020)  and  Hausa (Hedderich et al. , 2020). Additionally, the LORELEI lan guage packs  (Strassel and Tracey, 2016)  include some African languages (Yorùbá, Hausa, Amharic, Somali, Twi, Swahili, Wolof, Kinyarwanda, and Zulu), but are not publicly available. NER models Popular sequence labeling mod els for NER include the CRF  (Lafferty et al. , 2001) , CNNBiLSTM (Chiu and Nichols, 2016), BiLSTMCRF  (Huang et al. , 2015) , and CNN BiLSTMCRF  (Ma and Hovy, 2016). The tra ditional CRF makes use of handcrafted features like partofspeech tags, context words and word capitalization. Neural NER models on the other hand are initialized with word embeddings like Word2Vec  (Mikolov et al.
, 2013) , GloVe  (Penning ton et al. , 2014)  and FastText  (Bojanowski et al. , 2017). More recently, pretrained language mod els such as BERT  (Devlin et al. , 2019) , RoBERTa  (Liu et al. , 2019) , and LUKE  (Yamada et al. , 2020)  have been applied to produce stateoftheart re sults for the NER task. Multilingual variants of these models like mBERT and XLMRoBERTa  (Conneau et al. , 2020 ) make it possible to train NER models for several languages using transfer learning. Languagespecific parameters and adap tation to unlabeled data of the target language have yielded further gains  (Pfeiffer et al. , 2020a,b).
Named Entities Most of the work on NER is centered around En glish, and it is unclear how well existing mod els can generalize to other languages in terms of sentence structure or surface forms. In Hu et al. (  2020 )'s evaluation on crosslingual gener alization for NER, only two African languages were considered and it was seen that transformer based models particularly struggled to generalize to named entities in Swahili. To highlight the dif ferences across our focus languages, Table  2  shows an English 2 example sentence, with colorcoded PER, LOC, and DATE entities, and the correspond ing translations. The following characteristics of 2 Although the original sentence is from BBC Pidgin https://www. bbc. com/pidgin/tori-51702073 the languages in our dataset could pose challenges for NER systems developed for English: • Amharic shares no lexical overlap with the English source sentence.
Data and Annotation Methodology Our data was obtained from local news sources, in order to ensure relevance of the dataset for na tive speakers from those regions. The dataset was annotated using the ELISA tool  (Lin et al. , 2018)  by native speakers who come from the same re gions as the news sources and volunteered through the Masakhane community 3. Annotators were not  Corpus  (Tjong Kim Sang, 2002). We replaced the MISC tag with the DATE tag following  Alabi et al. (2020)  as the MISC tag may be illdefined and cause disagreement among nonexpert annotators. We report the number of annotators as well as gen eral statistics of the datasets in Table  3. For each language, we divided the annotated data into train ing, development, and test splits consisting of 70% training, 10%, and 20% of the data respectively.
A key objective of our annotation procedure was to create highquality datasets by ensuring a high annotator agreement. To achieve high agreement scores, we ran collaborative workshops for each language, which allowed annotators to discuss any disagreements. ELISA provides an entitylevel F1 score and also an interface for annotators to cor rect their mistakes, making it easy to achieve inter annotator agreement scores between 0. 96 and 1. 0 for all languages. We report interannotator agreement scores in Table  4  using Fleiss' Kappa  (Fleiss, 1971 ) at both the token and entity level. The latter considers each span an annotator proposed as an entity. As a result of our workshops, all our datasets have exceptionally high interannotator agreement. For Kinyarwanda, Luo, Swahili, and Wolof, we report perfect interannotator agreement scores (κ = 1).
NER baseline models To evaluate baseline performance on our dataset, we experiment with three popular NER mod els: CNNBiLSTMCRF, multilingual BERT (mBERT), and XLMRoBERTa (XLMR). The lat ter two models are implemented using the Hug gingFace transformers toolkit  (Wolf et al. , 2019). For each language, we train the models on the in language training data and evaluate on its test data. CNNBiLSTMCRF This architecture was pro posed for NER by  Ma and Hovy (2016). For each input sequence, we first compute the vec tor representation for each word by concatenating characterlevel encodings from a CNN and vector embeddings for each word. Following Rijhwani et al. (  2020 ), we use randomly initialized word embeddings since we do not have highquality pretrained embeddings for all the languages in our dataset.
Our model is implemented using the DyNet toolkit  (Neubig et al. , 2017). mBERT We finetune multilingual BERT  (De vlin et al. , 2019)  on our NER corpus by adding a linear classification layer to the pretrained trans former model, and train it endtoend. mBERT was trained on 104 languages including only two African languages: Swahili and Yorùbá. We use the mBERTbase cased model with 12layer Trans former blocks consisting of 768hidden size and 110M parameters. XLMR XLMR  (Conneau et al. , 2020)  was trained on 100 languages including Amharic, Hausa, and Swahili. The major differences be tween XLMR and mBERT are (1) XLMR was trained on Common Crawl while mBERT was trained on Wikipedia; (2) XLMR is based on RoBERTa, which is trained with a masked lan guage model (MLM) objective while mBERT was additionally trained with a next sentence prediction objective.
We make use of the XLMR base and large models for the baseline models. The XLMR base model consisting of 12 layers, with a hidden size of 768 and 270M parameters. On the other hand, the XLMRlarge has 24 layers, with a hid den size of 1024 and 550M parameters. MeanEBiLSTM This is a simple BiLSTM model with an additional linear classifier. For each input sequence, we first extract a sentence embedding from mBERT or XLMR language model (LM) before passing it into the BiLSTM model. Following Reimers and Gurevych (2019), we make use of the mean of the 12layer output embeddings of the LM (i. e MeanE). This has been shown to provide better sentence representations than the embedding of the [CLS] token used for finetuning mBERT and XLMR. Language BERT The mBERT and the XLM R models only supports two and three languages under study respectively.
Improving the Baseline Models In this section, we consider techniques to improve the baseline models such as utilizing gazetteers, transfer learning from other domains and lan guages, and aggregating NER datasets by regions. For these experiments, we focus on the PER, ORG, and LOC categories, because the gazetteers from Wikipedia do not contain DATE entities and some source domains and languages that we transfer from do not have the DATE annotation. We ap ply these modifications to the XLMR model be cause it generally outperforms mBERT in our ex periments (see Section 6).
Gazetteers for NER Gazetteers are lists of named entities collected from manually crafted resources such as GeoN ames or Wikipedia. Before the widespread adoption of neural networks, NER methods used gazetteersbased features to improve perfor mance  (Ratinov and Roth, 2009). These features are created for each ngram in the dataset and are typically binaryvalued, indicating whether that n gram is present in the gazetteer. Recently,  Rijhwani et al. (2020)  showed that augmenting the neural CNNBiLSTMCRF model with gazetteer features can improve NER per formance for lowresource languages. We con duct similar experiments on the languages in our dataset, using entity lists from Wikipedia as gazetteers. For Luo and NigerianPidgin, which do not have their own Wikipedia, we use entity lists from English Wikipedia.
Domain Adaptation from WikiAnn We make use of the WikiAnn corpus  (Pan et al. , 2017) , which is available for five of the languages in our dataset: Amharic, Igbo, Kinyarwanda, Swahili and Yorùbá. For each language, the corpus con tains 100 sentences in each of the training, develop ment and test splits except for Swahili, which con tains 1K sentences in each split. For each language, we train on the corresponding WikiAnn training set and either zeroshot transfer to our respective test set or additionally finetune on our training data.
Crosslingual transfer For training the cross lingual transfer models, we use the CoNLL2003 5 NER dataset in English with over 14K training sen tences and our annotated corpus. The reason for CoNLL2003 is because it is in the same news do main as our annotated corpus. We also make use of the languages that are supported by the XLM R model and are widely spoken in East and West Africa like Swahili and Hausa. The English cor pus has been shown to transfer very well to low re source languages (Hedderich et al. ,  2020; Lauscher et al. , 2020. We first train on either the English CoNLL2003 data or our training data in Swahili, Hausa, or NigerianPidgin before testing on the tar get African languages.
Baseline Models Table  5  gives the F1score obtained by CNN BiLSTMCRF, mBERT and XLMR models on the test sets of the ten African languages when training on our inlanguage data. We addition ally indicate whether the language is supported by the pretrained language models (). The per centage of entities that are of outofvocabulary (OOV; entities in the test set that are not present in the training set) is also reported alongside re sults of the baseline models. In general, the datasets with greater numbers of OOV entities have lower performance with the CNNBiLSTM CRF model, while those with lower OOV rates (Hausa, Igbo, Swahili) have higher performance. We find that the CNNBiLSTMCRF model per forms worse than finetuning mBERT and XLM R models endtoend (FTune). We expect perfor mance to be better (e.
g. , for Amharic and Nigerian Pidgin with over 18 F1 point difference) when us ing pretrained word embeddings for the initializa tion of the BiLSTM model rather than random ini tialization (we leave this for future work as dis cussed in Section 7). Interestingly, the pretrained language models (PLMs) have reasonable performance even on lan guages they were not trained on such as Igbo, Kin yarwanda, Luganda, Luo, and Wolof. However, languages supported by the PLM tend to have bet ter performance overall. We observe that fine tuned XLMRbase models have significantly bet ter performance on five languages; two of the languages (Amharic and Swahili) are supported by the pretrained XLMR. Similarly, finetuning mBERT has better performance for Yorùbá since the language is part of the PLM's training cor pus.
Although mBERT is trained on Swahili, XLMRbase shows better performance. This observation is consistent with  Hu et al. (2020)  and could be because XLMR is trained on more Swahili text (Common Crawl with 275M tokens) whereas mBERT is trained on a smaller corpus from Wikipedia (6M tokens 6  ). Another observation is that mBERT tends to have better performance for the nonBantu NigerCongo languages i. e. , Igbo, Wolof, and Yorùbá, while XLMRbase works better for Afro   Asiatic languages (i. e. , Amharic and Hausa), Nilo Saharan (i. e. , Luo) and Bantu languages like Kin yarwanda and Swahili. We also note that the writ ing script is one of the primary factors influenc ing the transfer of knowledge in PLMs with re gard to the languages they were not trained on. For example, mBERT achieves an F1score of 0.
0 on Amharic because it has not encountered the script during pretraining. In general, we find the fine tuned XLMRlarge (with 550M parameters) to be better than XLMRbase (with 270M parame ters) and mBERT (with 110 parameters) in almost all languages. However, mBERT models perform slightly better for Igbo, Luo, and Yorùbá despite having fewer parameters. We further analyze the transfer abilities of mBERT and XLMR by extracting sentence em beddings from the LMs to train a BiLSTM model (MeanEBiLSTM) instead of finetuning them end toend. Table  5  shows that languages that are not supported by mBERT or XLMR generally perform worse than CNNBiLSTMCRF model (despite being randomly initialized) except for kin. Also, sentence embeddings extracted from mBERT often lead to better performance than XLMR for languages they both do not support (like ibo, kin, lug, luo, and wol).
Lastly, we train NER models using language BERT models that have been adapted to each of the African languages via languagespecific finetuning on unlabeled text. In all cases, fine tuning language BERT and language XLMR mod els achieves a 1 -7% improvement in F1score over finetuning mBERTbase and XLMRbase respectively. This approach is still effective for small sized pretraining corpora provided they are of good quality. For example, the Wolof mono lingual corpus, which contains less than 50K sen tences (see Table  10  in the Appendix) still im proves performance by over 4% F1. Further, we obtain over 60% improvement in performance for Amharic BERT because mBERT does not recog nize the Amharic script.
Crossdomain Transfer We evaluate crossdomain transfer from Wikipedia to the news domain for the five languages that are available in the WikiAnn  (Pan et al. , 2017)  dataset. In the zeroshot setting, the NER F1score is low: less than 40 F1score for all languages, with Kin yarwanda and Yorùbá having less than 10 F1score. This is likely due to the number of training sen tences present in WikiAnn: there are only 100 sentences in the datasets of Amharic, Igbo, Kin yarwanda and Yorùbá. Although the Swahili cor pus has 1,000 sentences, the 35 F1score shows that transfer is not very effective. In general, cross domain transfer is a challenging problem, and is even harder when the number of training examples from the source domain is small. Finetuning on the indomain news NER data does not improve over the baseline (XLMRbase).
CrossLingual Transfer Zeroshot In the zeroshot setting we evaluated NER models trained on the English engCoNLL03 dataset, and on the NigerianPidgin (pcm), Swahili (swa), and Hausa (hau) annotated corpus. We ex cluded the MISC entity in the engCoNLL03 corpus because it is absent in our target datasets. Table  7  shows the result for the (zeroshot) transfer per formance. We observe that the closer the source and target languages are geographically, the bet CoNLL03 with an improvement of over 14 F1 on average. We found that, on average, transferring from Hausa provided the best F1, with an improve ment of over 16% and 1% compared to using the engCoNLL and swa data respectively. Perentity analysis in Table  8  shows that the largest improve ments are obtained for ORG. The pcm data was more effective in transferring to LOC and ORG, while swa and hau performed better when transferring to PER.
Regional Influence on NER We evaluate whether combining different language training datasets by region affects the performance for individual languages. Table  7  shows that all languages spoken in West Africa (ibo, wol, pcm, yor) except hau have slightly better performance (0. 1-2. 6 F1) when we train on their combined training data. However, for the EastAfrican lan guages, the F1 score only improved (0. 8-2. 3 F1) for three languages  (kin, lug, luo). Training the NER model on all nine languages leads to better performance on all languages except Swahili. On average over six languages (ibo, kin, lug, luo, wol, yor), the performance improves by 1. 6 F1.
Error analysis Finally, to better understand the types of entities that were successfully identified and those that were missed, we performed finegrained analysis of our baseline methods mBERT and XLMR us ing the method of  Fu et al. (2020) , with results shown in Table  9. Specifically, we found that across all languages, entities that were not con tained in the training data (zerofrequency entities), and entities consisting of more than three words (long entities) were particularly difficult in all lan guages; compared to the F1 score over all entities, the scores dropped by around 5 points when eval uated on zerofrequency entities, and by around 20 points when evaluated on long entities. Future work on lowresource NER or crosslingual repre sentation learning may further improve on these hard cases.
Conclusion and Future Work We address the NER task for African languages by bringing together a variety of stakeholders to create a highquality NER dataset for ten African languages. We evaluate multiple stateoftheart NER models and establish strong baselines. We have released one of our best models that can rec ognize named entities in ten African languages on HuggingFace Model Hub 7. We also investi gate crossdomain transfer with experiments on five languages with the WikiAnn dataset, along with crosslingual transfer for lowresource NER using the English CoNLL2003 dataset and other languages supported by XLMR. In the future, we plan to use pretrained word embeddings such as GloVe  (Pennington et al. , 2014)  and fastText  (Bo janowski et al. , 2017)  instead of random initializa tion for the CNNBiLSTMCRF, increase the num ber of annotated sentences per language, and ex pand the dataset to more African languages.
A. 3 Monolingual Corpora for Language Adaptive Finetuning Table  10  shows the monolingual corpus we used for the language adaptive finetuning. We pro vide the details of the source of the data, and their sizes. For most of the languages, we make use of JW300 8  and CC100 9. In some cases CCAligned (ElKishky et al. , 2020) was used, in such a case, we removed duplicated sen tences from CC100. For finetuning the language model, we make use of the HuggingFace  (Wolf et al. , 2019)  code with learning rate 5e5. How ever, for the Amharic BERT, we make use of a smaller learning rate of 5e6 since the multilin gual BERT vocabulary was replaced by Amharic vocabulary, so that we can slowly adapt the mBERT LM to understand Amharic texts. All lan guage BERT models were pretrained for 3 epochs ("ibo", "kin","lug","luo", "pcm","swa","yor") or 10 epochs ("amh", "hau","wol") depending on their convergence.