cwe also proposed three alternatives to learn multiple embeddings per character and resolve ambiguities: (i) position-based character embeddings where each character has different embeddings depending on the position it appears in a word, i.e., beginning, middle or end (ii) cluster-based character embeddings where a character can have $K$ different cluster embeddings, and (iii) position-based cluster embeddings (CWE-LP) where for each position $K$ different embeddings are learned. We use the latter in our experiments with CWE but no positional embeddings are used with fastText. Finally, we consider a contextualized embedding architecture, BERT . BERT is a masked language model based on the highly efficient and parallelizable Transformer architecture  known to produce very rich contextualized representations for downstream NLP tasks. The architecture is trained by jointly conditioning on both left and right contexts in all the transformer layers using two unsupervised objectives: Masked LM and Next-sentence prediction. The representation of a word is therefore learned according to the context it is found in. Training contextual embeddings needs of huge amounts of corpora which are not available for low-resourced languages such as Yorùbá and Twi.

Finally, we translate test sets and annotate corpora to evaluate the performance of both our models together with fastText and BERT pre-trained embeddings which could not be evaluated otherwise for Yorùbá and Twi. The evaluation is carried out in a word similarity and relatedness task using the wordsim-353 test set, and in a named entity recognition (NER) task where embeddings play a crucial role. Of course, the evaluation of the models in only two tasks is not exhaustive but it is an indication of the quality we can obtain for these two low-resourced languages as compared to others such as English where these evaluations are already available. The rest of the paper is organized as follows. Related works are reviewed in Section SECREF2 The two languages under study are described in Section SECREF3. We introduce the corpora and test sets in Section SECREF4. The fifth section explores the different training architectures we consider, and the experiments that are carried out. Finally, discussion and concluding remarks are given in Section SECREF6

Semantic Representations ::: Word Embeddings Architectures	Modeling sub-word units has recently become a popular way to address out-of-vocabulary word problem in NLP especially in word representation learning , , . A sub-word unit can be a character, character $n$-grams, or heuristically learned Byte Pair Encodings (BPE) which work very well in practice especially for morphologically rich languages. Here, we consider two word embedding models that make use of character-level information together with word information: Character Word Embedding (CWE)  and fastText . Both of them are extensions of the Word2Vec architectures  that model sub-word units, character embeddings in the case of CWE and character $n$-grams for fastText. CWE was introduced in 2015 to model the embeddings of characters jointly with words in order to address the issues of character ambiguities and non-compositional words especially in the Chinese language.

The 157-languages version is trained using CBOW with position-weights, in dimension 300, with character $n$-grams of length 5, a window of size 5 and 10 negatives. We want to compare the performance of these embeddings with the equivalent models that can be obtained by training on the different sources verified by native speakers of Twi and Yorùbá; what we call curated corpora and has been described in Section SECREF4 For the comparison, we define 3 datasets according to the quality and quantity of textual data used for training: (i) Curated Small Dataset (clean), C1, about 1.6 million tokens for Yorùbá and over 735 k tokens for Twi. The clean text for Twi is the Bible and for Yoruba all texts marked under the C1 column in Table TABREF7. (ii) In Curated Small Dataset (clean + noisy), C2, we add noise to the clean corpus (Wikipedia articles for Twi, and BBC Yorùbá news articles for Yorùbá).

Semantic Representations ::: Experiments ::: FastText Training and Evaluation	As a first experiment, we compare the quality of fastText embeddings trained on (high-quality) curated data and (low-quality) massively extracted data for Twi and Yorùbá languages. Facebook released pre-trained word embeddings using fastText for 294 languages trained on Wikipedia  (F1 in tables) and for 157 languages trained on Wikipedia and Common Crawl  (F2). For Yorùbá, both versions are available but only embeddings trained on Wikipedia are available for Twi. We consider these embeddings the result of training on what we call massively-extracted corpora. Notice that training settings for both embeddings are not exactly the same, and differences in performance might come both from corpus size/quality but also from the background model. The 294-languages version is trained using skipgram, in dimension 300, with character $n$-grams of length 5, a window of size 5 and 5 negatives.

Massive vs. Curated Word Embeddings for Low-Resourced Languages. The Case of Yor\`ub\'a and Twi	The success of several architectures to learn semantic representations from unannotated text and the availability of these kind of texts in online multilingual resources such as Wikipedia has facilitated the massive and automatic creation of resources for multiple languages. The evaluation of such resources is usually done for the high-resourced languages, where one has a smorgasbord of tasks and test sets to evaluate on. For low-resourced languages, the evaluation is more difficult and normally ignored, with the hope that the impressive capability of deep learning architectures to learn (multilingual) representations in the high-resourced setting holds in the low-resourced setting too. In this paper we focus on two African languages, Yor\`ub\'a and Twi, and compare the word embeddings obtained in this way, with word embeddings obtained from curated corpora and a language-dependent processing. We analyse the noise in the publicly available corpora, collect high quality and noisy data for the two languages and quantify the improvements that depend not only on the amount of data but on the quality too.

In these cases, training data is scarce, and the assumption that the capability of deep learning architectures to learn (multilingual) representations in the high-resourced setting holds in the low-resourced one does not need to be true. In this work, we focus on two African languages, Yorùbá and Twi, and carry out several experiments to verify this claim. Just by a simple inspection of the word embeddings trained on Wikipedia by fastText, we see a high number of non-Yorùbá or non-Twi words in the vocabularies. For Twi, the vocabulary has only 935 words, and for Yorùbá we estimate that 135 k out of the 150 k words belong to other languages such as English, French and Arabic. In order to improve the semantic representations for these languages, we collect online texts and study the influence of the quality and quantity of the data in the final models. We also examine the most appropriate architecture depending on the characteristics of each language.

Semantic Representations	In this section, we describe the architectures used for learning word embeddings for the Twi and Yorùbá languages. Also, we discuss the quality of the embeddings as measured by the correlation with human judgements on the translated wordSim-353 test sets and by the F1 score in a NER task.

That shows how pre-trained embeddings by themselves do not perform well in downstream tasks on low-resource languages. To address this problem for Yorùbá, we fine-tune BERT representations on the Yorùbá corpus in two ways: (i) using the multilingual vocabulary, and (ii) using only Yorùbá vocabulary. In both cases diacritics are ignored to be consistent with the base model training. As expected, the fine-tuning of the pre-trained BERT on the Yorùbá corpus in the two configurations generates better representations than the base model. These models are able to achieve a better performance on the NER task with an average F1 score of over 47% (see Table TABREF26 for the comparative). The fine-tuned BERT model with only Yorùbá vocabulary further increases by more than 4% in F1 score obtained with the tuning that uses the multilingual vocabulary. Although we do not have enough data to train BERT from scratch, we observe that fine-tuning BERT on a limited amount of monolingual data of a low-resource language helps to improve the quality of the embeddings.

Introduction	In recent years, word embeddings , ,  have been proven to be very useful for training downstream natural language processing (NLP) tasks. Moreover, contextualized embeddings ,  have been shown to further improve the performance of NLP tasks such as named entity recognition, question answering, or text classification when used as word features because they are able to resolve ambiguities of word representations when they appear in different contexts. Different deep learning architectures such as multilingual BERT , LASER  and XLM  have proved successful in the multilingual setting. All these architectures learn the semantic representations from unannotated text, making them cheap given the availability of texts in online multilingual resources such as Wikipedia. However, the evaluation of such resources is usually done for the high-resourced languages, where one has a smorgasbord of tasks and test sets to evaluate on. This is the best-case scenario, languages with tones of data for training that generate high-quality models. For low-resourced languages, the evaluation is more difficult and therefore normally ignored simply because of the lack of resources.

We also use different architectures that learn word representations both from surface forms and characters to further exploit all the available information which showed to be important for these languages. For the evaluation, we manually translate the wordsim-353 word pairs dataset from English into Yor\`ub\'a and Twi. As output of the work, we provide corpora, embeddings and the test suits for both languages.

Wikipedia is usually used as a high-quality freely available multilingual corpus as compared to noisier data such as Common Crawl. However, for the two languages under study, Wikipedia resulted to have too much noise: interference from other languages, text clearly written by non-native speakers, lack of diacritics and mixture of dialects. The JW300 corpus on the other hand, has been rated as high-quality by our native Yorùbá speakers, but as noisy by our native Twi speakers. In both cases, experiments confirm the conclusions.

A word or character embedding is learned in CWE using either CBOW or skipgram architectures, and then the final word embedding is computed by adding the character embeddings to the word itself: where $w_j$ is the word embedding of $x_j$, $N_j$ is the number of characters in $x_j$, and $c_k$ is the embedding of the $k$-th character $c_k$ in $x_j$. Similarly, in 2017 fastText was introduced as an extension to skipgram in order to take into account morphology and improve the representation of rare words. In this case the embedding of a word also includes the embeddings of its character $n$-grams: where $w_j$ is the word embedding of $x_j$, $G_j$ is the number of character $n$-grams in $x_j$ and $g_k$ is the embedding of the $k$-th $n$-gram.

In translating these, we picked the translation that best suits the context (other word in the pair). In two cases, the correlation is not fulfilled at all: soap–opera and star–movies are not related in the Twi language and the score has been modified accordingly.

Semantic Representations ::: Experiments ::: CWE Training and Evaluation	The huge ambiguity in the written Twi language motivates the exploration of different approaches to word embedding estimations. In this work, we compare the standard fastText methodology to include sub-word information with the character-enhanced approach with position-based clustered embeddings (CWE-LP as introduced in Section SECREF17). With the latter, we expect to specifically address the ambiguity present in a language that does not translate the different oral tones on vowels into the written language. The character-enhanced word embeddings are trained using a skipgram architecture with cluster-based embeddings and an embedding size of 300 dimensions, context window-size of 5, and 5 negative samples. In this case, the best performance is obtained with a minimum word count of 1, and that increases the effective vocabulary that is used for training the embeddings with respect to the fastText experiments reported in Table TABREF15. We repeat the same experiments as with fastText and summarise them in Table TABREF16.

Furthermore, we got two manually typed fully-diacritized Yorùbá literature (Ìrìnkèrindò nínú igbó elégbèje and Igbó Olódùmarè) both written by Daniel Orowole Olorunfemi Fagunwa a popular Yorùbá author. The number of tokens available from each source, the link to the original source and the quality of the data is summarised in Table TABREF7. The gathering of clean data in Twi is more difficult. We use as the base text as it has been shown that the Bible is the most available resource for low and endangered languages . This is the cleanest of all the text we could obtain. In addition, we use the available (and small) Wikipedia dumps which are quite noisy, i.e. Wikipedia contains a good number of English words, spelling errors and Twi sentences formulated in a non-natural way (formulated as L2 speakers would speak Twi as compared to native speakers). Lastly, we added text crawled from jw and the JW300 Twi corpus.

Semantic Representations ::: Experiments ::: BERT Evaluation on NER Task	In order to go beyond the similarity task using static word vectors, we also investigate the quality of the multilingual BERT embeddings by fine-tuning a named entity recognition task on the Yorùbá Global Voices corpus. One of the major advantages of pre-trained BERT embeddings is that fine-tuning of the model on downstream NLP tasks is typically computationally inexpensive, often with few number of epochs. However, the data the embeddings are trained on has the same limitations as that used in massive word embeddings. Fine-tuning involves replacing the last layer of BERT used optimizing the masked LM with a task-dependent linear classifier or any other deep learning architecture, and training all the model parameters end-to-end. For the NER task, we obtain the token-level representation from BERT and train a linear classifier for sequence tagging. Similar to our observations with non-contextualized embeddings, we find out that fine-tuning the pre-trained multilingual-uncased BERT for 4 epochs on the NER task gives an F1 score of 0. If we do the same experiment in English, F1 is 58.1 after 4 epochs.

Related Work	The large amount of freely available text in the internet for multiple languages is facilitating the massive and automatic creation of multilingual resources. The resource par excellence is Wikipedia, an online encyclopedia currently available in 307 languages. Other initiatives such as Common Crawl or the Jehovah’s Witnesses site are also repositories for multilingual data, usually assumed to be noisier than Wikipedia. Word and contextual embeddings have been pre-trained on these data, so that the resources are nowadays at hand for more than 100 languages. Some examples include fastText word embeddings , , MUSE embeddings , BERT multilingual embeddings  and LASER sentence embeddings . In all cases, embeddings are trained either simultaneously for multiple languages, joining high- and low-resource data, or following the same methodology. On the other hand, different approaches try to specifically design architectures to learn embeddings in a low-resourced setting. ChaudharyEtAl:2018 follow a transfer learning approach that uses phonemes, lemmas and morphological tags to transfer the knowledge from related high-resource language into the low-resource one.

Notice that the Bible text, is mainly written in the Asante dialect whilst the last, Jehovah's Witnesses, was written mainly in the Akuapem dialect. The Wikipedia text is a mixture of the two dialects. This introduces a lot of noise into the embeddings as the spelling of most words differs especially at the end of the words due to the mixture of dialects. The JW300 Twi corpus also contains mixed dialects but is mainly Akuampem. In this case, the noise comes also from spelling errors and the uncommon addition of diacritics which are not standardised on certain vowels. Figures for Twi corpora are summarised in the bottom block of Table TABREF7.

Languages under Study ::: Twi	is an Akan language of the Central Tano Branch of the Niger Congo family of languages. It is the most widely spoken of the about 80 indigenous languages in Ghana . It has about 9 million native speakers and about a total of 17–18 million Ghanaians have it as either first or second language. There are two mutually intelligible dialects, Asante and Akuapem, and sub-dialectical variants which are mostly unknown to and unnoticed by non-native speakers. It is also mutually intelligible with Fante and to a large extent Bono, another of the Akan languages. It is one of, if not the, easiest to learn to speak of the indigenous Ghanaian languages. The same is however not true when it comes to reading and especially writing. This is due to a number of easily overlooked complexities in the structure of the language. First of all, similarly to Yorùbá, Twi is a tonal language but written without diacritics or accents. As a result, words which are pronounced differently and unambiguous in speech tend to be ambiguous in writing.

Besides, most of such words fit interchangeably in the same context and some of them can have more than two meanings. A simple example is: Me papa aba nti na me ne wo redi no yie no. S wo ara wo nim s me papa ba a, me suban fofor adi. This sentence could be translated as (i) I'm only treating you nicely because I'm in a good mood. You already know I'm a completely different person when I'm in a good mood. (ii) I'm only treating you nicely because my dad is around. You already know I'm a completely different person when my dad comes around. Another characteristic of Twi is the fact that a good number of stop words have the same written form as content words. For instance, “na” or “na” could be the words “and, then”, the phrase “and then” or the word “mother”. This kind of ambiguity has consequences in several natural language applications where stop words are removed from text. Finally, we want to point out that words can also be written with or without prefixes.

Standard Yorùbá has 25 letters without the Latin characters c, q, v, x and z. There are 18 consonants (b, d, f, g, gb, j[dz], k, l, m, n, p[kp], r, s, ṣ, t, w y[j]), 7 oral vowels (a, e, ẹ, i, o, ọ, u), five nasal vowels, (an, $ \underaccent{\dot{}}{e}$n, in, $ \underaccent{\dot{}}{o}$n, un) and syllabic nasals (m̀, ḿ, ǹ, ń). Yorùbá is a tone language which makes heavy use of lexical tones which are indicated by the use of diacritics. There are three tones in Yorùbá namely low, mid and high which are represented as grave ($\setminus $), macron ($-$) and acute ($/$) symbols respectively. These tones are applied on vowels and syllabic nasals.

jiangEtal:2018 apply Positive-Unlabeled Learning for word embedding calculations, assuming that unobserved pairs of words in a corpus also convey information, and this is specially important for small corpora. In order to assess the quality of word embeddings, word similarity and relatedness tasks are usually used. wordsim-353  is a collection of 353 pairs annotated with semantic similarity scores in a scale from 0 to 10. Even the problems detected in this dataset , it is widely used by the community. The test set was originally created for English, but the need for comparison with other languages has motivated several translations/adaptations. In hassanMihalcea:2009 the test was translated manually into Spanish, Romanian and Arabic and the scores were adapted to reflect similarities in the new language. The reported correlation between the English scores and the Spanish ones is 0.86. Later, JoubarneInkpen:2011 show indications that the measures of similarity highly correlate across languages. leviantReichart:2015 translated also wordsim-353 into German, Italian and Russian and used crowdsourcing to score the pairs.

Mid tone is usually left unmarked on vowels and every initial or first vowel in a word cannot have a high tone. It is important to note that tone information is needed for correct pronunciation and to have the meaning of a word , , . For example, owó (money), ọw (broom), òwò (business), w (honour), ọw (hand), and w (group) are different words with different dots and diacritic combinations. According to Asahiah2014, Standard Yorùbá uses 4 diacritics, 3 are for marking tones while the fourth which is the dot below is used to indicate the open phonetic variants of letter "e" and "o" and the long variant of "s". Also, there are 19 single diacritic letters, 3 are marked with dots below (ẹ, ọ, ṣ) while the rest are either having the grave or acute accent. The four double diacritics are divided between the grave and the acute accent as well.

If we compare the relative numbers for the three datasets (C1, C2 and C3) we observe the same trends as before: the performance of the embeddings in the similarity task improves with the vocabulary size when the training data can be considered clean, but the performance diminishes when the data is noisy. According to the results, CWE is specially beneficial for Twi but not always for Yorùbá. Clean Yorùbá text, does not have the ambiguity issues at character-level, therefore the $n$-gram approximation works better when enough clean data is used ($\rho ^{C3}_{CWE}=0.354$ vs. $\rho ^{C3}_{fastText}=0.391$) but it does not when too much noisy data (no diacritics, therefore character-level information would be needed) is used ($\rho ^{C2}_{CWE}=0.345$ vs. $\rho ^{C2}_{fastText}=0.302$).

Data ::: Training Corpora	For Yorùbá, we use several corpora collected by the Niger-Volta Language Technologies Institute with texts from different sources, including the Lagos-NWU conversational speech corpus, fully-diacritized Yorùbá language websites and an online Bible. The largest source with clean data is the JW300 corpus. We also created our own small-sized corpus by web-crawling three Yorùbá language websites (Alàkwé, r Yorùbá and Èdè Yorùbá Rẹw in Table TABREF7), some Yoruba Tweets with full diacritics and also news corpora (BBC Yorùbá and VON Yorùbá) with poor diacritics which we use to introduce noise. By noisy corpus, we refer to texts with incorrect diacritics (e.g in BBC Yorùbá), removal of tonal symbols (e.g in VON Yorùbá) and removal of all diacritics/under-dots (e.g some articles in Yorùbá Wikipedia).

By carefully gathering high-quality data and optimising the models to the characteristics of each language, we deliver embeddings with correlations of $\rho $=$0.39$ (Yorùbá) and $\rho $=$0.44$ (Twi) on the same test set, still far from the high-resourced models, but representing an improvement over $170\%$ on the task. In a low-resourced setting, the data quality, processing and model selection is more critical than in a high-resourced scenario. We show how the characteristics of a language (such as diacritization in our case) should be taken into account in order to choose the relevant data and model to use. As an example, Twi word embeddings are significantly better when training on 742 k selected tokens than on 16 million noisy tokens, and when using a model that takes into account single character information (CWE-LP) instead of $n$-gram information (fastText). Finally, we want to note that, even within a corpus, the quality of the data might depend on the language.

The pre-trained Twi model has only 935 tokens. For Yorùbá, things are apparently better with more than 150 k tokens when both Wikipedia and Common Crawl are used but correlation is even lower. An inspection of the pre-trained embeddings indicates that over 135 k words belong to other languages mostly English, French and Arabic. If we focus only on Wikipedia, we see that many texts are without diacritics in Yorùbá and often make use of mixed dialects and English sentences in Twi. The Spearman $\rho $ correlation for fastText models on the curated small dataset (clean), C1, improves the baselines by a large margin ($\rho =0.354$ for Twi and 0.322 for Yorùbá) even with a small dataset. The improvement could be justified just by the larger vocabulary in Twi, but in the case of Yorùbá the enhancement is there with almost half of the vocabulary size. We found out that adding some noisy texts (C2 dataset) slightly improves the correlation for Twi language but not for the Yorùbá language.

An example is this same na and na which happen to be the same word with an omissible prefix across its multiple senses. For some words, the prefix characters are mostly used when the word begins a sentence and omitted in the middle. This however depends on the author/speaker. For the word embeddings calculation, this implies that one would have different embeddings for the same word found in different contexts.

Data ::: Evaluation Test Sets ::: Twi	Just like Yorùbá, the wordsim-353 word pairs dataset was translated for Twi. Out of the 353 word pairs, 274 were used in this case. The remaining 79 pairs contain words that translate into longer phrases. The number of words that can be translated by a single token is higher than for Yorùbá. Within the 274 pairs, there are 351 unique English words which translated to 310 unique Twi words. 298 of the 310 Twi words are single word translations, 4 transliterations and 16 are used as is. Even if JoubarneInkpen:2011 showed indications that semantic similarity has a high correlation across languages, different nuances between words are captured differently by languages. For instance, both money and currency in English translate into sika in Twi (and other 32 English words which translate to 14 Twi words belong to this category) and drink in English is translated as Nsa or nom depending on the part of speech (noun for the former, verb for the latter). 17 English words fall into this category.

As noted in Asahiah2014, most of the Yorùbá texts found in websites or public domain repositories (i) either use the correct Yorùbá orthography or (ii) replace diacritized characters with un-diacritized ones. This happens as a result of many factors, but most especially to the unavailability of appropriate input devices for the accurate application of the diacritical marks . This has led to research on restoration models for diacritics , but the problem is not well solved and we find that most Yorùbá text in the public domain today is not well diacritized. Wikipedia is not an exception.

Summary and Discussion	In this paper, we present curated word and contextual embeddings for Yorùbá and Twi. For this purpose, we gather and select corpora and study the most appropriate techniques for the languages. We also create test sets for the evaluation of the word embeddings within a word similarity task (wordsim353) and the contextual embeddings within a NER task. Corpora, embeddings and test sets are available in github. In our analysis, we show how massively generated embeddings perform poorly for low-resourced languages as compared to the performance for high-resourced ones. This is due both to the quantity but also the quality of the data used. While the Pearson $\rho $ correlation for English obtained with fastText embeddings trained on Wikipedia (WP) and Common Crawl (CC) are $\rho _{WP}$=$0.67$ and $\rho _{WP+CC}$=$0.78$, the equivalent ones for Yorùbá are $\rho _{WP}$=$0.14$ and $\rho _{WP+CC}$=$0.07$. For Twi, only embeddings with Wikipedia are available ($\rho _{WP}$=$0.14$).

This increases the number of training tokens for Twi to 742 k tokens and Yorùbá to about 2 million tokens. (iii) Curated Large Dataset, C3 consists of all available texts we are able to crawl and source out for, either clean or noisy. The addition of JW300  texts increases the vocabulary to more than 10 k tokens in both languages. We train our fastText systems using a skipgram model with an embedding size of 300 dimensions, context window size of 5, 10 negatives and $n$-grams ranging from 3 to 6 characters similarly to the pre-trained models for both languages. Best results are obtained with minimum word count of 3. Table TABREF15 shows the Spearman correlation between human judgements and cosine similarity scores on the wordSim-353 test set. Notice that pre-trained embeddings on Wikipedia show a very low correlation with humans on the similarity task for both languages ($\rho $=$0.14$) and their performance is even lower when Common Crawl is also considered ($\rho $=$0.07$ for Yorùbá). An important reason for the low performance is the limited vocabulary.

Languages under Study ::: Yorùbá	is a language in the West Africa with over 50 million speakers. It is spoken among other languages in Nigeria, republic of Togo, Benin Republic, Ghana and Sierra Leon. It is also a language of Òrìsà in Cuba, Brazil, and some Caribbean countries. It is one of the three major languages in Nigeria and it is regarded as the third most spoken native African language. There are different dialects of Yorùbá in Nigeria , , . However, in this paper our focus is the standard Yorùbá based upon a report from the 1974 Joint Consultative Committee on Education .

Acknowledgements	The authors thank Dr. Clement Odoje of the Department of Linguistics and African Languages, University of Ibadan, Nigeria and Olóyè Gbémisóyè Àrdèó for helping us with the Yorùbá translation of the WordSim-353 word pairs and Dr. Felix Y. Adu-Gyamfi and Ps. Isaac Sarfo for helping with the Twi translation. We also thank the members of the Niger-Volta Language Technologies Institute for providing us with clean Yorùbá corpus The project on which this paper is based was partially funded by the German Federal Ministry of Education and Research under the funding code 01IW17001 (Deeplee). Responsibility for the content of this publication is with the authors.

Data	We collect clean and noisy corpora for Yorùbá and Twi in order to quantify the effect of noise on the quality of the embeddings, where noisy has a different meaning depending on the language as it will be explained in the next subsections.

Data ::: Evaluation Test Sets ::: Yorùbá.	One of the contribution of this work is the introduction of the wordsim-353 word pairs dataset for Yorùbá. All the 353 word pairs were translated from English to Yorùbá by 3 native speakers. The set is composed of 446 unique English words, 348 of which can be expressed as one-word translation in Yorùbá (e.g. book translates to ìwé). In 61 cases (most countries and locations but also other content words) translations are transliterations (e.g. Doctor is dókítà and cucumber kùkúmbà.). 98 words were translated by short phrases instead of single words. This mostly affects words from science and technology (e.g. keyboard translates to pátákó ìtwé —literally meaning typing board—, laboratory translates to ìyàrá ìṣèwádìí —research room—, and ecology translates to ìm nípa àyíká while psychology translates to ìm nípa dá).

Finally, 6 terms have the same form in English and Yorùbá therefore they are retained like that in the dataset (e.g. Jazz, Rock and acronyms such as FBI or OPEC). We also annotate the Global Voices Yorùbá corpus to test the performance of our trained Yorùbá BERT embeddings on the named entity recognition task. The corpus consists of 25 k tokens which we annotate with four named entity types: DATE, location (LOC), organization (ORG) and personal names (PER). Any other token that does not belong to the four named entities is tagged with "O". The dataset is further split into training (70%), development (10%) and test (20%) partitions. Table TABREF12 shows the number of named entities per type and partition.

However, Google provided pre-trained multilingual embeddings for 102 languages including Yorùbá (but not Twi).

The Twi language benefits from Wikipedia articles because its inclusion doubles the vocabulary and reduces the bias of the model towards religious texts. However, for Yorùbá, noisy texts often ignore diacritics or tonal marks which increases the vocabulary size at the cost of an increment in the ambiguity too. As a result, the correlation is slightly hurt. One would expect that training with more data would improve the quality of the embeddings, but we found out with the results obtained with the C3 dataset, that only high-quality data helps. The addition of JW300 boosts the vocabulary in both cases, but whereas for Twi the corpus mixes dialects and is noisy, for Yorùbá it is very clean and with full diacritics. Consequently, the best embeddings for Yorùbá are obtained when training with the C3 dataset, whereas for Twi, C2 is the best option. In both cases, the curated embeddings improve the correlation with human judgements on the similarity task a $\Delta \rho =+0.25$ or, equivalently, by an increment on $\rho $ of 170% (Twi) and 180% (Yorùbá).

For Twi, the character-level information reinforces the benefits of clean data and the best correlation with human judgements is reached with CWE embeddings ($\rho ^{C2}_{CWE}=0.437$ vs. $\rho ^{C2}_{fastText}=0.388$).

The same observation holds true for high-resource languages like German and French .

Finally, jiangEtal:2018 translated with Google Cloud the test set from English into Czech, Danish and Dutch. In our work, native speakers translate wordsim-353 into Yorùbá and Twi, and similarity scores are kept unless the discrepancy with English is big (see Section SECREF11 for details). A similar approach to our work is done for Gujarati in JoshiEtAl:2019.

Question: What two architectures are used?
Answer: fastText, CWE-LP