Combining Pretrained High-Resource Embeddings and Subword
Representations for Low-Resource Languages
Machel Reid, Edison Marrese-Taylor, Yutaka Matsuo

Motivation

Data

The contrast between the need for large amountsof data for current Natural Language Processing(NLP) techniques, and the lack thereof, isaccentuated in the case of African languages,
most of which are considered low-resource.

For use on our downstream task: we use twodatasets:
1. The Multilingual Parallel Bible Corpus(Christodoulopoulos and Steedman 2015)
2. XhOxExamples - New dataset created fromexamples collected from the isiXhosa-English
Oxford Dictionary

To help circumvent this issue, we explore
techniques for combining qualities ofmorphologically rich languages (MRLs) andleveraging pretrained word vectors inwell-resourced languages (such as English).
We evaluate our ideas Xhosa-English translation
as downstream task for evaluation.
Proposed Approach

We assume the existence the following:
1. Low Resource Vocabulary: V = {v1, . . . , vT }2. The corresponding translations of the words in V
1in a high-resource language: D = {d1, . . . , dT }3. A pretrained embedding matrix in thehigh-resource language: EHR
1

Note that di can be comprised of a sequence of words in the case that vi
cannot be appropriately expressed in one word in the high resourcelanguage.

I To leverage the high-resource language, weembed the atomic elements (e.g. indoda → man)of D in EHR and map the resulting vectors to thecorresponding word in V.
I In the case of di being a sequence (e.g. bethuna[listen, everyone]), we take a similar approach toLazaridou et al. 2017 and sum the normalized
word vectors for each word in di to produce aword representation for the word vi . We refer tothe resulting embedding matrix as EV .
I We pretrain another embedding matrix EM ona corpus in our low resource language using
subword information to capture similarity
correlated with the morphological nature of words

Embedding Configurations

I Random Initialization
I XhPre - Initialization with EV . Words notpresent in EV are initialized with EM .
I XhSub - Initialization with EM only.
I VecMap - We learn cross-lingual wordembedding mappings by taking two sets ofmonolingual word embeddings, EV and EM , andmapping them to a common space following
Artexte et al., 2018.
I XhMeta - We compute meta-embeddings forevery word wi by taking the mean of EV (wi ) andEM (wi ), following Coates & Bollegala 2019.
Words not present in EV are associated with an
UNK token and its corresponding vector.
Evaluation

I On our collected datasets, we use
Xhosa-English translation as a downstream
task
I The generative task is evaluated using
sentence level BLEU, BLEU-4 and BEER.
ResultsBibleModel

XhOxExamples

BLEU BEER BLEU-4 BEER

Random Initializaton 21.79 21.84 16.08

25.30

VecMap22.46 22.03 16.38

25.42

XhSub

24.65 22.79 17.37

26.04

XhPre

27.67 22.40 17.06

25.70

XhMeta29.09 23.33 17.77

26.44

We show that both types of representations are importantin the context of the low-resourced MRL, Xhosa, and hopethat this research is able to assist others when doing NLP
for African languages.
machelreid2004@gmail.com, {emarrese,matsuo}@weblab.t.u-tokyo.ac.jp