# Creating Welsh Language Word Embeddings
## 2 Related Work  
This section reviews language resources that can support NLP in Welsh and, in particular, creation of word embeddings in this language. The first step in training word embeddings is to assemble a large corpus. Corpus-based language studies provide empirically based objective analyses of patterns of language as it is actually used, using evidence from a corpus (singular) or corpora (plural). CorCenCC (Corpws Cenedlaethol Cymraeg Cyfoes--the National Corpus of Contemporary Welsh) is a major corpus [3] containing 10 million words of written, spoken and digital (or 'e') Welsh language. It contains multiple language samples from real-life communication, allowing linguists to explore Welsh as it is actually used. It organises data into multiple facets, which can be used to study sub-languages as defined by [4]. All data are also annotated with different types of linguistic information including morphological units, tokens, part-of-speech (POS) [5] and semantic categories [6; 7]. In addition to linguistic research, the corpus can support a range of other applications such as learning and teaching of Welsh, but also NLP. In this study, together with other data sources, samples of data from an early release of CorCenCC were used to train word embeddings (see Section 3.1). Having identified the relevant sources of data, the first step towards the creation of word embeddings is the process of identifying individual words as discrete units of text, the process known as tokenisation. Welsh Natural Language Toolkit (WNLT) [8] implements a set of rule-based Welsh NLP tools for tokenisation, lemmatisation, POS tagging and named entity recognition (NER), which are embedded into the GATE framework [9]. Similar NLP capabilities have been implemented to support pre-processing of documents stored in CorCenCC, which are tokenised and tagged using CyTag, a rule-based POS tagger [5]. Another POS tagger, which can be used as a web service without the need to install it locally, can tag lexical categories (e.g., verbs and nouns) as well as the features specific to the Welsh language such as mutations [10]. The same team developed a lemmatiser, which can be used to normalise any inflected, mutated and/or conjugated word into its lemma [11]. Members of the CorCenCC team also developed downstream NLP methods for multi-word term recognition [12] and semantic tagging [6; 7]. These methods were originally developed for English and successfully adapted for Welsh [13; 14; 15]. These methods can be useful for improving the performance of the downstream task of machine translation methods. For example, verbatim translations often deviate from the established terminology in the target language. Therefore, high-quality translations, performed by either humans or machines, require management of terminologies. Most machine translation systems require a terminology dictionary, e.g., [16; 17] and/or the ability to extract terms dynamically [12] to support translations that use established terminology in the target language. In general, phrase-based statistical machine translation can improve the levels of translation quality where sufficiently large parallel corpora can be used for training as demonstrated in the case of English and Welsh [18]. In particular, the ability to align translated texts into paired sentences in the two languages [19] can support training of cross-lingual word embeddings [20], which can allow existing English language resources to be re-used for applications in Welsh. Welsh word embeddings were first described in a study that presented a general method for creating word embeddings that was tested across 157 languages [2]. They were since used to support a machine-learning approach to a joint task of POS and semantic tagging [21]. However, these embeddings were created using a generic approach, which does not take into account specific characteristics of the Welsh language.