# Creating Welsh Language Word Embeddings
## 3 Methods
### Training  
Traditionally, word embeddings are computed by minimising the distance between the words that appear in similar contexts. Prominent examples of such word embedding methods include word2 vec [34], GloVe [35] and fastText [36]. Contextual word embeddings take this approach to the next level by creating different embeddings for the same word used in different contexts to convey different meanings. For example, the word 'bank' can be interpreted as either 'financial bank' and 'river bank' depending on the context. Examples of such word embedding methods include ELMo [37] and BERT [38]. Finally, word embeddings can be enriched with different types of information. For example, sentiment embeddings [39] incorporate the sentiment of words into their embeddings. The benefit of sentiment embeddings over standard embeddings is that the distance between opposite words such as 'good' and 'bad' that tend to appear in similar contexts will become larger to reflect their semantics more appropriately. In this study, we focused solely on traditional word embedding methods, specifically word2 vec [34] and fastText [36]. These approaches produce one vector per word, which enabled us to make direct comparison to the existing baseline approach [2]. Word2 vec has two versions known as skip-gram and continuous bag of words (CBOW) respectively [34; 40]. Given a target word \(w_{t}\), skip-gram aims to predict its context. Formally, the objective of the skip-gram method is to maximize the log-likelihood defined in Equation (1), where \(\mathcal{C}_{t}\) is the set of indices of context words surrounding the target word \(w_{t}\). \[\sum_{t=1}^{T}\sum_{c\in\mathcal{C}_{t}}\log p(w_{c}|w_{t}) \tag{1}\]  
For each word \(w\), the skip-gram method defines two vectors \(u_{w}\) and \(v_{w}\) in \(\mathbb{R}^{n}\), which are learnt automatically. These vectors are commonly referred to as input and output vectors respectively [36]. Given this, the skip-gram version estimates the probability \(p(w_{c}|w_{t})\) using the SoftMax function defined in Equation (2) where \(s:\mathbb{R}^{n}\times\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}\) is the scoring function defined in Equation (3) and \(u_{t}^{\top}\) denotes the transpose of the vector \(u_{t}\). \[p(w_{c}|w_{t})=\frac{\exp(s(w_{t},w_{c}))}{\sum_{j=1}^{W}\exp(s(w_{t},w_{j}))} \tag{2}\]  
\[s(t,c)=u_{t}^{\top}v_{c} \tag{3}\]  
This formulation of \(p(w_{c}|w_{t})\) renders the learning of the vectors \(u_{w}\) and \(v_{w}\) impractical because the cost of computing derivatives is proportional to the size of the corpus \(W\). To overcome this challenge, [40] proposed two approximations known as hierarchical SoftMax and negative sampling. Conversely, given a context, CBOW aims to predict the target word. Formally, instead of modelling \(p(w_{c}|w_{t})\), CBOW models \(p(w_{t}|w_{c})\)[34]. The fastText method generalises the two versions of word2 vec, i.e., skip-gram and CBOW, by considering the subwords within the words [36]. The authors argue that this method is useful for morphologically rich languages such as Turkish and Finnish. Welsh is also morphologically rich, where inflection can give rise to multiple surface forms of a single word. Consider, for example, the word 'ci' (Engl. dog). In the phrase 'ei gi' (Engl. his dog), soft mutation applies to the word 'ci'. On the other hand, in the phrase 'ei chi' (Engl. her dog), aspirate mutation applies to the word 'ci' [41]. Therefore, both 'gi' and 'chi' correspond to the same lemma--'ci'. Mutations occur frequently in Welsh. Subword information has the potential to allow a word embedding method to relate different mutations of the same word. Therefore, fastText represents an appropriate choice of the word embedding method for this language. FastText inserts special boundary characters \(<\) and \(>\) at the beginning and end respectively of each word. This allows the model to distinguish prefixes and suffixes from other character sequences.