# Creating Welsh Language Word Embeddings
## 4 Results and Analysis
### Qualitative Evaluation  
We considered 30 nearest neighbours of a small set of prototype words. All prototype words were present in the text corpus used in training the word embeddings. Nearest neighbours were identified using the cosine similarity measure, which was calculated using the formula in Equation (9). \[\text{similarity}(\mathbf{u},\mathbf{v})=\frac{\mathbf{u}\cdot\mathbf{v}}{ \|\mathbf{u}\|\|\mathbf{v}\|} \tag{9}\]  
Given the cost of inspecting the neighbourhood manually, we limited this aspect of evaluation to comparison of a single model, namely the fastText skip-gram model with WNTL tokenisation, to the baseline. This model was chosen as it demonstrated the best results in our quantitative evaluation presented above. The following are a small selection of the words used for comparison and comments on the performance of the models:  
* _nofio_ (to swim): Both models list several sports and water-related activities as nearest neighbours although each model gives different words, including _beicio_ (cycling), _caiacio_ (kayaking), _syrffio_ (surfing), _plymio_ (diving), _nofuyr_ (swimmers) and _pwl_ (pool). [PERSON]'s model also lists several mutations and misspellings of _burccini_ (burkin), which may indicate a small or un-diverse training corpus. * _glaw_ (rain): Our model lists a variety of weather phenomena including _eira_ (snow), _guvnttoedd_ (winds), _cavodydydd_ (showers), _cenllysg_ (hail), _guzlyb_ (wet), _stormydd_ (storms), _corvyntoedd_ (hurricanes), and _tananu_ (thunder). [PERSON]'s model does list some related words such as _monswn_ (monsoon), but mainly lists derivations of the original word \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Tokenisation** & **Version** & **Accuracy** \\ \hline fastText & naive & skip-gram & 16.64\% \\ fastText & naive & CBOW & 23.67\% \\ fastText & Gensim & skip-gram & 18.75\% \\ fastText & Gensim & CBOW & 28.01\% \\ fastText & WNLT & skip-gram & 17.97\% \\ fastText & WNLT & CBOW & 25.37\% \\ word2 vec & naive & skip-gram & 16.66\% \\ word2 vec & naive & CBOW & 23.88\% \\ word2 vec & Gensim & skip-gram & 19.43\% \\ word2 vec & Gensim & CBOW & 28.22\% \\ word2 vec & WNLT & skip-gram & 18.21\% \\ word2 vec & WNLT & CBOW & 25.21\% \\ [PERSON]’s &... &... & 9.00\% \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The results achieved for the word analogy task. e.g., _glawiog_ (rainy), and other unrelated words such as _car-boot_ and _sgubai_ (sweep), although this may relate to rain sweeping across the land. * _hupus_ (happy): Our model lists several synonyms or related adjectives, including _lucus_ (lucky), _falch_ (glad), _ffdus_ (fortunate), and _bodlon_ (satisfied). [PERSON]'s model list some of these, but contains many other less similar words that could appear in the same context, such as _anhapus_ (unhappy), _eisiau_ (want), _teinlon_ (felt), and _grac_ (angry). It also lists words of similar spelling, but unrelated semantically: _siupus_ (shapely) and _nupus_ (_brassica napus_; a species of rapeseed), which may indicate their model is relying too heavily on subword information. * _meddalved_ (software): Both models list many words related to computing and technology here, including _salvedd_ (malware), _amgryptio_ (encrypting), _cyrifiatud_ (computational), _metadata_ (meta-data), _telegyfultrebu_ (telecommunication), and _rhugugwyneb_ (interface). Our model provides a greater variety of words, while [PERSON]'s model provides some English words and product names e.g., _DropBox_. This may be due to the fact that a more recent corpus was used by our model, as there will have been more technological articles published, and more technological terminology developed, in recent years. * _ffrangeg_ (the French language). There was a stark difference in the lists produced by the two models. Our model returned several other western European languages including _llydaweg_ (Breton), _isahmaeeg_ (Dutch), _galaweg_ (Gallo) and _sbaneeg_ (Spanish).