# Creating Welsh Language Word Embeddings
## 4 Results and Analysis
### Word Analogies  
Word analogies can be used to evaluate whether the semantic relationships between words correspond to the mathematical relationships between their respective embeddings. For example, the relationship between the word 'king' and 'queen' should be identical to the relationship between the words 'actor' and 'actress'. Therefore, if \(\mathbf{x}_{\text{king}}\), \(\mathbf{x}_{\text{queen}}\), \(\mathbf{x}_{\text{actor}}\) and \(\mathbf{x}_{\text{actress}}\) are the trained vectors for the words 'king', 'queen', 'actor' and 'actress', respectively, then we would expect:  
\[\mathbf{x}_{\text{actor}}-\mathbf{x}_{\text{king}}+\mathbf{x}_{\text{queen}} \approx\mathbf{x}_{\text{actress}} \tag{8}\]  
Given a set of examples of various language-specific relationships, the Gensim library allows us to measure the proportion for which the above equation holds, where the proximity is calculated using five nearest neighbours of the vector to the left. We  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **Word Embeddings** & & & **Euclidean Distance** & & **Cosine Distance** & \\
**Method** & **Tokenisation** & **Version** & **Purity** & **Rand Index** & **Entropy** & **Purity** & **Rand Index** & **Entropy** \\ \hline fastText & naive & skip-gram & 0.4860 & 0.8305 & 3.3298 & 0.6542 & 0.9178 & 2.2509 \\ fastText & naive & CBOW & 0.3832 & 0.7451 & 4.2892 & 0.5140 & 0.8826 & 3.4872 \\ fastText & Gensim & skip-gram & 0.5701 & 0.8811 & 2.8500 & 0.7242 & 0.9319 & 2.1092 \\ fastText & Gensim & CBOW & 0.4533 & 0.6958 & 3.9220 & 0.5140 & 0.8984 & 3.1224 \\ fastText & WNLT & skip-gram & 0.5794 & 0.8689 & 2.8448 & 0.6636 & 0.9225 & 2.2045 \\ fastText & WNLT & CBOW & 0.3972 & 0.7479 & 4.2486 & 0.5514 & 0.8692 & 3.2374 \\ Grave’s &... &... & 0.2383 & 0.5640 & 5.3069 & 0.4486 & 0.7965 & 3.9540 \\ \hline \hline \end{tabular}
\end{table}
Table 4: The results for the concept categorisation task.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Tokenisation** & **Version** & **\% Correct** \\ \hline fastText & naive & skip-gram & 38\% \\ fastText & naive & CBOW & 32\% \\ fastText & Gensim & skip-gram & 36\% \\ fastText & Gensim & CBOW & 30\% \\ fastText & WNLT & skip-gram & 42\% \\ fastText & WNLT & CBOW & 30\% \\ Grave’s &... &... & 28\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: The results achieved on the synonymy detection task.  
translated language-independent relationships such as those between nations and nationalities to Welsh. A dataset of grammatical relationships was constructed by a native Welsh speaking linguist and included adjective-opposite (435 pairs), adjective-comparative (55 pairs), adjective-superlative (55 pairs), adjectives-equative (55 pairs), nationalities (210 pairs), languages (120 pairs), noun-plural (6555 pairs), noun-singular (105 pairs), noun-gender (703 pairs), adjective-gender (105 pairs), adjective-plural (528 pairs), verb-nonfinite (325 pairs), verb-past-1 st-singular (45 pairs), verb-past-3 rd-singular (45 pairs), verb-past-impersonal (45 pairs), verb-present-impersonal (45 pairs), inflectional-preposition-1 st-singular (36 pairs), and inflectional-preposition-2 nd-plural (36 pairs).  
The accuracy over all 9503 pairs, for each of the models, are given in Table 6. All models perform much better than [PERSON]'s model. There are also obvious rankings between tokenisation and training methods: gensim tokenisation performed better than WNLT, which in turn performed better than naive tokenisation; CBOW yields much more accurate models than skip-gram here; and, surprisingly, word2 vec performs marginally better than fastText in all cases.