# Creating Welsh Language Word Embeddings
## 4 Results and Analysis
### Word Similarity  
Similarity-based methods for evaluating word embeddings use ground-truth pairwise semantic similarity of words, where semantic similarity is represented by a value within a fixed range with higher values indicating greater semantic similarity. The correlation of the semantic similarity of words and the cosine similarity of the corresponding word embeddings is used to gauge the utility of the word embeddings. The higher the correlation, the higher the utility of word embeddings. The ground-truth semantic similarity is typically estimated by native speakers. We considered two word similarity datasets. First, the WordSimilarity-353 dataset contains a total of 353 word pairs in English [45; 46]. Each word pair is associated with the mean taken from semantic similarity estimated independently by multiple individuals on a Likert scale from 0 to 10 inclusive. We adapted this dataset by translating it into Welsh using bilingual Welsh-English speakers. We used [PERSON]'s rank correlation coefficient to measure the correlation between semantic similarity and cosine similarity. Table 2 provides the results. The naive tokenisation combined with CBOW performed best. In fact, all CBOW methods regardless of tokenisation outperformed [PERSON]'s model. On the other hand, all methods based on skip-gram performed worse than [PERSON]'s model. The second dataset we considered was SimLex-999, which contains a total of 999 word pairs consisting of 666 noun pairs, 222 verb pairs and 111 adjective pairs in English [47,48]. Each word pair is associated with the mean semantic similarity estimated independently by 50 individuals on a Likert scale from 0 to 10 inclusive. We adapted this dataset by translating it into Welsh using bilingual Welsh-English speakers. Brute-force translation may introduce minor inaccuracies or other biases into the ground truth. For example, the word 'bank' has two interpretations, each being similar to the words 'river' and'money' respectively. However, this homonymy is not observed in Welsh. Other considerations are cultural. For example, references to American concepts such as 'baseball', 'dollar' and 'buck' may occur rarely if at all in the Welsh corpus. Two word pairs with no Welsh equivalents, 'football-soccer' and 'dollar-buck', were removed from the dataset. In addition to semantic similarity, SimLex-999 also provides the strength of free association represented as a value in the range 0 to 10 inclusive. For example, the words 'car' and 'petrol' are not semantically similar but have high free association. The strength of free association was calculated using the University of South Florida Free Association Dataset [49]. This dataset was generated by presenting human subjects with one of 5000 cue concepts and asking them to write the first word that comes to mind. Table 3 provides the results. The baseline performed the best, although all values are very low. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Word Embedding Method** & **Tokenisation** & **Version** & **Semantic Correlation** \\ \hline fastText & naive & skip-gram & 0.0495 \\ fastText & naive & CBOW & 0.1164 \\ fastText & Gensim & skip-gram & 0.0681 \\ fastText & Gensim & CBOW & 0.1326 \\ fastText & WNLT & skip-gram & 0.0632 \\ fastText & WNLT & CBOW & 0.1108 \\ word2 vec & naive & skip-gram & 0.0448 \\ word2 vec & naive & CBOW & 0.1157 \\ word2 vec & Gensim & skip-gram & 0.0692 \\ word2 vec & Gensim & CBOW & 0.1285 \\ word2 vec & WNLT & skip-gram & 0.0604 \\ word2 vec & WNLT & CBOW & 0.1067 \\ Grave’s & \(\cdots\) & \(\cdots\) & 0.0785 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The results achieved on the WordSimilarity-353 dataset.