# Creating Welsh Language Word Embeddings
## 4 Results and Analysis
### Word Clustering  
Concept categorisations is a common method for evaluating word embeddings [51]. It checks whether words can be grouped into natural categories from their vectors only. For example, the words 'bear' and 'bull' belong to an animal class, while cupboard and chair will belong to a furniture class.  
We adapted a concept categorisation dataset from [52] by translating it into Welsh using bilingual Welsh-English speakers. It consists of 214 words assorted into 13 categories, which are provided in the Appendix A of this article. The vectors for each of the 214 words were clustered using \(k\)-means clustering combined with cosine distance and Euclidean distance respectively, with \(k=13\) to match the number of categories. Ideally, the 13 clusters should map directly to the 13 categories. Three measures were used to evaluate the clustering results.  
* Purity measures the extent to which clusters contain words of the same category. It is calculated using Equation (5), where \(N\) is the number of words in total, \(M\) is the set of clusters and \(D\) is the set of known categories. It is calculated as the average count of the categories per cluster. Purity is commonly used to evaluate vector semantics, e.g., [51, 53]. Its main shortcoming is that it does not penalise a single category being distributed over more than one cluster. For example, words belonging to an education category being distributed over more than one cluster. \[P=\frac{1}{N}\sum_{m\in M}\max_{d\in D}|m\cap d|\] (5)
* Rand Index measures the extent to which pairs of words that do or do not belong to the same category end up in the same cluster or not. For each word pair, clustering can produce a true positive (the same category and the same cluster), true negative (different categories and different clusters), false positive (different categories, but the same cluster), or false negative (the same category, but different clusters). The counts are given by \(TP\), \(TN\), \(FP\) and \(FN\), respectively. These measures were used to measure accuracy in [52]. Rand index is calculated as the proportion of correctly predicted pairs as prescribed by Equation (6). \[RI=\frac{TP+TN}{TP+TN+FP+FN}\] (6)
* Entropy measures how words from the same categories are distributed across the clusters. Low entropy indicates that words of the same category tend to be grouped within the same cluster. This measure, used in [53], is given in Equation (7). \[E=-\sum_{m\in M}\frac{|m|\sum_{d\in D}\frac{|m\cap d|}{|m|}\ln\left(\frac{|m \cap d|}{|m|}\right)}{N\ln(|D|)}\] (7) Only the fastText models can be evaluated in this way. As word2 vec models do not capture subword information, vectors for unseen words cannot be implied, and so the \(k\)-means algorithm fails. The results for the fastText models are given in Table 4.  
WNLT tokenisation combined skip-gram performed best when Euclidean distance was used for clustering, while Gensim tokenisation combined with skip-gram performedbest when cosine distance was used for clustering. [PERSON]'s model performed the worst in both cases, indicating that a larger and more representative corpus yields word embeddings that perform better at concept categorisation tasks.