# Enhancing Word Embeddings for Improved Semantic Alignment
## 3 Material and Methods
### Method of Semantic Word Embedding Creation
#### 3.2.1 Embedding Layer Training
In the case of the mean sentence distance to a category embedding, geometric embeddings showed their full potential, since both the method for creating embeddings and the method for testing them were based on the geometric aspects of vectors in space. In contrast, the Random Forest classification is based on individual features (vector dimensions), and each decision tree in the ensemble model consists of a limited number of features, so it is important that individual features convey as much value as possible, rather than the entire vector as a whole. Again, weighted embeddings achieve the worse results, although in this case, the difference between weighted embeddings and their unweighted counterparts is much smaller, similar to the difference in the source [40]. However, it turns out that even weighted embeddings, for the neural and fine-tuned methods (95.56% and 93.5%), perform better than the basic version of geometric embeddings (92.39%). Original weighted embeddings perform the worst, with an accuracy drop of almost 12 percentage points (97.95% to 86.07%). Figure 1: Classification results for the nearest neighbor method. Figure 2: Classification results for the random forest classifier with mean sentence embedding. Table 1 shows a summary of the results for Random Forest classification and mean sentence embedding distance. ### Separability and Spatial Distribution

In the next test, the separability of the classes in space and the overall distribution of the vectors were tested. First, the distribution of keyword embeddings in space is shown using the PCA method to reduce dimensionality to two dimensions [43]. Figure 3 shows the distribution of the original embeddings. As can be seen, even here, the classes seem to be relatively well separated. The classes _technology_ and _vehicle_ seem to be far apart in space, while _meal_ and _animal_ are closer together. Note that the keyword _fish_ is much further away from the other keywords in the cluster, heading towards the category _meal_. Also, the word _kebab_ seems to be far away from the correct cluster, even though it refers to a meal and does not have a meaning correlated with animals. Figure 4 shows a visualization of the PCA for neural embeddings. In this case, the proximity of vectors within classes is visible, and each class becomes more separable. One can see a significant closeness to the corresponding classes of words that were previously more distant, such as _fish_ and _kebab_, although the latter is somewhere roughly between the centers of the two clusters, so clustering with the K-means algorithm later in this section may still be somewhat problematic. \begin{table}
\begin{tabular}{l l l} \hline \multirow{2}{*}{**Embeddings**} & **Classification with** & **Classification with Mean Sentence** \\  & **Random Forest** & **Embedding Distance** \\ \hline Original & 0.920 & 0.839 \\ Original Weighted & 0.861 & 0.775 \\ Neural & 0.966 & 0.875 \\ Neural Weighted & 0.935 & 0.836 \\ Fine-tuned & 0.979 & 0.878 \\ Fine-tuned Weighted & 0.956 & 0.826 \\ Geometrical & 0.924 & 0.938 \\ Geometrical Weighted & 0.878 & 0.873 \\ \hline \end{tabular}
\end{table} 
Table 1: Classification results for weighted and unweighted semantic embeddings. (Underline—best results). Figure 3: PCA visualization of original embeddings. Figure 5 shows how vectors decompose in two-dimensional space for fine-tuned embeddings, and this method seems to have the worst separability of all the methods so far. Once again, the words _fish_ and _kebab_, which are separated from their classes while being very close to each other, proved to be problematic. Also, the classes _technology_ and _vehicle_ seem to be more similar to each other, which will be checked with the metrics shown later. The last of the visualizations of vectors in two-dimensional space using the PCA method concerns geometrical embeddings, as illustrated in Figure 6. In this case, the classes are brilliantly separable, the words within a single cluster are very close to each other, and the problematic cases are resolved.