# Enhancing Word Embeddings for Improved Semantic Alignment
## 3 Material and Methods
### Method of Semantic Word Embedding Creation
#### 3.2.1 Embedding Layer Training
Initially, however, it would be better to test the capabilities of the embeddings themselves without further training on the new architecture. Thus, we used a method based on measuring the distance between embeddings. In this case, our approach consisted of the following steps:

1. For each sentence in the training set: 1.1 Preprocess the sentence. 1.2 Convert the words from the sentence into word embeddings. 1.3 Calculate the mean embedding from all the embeddings in the sentence. 1.4 Calculate the cosine distance between the mean sentence embedding and all category embeddings based on Equation (2): \[distance(\overline{e},c)=\cos(\overline{e},c)=\frac{\overline{e}\cdot c}{\| \overline{e}\|\|c\|}\] (2) where \(\overline{e}\) is the mean sentence embedding and \(c\) is the category embedding. 1.5 Select the assigned category by taking the category embedding with the smallest distance from the sentence embedding. 2. Calculate the accuracy score by taking the predictions for all sentences based on Equation (3): \[Accuracy=\frac{Number\ of\ correct\ predictions}{Total\ number\ of\ predictions}\] (3)

In addition, in modern research, weighted embeddings are typically used to improve the quality of the prediction [40; 41]. Latent semantic analysis uses a term-document matrix that takes into account the frequency of a term's occurrence. This is accomplished by using the TF-IDF weighting schema, which is used to measure the importance of a term by its frequency of occurrence in a document. To fully see the potential of word embeddings in classification, it is beneficial to combine them with a separate classification model, distinct from the DNN used to create the embeddings. We chose a KNN and Random Forest [42], which are non-neural classifiers, to avoid introducing any bias into the results. It should be noted that for the evaluation of the embeddings, any other classifier can be used. #### 3.3.2 Separability and Spatial Distribution

To measure the quality of the semantic word embeddings, we checked their spatial distribution by performing two tests. The first was an empirical test aimed at reducing the dimensionality of the embeddings and visualizing them in two-dimensional space using the Principal Component Analysis (PCA) method, which transforms multivariate data into a smaller number of representative variables, preserving as much accuracy as possible. In other words, the goal is to achieve a trade-off between the simplicity of the representation and accuracy [43]. Still considering the quality of the semantic word embeddings, we computed the relevant metrics, which prove the separability of the classes, as well as the correct distribution of the vectors in space. In view of this, we used the mean distance of all the keywords to their respective category words. The purpose of checking these distances is to test whether the average embedding of keywords in a given category yields a vector that is close to the category vector, evaluated based on Equation (4), with sample objects from the same semantic category. \[\frac{e(computer)+e(CPU)+e(keyboard)+e(monitor)+e(TV)+e(phone)}{6}\approx e(technology) \tag{4}\]

The formula for calculating the average distance of keyword embeddings from their category embeddings is based on Equation (5). \[meanDistanceToCategory(c,C)=\frac{1}{|C|}\sum_{k\in C}\cos(c,k)=\frac{1}{|C|} \sum_{k\in K}\frac{c\cdot k}{\|c|\|k\|} \tag{5}\]

where

* \(c\) is a category embedding;
* \(C\) is a list of all the keyword embeddings in the category. The local density of classes was another metric used. This is the mean distance of all the keywords to the mean vector of all the keywords in the category, calculated based on Equation (6). \[categoryDensity(C)=\frac{1}{|C|}\sum_{k\in C}\cos(k,\frac{1}{|C|}\sum_{l\in C}l) \tag{6}\]

where

* \(c\) is a category embedding;
* \(C\) is a list of all the keyword embeddings in the category. The Silhouette Coefficient was another metric used to examine class separability [44].