# Enhancing Word Embeddings for Improved Semantic Alignment
## 3 Material and Methods
### Method of Semantic Word Embedding Creation
#### 3.2.1 Embedding Layer Training
originally used embeddings from the SpaCy library as a base [38], even though they also used the GloVe method for training, the data they used differed. Therefore, it was necessary to modify how the vectors were loaded. While SpaCy provides a dedicated function for this task, it had to be replaced with an alternative. * Manual splitting of the training set into two smaller sets, one for training (60%) and one for validation (40%), aimed at proper hyperparameter selection. * Adapting other features: [PERSON] et al. created embeddings by combining the embedding of a word and the embedding of the meaning of that word, for example, the word _tree_ and its meaning _forest_, or _tree (structure)_. They called them _keyword embeddings_. In the case of the experiment conducted in this thesis, the embeddings of the word and the category were combined, like _cat_ and _animal_, or _airplane (technology)_. An embedding created in this way is called a keyword-category embedding. The term _keyword embedding_ is used to describe any word contained in categories, such as _cat_, _dog_, etc. The term _category embedding_ describes the embedding of a specific category word, namely _animal_, _meal_, _technology_, or _vehicle_. * Adding hyperparameter selection optimization: Finally, for the sake of clarity, the algorithm design was as follows: 1. Upload or recreate non-optimized keyword-category embeddings. For each keyword in the dataset and its corresponding category, we created an embedding, as shown in Equation (1), which included both sample objects and category items: \[k_{fish(animal)}=e(fish,animal)=\frac{e(fish)+e(animal)}{2}\] (1)
2. Load and preprocess the texts used to train the model. 3. For each epoch: 3.1. For each sentence in the training set: 3.1.1 Calculate the context embedding. The context itself is the set of words surrounding the keyword, as well as the keyword itself. We create the context embedding by averaging the embeddings of all the words that make up that context. 3.1.2 Measure the cosine distance between the computed context embedding and all category embeddings. In this way, we select the closest category embedding and check whether it is a correct or incorrect match. 3.1.3 Update keyword-category embeddings. The algorithm moves the keyword-category embedding closer to the context embedding if the selected category embedding turns out to be correct. If not, the algorithm moves the keyword-category embedding away from the context embedding. The alpha (\(\alpha\)) and beta (\(\beta\)) coefficients determine how much we manipulate the vectors. 4. Save the newly modified keyword-category embeddings in the _pickle_ format. The _pickle_ module is a method for serializing and deserializing objects, developed in the Python programming language [39]. The top-k accuracy on the validation set was used as a measure of the quality of the embeddings to be used in selecting appropriate hyperparameters and selecting the best embeddings. This is a metric that determines whether the correct category is in the top k categories, where the input data are context embeddings from the validation set. Due to the small number of categories (4 in total), it was decided to evaluate the quality of the embeddings using the top-1 accuracy. Finally, a random search yielded the optimal hyperparameters. ### Semantic Word Embedding Evaluation

We performed the evaluation of our method for building the word embeddings using two applications. We tested how they influence the quality of the classification, and we measured their distribution and plotted them using PCA projections. We also evaluated how the introduction of the embeddings influences the unsupervised processing of the data. #### 3.3.1 Text Classification

The first test for the newly created semantic word embeddings was based on a prediction of the category from the sentence. This can be carried out in many ways, for example, by using the embeddings as a layer in a deep neural network.