# Enhancing Word Embeddings for Improved Semantic Alignment
## 3 Material and Methods
### Method of Semantic Word Embedding Creation
#### 3.2.1 Embedding Layer Training  
For building word embeddings in the neural network, we use a hidden layer with initial weights set to pretrained embeddings. In our tests, the architectures with fewer layers performed best, likely due to the small gradient in the initial layers. As the number of layers increased, the initial layers were updated less effectively during backpropagation, causing the embedding layer to change too slowly to produce strongly semantically related word embeddings. As a result, we used a smaller network for the final experiments. The algorithm for creating semantic embeddings using a neural network layer is as follows:  
Load the pretrained embeddings. Preprocess the input data. Separate a portion of the training data to create a validation set. In our approach, the validation set was 20% of the training data. Next, the initial word vectors were combined with the pretrained embedding data. To achieve this, an embedding matrix was created, where the row at index \(i\) corresponds to the pretrained embedding of the word at index \(i\) in the vectorizer. Load the embedding matrix into the embedding layer of the neural network, initializing it with pretrained word embeddings as weights. Create a neural network-based embedding layer. In our experiments, we tested several architectures. The one that worked best was a CNN with the following configuration. ```
inputs=keras.Input(shape=(None,),dtype=\"int64\") x=word_embeddings_layer(int_sequences_input) x=ConvID(128,5,activation=\"relu\")(x) x=MaxPoolingID(5)(x) x=ConvID(128,5,activation=\"relu\")(x) x=GlobalMaxPoolingID()(x) x=Dense(128,activation=\"relu\")(x) x=Dropout(0.5)(x) outputs=layers.Dense(len(class_names),activation=\"softmax\")(x) model=Model(inputs=inputs,outputs=output)
```  
6. Train the model using the prepared data. Map the data into words in vocabulary embeddings created in hidden layer weights. ```
defget_word_embeddings_from_weights(model): weights=model.get_weights()[0]
#Emptydist



word: weights[word_index.get(word)] ) returnembedding_dict

8. Save the newly created embeddings in a pickle format. After testing several network architectures, we found that the models with fewer layers performed best. This is likely due to the small gradient in the initial layers. As the number of layers increases, the initial layers are less effectively updated during backpropagation, causing the embedding layer to change too slowly to produce strongly semantic word embeddings. For this reason, a smaller network was used for the final experiments, with the following configuration. ```
inputs=keras.Input(shape=(None,),dtype=\"int64\") x=word_embeddings_layer(int_sequences_input) x=ConvD(128,5,activation=\"relu\")(x) x=GlobalMaxPoolingID()(x) x=Dropout(0.5)(x) outputs=layers.Dense(len(class_names),activation=\"softmax\")(x) model=Model(inputs-inputs,outputs=output)
```

#### 3.2.2 Fine-Tuning of Pretrained Word Embeddings

This step uses word embeddings created in a self-supervised manner by fine-tuning pretrained embeddings. The entire process for training embeddings is encapsulated in the following steps:

1. Load the pretrained embeddings. The algorithm uses GloVe embeddings as input, so the embeddings need to be converted into a dictionary. 2. Preprocess the input data (described in Section 3.1). 3. Prepare a co-occurrence matrix of words using the GloVe mechanism. 4. Train the model using the function provided by Mittens for fine-tuning GloVe embeddings [33]. 5. Save the newly created embeddings in a pickle format. #### 3.2.3 Embedding Shifts

The goal of this step is to adjust word vectors in the representation space, enabling geometric operations on pretrained embeddings that incorporate semantics from the WordNet network. To achieve this, we used the implementation by [PERSON] et al., available on GitHub [37]. However, to adapt the code to the planned experiments, the following modifications were introduced:

* Modification of the pretrained embeddings: Since [PERSON] et al.