# Enhancing Word Embeddings for Improved Semantic Alignment
## 2 Related Works  
Word embeddings, also known as distributed word representations, are typically created using unsupervised methods. Their popularity stems from the simplicity of the training process, as only a large corpus of text is required. The embedding vectors are first constructed from sparse representations and then mapped to a lower-dimensional space, producing vectors that correspond to specific words. To provide context for our research, we briefly describe the most popular approaches to building word embeddings. After the pioneering work of [PERSON] et al. on neural language models [11], research on word embeddings stalled due to limitations in computational power and algorithms being insufficient for training large vocabularies. However, in 2008, [PERSON] and [PERSON] showed that word embeddings trained on sufficiently large datasets capture both syntactic and semantic properties, improving performance in subsequent tasks. In 2011, they introduced SENNA embeddings [12], based on probabilistic models that allow for the creation of word vectors. For each training iteration, an \(n\)-gram was used by combining the embeddings of all \(n\) words. The model then modified the \(n\)-gram by replacing the middle word with a random word from the vocabulary. It was trained to recognize that the intact \(n\)-gram was more meaningful than the modified (broken) one, using a hinge loss function. Shortly thereafter, [PERSON] et al. replicated this method, but instead of replacing the middle word, they replaced the last word [13], achieving slightly better results. These embeddings have been tested on a number of NLP tasks, including Named Entity Recognition (NER) [14], Part of Speech (POS) [15], Chunking (CHK) [16], Syntactic Parsing (PSG) [17], and Semantic Role Labeling (SRL) [18]. This approach has shown improvement in a wide number of tasks [19] and is computationally efficient, using only 200 MB of RAM. Word2 vec is a set of models introduced by [PERSON] et al. [5] that focuses on word similarity. Words with similar meanings should have similar vectors, while words with different vectors do not have similar meanings. Considering the two sentences _\"They like watching TV\"_ and _\"They enjoy watching TV\"_, we can conclude that the words _like_ and _enjoy_ are very similar, although not identical. Primitive methods like one-hot encoding would not be able to capture the similarity between them and would treat them as separate entities, so the distance between the words _(like, TV)_ would be the same as the distance between _(like, enjoy)_. In the case of word2 vec, the distance between _[like, TV)_ would be greater than the distance between _[like, enjoy]_. Word2 vec allows one to perform basic algebraic operations on vectors, such as addition and subtraction. The GloVe algorithm [6] integrates word co-occurrence statistics to determine the semantic associations between words in the corpus, which is somewhat the opposite to Word2 vec--which depends only on local features in words. GloVe implements the global matrix factorization technique, which uses a matrix to encode the presence or absence of word occurrences. Word2 vec is commonly known as neural word embeddings; hence, it is a feedforward neural network model, whereas GloVe is commonly known as a count-based model, also called a log-bilinear model. By analyzing how frequently words co-occur in a corpus, GloVe captures the relationships between words. The co-occurrence likelihood can encode semantic significance, which helps improve performance on tasks like the word analogy problem. FastText is another method developed by [PERSON] et al. [20] and further improved [21] by introducing an extension of the continuous skipgram model. In this approach, the concept of word embedding differs from Word2 Vec or GloVe, where words are represented by vectors. Instead, word representation is based on a bag of character \(n\)-grams, with each vector corresponding to a single character \(n\)-gram.