# Enhancing Word Embeddings for Improved Semantic Alignment
## 2 Related Works
By summing the vectors of the corresponding \(n\)-grams, we obtain the full word embedding. As the name suggests, this method is fast. In addition, since the words are represented as a sum of \(n\)-grams, it is possible to represent words that were not previously added to the dictionary, which increases the accuracy of the results [7; 22]. The authors of this approach have also proposed a compact version of fastText, which, due to quantization, maintains an appropriate balance between model accuracy and memory consumption [23]. [PERSON] et al. [24] introduced the ELMo embedding method, which uses a bidirectional language model (biLM) to derive word representations based on the full context of a sentence. Unlike traditional word embeddings, which map each token to a single dense vector, ELMo computes representations through three layers of a language model. The first layer is a Convolutional Neural Network (CNN), which produces a non-contextualized word representation based on the word's characters. This is followed by two bidirectional Long Short-Term Memory (LSTM) layers that incorporate the context of the entire sentence. By design, ELMo is an unsupervised method, though it can be applied in supervised settings by passing pretrained vectors into models like classifiers [25]. When learning a word's meaning, ELMo uses a bidirectional RNN [26], which incorporates both preceding and subsequent context [24]. The language model learns to predict the likelihood of the next token based on the history, with the input being a sequence of \(n\) tokens. Bidirectional Encoder Representations of Transformers, often referred to as BERT [27], is a transformer-based architecture. Transformers were first introduced relatively recently, in 2017 [28]. This architecture does not rely on recursion, as previous state-of-the-art systems (such as LSTM or ELMo) did, significantly speeding up training time. This is because sentences are processed in parallel rather than sequentially, word by word. A key unit, self-attention, was also introduced to measure the similarity between words in a sentence. Instead of recursion, positional embeddings were created to store information about the position of a word/token in a sentence. BERT was designed to eliminate the need for decoders. It introduced the concept of masked language modeling, where 15% of the words in the input are randomly masked and predicted using positional embeddings. Since predicting a masked token only requiresthe surrounding tokens in the sentence, no decoder is needed. This architecture set a new standard, achieving state-of-the-art results in tasks like question answering. However, the model's complexity leads to slow convergence [27]. [PERSON] et al. [10] proposed an approach to represent unambiguous words based on their meaning in the context [29]. In this method, word vector representations are adjusted according to polysemy: vectors of synonyms are brought closer together in the representation space, while vectors of homonyms are separated. The supervised iterative optimization process leads to the creation of vectors that offer more accurate representations compared to traditional approaches like word2 vec or GloVe. Pretrained models can be used across a wide range of applications. For instance, the developers of Google's Word2 Vec trained their model on unstructured data from Google News [30], while the team at Stanford used data from Wikipedia, Twitter, and web crawling for GloVe [31]. Competing with these pretrained solutions is challenging; instead, it is often more effective to leverage them. For example, [PERSON] and [PERSON] applied the fine-tuning of pretrained vectors for sentiment analysis and emotion detection tasks [32]. To further leverage the power of pretrained embeddings, [PERSON] and [PERSON] developed the Mittens tool for fine-tuning GloVe embeddings [33]. The main goal is to enrich word vectors with domain-specific knowledge, using either labeled or unlabeled specialized data.