# Contemporary Approaches in Evolving Language Models
## 3 Neural Network Models
### Word Representation  
#### 3.1.1 Word2 Vec  
Word2 Vec is a neural model designed for deriving vector representations (embeddings) of words from vast textual corpora which was introduced by [PERSON] and his colleagues [18; 19]. It was conceived with the intent of capturing the semantic properties of words, facilitating their representation as numerical vectors. Notably, there is no need for corpus annotation as the Word2 Vec application entails training on raw, unlabeled data. In the Word2 Vec model, there is an encapsulation of inter-relational nuances and similarities among words. Distinguishing it from other neural network architectures tailored for learning word vector representations, Word2 Vec proved to be highly computationally efficient. It showed the capacity to handle vast lexicons encompassing millions or even billions of words while evading dense matrix multiplications. In collaboration with his peers, [PERSON] introduced two training methodologies based on hierarchical softmax and negative sampling. To address the inherent complexity of words, word embeddings offer a solution by linking each word to a dense vector of real numbers usually falling within the range of 50 to 300 dimensions. Throughout the training process, the model fine-tunes the embedding parameters, iteratively adapting them to minimize the loss function. This function measures the difference between the predicted outcome and the ground truth, providing guidance for the optimization process. A salient property of vectors discerned through Word2 Vec models pertains to their translational characteristics, furnishing crucial linguistic and relational parallels. Specifically, [PERSON] and his team's investigations illuminated that vector arithmetic can engender qualitative analogies and affinities between words. Word2 Vec retains the property of preserving semantic relationships among words in their vector representations. Semantically congruent words are proximal in the embedding space, rendering them invaluable for a plethora of tasks in natural language processing. There exist two primary variants of the Word2 Vec model: Skip-gram and Continuous Bag of Words (CBOW). Both variants employ neural networks with one or two hidden layers for training embeddings. Continuous Bag of Words (CBOW) Model  
In the CBOW model, context words are utilized to predict a target word. For every target word, a window of context words is chosen, and the neural network is trained to predict the target word based on these context words. The CBOW architecture employs a projection layer designed to forecast the target word when presented with a context window that contains \(c\) words to both the left and right of the target word (Figure 3). At the input layer, each context word is transformed by an embedding matrix \(W\) into a dense vector of dimension \(k\), and these resulting vectors represent context words are then averaged dimension-wise, culminating in a single \(k\)-dimensional vector. The embedding matrix \(W\) is shared across all context words. Notably, due to the aggregation of context words, their sequence does not influence the summation process, rendering this model akin to the bag-of-words approach, albeit with continuous representation. The primary objective of the CBOW model is to maximize the average logarithmic likelihood (2). \[\frac{1}{T}\sum_{t=1-q<t_{j},\ j\
eq 0}^{T}\sum_{\log}(p(w_{t}|w_{t+j})) \tag{2}\]  
Within this context, the variable \(q\) denotes the number of context words positioned on both sides of the target word. In the fundamental CBOW model, the resultant representation of the average vector, derived from the outputs of the projection layer, is passed through a softmax function. This softmax function makes predictions across the entire vocabulary of the corpus, facilitated by the use of backpropagation to optimize the objective of maximizing the logarithmic likelihood (3).