# Towards AI-Generated Essay Classification Using Numerical Text Representation
## 3 Natural Language Processing
### Numerical Text Representations
#### 3.1.2 Embedding Techniques  
An alternative to high-dimensional data generation methods is word embedding, also known as vector-based distributional word representation [19]. This technique allows for the creation of dense vectors, that is, low-dimensional vectors. These representations are created by algorithms that assign each word in the corpus a set of numerical values. The advantage of word embedding techniques is that they reflect semantic relationships between words, enabling models to process text more efficiently [8]. Words with similar meanings are represented by vectors that are close to each other in space, which not only helps the models detect patterns in the text but also alleviates the problem of sparse matrices, thus increasing the efficiency of the model [19]. A popular algorithm for creating word vector representations is Word2 vec, based on neural networks, which was proposed by [PERSON] et al. [19]. It was developed in two architectures aimed at minimizing computational complexity. The authors concluded that such complexity mainly arises from the non-linear hidden layer in neural networks. Therefore, they decided to explore simpler models and their effectiveness [19]. The first architecture is the Continuous Bag of Words (CBoW). Its name stems from the fact that the order of words does not affect the results, similar to the classical Bag of Words. This is mainly because a single weight matrix is used for all words during training instead of creating separate weights for each word. The term 'continuous' refers to the continuous and distributed representation of the surrounding context [19]. In the standard Bag of Words model, word representation is discrete and sparse. Each word has a separate index in the vocabulary, the vector length equals the vocabulary length, and most elements are zeros. Continuity and dispersion in continuous bags of words imply that words are represented by dense vectors of a specified length, with each element potentially containing a real floating-point number, indicating the context of the given word. Additionally, in the CBoW method, both preceding and succeeding words of the target word are used. This is achieved by using a logistic classifier that takes four preceding and four succeeding words as input and attempts to correctly classify the target word [19]. The second architecture of the Word2 vec algorithm is Skip-gram. It works in the opposite way to the CBoW method, where the target word is predicted based on its surrounding context. In this case, contextual words are predicted, that is, words surrounding the target word. In other words, Skip-gram analyses a specific word and tries to predict which words might surround it. Given a sequence of training words \(w_{1},w_{2},\ldots,w_{T}\), the goal of the Skip-gram model is to maximize the average log probability [19]:  
\[\frac{1}{T}\sum_{t=1}^{T}\sum_{-c\leq j\leq c,j\
eq 0}\log p(w_{t+j}|w_{t}), \tag{4}\]  
where  
* \(c\)--the size of the training context (number of words surrounding the target word). A higher value of \(c\) means a larger training context, resulting in a more complex model. A lower value of \(c\) results in faster training, but the model may be less precise in predicting the context. FastText is an extension of [PERSON]'s algorithm. This approach is based on the Skip-gram model, whose original concept was word-based. In the case of fastText,an innovative approach using character n-grams was introduced, where each n-gram is treated as a separate \"word-like\" fragment [20]. This allows the model to flexibly recognize words, including rare ones, those with typos, or even those not present in the training set, by analyzing their n-gram fragments [21]. It can be trained on custom corpora, providing specialized embeddings tailored to specific data. However, this requires a sufficiently large dataset to ensure robustness. Alternatively, pre-trained models on large corpora offer general embeddings suitable for various tasks, but may lack domain-specific nuances.