# Towards AI-Generated Essay Classification Using Numerical Text Representation
## 4 Research Methodology
### Text Representations  
Further analysis requires transforming textual data into appropriate numerical structures. After lemmatization, the data consist of lists of tokens for individual essays. To proceed, these token lists are converted back into sequences of text, allowing for the restoration of sentence structure.  
The first technique used for numerical text representation was the Bag of Words (BoW). The texts were converted into a matrix where each column represented a unique word in the corpus, and each row represented an essay, showing the frequency of each word. This simple and commonly used technique in natural language processing tasks was applied only to training data at this stage. Transformation of the test data occurred directly before evaluating the classifiers due to the method's speed.  
Next, the TF-IDF (term frequency-inverse document frequency) technique was applied. Similar to BoW, it represents texts as vectors, but it accounts for both the frequency of words and their significance across the entire corpus. In addition to the basic TF-IDF matrix, a modified version with selected parameters was configured to improve text representation:  
* L2 normalization: adjusts vectors so their Euclidean length is 1, making vector components proportionally scalable, which is especially useful for documents of varying lengths;
* IDF smoothing: adds 1 to the frequency of each term, preventing division-by-zero errors when calculating IDF values for terms not present in any training document;
* Sublinear term frequency transformation: uses a logarithmic scale: \[TF=1+\log(TF)\] (7) to reduce the influence of terms that appear very frequently, making them less informative.  
Like BoW, this transformation was applied only to the training data. The same vectorizers will be used to transform test data during classification.  
The fastText model was used to create the first numerical representations in the form of word embeddings. Two models-Skip-gram and Continuous Bag of Words (CBoW)--were trained using the lemmatized training data with the gensim library. Training parameters included:* vector dimensionality (vector_size) of 300, based on common values in pre-trained models;
* context window size (window) of 5, considering the context of five words before and after a given word;
* minimum word occurrence (min_count) of 3 to exclude rare words;
* training algorithm (sg): 0 for CBoW and 1 for Skip-gram;
* range of n-gram sizes (min_n and max_n) for constructing word vectors, improving model handling of rare words or spelling errors;
* number of buckets (bucket) for hashing n-grams during training, affecting model accuracy and training time.  
Additionally, a pre-trained fastText model was used, which was trained on Wikipedia and Common Crawl data. The English model (cc.en.300) uses CBoW with position weighting, considering both context words and their positions. This resulted in three forms of word embeddings, transformed first on the training set and similarly on the test set.  
The final stage involved creating sentence embeddings using transformer-based neural network models. Models were accessed via Hugging Face using API interfaces like transformers and sentence_transformers. Selected models were MiniLM (all-MiniLM-L6-v2) and distilBERT (distilbert-base-nli-stsb-mean-tokens), both modified versions of Sentence-BERT (sBERT), designed for semantic sentence embeddings.  
MiniLM focuses on minimizing model size while maintaining efficient embeddings. It employs knowledge distillation from a larger model, enabling faster data processing with lower computational requirements [34]. DistilBERT, also a result of BERT model distillation, retains similar-quality embeddings with faster processing and lower resource demands. These models were trained on tasks solved by the larger model, effectively transferring knowledge to a smaller, more efficient model [35].