## 3 Developing Models: NLP and Word Embeddings
### Training the Models  
The heart of DL involves training models on a subset of data, allowing them to discover data features and apply this knowledge to new, unseen data. In our specific scenario, the models have the responsibility of categorizing text into either the \"spam: 0\" or \"quality: 1\" categories. As depicted in Figure 4, passing embedding layer representations into the architectures provides the foundation for acquiring this knowledge. Specifically, recurrent networks like LSTMs effectively model sequential dependencies in language. The input, output, and forget gates of LSTM cells allow capturing long-range contexts and nuanced relationships within textual data. On the other hand, contextualized techniques like BERT and ELMo generate dynamic token embeddings based on the surrounding context. Pretraining them on our diversified corpora (Twitter(X)/YouTube) imparts semantic encoding capabilities. Our approach loads these pretrained models via TensorFlow-Hub/Text and generates contextual word vectors to feed into the classification layers after fine-tuning, enhancing performance over static embeddings. We employ binary classification layers tailored for spam detection--logistic regression for ELISA and feedforward network for BERT. The training process involves iterating over text sequences in the Twitter/YouTube datasets and updating model parameters to minimize the loss function chosen. Tracking validation accuracy and loss metrics over epochs then allows us to quantitatively evaluate model effectiveness on unseen data. Embedding layers, such as those derived from ELMo and BERT, are pivotal in text classification models by enabling the extraction of dynamic and context-aware word representations. These embeddings are instrumental in analyzing textual characteristics, thus informing the model's ability to categorize text. The model weights are optimized through a process aimed at minimizing the loss function, represented as\[\text{Loss}_{\text{model}}=-\frac{1}{N}\sum_{i=1}^{N}(y_{i}\log(\hat{y}_{i})+(1-y_{ i})\log(1-\hat{y}_{i})) \tag{1}\]  
where \(N\) is the number of samples, \(y_{i}\) is the actual label, and \(\hat{y}_{i}\) is the predicted label by the model. This equation exemplifies the cross-entropy loss, a common choice for binary classification tasks. Following our training phase, the effectiveness of the models is assessed on a test set, focusing on metrics such as accuracy and loss, which provide quantitative insights into the models' performance. ELMo and BERT contribute significantly to this approach by offering rich, pretrained word embeddings that enhance the model's understanding and classification of text. These embeddings are sourced from TensorFlow-Hub and TensorFlow-Text, facilitating the generation of deep contextualized word representations that improve the performance of downstream NLP tasks [55]. For ELMo, we embed a generation process for each text in the Twitter and YouTube datasets, and it is formalized as follows:  
\[\text{ELMo embeddings}=\text{Function (Text)} \tag{2}\]  
These embeddings serve as input features for the classification model, which undergoes training on the derived ELMo embeddings. The training regimen entails multiple iterations over the dataset (epochs), with the objective of refining model weights to reduce the loss function further. The model's post-training evaluation on the test set involves calculating both the loss and accuracy, furnishing a quantitative measure of its efficacy. RNNs, and by extension LSTMs, offer a solution to the limitations of traditional neural networks in processing sequential data. The construction of RNN/LSTM models, facilitated by frameworks such as Keras, typically involves layers conducive to binary classification tasks. The architecture of an NN/LSTM distinguishes itself by allowing variable connections between neurons, as opposed to the static, feedforward nature of traditional neural networks.