## 3 Proposed Methodology
### Algorithm Flow of TextCNN + RoBERTa and RNN_Bert_Based
```
1:Data: X_train_text, y_train, X_test_text
2:Result: Trained Hybrid Model, Predictions
3:Initialize TextCNN model:
4: TextCNN (The size of word embeddings, 2D convolutional layers, classification output)
5:Initialize RoBERTa model:
6:roberta_model = loads a pre-trained RoBERTa model
7: hybrid_model = creates a hybrid model by combining a TextCNN mode
8:Training parameters:
9: Learning rate = \(2\times 10^{-5}\), Epochs = 5, and Batch_size = 16, 32
10:Loss function and optimizer:
11: criterion = Binary Cross-Entropy Loss ()
12: optimizer = [PERSON]
13:Training loop:
14:for epoch in range(epochs): do
15:hybrid_model.train0
16:For
17:\(i\) in tqdm(range(0, len(X_train_text), batch_size)): Do
18: Text input batch = X_train_text, (\(i\): \(i\)+batch_size)
19: Inputs for the RoBERTa batch = additional input features for RoBERTa
20: labels_batch = True labels for the current batch of data. 21: clears the old gradients from the previous iteration()
22: Runs the current batch through the hybrid model, producing predictions. 23: loss = computes the loss via Binary Cross-Entropy Loss ()
24: backpropagation ()
25: optimizer.step()
26: end
27:end
28:Evaluate the hybrid model:
29: hybrid model:
30: Disables gradient calculation to save memory and computations ():
31: Extracts the input_ids from the test data
32: inputs other required parameters by the RoBERTa model (like attention masks or token type IDs)
33: Converts the binary predictions to integer values 0 (negative) or 1 (positive)
34:Metrics calculation
```  
**Algorithm 1** TextCNN + RoBERTa  
In Algorithm 2, the objective is to leverage the strengths of both the Bert model and the RNN by integrating them into a unified framework, namely RNN_Bert_based. This approach capitalizes on the deep contextual embeddings of the Bert model and the recurrent nature of the RNN, combining the two into a single, more powerful model. This configuration capitalizes on the respective strengths of both models. Bert is employed for the purpose of capturing rich contextual embeddings, while RNN is utilized for the handling of sequential dependencies over these embeddings. The Bert model is a transformer-based model that generates robust contextualized embeddings for text by processing the entire sequence of tokens simultaneously and capturing bidirectional context. RNN is a type of artificial neural network designed to process sequential data. It maintains a hidden state that captures information about previous inputs, making it ideal for modeling dependencies over time or sequences. The dataset is divided into the following categories: The dataset, designated as \"X_train_text\", is comprised of the input training texts and is represented by the variable y_train (the aforementioned training data is accompanied by the corresponding labels); and the X_test_text category is comprised of the input test texts, which the model will subsequently predict following the completion of the training phase. [PERSON] tokenizes the input text, producing embeddings for each token in the sequence. The resulting embeddings are context-rich, as [PERSON] considers the entire sequence of words when generating each token's embedding. The output from Bert can be either the hidden states of all tokens or, alternatively, the hidden state corresponding to the [CLS] token, which often serves as a summary of the entire sequence. The RNN layer has a function for passing the embeddings from Bert through RNN. This could be a basic RNN, such as a long short-term memory (LSTM) network. The basic recurrent network has the potential for vanishing gradients, which is a mechanism for retaining information over long sequences. This mitigates issues in the proposed model. The RNN processes the sequence of Bert embeddings, updating its hidden state with each token, and captures sequential dependencies that [PERSON] might not explicitly model, given that Bert is bidirectional but not inherently sequential. The final hidden state of the RNN (or a combination of hidden states, such as the output at each time step) is then extracted.