## 3 Proposed Methodology
### Algorithm Flow of TextCNN + RoBERTa and RNN_Bert_Based
The aforementioned hidden state represents the entirety of the sequence as processed by the RNN, incorporating both the contextual information derived from Bert and the sequential dependencies captured by the RNN. ```
1:Data:X_train_text,y_train,X_test_text
2:Result:Trained RNN_Bert_based, Predictions
3:Initialize RNN_Bert_based model:
4:Bert (Setting up parameters for Bert model)
5:Initialize Bertmodel:
6:Bert_model = loads a pre-trained Bert model
7:Integrated_model = Integrated_Model l (Bert_model, RNN_model)
8:Training parameters:
9:Learning rate = 2 \(\times\) 10\({}^{-5}\), Epochs = 5, and Batch_size = 16, 32
10:Loss function and optimizer:
11:criterion = Cross Entropy Loss ()
12:optimizer = Adam
13:Training loop:
14:for epoch in range(epochs): do
15:Model Training
16:For (Trains the model for a specified number of epochs until validation accuracy improves.) 17:\(i\) in tddm (range(0, len(X_train_text), batch_size)): Do
18:Data: An RNN model \(M\) with \(L\) layers, Weight \(W\), pruning rate \(P\), training epoch \(n\)
19:For \(i\)\(\leftarrow\) 0 to \(L\) & epoch \(\leftarrow\) 0 \(to\) \(n\) to do
20:W \(\leftarrow\) Initialization Weight
21:W \(\leftarrow\) normal training (W)
22:% Weight prunning
23:labels_batch = True labels for the current batch of data. 24:labels_batch = y_train[\(i\): \(i\) + batch_size]
25:Optimizer = Adam
26:loss = Cross Entropy Loss ()
27:loss.backward()
28:optimizer.step()
29:end
30:end
31:end
32:Metrics calculation: outputs = binary predictions to integer values 0 (negative) or 1 (positive)The hidden state of the RNN is passed through a fully connected dense layer. This layer serves to map the high-level features extracted by the RNN to the output space. The final output is a sigmoid-activated unit, given that the loss function is the Binary Cross-Entropy Loss. This unit produces a probability for the binary classification task. A learning rate (\(lr\)) of 0.00002 is employed, which is a standard value for fine-tuning transformer-based models such as RoBERTa. This parameter determines the extent to which the model's weights are updated during the training phase. The model will be trained on the entire training dataset, comprising five iterations or epochs. A batch size of 16 or 32 represents the number of samples that the model processes prior to updating its weights. It is typically observed that transformer-based models exhibit enhanced performance when trained with larger batch sizes, for example 32. However, this is contingent upon the availability of sufficient computational resources. The binary cross-entropy loss function is utilized, which represents a conventional methodology for binary classification tasks. It quantifies the discrepancy between the predicted probabilities and the actual binary labels. The provided parameters (learning rate, epoch, batch size, etc.) are set with the objective of optimizing the training process, with the aim of achieving good generalization and performance on the test set. ## 4 Experiment Result and Analysis

To ensure the effective implementation of the RNN_Bert_based algorithm, it is essential to employ the traditional Bert model in conjunction with the RNN. The objective of this integration is to enhance the model's capacity to identify and capture sequential dependencies within textual data. The SST-2 dataset which is widely recognized for its binary sentiment classification tasks will serve as the primary benchmark for evaluating the algorithm's performance. The objective is to enhance the model's accuracy and robustness in sentiment analysis by merging Bert's contextual embeddings with RNN's sequential learning capabilities. The following section provides a detailed explanation and analysis of the evaluation metrics, the proposed method, and a comprehensive overview of existing work on text classification models.