## 3 Proposed Methodology
### Algorithm Flow of TextCNN + RoBERTa and RNN_Bert_Based
This is achieved by factorizing the embedding parameters and applying cross-layer parameter sharing. ALBERT also introduces two novel training strategies: Sentence Order Prediction and Cross-Example Loss, which are intended to further enhance its performance. ALBERT reduces the number of parameters compared to Bert by sharing parameters across layers. In Bert, each layer has its own set of parameter weights, but ALBERT reuses the same parameters for each layer. This significantly reduces the total number of parameters, making the model smaller and faster. It splits the large vocabulary embedding matrix into two smaller matrices, which also helps in reducing the model size. This factorization decouples the size of the hidden layers from the size of the vocabulary embeddings, allowing for a smaller overall footprint. ### Experimental Results

The experiment is conducted under a variety of conditions, including the Bert-based method with different learning rates and batch sizes, the KNN_Bert_based method, the proposed RNN_Bert_based method, and other existing text classification methods for comparison with the proposed model. #### 4.3.1 Bert-Based Method with Different Learning Rates and Batch Sizes

The following results present an evaluation of the Bert model's accuracy for text classification, with varying learning rates and total batch sizes of 16 for each experiment. The results demonstrate that with a total batch size of 16 and learning rates of \(2\times 10^{-5}\), \(3\ \times\ 10^{-5}\), and \(5\times 10^{-5}\), accuracies of 92.75%, 92.20%, and 92.31% can be achieved, respectively. Figure 6(a) presents a comparison of accuracy across different learning rates with a batch size of 16. The next results present an evaluation of the Bert model's accuracy for text classification, with varying learning rates and total batch sizes of 32 for each experiment. The results demonstrate that with a total batch size of 32 and learning rates of \(2\times 10^{-5}\), \(3\ \times\ 10^{-5}\), and \(5\times 10^{-5}\), accuracies of 90.48%, 90.25%, and 89.79% can be achieved, respectively. Figure 6(b) presents a comparison of accuracy across different learning rates with a batch size of 32. The outcome of the experiment, conducted with a batch size of 32, indicates that an increase in the learning rate is associated with a decline in the performance of the model. The results indicate that the learning rate of \(2\times 10^{-5}\) with a batch size of 16 represents the optimal Bert hyperparameters. This configuration will serve as the primary approach in the integration with RNN, with the objective of achieving enhanced performance in model accuracy for classification tasks. #### 4.3.2 KNN_Bert_Based Method

The integration of KNN with the Bert model leverages the strengths of both techniques, enhancing semantic understanding, improving performance, providing robustness, and maintaining simplicity and efficiency. This integration is particularly powerful for tasks that require high-quality text representations and flexible, interpretable classification or retrieval methods. The experiment involved training a Bert model and a K-Nearest Neighbors (KNN) model with the following key parameters for the Bert model part: max_seq_len = 50, epochs = 5, batch_size = 16, learning rate \((lr)=2\times 10^{-5}\), and patience = 1. For the KNN model part, the number of neighbors to use for the K-Nearest Neighbors algorithm is 5. This is a critical hyperparameter for KNN that determines how many nearest neighbors are considered when making a prediction. Here, n_neighbors = 5 means that the KNN model will look at the 5 nearest points in the training data when predicting the label for a new data point. The following results present an evaluation of the KNN_Bert_based model's accuracy for text classification, with a learning rate \((lr)\) of \(2\times 10^{-5}\) and a total batch size of 16 in five episodes. This is a hybrid model that combines KNN with Bert, a powerful pre-trained transformer model for natural language processing.