# RoBERTa-GRU: A Hybrid Deep Learning Model for Enhanced Sentiment Analysis
## 5 Hyperparameter Tuning  
The hyperparameters of the proposed RoBERTa-GRU model were carefully selected to optimize its performance. To determine the optimal hyperparameters, grid search was conducted on the smallest dataset, i.e., the Twitter US Airline dataset after data augmentation. The grid search involved evaluating the model performance for different  
Figure 3: The sample distributions of the datasets.  
combinations of hyperparameters and selecting the combination that results in the highest accuracy. The hyperparameters and their tested values are summarized in Table 2.  
Table 3 presents the results of the hyperparameter tuning for the number of GRU in the RoBERTa-GRU model. The number of GRUs plays a crucial role in determining the capacity of the model to capture the long-range dependencies of the RoBERTa embedding. By increasing the number of GRUs, the model can learn more complex relationships between the input and output. However, using a large number of GRUs might also lead to overfitting, especially when the training data is limited. The results in Table 3 suggest that the optimal number of GRUs is 256. At this number of units, the RoBERTa-GRU model achieved the highest accuracy in the sentiment classification task. This indicates that a hidden layer with 256 GRUs is capable of capturing the complex relationships between the input and output while avoiding overfitting.  
The grid search for different optimizers was an important step in determining the optimal hyperparameters for the RoBERTa-GRU model. The results in Table 4 indicate that the Nadam optimizer is the most effective optimizer for the RoBERTa-GRU model as it not only yields the highest accuracy but also allows for the quickest convergence during the training process. By choosing the Nadam optimizer, the model can effectively learn the underlying relationships in the data, leading to better performance in sentiment analysis tasks.  
The results of the grid search for the learning rate are shown in Table 5. The learning rate, as a hyperparameter, has a crucial role in determining the convergence speed and the final performance of the model. The experiment results indicate that the optimal value for the learning rate was found to be 0.00001. This value balances the trade-off between underfitting and overfitting. If the learning rate is too high, the model may underfit, meaning that it will not be able to capture the underlying patterns in the data and may result in poor performance. On the other hand, if the learning rate is too low, the model may overfit, which means that it will learn the noise in the data and may perform poorly on unseen data. The learning rate of 0.00001 provides a good balance between these two extremes and results in the best performance for the RoBERTa-GRU model.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Optimizer** & **Accuracy (\%)** & **Execution Time (s)** \\ \hline Adam & 90.72 & 1821.97 \\ Nadam & 91.52 & 1584.04 \\ SGD & 84.66 & 4520.34 \\ \hline \hline \end{tabular}
\end{table}
Table 4: The experimental results of different optimizers (GRU = 256, learning rate = 0.00001).  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Hyperparameter** & **Tested Values** & **Optimal Value** \\ \hline GRU & 64, 128, 256, 512 & 256 \\ Optimizer & Adam, Nadam, SGD & Nadam \\ Learning rate & 0.00001, 0.0001, 0.001 & 0.00001 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The hyperparameter tuning of RoBERTa-GRU model.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**GRU** & **Accuracy (\%)** \\ \hline
64 & 91.07 \\
128 & 91.41 \\
256 & 91.52 \\
512 & 90.83 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The experimental results of different GRUs (optimizer = Nadam, learning rate = 0.00001).