# RoBERTa-GRU: A Hybrid Deep Learning Model for Enhanced Sentiment Analysis
###### Abstract  
This paper proposes a novel hybrid model for sentiment analysis. The model leverages the strengths of both the Transformer model, represented by the Robustly Optimized BERT Pre-training Approach (RoBERTa), and the Recurrent Neural Network, represented by Gated Recurrent Units (GRU). The RoBERTa model provides the capability to project the texts into a discriminative embedding space through its attention mechanism, while the GRU model captures the long-range dependencies of the embedding and addresses the vanishing gradients problem. To overcome the challenge of imbalanced datasets in sentiment analysis, this paper also proposes the use of data augmentation with word embeddings by over-sampling the minority classes. This enhances the representation capacity of the model, making it more robust and accurate in handling the sentiment classification task. The proposed RoBERTa-GRU model was evaluated on three widely used sentiment analysis datasets: IMDb, Sentiment140, and Twitter US Airline Sentiment. The results show that the model achieved an accuracy of 94.63% on IMDb, 89.59% on Sentiment140, and 91.52% on Twitter US Airline Sentiment. These results demonstrate the effectiveness of the proposed RoBERTa-GRU hybrid model in sentiment analysis.  
sentiment analysis; deep learning; Transformer; RoBERTa; GRU +
Footnote †: journal: _Applied_  
dependencies more effectively. In particular, the LSTM model includes a memory cell that can maintain information over long time periods, while the GRU model has a simpler gating mechanism that can be more computationally efficient.  
Recently, the Transformer model, which leverages the attention mechanism, has shown excellent performance in natural language processing. The attention mechanism [2] allows the model to selectively weigh different parts of the input sequence to create informative embeddings. Specifically, the attention mechanism computes a weighted sum of the input embeddings, where the weights are determined by a learned compatibility function between the query and the key embeddings. This enables the model to effectively capture long-range dependencies in the input sequence and create more informative representations of the input. In light of this, a hybrid model that combines the strengths of the Transformer and GRU models is proposed in this paper. The hybrid model uses the Robustly Optimized BERT Pretraining Approach (RoBERTa) Transformer model as the encoder, tokenizing the input sequence and encoding it into a discriminative word embedding. The word embedding is then passed into the GRU, capturing the long-range dependencies of the word embedding. A dense layer is incorporated to learn the relationships between the GRU output and the class labels, followed by a classification layer with a softmax function that estimates the probability distributions of the classes. Our main contributions are:  
* A RoBERTa-GRU hybrid model that takes advantage of the best features of both the Transformer and GRU models. The RoBERTa model is applied to generate representative word embeddings that capture the unique characteristics of the text. Dynamic attention masking is used in the RoBERTa model to enable the model to learn from a more diverse set of input sequences.
* A GRU model that is utilized to effectively capture long-range dependencies, which are critical for text analytics. The GRU model incorporates update and reset gates to retain long-term memory and mitigate the vanishing gradient problem.
* To tackle the challenge of imbalanced datasets, the paper employs data augmentation with word embeddings. This technique synthesizes additional samples for the minority classes, leading to improved model performance and generalization ability.