# RoBERTa-GRU: A Hybrid Deep Learning Model for Enhanced Sentiment Analysis
## 3 Sentiment Analysis with RoBERTa-GRU
### RoBERTa-GRU
#### 3.3.1 RoBERTa  
The proposed model employs a Robustly Optimized BERT Pretraining Approach (RoBERTa) as its first layer. The RoBERTa is an improved version of the pretrained Bidirectional Encoder Representations from Transformers (BERT) model. Both the BERT and RoBERTa are based on the Transformer architecture.  
The Transformer model was designed for sequence-to-sequence tasks where long-range dependencies are important. It uses self-attention mechanisms instead of recurrence or convolution to identify dependencies between inputs and outputs. The self-attention mechanism assigns higher weights to relevant inputs, thus reducing the length of the sequence.  
The Transformer consists of two main components: the encoder and the decoder. The encoder is made up of a self-attention layer and a feed-forward network, and the decoder consists of a self-attention layer, an encoder-decoder attention layer, and a feed-forward network. The encoder is responsible for reading the input text, while the decoder focuses on making predictions. In the proposed model, only the encoder part of the RoBERTa is used as the text encoder.  
BERT was created to overcome the limitations of unidirectional methods that limit context learning. It was originally used for two tasks: Masked Language Modeling and Next Sentence Prediction. In Masked Language Modeling, the BERT uses masked token features to predict missing words in context, while in Next Sentence Prediction, the BERT uses semantic meaning from different text segments or documents to predict the next sentence.  
The RoBERTa has some advantages over BERT. For instance, RoBERTa uses byte-level Byte Pair Encoding for tokenization, resulting in a smaller vocabulary and less computational resources compared with BERT's character-level Byte Pair Encoding. Additionally, BERT uses static masking where the masking is performed once during data preprocessing, while RoBERTa uses dynamic masking, where the input sequence is duplicated and different attention masks are applied, enabling the RoBERTa model to learn from different input sequences. The RoBERTa is also trained on much larger datasets using a larger batch size and longer sequence for a longer time. It was trained on four datasets: BookCorpus + English Wikipedia (16 GB), CC-News (76 GB), OpenWebText (38 GB), and Stories (31 GB).  
Figure 2: The architecture of the proposed RoBERTa-GRU model.  
To prepare the input data for the RoBERTa model, the pretrained RoBERTa tokenizer is employed in this study. The tokenizer breaks down the raw text into a sequence of subword tokens, leveraging the large corpus of text on which the RoBERTa was trained to learn a robust vocabulary. This process helps preserve the semantic meaning of the text while minimizing the impact of out-of-vocabulary (OOV) words. Following tokenization, each token is assigned a unique input ID, representing its index in the RoBERTa vocabulary. An attention mask is also assigned to each token, indicating its relevance in the context of the input sequence. This mechanism enables the model to focus on the most salient tokens while disregarding those that are less important, thus improving the model's performance on downstream tasks. The resulting input IDs and attention masks are then fed into the RoBERTa model comprising 12 layers, each with 768 hidden states. The RoBERTa model processes the input sequence in a self-attentive manner, capturing contextual information at multiple levels of abstraction. To further refine the output of the RoBERTa model, a GRU is employed as a feature extractor. By incorporating the GRU, the model can leverage both the contextual information captured by the RoBERTa and the long-range dependencies between tokens to generate more accurate predictions.