# Examining Attention Mechanisms in Deep Learning Models for Sentiment Analysis
## 5 Experimental Study
### Results  
To evaluate the performance of the proposed models on a sentiment classification task, we compare every attention-based approach to equivalent baseline models that do not utilize attention mechanisms. Regarding the preprocessing of the data, the representation (embeddings) of the input documents is described in Section 3. The word2 vec embedding vectors are pre-trained on an unlabeled corpus whose size is about 100 billion words. We trained all models with a batch size of 256 samples, a hidden neuron count of 256 and an initial learning rate of 0.001 for the Adam optimizer.  
Overall, to implement the proposed models, we utilized the deep learning framework named Tensorflow [51]. We trained the models on an Nvidia (Santa Clara, CA, USA) RTXTM 2080 Ti since Tensorflow supports GPU computations, which heavily speed up the training process.  
To evaluate our models, we utilized k-fold cross-validation, which splits the data into k subsets of equal size. Then the entire process is executed k times in a row by selecting 1 subset as the test set and the remaining as the training set. This leads to k non-overlapping test data sets and k different accuracies for the model. Cross-validation ensures fairness in the testing process and reduces the probability of randomly selecting a biased/bad test set. We decided on a 10-fold cross-validation split, which is a rather popular configuration.  
The models are evaluated on the commonly used performance metrics of accuracy and F1-score. They can be calculated as follows: In our experiment, if we define TP as the number of \"true-positive\" samples predicted as positive, FN as the number of true-positive samples predicted as negative, FP as the number of true-negative samples predicted as positive, and TN as the number of true-negative samples predicted as negative, then the metrics can be expressed as follows:  
After various experiments, we come to certain conclusions regarding the best hyperparameters and settings. One of them is that using a pre-trained 300-dimensional word2 vec and further refining it through additional training achieves the highest performance. Two experiments were designed and conducted, and each one examined the performance of different types of models.  
The first experiment assessed the performance of global and self-attention mechanisms using various models and parameters. Specifically, attention-based LSTMs with global attention (Global-Att-LSTM) and attention-based LSTMs with self-attention (Self-Att-LSTM) are implemented and assessed; their performance is compared to baseline models like CNNs, LSTMs as well as more sophisticated models like multiplicative and bidirectional LSTM models. In Figure 5, we illustrate the performance results of all the models examined in the context of the first experiment, thus illustrating a comparison between the different neural network models that were implemented and examined.  
Figure 5: Performance of the models on the SUBJ, MR and IMDB datasets.  
The second experiment assessed the performance of hierarchical-attention mechanisms compared to baseline neural network models. Specifically, attention-based GRUs with hierarchical attention (Hierarchical-Att-GRU) are implemented and their performance is evaluated against baseline GRU models that do not feature an attention mechanism. In Figure 6, the performance results of all the aforementioned models are illustrated. Finally, an attention-based GRU that also incorporates dropout layers is formulated (Hierarchical-Att-GRU+Dropout).  
Both experiments showcase quite interesting findings. The experimental results on the three datasets indicate that the attention-based models achieve very promising performance as they outperform the baseline neural networks that do not make use of an attention mechanism, to a great degree in all experiments. As a result, the examined self-attention, global-attention and hierarchical-attention models lead to a performance increase of up to 3.5%.