# Exploiting Stacked Autoencoders for Improved Sentiment Analysis
## 5 Experimental Results and Evaluation
### Baseline  
The proposed model is compared to the following baseline methods:  
* SAE: The SAE method comprises of four cascaded autoencoders. The autoencoders combine two neural networks: an encoder and a decoder.
* SVM: The support vector machine is used as a baseline method for this research because it is one of the most widely used ensemble methods.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Hyperparameters** & **Model 1** & **Model 2** & **Model 3** & **Model 4** \\ \hline Training Epoch & 100 & 125 & 125 & 150 \\ Batch Size & 32 & 64 & 64 & 32 \\ L2 Regularization & 0.05 & 0.03 & 0.02 & 0.001 \\ Momentum & 0.93 & 0.97 & 0.96 & 0.95 \\ Learning rate & 0.05 & 0.03 & 0.02 & 0.001 \\ Training Epoch & 100 & 125 & 125 & 150 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Experimental hyperparameters of GA(SAE)-SVM.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Hyperparameters** & **Model 1** & **Model 2** & **Model 3** & **Model 4** \\ \hline Ae1 Hidden Neurons & 20 & 20 & 20 & 20 \\ Ae2 Hidden Neurons & 10 & 10 & 10 & 10 \\ Ae3 Hidden Neurons & 8 & 8 & 8 & 8 \\ Ae4 Hidden Neurons & 4 & 4 & 4 & 4 \\ Loss Function & Cross-entropy & Cross-entropy & Cross-entropy & Cross-entropy \\ Optimizers & Adadelta & Adadelta & Adam & Adam \\ Activation Function for encoder & Swish & Swish & Relu & Relu \\ Activation function for decoder & Swish & Tanh & Relu & Sigmoid \\ Bias & 1 & 1 & 1 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Detailed information regarding the layers and parameters of the SAE.  
\begin{table}
\begin{tabular}{l l l} \hline \hline
**Dataset** & **Total Tweets/Reviews** & **Classes** \\ \hline Sentiment140 (10\%) [79] & 125,000 & Positive, Negative, Neutral \\ Tweets Airline [80] & 25,000 & Positive, Negative \\ Tweets SemEval [81] & 17,750 & Positive, Negative, Neutral \\ IMDb movie reviews (1) [82] & 14,640 & Positive, Negative, Neutral \\ IMDb movie reviews (2) [83] & 10,662 & Positive, Negative \\ Cornell movie reviews 6 [84] & 160,000 & Positive, Negative \\ Book reviews [85] & 2000 & Positive, Negative \\ Music reviews [86] & 2000 & Positive, Negative \\ \hline \hline \end{tabular}
\end{table}
Table 5: Dataset description.  
SAE-SVM: A hybrid model, SAE-SVM, was also tested for all datasets. The features of the acquired dataset are extracted using the SAE, and then the features are fed into the SVM in this hybrid model.
* The performance of the deep learning algorithm can be improved by hyperparameter optimization. For this purpose, three different hyperparameter tuning techniques are tested in this research. Values of the learning rate, momentum, epoch, and L2 regularization of SAEs are optimized. Tested parameter optimization techniques include:
* RS(SAE)-SVM: In this model, random search is used to tune the hyperparameters of the SAE.
* GS(SAE)-SVM: In this model, grid search is used to tune hyperparameters of the SAE.
* GA(SAE)-SVM: Using this model, hyperparameters of the SAE are optimized by GA. SAE hyperparameters such as learning rate, L2 regularization, momentum, and epoch are tuned, and the best values of these parameters are selected. The final classification is performed by SVM.