## 3 Methodology
### The Proposed Ensemble Model  
Instead of employing a single model, the ensemble approaches improve model accuracy by mixing numerous models. The combined models increase the outcome's accuracy considerably [60]. A stacking ensemble is an ensemble method in which a new model learns how to blend predictions from numerous existing models as best as possible. It combines predictions from a number of different trained models [61]. CNN models and hybrid models of CNN and other DL models, such as long short-term memory (LSTM), provided significant improvements in performance in sentiment analysis [4; 5]. CNN uses different deep layers to extract more deep and important features from text data. LSTM has a memory state that is effective at memorizing important information in the text and understanding the meaning of the whole sentence. Therefore, we proposed an optimized heterogeneous ensemble stacking model based on the best combination of CNN, hybrid CNN-LSTM, and CNN-GRU for Arabic sentiment analysis with SVM as a meta-learner. In our work, our model is developed in many steps, as shown in Figure 2.  
Figure 4: Representation for GRU [59].  
Figure 3: LSTM representation.  
* The pre-trained models of DeepCNN, CNN-LSTM, and CNN-GRU that are described in Section 3.5 are loaded, and all layers of the model are frozen without the output layers.
* The output prediction of the training set for each pre-trained model are combined in the training stacking. Then, the stacking is used to train and optimize the meta-learner ([PERSON] in our case). SVM as a meta-learner is optimized using grid search.
* The output predictions of the testing set for each pre-trained model are combined in the testing stacking. Then, the testing stacking is used to evaluate the meta-learner ([PERSON]) using accuracy, precision, recall, f1-score and ROC.  
Three datasets were divided into two parts: 80% training and 20% testing. Models were optimized using the training set. To extract features and generate feature matrices for DL models, CBOW and SkipGram word embedding were utilized. The final values of each parameter of DeepCNN, CNN-LSTM and CNN-GRU were applied with CBOW and SkipGram word embedding for each dataset, as shown in Tables 2 and 3, respectively.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multirow{2}{*}{**Parameters**} & \multicolumn{2}{c}{**Values for Main-AHS**} & \multicolumn{2}{c}{**Values for Sub-AHS**} & \multirow{2}{*}{**Values for ASTD Dataset**} \\ \cline{3-4}  & & **Dataset** & & **Dataset** \\ \hline \multirow{5}{*}{CNN} & Num\_filters & [256; 128; 128] & [128; 256; 256] & [256; 128; 128] \\ \cline{2-4}  & Kernel\_size & [4; 5; 2] & [4; 5; 4] & [3; 5; 4] \\ \cline{2-4}  & Pool\_Size & [4; 5; 3] & [2; 4] & [2; 5] \\ \cline{2-4}  & Dense\_Unit & 300 & 150 & 500 \\ \cline{2-4}  & learning\_rate & 0.0012 & 0.00152 & 0.0007 \\ \hline \multirow{5}{*}{CNN-LSTM} & Num\_filters & 128 & 256 & 256 \\ \cline{2-4}  & Kernel\_size & 4 & 4 & 3 \\ \cline{2-4}  & Pool\_Size & 5 & 2 & 2 \\ \cline{2-4}  & LSTM\_Unit & 400 & 500 & 550 \\ \cline{2-4}  & Dense\_Unit & 800 & 700 & 300 \\ \cline{2-4}  & learning\_rate & 0.0012 & 0.0025 & 0.0046 \\ \hline \multirow{5}{*}{CNN-GRU} & Num\_filters & 256 & 64 & 128 \\ \cline{2-4}  & Kernel\_size & 5 & 3 & 5 \\ \cline{2-4}  & Pool\_Size & 2 & 3 & 2 \\ \cline{2-4}  & GRU\_Unit & 950 & 150 & 250 \\ \cline{2-4}  & Dense\_Unit & 150 & 900 & 300 \\ \cline{2-4}  & learning\_rate & 0.0012 & 0.0059 & 0.0021 \\ \hline \end{tabular}
\end{table}
Table 2: Values of parameters after applying DeepCNN, CNN-LSTM and CNN-GRU with CBOW.