## 4 Evaluation
### Evaluation of Deep Learning Models  
Although the simple machine classifiers performed well as per our experimental settings, they still can be improved. If an AI algorithm returned an incorrect prediction \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Feature Set** & **LR** & **MLP** & **SVC** & **RF** & **NB** & **AB** & **GB** \\ \hline Baseline & 0.72 & 0.71 & 0.68 & 0.69 & 0.69 & 0.68 & 0.70 \\ +transitivity & 0.72 & 0.71 & 0.68 & 0.70 & 0.69 & 0.68 & 0.70 \\ +modality & 0.72 & 0.69 & 0.69 & 0.70 & 0.69 & 0.69 & 0.70 \\ +polarity & 0.72 & 0.70 & 0.69 & 0.72 & 0.69 & 0.69 & 0.70 \\ +syntax & 0.71 & 0.71 & 0.68 & 0.70 & 0.69 & 0.69 & 0.68 \\ +doc2 vec & 0.73 & 0.72 & 0.69 & 0.75 & 0.69 & 0.75 & 0.67 \\ +word2 vec & 0.75 & 0.71 & 0.72 & 0.79 & 0.69 & 0.79 & 0.75 \\ +word2 vecSmoothed & 0.75 & 0.70 & 0.72 & 0.79 & 0.69 & 0.81 & 0.75 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Classification of solutions from non-solutions using ML classifiers by 10-fold cross validation. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Feature Set** & **LR** & **MLP** & **SVC** & **RF** & **NB** & **AB** & **GB** \\ \hline Baseline & 0.71 & 0.70 & 0.65 & 0.65 & 0.67 & 0.67 & 0.66 \\ +transitivity & 0.70 & 0.70 & 0.68 & 0.69 & 0.68 & 0.66 & 0.68 \\ +modality & 0.70 & 0.70 & 0.68 & 0.67 & 0.67 & 0.66 & 0.68 \\ +polarity & 0.71 & 0.70 & 0.68 & 0.69 & 0.68 & 0.67 & 0.69 \\ +syntax & 0.74 & 0.73 & 0.74 & 0.76 & 0.67 & 0.71 & 0.74 \\ +doc2 vec & 0.76 & 0.75 & 0.75 & 0.75 & 0.68 & 0.76 & 0.64 \\ +word2 vec & 0.76 & 0.73 & 0.71 & 0.79 & 0.68 & 0.80 & 0.73 \\ +word2 vecSmoothed & 0.77 & 0.72 & 0.70 & 0.78 & 0.66 & 0.80 & 0.71 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Classification of solutions from non-solutions using ML classifiers by training data iteration. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Feature Set** & **LR** & **MLP** & **SVC** & **RF** & **NB** & **AB** & **GB** \\ \hline Baseline & 0.71 & 0.71 & 0.68 & 0.71 & 0.66 & 0.70 & 0.71 \\ +transitivity & 0.70 & 0.71 & 0.67 & 0.70 & 0.66 & 0.70 & 0.70 \\ +modality & 0.70 & 0.71 & 0.67 & 0.69 & 0.66 & 0.70 & 0.70 \\ +modality & 0.70 & 0.71 & 0.67 & 0.69 & 0.66 & 0.69 & 0.67 \\ +polarity & 0.70 & 0.71 & 0.67 & 0.69 & 0.66 & 0.70 & 0.70 \\ +modality & 0.70 & 0.71 & 0.67 & 0.69 & 0.66 & 0.69 & 0.67 \\ +polarity & 0.70 & 0.71 & 0.67 & 0.69 & 0.66 & 0.70 & 0.67 \\ +modality & 0.71 & 0.71 & 0.68 & 0.70 & 0.69 & 0.69 & 0.68 \\ +doc2 vec & 0.73 & 0.72 & 0.69 & 0.75 & 0.69 & 0.75 & 0.67 \\ +word2 vec & 0.75 & 0.71 & 0.72 & 0.79 & 0.69 & 0.79 & 0.75 \\ +word2 vecSmoothed & 0.75 & 0.70 & 0.72 & 0.79 & 0.69 & 0.81 & 0.75 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Classification of problems from non-problems using ML classifiers by 10-fold cross validation. for a few scenarios, we had to interfere. A deep learning model can efficiently improve our results because a deep learning model may use neural networks to figure out on its own whether a prediction is correct or not [75; 76]. Hence, we compared the classifications between the ML classifiers and the deep learning models. A deep learning model's reasoning structure is close to how a person can conclude. This is achieved using a layered system of algorithms called an artificial neural network. An artificial neural network's architecture was inspired by the human brain's neural network [77; 78], contributing to a learning mechanism that is far more capable than the machine learning classifiers discussed earlier. For the first method, we applied three deep learning models to train the dataset. We divided the data into 67% training and 33% testing. We trained the data in 5-10 iterations and applied the deep learning models: Long short-term memory (LSTM), neural network (NN), and convolutional neural network (CNN) [79; 80]. As with the ML classifiers, we started with BOW, added the other features in the same order as before, and calculated the accuracy. We used Keras, an open-source library, for the application of the deep learning models [74; 81].