## 4 Evaluation
### Evaluation of Machine Learning Classifiers  
Machine learning is all about discovering patterns and applying those to new datasets. To evaluate an algorithm, we can divide the dataset into two parts: train and test. In the first method, we divided the data into 67% training and 33% testing. We trained the data in 5-10 iterations and applied ML classifiers, such as logistic regression (LR), multilayer perceptron (MLP), support vector classifier (SVC), random forest (RF), Naive Bayes classifier (NV), AdaBoost (AB), and gradient boosting (GB). We used the scikit-learn machine learning library for our experiments [73]. It is a free, open-source, and trendy machine learning package for Python [74].  
We used a bag of words (BOW) for the first feature, which is also our baseline feature. We applied the ML classifiers and checked the accuracy. In the next step, we added the transitivity feature on top of the baseline feature. Similarly, we kept adding extra features in the given order: bag of words (BOW), transitivity, modality, polarity, syntax, doc2 vec, word2 vec, and word2 vec smoothed [48]. We calculated the disambiguation capacity of the problem/solution through these ML classifiers. The accuracy of each model for both scenarios is shown in Tables 1 and 2 below.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline
**Feature Set** & **LR** & **MLP** & **SVC** & **RF** & **NB** & **AB** & **GB** \\ \hline Baseline & 0.69 & 0.70 & 0.66 & 0.68 & 0.66 & 0.68 & 0.67 \\ +transitivity & 0.69 & 0.69 & 0.64 & 0.68 & 0.65 & 0.68 & 0.66 \\ +modality & 0.69 & 0.68 & 0.66 & 0.68 & 0.65 & 0.67 & 0.67 \\ +polarity & 0.70 & 0.70 & 0.65 & 0.69 & 0.65 & 0.67 & 0.67 \\ +syntax & 0.76 & 0.75 & 0.75 & 0.77 & 0.65 & 0.73 & 0.73 \\ +doc2 vec & 0.77 & 0.77 & 0.76 & 0.76 & 0.66 & 0.75 & 0.68 \\ +word2 vec & 0.73 & 0.69 & 0.61 & 0.70 & 0.66 & 0.72 & 0.72 \\ +word2 vecSmoothed & 0.76 & 0.74 & 0.75 & 0.77 & 0.65 & 0.73 & 0.72 \\ \hline \end{tabular}
\end{table}
Table 1: Classification of problems from non-problems using ML classifiers by training data iteration.  
For method one, the three most effective ML classifiers for the classification of the problems from the non-problems by training data iteration were **logistic regression (LR), multilayer perceptron (MLP)**, and **random forest (RF)**, giving an accuracy of **77%**.  
For the classification of the solutions from the non-solutions, the most effective ML model in method one was **AdaBoost**, giving an accuracy of **80%**.  
In the second method, we applied a 5-10-fold cross-validation for the same ML models. In a similar way to method 1, we started with the BOW, added the other features in the same order as before, and calculated the accuracy. We used the same scikit-learn machine learning library for method two as well. The results show that for most cases, the accuracy in method 2 decreases when compared to that of method 1. Moreover, the accuracy of the models increases from 5-fold validation to 10-fold validation. Hence, we present the results from the 10-fold validation. The accuracy of each model for method two is shown in Tables 3 and 4 below.  
For method two, the most effective ML model for classifying problems from non-problems was **random forest (RF)**, giving an accuracy of **76%**.  
For the classification of solutions from non-solutions, the most effective ML model for method two was **AdaBoost**, giving an accuracy of **81%**.