## 4 Rule-Based Reasoning
### Multiple Classifiers by Ensemble Learning  
The three individual classifiers mentioned previously could obtain a predictive precision ranging from 0.79 to 0.88. The research team further considered constructing a group of individual classifiers that can often receive a better predictive precision than any of its members, as suggested by the so-called ensemble learning [28]. The benefit of the ensemble method is that it combines different classifiers into a multiple classifier and thus has a better prediction performance than individual classifiers alone. The three ensemble learning classifiers implemented in this study are based on different strategies, including majority voting, bagging, and random forest. 1. Majority Voting Classifier  
As stated earlier, during the ensemble learning, a group of individual classifiers will be involved in the prediction process. Majority voting means that we combine different types of classifiers for prediction and select the answer that is predicted by the majority of classifiers [28]. The majority voting classifier implemented by this study comprises all three individual classifiers introduced in Section 5.3. Since the DT classifier has the best predictive performance among the three, the voting weights of the DT, SVM, and k-NN classifier is assigned as 5:1:1, respectively, indicating that the prediction made by the DT classifier has five times more weight than the predictions by other two classifiers. Table 9 and Figure 10 present one of the predictions made by the majority voting classifier that we implemented using training dataset #1, which was the one without the feedback from rule-based reasoning. Its overall predictive precision, recall, and f1-score are 0.86, 0.84, and 0.84. Again, we repeated the train-then-test cycle 30 times and obtained an average precision of 0.89, recall of 0.88, and f1-score of 0.88. The predictive performance of the majority voting classifier is close to the individual DT classifier, indicating that the ensemble learning effect is not significant. Although this result can be further improved by choosing a new set of individual classifiers or tuning the hyper-parameters such as weights, we still stopped here and moved forward to implement other ensemble learning classifiers. 2. Bagging  
Bagging is another ensemble strategy that is similar to majority voting that includes multiple classifiers and casts votes to the final decision. Instead of using different types of individual classifiers and the same training dataset to fit the classifiers, a bagging classifier only employs the same type of individual classifier and draws random samples from the training set to train the classifiers. It is a useful technique to reduce the model variance [28]. The bagging classifier we implemented employed 100 DT classifiers. Using training dataset #1 and repeating 30 train-then-test cycles, the bagging classifier obtains an average predictive performance of 0.91. Comparing with the previous individual classifiers mentioned in Section 5.3, this result moderately demonstrates the effect of ensemble learning. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Clash Types** & **Precision** & **Recall** & **F1-Score** & **Cases** \\ \hline Errors & 0.61 & 0.81 & 0.69 & 21 \\ Pseudo clashes & 0.90 & 0.78 & 0.84 & 23 \\ Deliberate clashes & 1.00 & 0.95 & 0.98 & 42 \\ Unknown & 0.70 & 0.58 & 0.64 & 12 \\ \hline Average/Total & 0.86 & 0.84 & 0.84 & 98 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: One of the predictive results of the majority voting classifier using training dataset #1. 3. Random Forest  
The last ensemble learning classifier we implemented is the random forest, which is a special case of bagging. Similar to bagging that only employs one type of individual classifier, which is DT, and draws random samples from the training set, the random forest also randomly selects feature subsets to fit the individual decision trees [28].