## 6 Results and Evaluation
### Evaluation  
The effect of the hybrid method on predictive performances obviously comes from the contribution of preliminary results by rule-based reasoning. Without this feedback, though it is not very accurate, the machine learning algorithms require more cases to reach a better performance. The improvement itself is not beyond our expectation, but what amazed us was that such a tiny rule base with five simple rules, possessing a relatively low predictive accuracy of 0.6, can still make contributions to those classifiers with a much higher predictive accuracy of 0.8. Figures 13-15 illustrate the feature importance of the decision tree, bagging, and random forest classifiers. The feature importance reveals the contribution of each feature in deriving the prediction results. The left histogram of these figures shows the results using training dataset #1, whereas the right histogram uses training dataset #2. As shown in the left histograms of Figures 13-15, the most discriminative feature processed by training dataset #1 is \"Framing\", which implies that if the type of  
Figure 11: Predictive performance improvement by overall f1-scores. Figure 12: Predictive performance improvement by clash types. clashing item is a beam, it can contribute nearly 40% of correct prediction. However, among those features processed by training data with the feedback of rule-based reasoning, or training dataset #2, the rule-based tag with \"errors\" becomes the most contributing feature to the prediction results. The shift of the most discriminative feature between the two experiments responds and explains the above-mentioned prediction improvement. According to Table 2, the proportion of actual cases that are \"errors\" is around 30%, representing the second-largest label cluster in the training data and reflecting the character of DT classifiers that tend to make a prediction to those labels with more cases. The feature importance of both \"framing\" and \"rule_errors\" in our experiments may reflect the effect of the beam rule, which was introduced in Section 4. Besides, the feature \"rule_pseudo\" also outperforms other features. These two features could be the reason why adding the feedback of rule-based reasoning can improve the prediction performance of machine learning. On the contrary, the label with the largest cluster, \"deliberate clashes\", does not appear in feature important ranking list, which needs more investigation in the future. Figures 16 and 17 show the learning curves of six classifiers in the two experiments. As shown in Figure 16, the training curves and validation curves converge with a large gap, indicating that the models processed by training dataset #1 may suffer from a small degree of overfitting. This problem is significantly improved in the second experiment when the feedback of rule-based reasoning is added in the training data. According to Figure 17, all the curves converge within a small gap with an f1-score higher than 0.90. Figure 14: Feature importance suggested by the bagging classifier. Figure 13: Feature importance suggested by the DT classifier. Figure 16: Learning curves of classifiers using training dataset #1. Figure 15: Feature importance suggested by the random forest classifier. Simply looking at the numbers, it may indicate that the models processed by training dataset #2 can achieve a satisfactory f1-score with a low bias and variance. Nevertheless, the predictive performance may be still under the influence of the nature of the training dataset we collect. For instance, among those clashes in our training data, \"errors\" accounted for nearly 50% of all the 326 cases. Histograms of feature importance shown in Figures 13-15 reveal the influence of this distribution on final prediction. It can be reasonably expected that the predictive performance of the same models may vary from one training dataset to another. More experiments with different training data from similar architectural projects are still needed.