# Combining Machine Learning and Logical Reasoning to Improve Requirements Traceability Recovery
## 4 Evaluation
### Results and Analysis
Figure 12: Results of using the k-nearest neighbors (KNN) classification model for traceability recovery with different \(k\) values on the eANCI dataset. Figure 13: Results of using the support vector machine (SVM) classification model for traceability link recovery with different kernel functions on the SMOS dataset. Therefore, we can answer the _RQ3_. We propose that using logical reasoning rules based on the features of implement, inherit, return, attribute, method call can effectively capture the traceability links. Implement, inherit, return, attribute, and method call are all important features for traceability recovery. According to the _Experiment 4_ results in Figure 15, it can be seen that after applying logical reasoning rules based on use case relationship features to capture more traceability links, the precision, recall, and F1-score of traceability recovery are further improved. In response to _RQ4_, we propose that the relationship features of use cases can help to improve traceability recovery. Using the rules based on the use case relationship features can further capture more traceability links. The use case relationship features are significant for traceability recovery. The result of _Experiment 5.1_ can be seen from the Figure 15. Using logical reasoning rules based on source code structural information features to perform traceability recovery, the improvement of the traceability recovery is more obvious, when compared to using logical reasoning rules based on use case relationship features. Figure 14: Results of using various structural information features of source code to perform traceability link recovery. Figure 15: Comparison of different combinations of our proposal on the SMOS dataset. Logical reasoning rule 1–5 are designed based on source code interrelationships, while logical reasoning 6–8 are designed based on use case interrelationships. Therefore, as the answer to **RQ5.1**, we believe that the features based on structural information between source code classes are more important than the relationship features between use cases, and can play a greater role. We speculate that this is because in the project system, the source code structural information is larger and more complex than for the use case relationships. The result of _Experiment 5.2_ can be seen from Figure 15. The experimental results shows that using the source code structural information features and use case relationship features in combination can further improve traceability recovery, particularly the recall. In response to **RQ5.2**, we propose that the source code structural information features and use case relationship features have gain effect on each other, and the combination of using the source code structural information features and use case relationship features can capture more traceability links. In conclusion for **RQ5**, the structural information is more important than the relationship between use cases, but they can improve each other. The result of _Experiment 6.1_ can be seen from Figure 16 that the precision, recall, and F1-score obtained by our approach are higher than other research methods [2; 3; 4; 17] on the eTour data set. According to the result of _Experiment 6.2_ shown in Figure 17, the recall obtained by our method is lower than the VSM-based method [14], but the precision, F1-score is higher than the VSM-based method on the Albergate dataset. According to the result of _Experiment 6.3_ shown in Figure 18, on the SMOS dataset it can be seen that the precision, recall, and F1-score obtained by our method are higher than the UD-CSTI method [3], especially in precision and F1-score. Therefore, as the answer of **RQ6**, the method proposed in this paper explore features of artifacts effectively which can achieve very high evaluation results. Our approach is an effective method for traceability recovery, and surpasses the methods proposed in related research.