# Combining Machine Learning and Logical Reasoning to Improve Requirements Traceability Recovery
## 4 Evaluation
### Results and Analysis  
In this section, we present and analyze the results of our experiments. Figure 10 shows the result of _Experiment 1_. In our previous research [10], we utilized the class name, class comment, method name and method comment source code features. In this paper, we further utilize tje class attribute, method parameter and method return features. As can be seen in Figure 10, after adding the class attribute, method parameter and method return features for the traceability link vector definition, the precision, recall, and F1-score are all slightly improved. Therefore, we can answer the **RQ1**. We propose that adding class attribute, method parameter, and method return features to the traceability link vector definition can improve traceability recovery. Class attribute, method parameter, and method return are meaningful features for traceability recovery. Next we conduct **Experiment 2**. Figure 11 shows the classification result on eTour. It can be seen from Figure 11 that Decision Tree, KNN, Random Forest, and GBDT can achieve good performance, and their performance is similar. In this paper, we combine using the decision tree, KNN, random forest and GBDT classification models for traceability recovery, merging the results of these classifier model. The SVM algorithm can achieve high precision, but its recall and F1-score are low. The results obtained by logistic regression and multinomial native bayes are low. Therefore, we can answer the **RQ2** and **RQ2.1**. We find that decision tree, KNN, random forest and GBDT are all suitable classification algorithms for traceability recovery. Logistic regression, multinomial native bayes are not suitable for the traceability recovery task. If the traceability recovery task needs to achieve high precision without considering recall, an SVM classification model is a good choice. In our study, we combine the decision tree, KNN, random forest and GBDT classification models for traceability recovery, merging the output links of these classifier models. Next, we explore the suitable value of the \(k\) parameter for KNN. Generally, if the value of \(k\) is too small, noise will have a greater impact on the prediction, which will cause deviations. A decrease in \(k\) value means that the overall model becomes more complex and prone to overfitting. If the value of \(k\) is too large, the approximate error of learning will increase, and instances far from the input target point will have an detrimental effect on the prediction. It can be seen from Figure 12 that as the value of \(k\) increases, precision improves slightly, and recall and F1-score decrease. When the parameter \(k\) is set to a smaller value, a better classification effect can  
Figure 11: Results of various classification models that are applied in the machine learning recovery phase. Figure 10: Comparison of before and after adding features (class attribute, method parameter and method return) to the traceability link vector definition. be achieved. Therefore, when the value of \(k\) is smaller, the classification effect is better. In response to **RQ2.2**, the value of \(k\) is 5 in our study. As can be seen from the Figure 13, the experimental results of the SVM classification algorithm are highest when using an rbf kernel, and lowest when using a sigmoid kernel. As the answer to the **RQ2.3**, we propose using rbf as the kernel when using SVM. Although the recall and F1-score obtained by using SVM for traceability recovery are low, it can achieve a high precision. When it is necessary to obtain a high precision without considering recall, SVM is a suitable choice. The results of _Experiment 3_ can be seen in Figure 14. By applying the implement, inherit, return, attribute and method call rules to capture traceability links after machine learning recovery, our approach can achieve higher results, especially for the implement, inherit and method call features. Using these source code structural information feature rules in combination can further improve the results.