# Combining Machine Learning and Logical Reasoning to Improve Requirements Traceability Recovery
## 4 Evaluation
### Experiment Design  
In this section, we design a series of experiments in response to the research questions we have proposed. We implement our approach on four data sets that are eTour, SMOS, Albergate and eANCL, as discussed in Section 4.1. We will calculate and present the average results obtained from all the four datasets.  
_Experiment 1_: For the research question _RQ1_, in phase 1 machine learning recovery, we respectively input the traceability link vector with and without the feature dimensions of class attribute, method parameter, and method return into the classification model for traceability recovery.  
_Experiment 2_: For the research questions _RQ2_, _RQ2.1_, _RQ2.2_, and _RQ2.3_, we use different classification algorithms and set different parameters to observe which classification algorithm with specific parameters can achieve better results. We will use the results of Experiment 2 to choose the suitable algorithms with proper parameters for later experiments.  
_Experiment 3_: For the research question _RQ3_, in Phase 2 Logical Reasoning Recovery, for each structural information feature between source code classes, we use the corresponding logical reasoning rule separately to capture traceability links.  
_Experiment 4_: For the research question _RQ4_, we have designed a series of experiments to measure the effectiveness of capturing relationships among use cases when defining the inference rules. Specifically, we first compare the following two cases to see whether use case relationships contribute to the original approach: (1) applying only the machine learning approach; (2) applying the machine learning approach and inference rules derived from use case interrelationships. Then, we compare the following two cases to see whether use case relationships can render additional values on the top of the initial hybrid approach: (1) applying the machine learning approach and inference rules derived from source code classes; (2) applying the machine learning approach and inference rules derived from both source code classes and use cases.  
_Experiment 5.1_: For the research question _RQ5.1_, we observed the experimental results in the following two cases: (1) After phase 1 machine learning recovery, additionally using logical reasoning rules based on source code structural information features to perform traceability recovery. (2) After phase 1 machine learning recovery, additionally using logical reasoning rules based on use case relationship features to perform traceability recovery.  
_Experiment 5.2_: For the research question _RQ5.2_, we observe the experimental results of the following cases: (1) Using logical reasoning rules based on source code structural information features to perform traceability recovery. (2) Using logical reasoning rules based on use case relationship features to perform traceability recovery. (3) Combine using logical reasoning rules based on source code structural information features and use case relationship features.  
In order to answer the research question _RQ6_, we compared our approach with other research methods on multiple datasets.  
_Experiment 6.1_: On the eTour data set, we compare our approach with Combine IR [17], Relational Topic Modeling [2], UD-CSTI [3], and R2C [4].  
_Experiment 6.2_: On the Albergate dataset, we compare the proposed method with the VSM-based method [14].  
_Experiment 6.3_: On the SMOS dataset, we compare the proposed in this paper with the UD-CSTI method [3].