# Automated Conditional Statements Checking for Complete Natural Language Requirements Specification
## 4 Empirical Evaluation
### Results and Analysis
#### 4.4.2 RQ2: How Does the Performance of Our Approach Perform?  
We have selected the Skip-Thought method for the conditional statements embedding and the affinity propagation method for the conditional statements clustering. Based on the dataset used for the evaluation, the performance of the proposed approach is shown in Table 5. From Table 5, it can be seen that the proposed approach reaches 42.31% in precision and 68.75% in recall.  
From the results shown in Table 5, it can be seen that the proposed approach has achieved a low precision. Compared with the precision, the recall is higher. This is partly because that in most classification problems, high recall rates come at the cost of the decrease of precision [55]. By analyzing the results of the proposed approach, we found that there are several factors that affect the performance of the proposed approach.  
1. _Template non-conformance_: The proposed approach finds the opposite conditions by using the negative particles and the antonyms. This means that when the conditional statements have not used the negative particles and antonyms, the opposite conditions will not be found. For example, for the conditional statement _\"When there are GPS events to report\"_, it is difficult for our approach to find its opposite _\"When there are none GPS events\"_.
2. _Unknown words_: We found that there are a lot of proper nouns used in the requirements specification. In most cases, they appear in the form of abbreviations. For a part of the speech tagging of these words, low accuracy is often achieved. This will affect the grouping of the conditional statements.
3. _Incorrect keywords_: When specifying requirements, the analysts may use some words to describe some objects such as a button. For example, for the conditional statement _\"When the screen prompts 'DORMANCY LIGHT: ON\"_, the word of DORMANCY here refers to a light and not a state. In this case, it is incorrect to detect the incompleteness by finding the antonym of this word. However, our approach cannot identify these cases and this results in some wrongly reported detections.
4. _Ill-formed sentences_: After clustering the conditional statements, the proposed approach groups the conditional statements according to the subjects of the statements. However, in practice, the subjects may be missing and the statements are not complete. For example, for the conditional statement _\"When transmitted\"_, it is obvious that the subject is missing. This will lower the precision of the proposed approach.  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **Rand Index** & **Mutual Information** \\ \hline K-Means & 0.1265 & 0.7334 \\ Spectral Clustering & 0.0658 & 0.6814 \\ Agglomerative Clustering & 0.1215 & 0.7331 \\ DBSCAN & 0.1979 & 0.8519 \\ Affinity Propagation & **0.3107** & **0.8667** \\ \hline \hline \end{tabular}
\end{table}
Table 4: The comparison results between the clustering methods.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Precision** & **Recall** & **F1** \\ \hline Our approach & 0.4231 & 0.6875 & 0.5238 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The performance of the proposed approach described in Section 3.  
Among these factors, the cases of _template non-conformance_ and _unknown words_ collectively account for approximately 90 percent of the values of FN and FP. For these factors, more improvements should be adopted to address them in the future.