## 5 The Validation of the Proposed Approach With Expert Judgment
Based on the provided data, it is also possible to calculate accuracy, precision, recall and F-Measure, presented with Equations (5)-(8). Accuracy measures how a prediction matches the reality [56], whereas precision and recall present values for how well relevant entities are retrieved [57]. Finally, the F-Measure presents information retrieval performance [58]. \[Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \tag{5}\]  
\[Precision=\frac{TP}{TP+FP} \tag{6}\]  
\[Recall=\frac{TP}{TP+FN} \tag{7}\]  
\[F\text{-}Measure=2*\frac{Precision*Recall}{Precision+Recall} \tag{8}\]  
To calculate accuracy, precision, recall and F-Measure for the proposed identification based on a combination of software metrics and by applying the majority function, we used the results of the validation of the proposed identification, presented in Section 5. We considered only the part of the study where the evaluation of Java classes was performed. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multicolumn{2}{c}{**Population: X**} & \multicolumn{3}{c}{**True Condition**} \\  & & **Positive** & **Negative** \\ \hline \multirow{2}{*}{**Predicted state**} & **Positive** & true positive & false positive \\  & **Negative** & false negative & true negative \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Confusion matrix. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Entity** & **\# ev.** & **Pairs** & **(1)** & **(2)** & **(3)** & **(4)** & **(5)** \\ \hline C\#201-1 & 5 & 2 & ✓ & ✓ & ✓ & & ✓ \\ C\#201-2 & 5 & 2 & ✓ & ✓ & ✓ & ✓ & \\ C\#201-3 & 5 & 2 & ✓ & ✓ & ✓ & ✓ & \\ C\#201-4 & 5 & 2 & ✓ & ✓ & ✓ & ✓ & \\ C\#202-5 & 5 & 2 & ✓ & ✓ & ✓ & & ✓ \\ C\#202-6 & 4 & 2 & ✓ & ✓ & ✓ & ✓ & \\ C\#202-7 & 5 & 2 & ✓ & ✓ & ✓ & & ✓ \\ C\#202-8 & 5 & 2 & ✓ & ✓ & ✓ & ✓ & \\ C\#202-9 & 4 & 2 & ✓ & ✓ & ✓ & & ✓ \\ C\#202-10 & 5 & 2 & ✓ & ✓ & ✓ & ✓ & \\ C\#202-11 & 5 & 2 & ✓ & ✓ & ✓ & & ✓ \\ C\#203-12 & 5 & 2 & ✓ & ✓ & ✓ & & ✓ \\ C\#203-13 & 5 & 2 & ✓ & ✓ & ✓ & & ✓ \\ C\#203-14 & 4 & 2 & ✓ & ✓ & ✓ & ✓ & \\ C\#203-15 & 4 & 2 & ✓ & ✓ & ✓ & & ✓ \\ C\#203-16 & 4 & 2 & ✓ & ✓ & ✓ & ✗ & \\ C\#203-17 & 5 & 2 & ✓ & ✓ & ✓ & ✓ & \\ C\#203-18 & 5 & 2 & ✓ & ✓ & ✓ & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table} 
Table 12: Results of expert judgment for classes in C# programming language, with assessed quality aspects, overall quality (1); source code size (2); class complexity (3); class cohesion (4) and class coupling (5); based on an evaluation using a combination of software metrics. Table 14 shows the confusion matrix and the calculated measures of predictive performance. 28 classes were identified as true positive and five classes as true negatives. No classes were detected as a false positive or false negative. Consequently, the accuracy, precision and recall for the presented identification is 100%. The same is also true for the value of the F-Measure. The proposed approach addresses multiple quality aspects of the evaluated program entities. On the other hand, if the quality of a program entity is evaluated using an individual software metric, only one aspect of the program entity is assessed. If the entity exceeds the threshold according to an individual software metric and is in good shape according to other metrics, it is hard to generalize that the evaluated entity really contains deficient code. This can result in many false positive and false negative results. If we assume that a class that exceeds the threshold value of at least one of the evaluated software metrics is rated as poor, the number of potentially deficient classes would be very large. This can be seen from Tables 3-5. For the purposes of studying the reliability and occurrence of false positive and false negative results, we selected 40 software classes in the programming language Java for another expert judgment with 6 other Java experts with multiple years of experiences. Detailed profiles of the participants are presented in Table 15 and are similar to the profiles of participants in the previously presented assessment.