## 6 Limitations  
Despite careful planning of the presented study and its previous research, the results can be affected by different factors. Hereinafter, the limitations and potential threats to validity are presented. The study was limited to object-oriented programming languages and evaluating quality at the class level. We did not identify specific types of code smells defined by [PERSON] et al. [3] and classified by [PERSON] et al. [59], but deficient source code in general. We are aware of the definitions of different code smells and their detection, but since they are not unified they were not used in the presented \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Entity** & **\# ev.** & **Pairs** & **Overall Quality** & **Entity** & **\# ev.** & **Pairs** & **Overall Quality** \\ \hline Java201-34 & 4 & 2 & ✓ & Java201-54 & 4 & 2 & ✓ \\ Java201-35 & 4 & 2 & ✗ & Java201-55 & 4 & 2 & ✗ \\ Java201-36 & 4 & 2 & ✓ & Java201-56 & 4 & 2 & ✗ \\ Java201-37 & 4 & 2 & ✗ & Java201-57 & 4 & 2 & ✓ \\ Java201-38 & 4 & 2 & ✓ & Java202-58 & 4 & 2 & ✗ \\ Java201-39 & 4 & 2 & ✗ & Java202-59 & 4 & 2 & ✗ \\ Java201-40 & 4 & 2 & ✓ & Java202-60 & 4 & 2 & ✗ \\ Java201-41 & 4 & 2 & ✗ & Java202-61 & 4 & 2 & ✓ \\ Java201-42 & 4 & 2 & ✗ & Java202-62 & 4 & 2 & ✓ \\ Java201-43 & 4 & 2 & ✗ & Java202-63 & 4 & 2 & ✗ \\ Java201-44 & 4 & 2 & ✓ & Java202-64 & 4 & 2 & ✓ \\ Java201-45 & 4 & 2 & ✓ & Java203-65 & 4 & 2 & ✗ \\ Java201-46 & 4 & 2 & ✓ & Java203-66 & 4 & 2 & ✗ \\ Java201-47 & 4 & 2 & ✓ & Java203-67 & 4 & 2 & ✓ \\ Java201-48 & 4 & 2 & ✗ & Java204-68 & 4 & 2 & ✗ \\ Java201-49 & 4 & 2 & ✓ & Java204-69 & 4 & 2 & ✓ \\ Java201-50 & 4 & 2 & ✗ & Java204-70 & 4 & 2 & ✓ \\ Java201-51 & 4 & 2 & ✓ & Java204-71 & 4 & 2 & ✗ \\ Java201-52 & 4 & 2 & ✗ & Java204-72 & 4 & 2 & ✓ \\ Java201-53 & 4 & 2 & ✓ & Java204-73 & 4 & 2 & ✓ \\ \hline \hline \end{tabular}
\end{table} 
Table 16: Results of expert judgment of assessed Java classes evaluated based on an individual software metric. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multicolumn{2}{c}{**Population: 40**} & \multicolumn{2}{c}{**Actual State**} \\  & & **Poor Class** & **Good Class** \\ \hline \multirow{2}{*}{**Predicted state**} & **Poor class** & 14 & 16 \\  & **Good class** & 3 & 7 \\ \hline \multirow{4}{*}{} & Accuracy & **0.525** \\  & Precision & **0.467** \\  & Recall & **0.824** \\ \hline \multicolumn{2}{c}{F-Measure} & **0.596** \\ \hline \hline \end{tabular}
\end{table} 
Table 17: Reliability analysis of identification based on an individual software metric. study. We identified classes as poor or good, where poor classes are classes that are exposed to a high risk for containing deficient source code. The results can be affected by the derived threshold values of the software metrics. We calculated thresholds by ourselves, using a carefully selected method, systematically collected benchmark data and a single tool used for collecting metric values. The results can also depend on the chosen software metrics that evaluate each quality aspect. Calculated F-Measure, accuracy, precision and recall may be affected by the selection of software classes. Two different expert judgments were performed in order to evaluate the proposed novel identification approach. Number of identified potentially deficient classes differ if a majority rule is applied on a combination of software metrics or identification is based on an individual software metric. For example, within Alfresco Repository, 441 out of 3314 classes were identified as potentially deficient, when considering the combination of all software metrics, which were evaluated as poor when applying the majority function. On the other hand, 2293 out of 3314 classes were identified as potentially problematic when evaluation was done using an individual software metric. Since each expert judgment assessed only a subset of identified classes using a certain approach, the used program entities vary. The execution and results of the expert judgment can also be influenced by the expertise of the participating experts.