## 4 The Case: Expert Judgment in Software Quality Domain
### Results Analysis  
As we are interested in examining and comparing the student's and expert's judgments of the source code quality and software metrics, computing the agreement between participants (in our case, this would be an agreement in the judgment of pairs of students and experts) would be most suitable approach for an analysis. Several measures of inter-participant agreement exist. However, they are mainly limited to estimating the agreement between the two participants. In our case, more than one participant gave their assessment of the source code. Furthermore, they assessed it on a dichotomous scale. To enable the estimation of inter-participant agreement in such cases (more than \begin{table}
\begin{tabular}{l l l} \hline \hline  & **Students** & **Experts** \\ \hline Number of participants & 54 & 11 \\ \hline Experiences with software development (1–10) & 4.0 & 8.1 \\ Knowledge of Java (1–10) & 4.9 & 8.6 \\ Years of experiences with Java & between 3 and 6 years (31\%) & more than 10 year (81\%) \\  & less than 3 years (69\%) & between 6 and 10 years (19\%) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Profile of participating students and experts. Figure 4: Source Code Quality Judgment Tool Architecture. two participants), the Fleiss's kappa was used, which was first developed in 1971 [22]. Fleiss's kappa has since proven itself to be a useful tool to estimate inter-participant agreement between several participants in the literature [23]. To answer our research questions, we followed the example of [PERSON] [23], and first calculated the agreement proportions for each entity separately using the following equation:  
\[P_{i}=\frac{1}{n(n-1)}\sum_{i=1}^{k}n_{\mathrm{ij}}(n_{\mathrm{ij}}-1),\]  
where \(i\) represents the number of subjects (in case of calculations for a specific entity, this would be 1), \(n\) represents the number of participants, and \(n_{\mathrm{ij}}\) represents the number of participants assigned to the \(i\)th subject in the \(j\)th category [22,23]. Specifically, [PERSON]'s kappa was calculated for each entity on the expert's and student's general assessments of the source code, the assessment of source code size, class complexity, class cohesion, and coupling with other classes. Furthermore, the generalized [PERSON]'s kappa for each assessment category was calculated, representing the mean category assessment across all entities. The equation, provided by [PERSON] [23] was applied,  
\[\frac{1}{N}\sum_{i=1}^{N}P_{i}\]  
where \(N\) represents the total sum of subjects (e.g., entities) and \(P_{i}\) represents the kappa value for the \(i\)th entity. The results can be interpreted in terms of the level of agreement between participants. Recommendations on the interpretation of the kappa that are given by [PERSON] and [PERSON] [24], can be found in Table 5.  
We calculated [PERSON]'s kappa values for each program entity for both students and professionals. The data, interpreted as shown in Table 5, is collected in Table 6. Please note that the table shows the agreements of professional or student groups on a specific aspect of the given entity. The \"n/a\" value in the table means, that the group did not assess the given aspect of the entity. The \"-\" value in the table means that there was only one evaluation of the given aspect, which is why the calculation of [PERSON]'s kappa value was not performed.