# Quality Properties of Execution Tracing, an Empirical Study
## 4 Results and Discussion
### Data Collection from Software Professionals  
The data collection was conducted in seven focus groups which involved 49 participants, software architects, software developers, software maintainers, and software testers who are affected by the use of execution tracing in the course of their work. During the evaluation phase in a given focus group, each participant received 20 scores which had to be distributed among the quality property candidates identified. After concluding the focus groups, collected items were compared to the results of the previous focus groups, number and score value of newly identified items were computed. The normalised importance score value of the identified new quality property candidates dropped below a threshold score 2, which was defined as 10% of the individual score to distribute, in the last two focus groups; thus, the importance score of the new items identified approximated zero. Consequently, the new items identified were not any more important in the sight of the focus group; therefore, no further focus groups were organised. The distribution of the quality property candidates identified, the new quality property candidates identified, the importance score assigned, the number of participants and the normalised importance score per focus group are highlighted in Table 2 on page 10. The collected execution tracing quality property candidates were analysed and the same items identified in more focus groups were matched; moreover, their score values were summed as described in Section 3. This way, a common list of 42 items was produced with cumulated score values for each item after the data collection was completed. The data coding process had three stages: (1) each item was labelled with a short term possibly contained in the item description, (2) the similar labels were grouped in one cluster, and (3) the clusters created including their scores of importance were analysed, the similar ones merged while the dissimilar ones kept. The last stage was repeated in three cycles. The clusters, based on the output of the focus groups, constitute the quality properties of execution tracing, which are the variables on which execution tracing quality depends. The common list of the execution quality property candidates with the importance scores, the labels assigned and the clusters, in which the items were grouped at the end of data coding cycle two, we added to Table A1 in Appendix B; moreover, we presented the merged item descriptions by the clusters in Table A2 in Appendix B. The quality properties of execution tracing based on the output of the focus groups, we summarised after data coding cycle three in Table A3 in Appendix B and listed all the items assigned to the the given cluster. In summary, the research in the focus groups delivered the variables below, on which execution tracing quality depends. We indicate the score of importance for each variable beside its definition; moreover, we ordered the variables by their importance score values in descending order. In the sight of the focus groups by far the quality property _accuracy and consistency_ is the most important, while the quality property _security_ possesses the least importance, which is understandable from the point of view of the goal of execution tracing: localise errors in the application. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Group**} & **Number of** & **Number** & **Score Value** & \multirow{2}{*}{**Number of**} & **Normalized** \\  & **Items** & **New Items** & & **Assigned to** & **Number of** & **Score Value of** \\  & **Identified** & **Identified** & **New Items** & & **Participants** & **New Items for** \\  & & & & & **One Participant** \\ \hline
1. & 15 & 15 & 160 & 8 & 20 \\
2. & 14 & 9 & 76 & 5 & 15.2 \\
3. & 19 & 12 & 62 & 7 & 8.86 \\
4. & 8 & 1 & 0 & 8 & 0 \\
5. & 11 & 4 & 30 & 8 & 3.75 \\
6. & 6 & 0 & 0 & 6 & 0 \\
7. & 7 & 1 & 7 & 7 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Number of Items and Distribution of Importance Score in the Focus Groups.