## 5 Case Study and Bug Classification
### Manual Bug Classification  
In the manual bug classification approach, no automated tool is used, but the categorization relies entirely on the decision of human experts. According to the general flow of bug processing in Figure 1, the bug reports are first reviewed by a reviewer/triager. The reviewer/triager inspects the bug reports and classifies the bugs concerning the bug classification schema used in the project. In this classification, since the reviewer/triager does not know the root cause of the problem, the decision is based on the reports' explanations only. If the report type is an unstructured bug report, it is more challenging to make this classification, especially for the reports of black-box tests. As stated previously, we have classified 504 bugs manually before introducing machine learning algorithms. In our case study, the bugs were classified according to [PERSON]'s bug categories [11]. The classification was performed independently by two expert senior software developers ESE1 and ESE2. Since ESE1 and ESE2 are fluent in English and native in Turkish, bilingual bug reports are not an issue in manual bug classification. Both ESE1 and ESE2 are senior software developers having more than ten years of experience in software development and six years of experience in the case study's domain. The manual classification of software bugs in the case study took five days for ESE1 and ESE2, but the overall time for the case study took longer. The other tasks for the overall case study included the preparation of the set-up (two days), developing and running the test cases (seven days), developing the bug reports (three days), and the overall reporting (three days), which took fifteen days in total. Two expert software engineers were involved in the bug classification, but more than twenty people were involved in the whole process, including project managers, developers, testers, and clients. In the paper, we have only focused on the time for classifying the bugs. These were then compared with the ground truth classification derived after thoroughly checking the bugs in the source code and the joint discussion with all the experts. Finally, the results of manual classification are given in Table 4. \begin{table}
\begin{tabular}{l l} \hline \hline
**Case Study Design Activity** & **Case Study** \\ \hline \multirow{3}{*}{Goal} & Assessing the effectiveness of ML-based automated bug \\  & classification over the manual bug classification utilizing F1 \\  & measure. \\ \hline \multirow{3}{*}{Research Questions} & RQ 1: To what extent is the presented automated bug \\  & classification approach effective? \\  & RQ 2: How does the automated bug classification approach \\  & compare to the manual bug classification approaches? \\ \hline \multirow{3}{*}{Background and Source} & Software testers \\  & Software developers \\ \cline{1-1}  & Bug classifiers \\ \cline{1-1}  & Bug reports \\ \cline{1-1}  & Source code \\ \hline \multirow{3}{*}{Data Collection} & Direct data collection of bug reports \\ \cline{1-1}  & Indirect data collection based on source code analysis \\ \hline \hline Data Analysis & Qualitative and quantitative data analysis \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Case study design of evaluation protocol adopted from [16]. To evaluate the results, we have considered two issues. First of all, to what extent are the classification of the expert software engineers aligned with each other? Secondly, to what extent are the results of these first bug classifications accurate and reliable? To address the first question, which is the inter-rating differences, we performed a Kappa test. The Kappa (K) coefficient of agreement [17] measures inter-rater and intra-rater reliability for categorical items. The Kappa coefficient is more robust than a simple agreement calculation. K values are between 0 and 1, where higher K values indicate higher and lower values indicate lower agreement between raters.