# A Weighted PageRank-Based Bug Report Summarization Method Using Bug Report Relationships
## 4 Experimental Set-Up
### Experimental Subjects  
The experimental data commonly used in existing bug report summarization methods are BRC data, which are described in [PERSON] et al. [2]. Based on 36 bug reports and a summary of three commenters, BRC selects approximately 25% of sentences per bug report to form a summary and defines the summary as the golden summary. However, out of the 36 bug reports in the BRC data, only 13 bug reports have relationships with each other. [PERSON] et al. [6] noted this problem and built MBRC(Modified BRC) and OSCAR data. In this paper, we propose a summary method that uses all three associations of bug reports. Therefore, we investigated 50,000 bug reports created in 2018 for the Mozilla project. Based on these Mozilla project's one-year bug reports, we selected 36 bug reports that satisfy the following conditions (the 36 bug reports are in [16]):  
1. A bug report has a duplicates relationship with other bug reports.
2. A bug report has a blocks or depends-on relationship with other bug reports.
3. A bug report is longer than 300 words.  
In the case of condition 3, since a short report does not need summarization, a bug report having a length of at least 300 words is generally selected as an experimental subject. The information on selected bug reports is shown in Table 2. For example, the total number of bug reports that have duplicates relationships is 36. This means that each bug report has one or more duplicates relationships. The total number of duplicates relationships in the bug reports is 47, and the average number of duplicates relationships per bug report is 1.31.  
\begin{table}
\begin{tabular}{c c c c c} \hline \multirow{2}{*}{**Relationships**} & \multicolumn{2}{c}{**The Number of Bug Reports that Have**} & \multicolumn{2}{c}{**The Number of Relationships in the**} \\  & \multicolumn{2}{c}{**Relationships**} & \multicolumn{2}{c}{**Bug Reports**} \\ \cline{2-5}  & **Total Number** & **Average Number** & **Total Number** & **Average Number** \\ \hline Duplicates & 36 & 1.0 & 47 & 1.31 \\ Depends-On & 18 & 0.5 & 24 & 0.67 \\ Blocks & 26 & 0.72 & 36 & 1.0 \\ \hline \end{tabular}
\end{table}
Table 2: Information about the relationships of 36 selected bug reports.  
After selecting 36 bug reports as an experimental subject, an annotation process is performed to generate summary data for our experiment. The first author of this paper selects approximately 25% of the important sentences for all 36 bug reports. As the commenting work, the first author proceeds in the same way that is described by [PERSON] et al. [2] for building the BRC data. Commenting work follows the steps below:  
1. The participant reads and understands a bug report.
2. The participant writes his/her own summary statements with 25% of the total number of sentences in the bug report.
3. The participant associates each sentence of the summary with the original sentences.
4. The participant selects the linked sentences as the golden summary.  
The difference of our commenting work from [PERSON] et al. [2] is that the first author of this paper creates the golden summary, while [PERSON] et al. [2] recruited three participants to perform the commenting work and create the golden summaries consisting of the sentences that two participants selected. Due to the difference, our commenting work can be more subjective than [PERSON] et al.'s work [2]. To mitigate this risk, the first author carefully reads each bug report and, after feeling confident that he completely understands the report, creates summaries. We also conducted an additional experiment, described in Section 6, by recruiting three participants.  
We completed the annotation work to build the relationship-based bug report summary data. The average number of sentences in the summary data accounts for 24% of the original text.