# Automatic Bug Triaging via Deep Reinforcement Learning
## 4 Bug Triaging via Deep Reinforcement (BT-RL) Model
### Overview  
As mentioned above, most state-of-the-art bug triaging models are offline. Hence, the models cannot be adjusted based on the situation presented by the bug reports and developers; as such, the performance yielded may be unsatisfactory. Conversely, BT-RL is trained for each new task, and it fully learns the relevant information of the task. In other words, BT-RL explores and learns from the real environment by addressing bug triaging problems. As illustrated in Figure 3, the BT-RL method comprises two sections: (a) a DMSF model and (b) an ODM model. The DMSF model aims to extract high-level features of multidimensional information, whereas the ODM model aims to identify the most suitable developer to fix the bug by analyzing relevant information from the bug reports.  
The details of BT-RL are described as Algorithm 1. First, we initialized the environment, including the replay memory pool \(D\), learning rate \(L\), batch size \(B\), step-threshold \(C\), and preprocessed bug reports. Subsequently, we used the \(\varepsilon-greedy\) strategy to select the developer, evaluate the selection, store it in \(D\) (i.e., line 10), and use it as a sample. During agent exploring after \(C\) steps, we selected samples \(B\) from \(D\) randomly for training. We obtained different rewards (\(r\)) based on different results and updated the network parameters based on the gap between reality and expectation. Finally, we updated \(\theta\) via linear interpolation.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Project** & **Period** & **Bug Reports** & **Developers** & **Subprojects** & **Components** & **Avg\_dev\_fix** & **Avg\_mon\_bug** \\ \hline NetBeans & 1 January 2008 to 11 January 2010 & 22,691 & 226 & 33 & 148 & 85.30 & 961 \\ OpenOffice & 1 March 2007 to 7 April 2013 & 23,491 & 553 & 36 & 116 & 42.48 & 341 \\ Mozilla & 23 June 2009 to 3 June 2010 & 18,793 & 1021 & 12 & 167 & 18.40 & 1705 \\ Eclipse & 1 January 2008 to 23 July 2009 & 41,830 & 772 & 7 & 114 & 54.25 & 2418 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of collected bug reports.  
Figure 2: The statistics of the average number of different developers participating in each of the four datasets per month. (**a**) Mozilla; (**b**) Eclipse; (**c**) Netbeans; (**d**) OpenOffice.  
Figure 3: The framework of BT-RL.  
```
Input: input \(D\) to capacity \(N\), \(B\), \(C\), \(L\) Out-put: Top-K developers for each bug task Initialize \(Q\) network with weights \(\theta\);  Initialize \(Q_{t}\) network with weights \(\theta_{t}=\theta\); For episode \(=1\), \(M\) do;  Initialize and preprocessed \(s_{1}\) \(\phi_{1}=\phi(s_{1})\); For t = 1, \(T\) do;  With probability \(\varepsilon\) select random action \(a_{t}\);  Otherwise select \(a_{t}=\) argmax \(Q(\phi(s_{t}),a_{t};\theta)\);  Execute action \(a_{t}\) in emulator and observe  reward \(r_{t}\) and \(s_{t}\);  Set \(s_{t}=s_{t+1}\) and preprocess \(\phi_{t+1}=\phi(s_{t+1})\);  Store transition \(\phi_{t}\), \(a_{t},r_{t},\phi_{t+1}\) in \(D\);  Set \(y_{i}=\left\{\begin{array}{l}r_{if}\text{terminal}\\ r_{j}+\gamma\max_{a}Q_{t}(\phi_{j+1},a^{\prime};\theta_{t})\end{array}\right.\);  Perform a gradient descent step;  Update network parameter \(Q_{t}=Q\); End For; End For;
```  
**Algorithm 1**BT-RL Algorithm via DQN