# Code Edit Recommendation Using a Recurrent Neural Network
## 6 Evaluation Results
### RQ1: Recommendation Accuracies of CERNN and MI-EA  
We compared the recommendation accuracy of our CERNN approach with that of the state-of-the-art approach MI-EA. The main difference between CERNN and MI-EA is that CERNN maintains sequential information related to developers' operations and uses a deep learning technique, LSTM. To compare the accuracies of CERNN and MI-EA under similar conditions, we chose 3 as the size of the sliding window. The size of 3 in CERNN means that the sliding window maintains three sequentially viewed or edited elements. Because recommendation occurs at an edit event, one edited element is necessary in the context. The sliding window used in MI-EA maintains three viewed elements and one edited element. Therefore, the configuration that sets the size of the window to three in CERNN is similar to the configuration of the sliding window in MI-EA. Table 3 shows the recommendation results made by CERNN and MI-EA. The last two rows of Table 3 show the averaged precision, recall, and F1 score of the projects, Mylyn, Platform, PDE, ECF, and MDT. The second last row shows simple averages, where CERNN yielded 0.63 precision, 0.48 recall, and 0.54 F1-score with the five projects. The last row shows weighted averages by the numbers of recommendations, where CERNN yielded 0.57 precision, 0.38 recall, and 0.45 F1-score with 8307 recommendations. In contrast to our expectation, CERNN did not yield higher recommendation accuracy than MI-EA. While the averaged precision of MI-EA is 0.71, that of CERNN is 0.63. While the averaged recall of MI-EA is 0.51, that of CERNN is 0.48. Therefore, while the averaged F1 score of MI-EA is 0.55, that of CERNN is 0.54. The F1-score of CERNN is at least 1% lower than that of MI-EA. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Project**} & \multicolumn{5}{c}{**CERNN (W = 3)**} & \multicolumn{5}{c}{**MI-EA**} \\ \cline{2-9}  & **P** & **R** & **F1** & **\#R** & **P** & **R** & **F1** & **\#R** \\ \hline Mylyn & 0.53 & 0.35 & 0.41 & 6312 & 0.79 & 0.62 & 0.69 & 1096 \\ Platform & 0.71 & 0.47 & 0.57 & 1647 & 0.85 & 0.34 & 0.48 & 977 \\ PDE & 0.46 & 0.35 & 0.40 & 167 & 0.52 & 0.59 & 0.55 & 144 \\ ECF & 0.48 & 0.41 & 0.44 & 78 & 0.39 & 0.58 & 0.46 & 45 \\ MDT & 1.00 & 0.81 & 0.90 & 103 & 1.00 & 0.41 & 0.58 & 19 \\ Avg./Total & 0.63 & 0.48 & **0.54** & 8307 & 0.71 & 0.51 & **0.55** & 2281 \\ Weighted & 0.57 & 0.38 & **0.45** & 8307 & 0.79 & 0.50 & **0.59** & 2281 \\ Avg./Total & & & & & & & & \\ \hline \hline \end{tabular}  
* P denotes Precision, R Recall, F1 F-score, and #R the number of recommendations.  
\end{table} 
Table 3: Comparison of the recommendation accuracies of CERNN (W = 3) and MI-EA. When we considered the number of recommendations per project and calculated the weighted averaged precision, recall, and F1-score, the gap between CERNN and MI-EA becomes larger. As shown in the last row of Table 3, the weighted average precision of CERNN is 0.57, while that of MI-EA is 0.79. The weighted average recall of CERNN is 0.38, while that of MI-EA is 0.50. The weighted average F1 score of CERNN is 0.45, while that of MI-EA is 0.59. In this case, the F1-score of CERNN is 14% lower than that of MI-EA. This is mainly because a large number of recommendations occur in the Mylyn project. RQ2: Recommendation Accuracies of CERNN and MI-EA When Stopping Recommendations if the First Edit Is Found to Be False  
As we explained in Section 5.3.4, we observed that the recommendations occurring in the same interaction traces maintain similar recommendation accuracy. Therefore, we decided to stop recommendations when the first edit in an interaction trace was found to be false in our simulation. Now, the main difference between CERNN and MI-EA is that CERNN maintains sequential information related to developers' operations, uses a deep learning technique, and uses the result of the first edit at each interaction trace. Table 4 shows the recommendation results made by CERNN and MI-EA.