## 6 Experimental Results and Evaluations
### Source Code Classification  
In this section, we present the source code classification performance of the proposed LSTM-AttM and existing state-of-the-art models. We considered various kinds of error in source code, including semantic, syntax, logic, and communication errors. We evaluated the source code classification performance of the proposed model and state-of-the-art models by considering error occurrences in the source code. The proposed model calculated the error probability of each error candidate word to classify the source code.  
In our model, each variable, keyword, operator, operand, class, function, etc. in the source code was considered as a normal word. The model generated the error probability for each error candidate  
\begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Problem**} & \multicolumn{3}{c}{**Accuracy (\%)**} \\ \cline{2-4}  & **RNN** & **LSTM** & **LSTM-AttM** \\ \hline Insertion Sort & 60 & 75 & 95 \\ Greatest Common Divisor & 57 & 81 & 96 \\ Prime Numbers & 63 & 77 & 90 \\ Bubble Sort & 65 & 80 & 91 \\ Selection Sort & 56 & 78 & 89 \\
**Average** & **60.2** & **78.2** & **92.2** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Assessment results of logic error detection for erroneous source code.  
Figure 9: Logic error assessment and detection for source code evaluated using LSTM-AttM.  
word followed by the softmax layer. In general, our model detects error candidate words and estimates the corresponding error probability for each one. If the estimated error probability for any word is greater than 0.90, the source code is classified as incorrect. To evaluate the classification performance, we compared our model with some baseline methods, namely standard LSTM, RNN, and the random forest (RF) method with a deep belief network (DBN) [43].  
The performance of classification was evaluated in terms of precision, recall, and F-measure indices, respectively expressed as follows:  
\[Precision\ (P_{i})=\frac{TP_{i}}{TP_{i}+FP_{i}} \tag{17}\]  
\[Recall\ (R_{i})=\frac{TP_{i}}{TP_{i}+FN_{i}} \tag{18}\]  
\[F-measure=\frac{2*P_{i}*R_{i}}{P_{i}+R_{i}} \tag{19}\]  
where \(TP_{i}\) is the true positive rate (erroneous source code classified as erroneous), \(FP_{i}\) is the false positive rate (correct source code classified as erroneous), and \(FN_{i}\) is the false negative rate (erroneous source code classified as correct). F-measure is the harmonic mean between precision and recall. Usually, it is difficult to always obtain excellent precision and recall. If all samples are classified as erroneous, the recall will be high but precision will be low. F-measure is a balance between precision and recall. The F-measure value is between 0 and 1, where a higher value indicates better classification.  
We evaluated the model performance in terms of classification accuracy using source code samples with logic and syntax errors. Figure 10 shows the classification results for source code with syntax errors. The results show that the precision and recall values for the proposed model are better than those for the other models.  
Figure 10: Comparison of precision and recall values for classification of source code with syntax errors for (**a**) Insertion Sort (IS), (**b**) Greatest Common Divisor (GCD), (**c**) Prime Numbers (PN), (**d**) Bubble Sort (BS), and (**e**) Selection Sort (SS).