## 6 Experimental Results and Evaluations
### Perplexity, Training, and Hidden Layer Selection  
The performance of a language model strongly depends on training time and perplexity. Perplexity also determines how good a model training process as well as calculates the model loss function. During training with various numbers of hidden layers, we calculated the epoch-wise perplexity to determine the optimal number of hidden layers. Correct source code samples were selected from the AOJ system for training. The perplexity at the last epoch (30 th) of training for each type of program is shown in Figure 5.  
The figure shows that the 200-layer model had the lowest perplexity during the training period. The epoch-wise perplexity for the 200-layer model for various problem sets is shown in Figure 6.  
Figure 5: Perplexity values at last epoch of proposed and other state-of-the-art models during the training period for various problem sets.  
Based on these results, we selected 200 hidden layers for LSTM-AttM and the other state-of-the-art models for all subsequent experiments. The training accuracies for the RNN, LSTM, and LSTM-AttM models are listed in Table 2.  
After model training, we evaluated the performance of the proposed model in terms of the detection of syntax, logic, and other errors as well as source code classification (correct or incorrect). We selected source code with errors for model validation and testing. Our goal was to evaluate the performance of the proposed model in terms of how accurately it assesses and detects errors in source code. To evaluate model performance, we adopted three evaluation indices, namely error detection accuracy (\(EDA\)), error prediction accuracy (\(EPA\)), and model accuracy (\(MA\)), respectively defined below.  
\[EDA=\frac{Actual\ Error\ Word\ (AEW)}{Total\ Detected\ Errors\ (TDE)}\times 100\% \tag{14}\]  
\[EPA=\frac{Actual\ Correct\ Word\ (ACW)}{Total\ Predicted\ Words\ (TPW)}\times 100\% \tag{15}\]  
\[MA=\frac{EDA+EPA}{2} \tag{16}\]  
The proposed model detects errors in source code by utilizing the trained correct source code corpus. Of the detected errors, there are some true errors, which are called actual error words (\(AEWs\)). Of the predicted words, there are some true correct words, which are called actual correct words  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Problem**} & \multicolumn{6}{c}{**Training Accuracy (\%)**} \\ \cline{2-7}  & \multicolumn{3}{c}{**Correct Source Codes**} & \multicolumn{3}{c}{**Incorrect Source Codes**} \\ \cline{2-7}  & **RNN** & **LSTM** & **LSTM-AttM** & **RNN** & **LSTM** & **LSTM-AttM** \\ \hline Insertion Sort & 70 & 81 & 94 & 68.3 & 82.4 & 93.6 \\ Greatest Common Divisor & 68 & 80 & 92 & 71.5 & 81 & 92.3 \\ Prime Numbers & 75 & 83 & 90 & 73.5 & 84.5 & 90 \\ Bubble Sort & 72 & 79 & 89 & 73.2 & 81.7 & 90 \\ Selection Sort & 65 & 78 & 87 & 66 & 80 & 89.6 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Average training accuracy of models for various problem sets.  
Figure 6: Epoch-wise perplexity for LSTM and LSTM-AttM models with 200 hidden layers for various problems.  
(_ACWs_). It is noted that the estimated probabilities of _AEW_ and _ACW_ should be more than 0.90. We used the above-mentioned evaluation indices to measure the performance of the models in terms of syntax and logic error assessment and detection.