# An Empirical Study on Software Defect Prediction Using CodeBERT Model
## 6 Experiments and Results
### RQ1: Does the Pre-Trained CodeBERT Model Outperform a Newly-Trained CodeBERT Model? We decompose RQ1 into two sub-research questions, RQ1a and RQ1b, that use the CVPSC and the CPPSC dataset for cross-version and cross-project defect prediction, respectively. 2.1 RQ1a: Does the Pre-Trained CodeBERT Model Outperform a Newly-Trained CodeBERT Model in Cross-Version Defect Prediction? To answer RQ1a, we perform cross-version defect prediction experiments on the CVPSC dataset to compare the prediction performance of the CodeBERT-NT, CodeBERT-PT model, and RANDOM model in terms of F-measure, as shown in Table 3. For example, we get an F-measure of 0.509 and 0.493 on the CodeBERT-NT and CodeBERT-PT models for the Camel project. Compared to the F-measure of 0.287 on the RANDOM model, CodeBERT-NT and CodeBERT-PT model outperform the trivial RANDOM baseline. Considering the average results of the F-measure, the CodeBERT-PT model outperforms the CodeBERT-NT model by 2.1%. Both models significantly outperform the RANDOM baseline at 0.397. Therefore, the CodeBERT-PT model outperforms the CodeBERT-NT model on the CVPSC dataset for cross-version defect prediction. Apart from the prediction performance of the two models, the time cost of the two models also varies. We investigate the fluctuating level of F-measure on CodeBERT-NT and CodeBERT-PT for each project, as shown in Figure 6. Despite the initial epochs, both models have good convergence. Specifically, the CodeBERT-PT model in solid blue line converges more quickly and has fewer fluctuations after several epochs, which shows less time cost. According to our statistics, training an effective CodeBERT-PT model could be four times faster than training an effective CodeBERT-NT model. This is easy to comprehend, as the CodeBERT-PT model is pre-trained by a large corpus. It greatly reduces the cost of learning syntactic and semantic rules of programming languages. Summary of RQ1a: The prediction performance of CodeBERT-PT outperforms the CodeBERT-NT by 2.1% in cross-version software defect prediction. The CodeBERT-PT also trains four times faster than the CodeBERT-NT with lower fluctuations. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Training Set** & **Test Set** & **CodeBERT-NT** & **CodeBERT-PT** & **RANDOM** \\ \hline Camel 1.4 & Camel 1.6 & **0.509** & 0.493 & 0.287 \\ JEdit 4.0 & JEdit 4.1 & 0.567 & **0.642** & 0.267 \\ Lucene 2.0 & Lucene 2.2 & 0.653 & **0.707** & 0.546 \\ Xalan 2.5 & Xalan 2.6 & 0.581 & **0.627** & 0.484 \\ Xerces 1.2 & Xerces 1.3 & **0.349** & 0.321 & 0.231 \\ Synapse 1.1 & Synapse 1.2 & **0.445** & 0.443 & 0.402 \\ Poi 2.5 & Poi 3.0 & 0.702 & **0.72** & 0.562 \\ \hline \hline Average & & 0.544 & **0.565** & 0.397 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Performance comparison of CodeBERT-NT, CodeBERT-PT, and RANDOM models on the CVPSC dataset for cross-version defect prediction. The float values are the average F-measure values. The best F-measure values are highlighted in bold. Figure 6: The fluctuating level of F-measure on CodeBERT-PT and CodeBERT-NT on the CVPSC dataset for cross-version defect prediction. 2.2 RQ1b: Does the Pre-Trained CodeBERT Model Outperform a Newly-Trained CodeBERT Model in Cross-Project Defect Prediction? We perform cross-project experiments on the CPPSC dataset to compare the prediction performance of the CodeBERT-NT and CodeBERT-PT model on F-measure, as illustrated in Table 4. For instance, we achieve an F-measure of 0.293 and 0.354 on the CodeBERT-NT and CodeBERT-PT model for the JEdit-Camel project. The RANDOM model achieves an F-measure of 0.255, which is lower than the CodeBERT-NT and CodeBERT-PT models. The CodeBERT-PT model outperforms the CodeBERT-NT model by 0.9% in terms of average F-measure. Both models also significantly outperform the RANDOM baseline at 0.402. Therefore, the CodeBERT-PT model outperforms the CodeBERT-NT model on the CVPSC dataset for cross-project defect prediction. Apart from the prediction performance of the two models, the time cost of the two models also varies.