# An Empirical Study on Software Defect Prediction Using CodeBERT Model
## 7 Discussion  
### Assumptions That Influence Software Defect Prediction  
The empirical results of RQ1 indicate that using a pre-trained CodeBERT language model predicts defects more accurately and is more time-efficient than using a newly-trained model with the same architecture. The results of the time efficiency part are easy to understand, while the conclusions of the prediction performance part are more profounding. The result demonstrates that the naturalness assumption regarding software defect prediction can be proven by both statistics-based metrics like cross-entropy and deep learning models like neural language models. The advantage of neural language models over cross-entropy is that neural language models are more capable of characterizing the naturalness of code. The problem arises: _How can the naturalness assumption be combined with other assumptions to enhance prediction performance?_  
To solve the problem, we summarize and list the assumptions that influence software defect prediction in Figure 8. The assumptions include:  
1. Product-based assumptions. Product-based assumptions focus on the scale and complexity of software. If the source code is large in size or complex, it is more likely to be buggy. Related metrics include LOC (lines of code) and CC (cyclomatic \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Training Set** & **Test Set** & **CodeBERT-PS** & **CodeBERT-PK** & **CodeBERT-PT** & **RANDOM** \\ \hline \(\mathrm{JEdit}\) 4.1 & Camel 1.4 & **0.357** & **0.357** & 0.354 & 0.255 \\ Camel 1.4 & JEdit 4.1 & 0.543 & **0.556** & 0.36 & 0.267 \\ Lucene 2.2 & Xalan 2.5 & **0.672** & 0.662 & 0.66 & 0.502 \\ Xerces 1.3 & Xalan 2.5 & 0.651 & 0.65 & **0.659** & 0.502 \\ Xalan 2.5 & Lucene 2.2 & **0.745** & 0.714 & 0.648 & 0.546 \\ Poi 2.5 & Synapse 1.1 & 0.482 & **0.516** & 0.431 & 0.349 \\ Poi 3.0 & Synapse 1.2 & 0.533 & **0.545** & 0.531 & 0.402 \\ Synapse 1.2 & Poi 3.0 & **0.682** & 0.568 & 0.587 & 0.562 \\ Xalan 2.5 & Xerces 1.3 & **0.463** & 0.392 & 0.445 & 0.231 \\ \hline \multicolumn{5}{c}{Average} & **0.57** & 0.551 & 0.519 & 0.402 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Performance comparison of CodeBERT-PS, CodeBERT-PK, CodeBERT-PT, and RANDOM models on the CPPSC dataset for cross-project defect prediction. The float values are the average F-measure values. The best F-measure values are highlighted in bold. complexity) for source code, WMC (weighted method per class) and DIT (depth of inheritance) for object-oriented software, Degree and Betweenness Centrality for software complex networks. 2. Process-based assumptions. Process-based assumptions focus on history and experience during the development process of software. If the historical version of the source code is buggy, or the source code is developed by inexperienced programmers, it is more likely to be buggy. Related metrics include code churn, fix, developer experience, developer interaction, ownership between software modules and developers, and so on. 3. Machine learning-based assumptions. Machine learning-based assumptions focus on code patterns and naturalness. If the source code includes frequent patterns in buggy source code or the source code is unnatural, i.e., not following patterns followed by the majority of the developers, it is more likely to be buggy. Related metrics include semantic features generated from source code ASTs and neural language models. As we use generated features for software defect prediction, and because the product-based and process-based features are mostly hand-crafted features, their combination does not seem to improve prediction performance that much. For example, [PERSON] et al. [4] tried to combine generated features and hand-crafted features and gain tiny improvements. Therefore, we infer that effective combinations of generated features should come from the machine learning-based assumption.