# An Empirical Study on Software Defect Prediction Using CodeBERT Model
## 5 Experimental Setup
### Baseline Models  
Our paper aims to explore empirical findings on neural programming language models, i.e., CodeBERT models. As the CodeBERT model takes BertTokenizer to tokenize raw source code, which is very rare in software defect prediction, it is inappropriate to  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Training Set** & **Test Set** & **No. Files** & **No. Defects** & **Avg. Buggy Rate** \\ \hline Camel 1.4 & Camel 1.6 & 847/934 & 145/188 & 17.1\%/20.1\% \\ JEdit 4.0 & JEdit 4.1 & 184/367 & 67/67 & 36.4\%/18.3\% \\ Lucene 2.0 & Lucene 2.2 & 186/238 & 91/143 & 48.9\%/60.1\% \\ Xalan 2.5 & Xalan 2.6 & 755/877 & 380/411 & 50.3\%/46.9\% \\ Xerces 1.2 & Xerces 1.3 & 436/446 & 70/67 & 16.1\%/15\% \\ Synapse 1.1 & Synapse 1.2 & 205/256 & 55/86 & 26.8\%/33.6\% \\ Poi 2.5 & Poi 3.0 & 380/438 & 248/281 & 65.3\%/64.2\% \\ \hline \hline \multicolumn{2}{c}{Total} & 6549 & 2299 & 35.1\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: CVPSC dataset description. The number of files, number of defects, and average buggy rate are listed separately for the training set and the test set.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Training Set** & **Test Set** & **No. Files** & **No. Defects** & **Avg. Buggy Rate** \\ \hline JEdit 4.1 & Camel 1.4 & 367/847 & 67/145 & 18.3\%/17.1\% \\ Camel 1.4 & JEdit 4.1 & 847/367 & 145/67 & 17.1\%/18.3\% \\ Lucene 2.2 & Xalan 2.5 & 238/755 & 143/380 & 48.9\%/50.3\% \\ Xerces 1.3 & Xalan 2.5 & 446/755 & 67/380 & 15\%/50.3\% \\ Xalan 2.5 & Lucene 2.2 & 755/238 & 380/143 & 50.3\%/48.9\% \\ Poi 2.5 & Synapse 1.1 & 380/205 & 248/55 & 65.3\%/26.8\% \\ Poi 3.0 & Synapse 1.2 & 438/256 & 281/86 & 64.2/33.6\% \\ Synapse 1.2 & Poi 3.0 & 256/438 & 86/281 & 33.6\%/64.2\% \\ Xalan 2.5 & Xerces 1.3 & 755/446 & 380/67 & 50.3\%/15\% \\ \hline \hline \multicolumn{2}{c}{Total} & 8589 & 3401 & 39.6\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: CPPSC dataset description. The number of files, number of defects, and average buggy rate are listed separately for the training set and the test set.  
compare the results with other software defect prediction models based on ASTs, CFGs, and so on because their inputs vary. Since it is extremely time-consuming to train a new programming language model from scratch and published large-scale programming language models are very scarce, we do not take other programming language models into account when choosing baseline models. We choose the following six baseline models:  
* Pre-trained CodeBERT sentence model (CodeBERT-PS). CodeBERT-PS uses the pre-trained CodeBERT model available on HuggingFace and predicts the answer of a declarative sentence, \"The code is buggy\", regarding a specific piece of code. If the output is 1, the code is buggy. Otherwise, the code is clean.
* Pre-trained CodeBERT keyword model (CodeBERT-PK). CodeBERT-PK uses the pre-trained CodeBERT model available on HuggingFace and predicts the relationship between a specific piece of code and a set of keywords, including bug, defect, error, fail, and patch. If the output is 1, the code is buggy. Otherwise, the code is clean.
* Pre-trained CodeBERT traditional model (CodeBERT-PT). CodeBERT-PT uses the pre-trained CodeBERT model available on HuggingFace and predicts if the source code is buggy, which is adopted by most researchers. If the output is 1, the code is buggy. Otherwise, the code is clean.
* Newly-trained CodeBERT traditional model (CodeBERT-NT). CodeBERT-NT uses the architecture of the CodeBERT model but discards the existing weights and trains from scratch. The model predicts if the source code is buggy, which is adopted by most researchers. If the output is 1, the code is buggy. Otherwise, the code is clean.
* RANDOM. The RANDOM model predicts if a source file is buggy randomly. A model that performs worse than RANDOM is no better than random guessing and, therefore, no practical value.