# An Empirical Study on Software Defect Prediction Using CodeBERT Model
## 2 Background  
### Deep Learning Based Software Defect Prediction  
Figure 1 shows a typical workflow of deep learning-based software defect prediction. The first step is to extract software modules from open-source repositories. A software module could be a method, a class, a file, a code change, etc. The second step is to mark software modules as buggy/clean. The bug information is extracted from post-release defects recorded in bug tracking systems, e.g., Bugzilla. If a software module contains bugs found in later releases, the module is marked as buggy. The third step is to extract code features from software modules. In deep learning-based software defect prediction, typical code features include character-based, token-based, AST-node-based, AST-tree-based, AST-path-based, and AST-graph-based features. The fourth step is to build a deep learning model to generate features and train the instances. Frequently used deep learning models in software defect prediction include CNN, LSTM, Transformers, etc. The last step is to use the trained deep learning model for inference, i.e., predict whether a software module is buggy or clean.  
Deep learning-based software defect prediction can be organized into two categories: **Software defect prediction based on deep learning and hand-crafted features.** Deep learning is expert at combining hand-crafted features. Therefore, hand-crafted features can be fed into deep learning models to improve prediction performance. [PERSON] et al. (2019) are the first to use fully connected neural networks for hand-crafted feature combinations. Other deep learning models include autoencoders ([PERSON] et al., 2019; [PERSON] et al., 2020) and ladder networks ([PERSON] et al., 2020). The results show that deep representations of hand-crafted features could further improve prediction performance compared to experiments using traditional machine learning models and hand-crafted features.  
**Software defect prediction based on deep learning and generated features.** With the rapid progress in the natural language processing domain, deep learning models are capable of processing long texts, e.g., source code. Unlike designed top-down hand-crafted features, the features are generated bottom-up from the source code, which could represent structural and semantics information of the source code. Recently, many deep learning models, including Deep Belief Networks ([PERSON] et al., 2018; [PERSON] et al., 2018), CNN ([PERSON] et al., 2018; [PERSON] et al., 2018; [PERSON] et al., 2018; [PERSON] et al., 2018; [PERSON] et al., 2018; [PERSON] et al., 2020; [PERSON] et al., 2020; [PERSON] et al., 2020), LSTM ([PERSON] et al., 2018; [PERSON] et al., 2018; [PERSON] et al., 2020; [PERSON] et al., 2020), Transformers ([PERSON] et al., 2018), and other deep learning models ([PERSON] et al., 2020; [PERSON] et al., 2020) are used in software defect prediction.  
Researchers also explore more efficient ways of source code representations. Most researchers use AST sequence to represent source code ([PERSON] et al., 2018; [PERSON] et al., 2018; [PERSON] et al., 2018) which balances the information density represented and the training difficulty of deep learning models given insufficient datasets. Some researchers also present representations based on AST paths, e.g., PathPair2 Vec ([PERSON] et al., 2018) and Code2 Vec ([PERSON] et al., 2019). AST paths could extract node-path information that is not shown in AST sequences at the cost of the explosion of path numbers and training efforts. Other researchers explore context information based on AST sequences to focus on defect characteristics ([PERSON] et al., 2020).