# An Empirical Study on Software Defect Prediction Using CodeBERT Model
## 2 Background
### Deep Transfer Learning  
In some domains, including software defect prediction, it is challenging to construct a large-scale, high-quality dataset due to the high cost of labeling data samples. Transfer learning, which assumes that training data do not have to be identically distributed (i.i.d.) with test data, could mitigate the problem of insufficient training data ([PERSON] et al., 2018).  
The definition of transfer learning is proposed by [PERSON] et al. (2019) as below. Given a learning task \(T_{t}\) based on \(D_{t}\), and we can get help from \(D_{s}\) for the learning task \(T_{s}\). Transfer learning aims to improve the performance of predictive function \(f_{T}(\cdot)\) for the learning task \(T_{t}\) by discovering and transferring latent knowledge from \(D_{s}\) and \(T_{s}\), where \(D_{s}\
eq D_{t}\) and/or \(T_{s}\
eq T_{t}\). In addition, in most cases, the size of \(D_{s}\) is much larger than the size of \(D_{t}\), \(N_{s}\gg N_{t}\).  
Deep transfer learning is a special form of transfer learning that uses a non-linear deep learning model for transfer learning ([PERSON] et al., 2018). Deep transfer learning can be organized into four categories: instance-based deep transfer learning, which uses instances in source  
Figure 1: Workflow of deep learning based software defect prediction.  
domain by appropriate weights; mapping-based deep transfer learning, which maps instances from two domains into a new data space with better similarity; network-based deep transfer learning, which reuses the parts of the network pre-trained in the source domain; and adversarial-based deep transfer learning, which uses adversarial learning to find transferable features that are both suitable for both domains. Among the four methods, network-based deep transfer learning is generally accepted by researchers and has been practically used in many domains. For example, researchers could use pre-trained models published by large IT companies like Google and Facebook and fine-tune the model via deep transfer learning to adapt to downstream tasks. More recently, pre-training language models with large amounts of unlabeled data and fine-tuning in downstream tasks have made a breakthrough in the natural language processing domain, such as OpenAI GPT and BERT [28].  
A typical process of network-based deep transfer learning is shown in Figure 2. First, a pre-trained deep learning model is downloaded and available for fine-tuning. The weights are reserved in a pre-trained model. Second, the last layers of the pre-training deep learning models are identified, and the existing weights of these layers are removed. It is assumed that a pre-trained model contains two parts: general knowledge embedding layers and classification layers that target a specific upstream task, e.g., masked language model for text processing. It is generally accepted to reset the weights of the classification layers so that the model remains common knowledge, whereas it could be generalized to downstream tasks, e.g., text classification. Third, the downstream training data, which is usually much small than the dataset to train the pre-trained model up to several orders of magnitude, are fed to the model, which is called the fine-tuning process. During the process, the model learns how to perform downstream prediction tasks based on both pieces of knowledges from the pre-trained model and the downstream training data. Last, the fine-tuned model is available for downstream prediction tasks.  
Deep transfer learning requires that the upstream and downstream tasks are transferable. For example, suppose we use a pre-trained model on text generation (e.g., text corpora form Wikipedia) and use the model for text sentiment classification (e.g., whether a sentence is positive or negative). In that case, the tasks are transferable because such models will classify a sentence as positive or negative based on both keywords like fantastic and awful, and the whole meaning of the sentence, e.g., a rhetorical question could entirely divert the meaning of a sentence.