# Improving Software Defect Prediction in Noisy Imbalanced Datasets
## 3 Materials and Methods
### Predictor Construction
## 4 Experiment

We conducted three experiments to answer the three research questions raised in Section 4.1 separately:

* defect prediction in SDP datasets without noise (for RQ1);
* defect prediction in SDP datasets in different noise environments (for RQ2);
* validation of the proportion of labeled noisy samples in the dataset before and after the use of the data pre-processing method (for RQ3). The datasets used in different experiments are described in Section 4.2, the selected ML algorithms for the aggregated model and experiment environment are shown in Section 4.3, and the SOTA methods compared to the proposed method in a different experiment is shown in Section 4.4. ### Research Questions

The experiment sought to answer the following three research questions:

* RQ1: Is US-PONR effective in SDP datasets? * RQ2: Can US-PONR perform better compared to the benchmark methods in unbalanced SDP datasets with noise? * RQ3: Is US-PONR especially good at eliminating label noise samples? RQ1 aims to validate the effectiveness of the proposed method in SDP datasets. RQ2 focuses on the effectiveness of US-PONR in noisy environments. And RQ3 further validates the effectiveness of the proposed method in noise reduction. ### Datasets

#### 4.2.1 PROMISE Public Dataset

To create the US-PONR method, we used the data of 24 versions of software from \(11\) different software projects in the PROMISE public dataset [57]. Table 5 provides an overview of these projects and their software, which represent the origin dataset before any pre-processing. The selected datasets cover different scales of data and different degrees of class imbalance. The CK and process metrics used in the experiment were directly obtained from the dataset. The net metrics were obtained by converting the dataset into a network diagram through an ISEE platform developed by our laboratory and by integrating measurement elements proposed by [PERSON] et al. [11] and by [PERSON] and [PERSON] [13]. PROMISE datasets, comprising software donations from NASA and other sources, have become one of the main public datasets in the SDP field, and they have been modified by researchers to mitigate various problems [58]. As [PERSON] et al. pointed out, the quality of the PROMISE dataset is now better than the noise-reduced NASA dataset [59]. Therefore, this article assumes that the PROMISE dataset used is clean and noise-free. #### 4.2.2 Noise Dataset Generation

In order to generate datasets with label noise, 18 researchers in our lab were selected to re-generate labels for the PROMISE dataset. After they were generated, labels that differed from the PROMISE labels were considered samples with label noise. We separately replaced 10%, 20%, and 30% samples of the training set with noise samples and generated three training sets with different numbers of noise samples to test the performance of the model in different noise environments. Meanwhile, the validation and test sets were kept the same as the PROMISE dataset. The generated noisy datasets were used to answer RQ2 and RQ3. ### Experiment Settings

As shown in Table 4 above, 14 machine learning algorithms were selected as candidates for model aggregation. Python's scikit-learn was then used to process any algorithms that required hyperparameter optimization and feature selection. The experiments were carried out using Ubuntu20.04 and Python 3.8 on an Intel E5-2620 CPU with 32 GB of memory and a NVidia GeForce RTX 2080 with 16 GB of RAM. ### Comparison Methods

For RQ1, the SOTA oversampling method SMOTUNED [46] was used to verify the effectiveness of US-PONR in non-noisy SDP datasets. For RQ2 and RQ3, to compare US-PONR with current mainstream methods of oversampling and noise removal, 12 methods were selected to join US-PONR in the noise experiment following undersampling, including SOMO [46], MWMOTE [48], SMOTE-IPF [51],