## 4 Experiments
### Research Question and Results
\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{**Datasets**} & \multicolumn{3}{c}{**ILMCR**} & \multicolumn{3}{c}{**SMOE**} & \multicolumn{3}{c}{**IHT**} \\ \cline{2-10}  & **GNB** & **DTC** & **ABC** & **GNB** & **DTC** & **ABC** & **GNB** & **DTC** & **ABC** \\ \hline synapse-1.2 & 0.721 & 0.688 & 0.689 & 0.698 & 0.713 & 0.717 & 0.714 & 0.66 & 0.66 \\ jedit-3.2 & 0.692 & 0.74 & 0.73 & 0.669 & 0.64 & 0.685 & 0.703 & 0.725 & 0.73 \\ synapse-1.1 & 0.672 & 0.639 & 0.69 & 0.677 & 0.511 & 0.638 & 0.643 & 0.552 & 0.734 \\ log4j-1.0 & 0.76 & 0.732 & 0.724 & 0.619 & 0.745 & 0.759 & 0.595 & 0.745 & 0.677 \\ jedit-4.0 & 0.755 & 0.632 & 0.734 & 0.702 & 0.623 & 0.622 & 0.638 & 0.656 & 0.709 \\ ant-1.7 & 0.78 & 0.772 & 0.717 & 0.749 & 0.723 & 0.671 & 0.71 & 0.766 & 0.778 \\ camel-1.6 & 0.555 & 0.592 & 0.594 & 0.578 & 0.606 & 0.644 & 0.589 & 0.599 & 0.638 \\ camel-1.4 & 0.65 & 0.522 & 0.514 & 0.663 & 0.559 & 0.586 & 0.653 & 0.591 & 0.587 \\ Xerces-1.3 & 0.675 & 0.657 & 0.666 & 0.786 & 0.687 & 0.719 & 0.732 & 0.682 & 0.678 \\ Xalan-2.4 & 0.665 & 0.613 & 0.542 & 0.635 & 0.645 & 0.671 & 0.63 & 0.682 & 0.724 \\ jedit-4.2 & 0.757 & 0.674 & 0.636 & 0.759 & 0.743 & 0.73 & 0.673 & 0.686 & 0.725 \\ arc & 0.558 & 0.604 & 0.538 & 0.649 & 0.501 & 0.647 & 0.581 & 0.506 & 0.492 \\ synapse-1.0 & 0.741 & 0.631 & 0.647 & 0.749 & 0.666 & 0.678 & 0.711 & 0.721 & 0.753 \\ tomcat & 0.735 & 0.639 & 0.723 & 0.857 & 0.776 & 0.786 & 0.604 & 0.746 & 0.777 \\ camel-1.0 & 0.635 & 0.75 & 0.686 & 0.521 & 0.781 & 0.582 & 0.598 & 0.737 & 0.778 \\ \hline Average & 0.69 & 0.659 & 0.655 & 0.687 & 0.661 & 0.676 & 0.652 & 0.67 & **0.696** \\ Rank & 3.933 & 6 & 6.1 & 4.2 & 5.433 & 4.733 & 5.8 & 4.933 & **3.867** \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Balancedscore of matching of classifiers with resampling methods on small datasets. \begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{**Datasets**} & \multicolumn{3}{c}{**ILMCR**} & \multicolumn{3}{c}{**SMOE**} & \multicolumn{3}{c}{**IHT**} \\ \cline{2-10}  & **GNB** & **DTC** & **ABC** & **GNB** & **DTC** & **ABC** & **GNB** & **DTC** & **ABC** \\ \hline synapse-1.2 & 0.721 & 0.688 & 0.689 & 0.698 & 0.713 & 0.717 & 0.714 & 0.66 & 0.66 \\ jedit-3.2 & 0.692 & 0.74 & 0.73 & 0.669 & 0.64 & 0.685 & 0.703 & 0.725 & 0.73 \\ synapse-1.1 & 0.672 & 0.639 & 0.69 & 0.677 & 0.511 & 0.638 & 0.643 & 0.552 & 0.734 \\ log4j-1.0 & 0.76 & 0.732 & 0.724 & 0.619 & 0.745 & 0.759 & 0.595 & 0.745 & 0.677 \\ jedit-4.0 & 0.755 & 0.632 & 0.734 & 0.702 & 0.623 & 0.622 & 0.638 & 0.656 & 0.709 \\ ant-1.7 are all larger than 0.05, which suggests that for SMOE and IHT, there are no significant difference among different combinations of them with different classifiers on the perspective of Balancedscore and G-mean. We can draw a conclusion from the experiment, LIMCR, SMOE and IHT all perform no significant difference on some metrics by being combined with different classifiers but inversely on other metrics. We take more consideration on the difference, performance of data resampling methods may change by using different classifiers in imbalance learning, therefore, when the dataset has a small sample size, it is necessary to choose GNB or Naive Bayes as the basic classifier in imbalance learning. RQ4: Does the number of features of datasets have an influence in LIMCR? Motivation: Our proposed method LIMCR is strongly related to the features of datasets. In former experiments we use datasets with the same feature dimension and have no idea if the number of features have influence on the performance of LIMCR. Approach: In this experiment, we retain \(k\) (\(k\) = 4, 8, 12, 16 and 20) highest scoring features to observe the variation of performance on LIMCR. We exploit a feature selection method named SelectKBest from model feature selection in scikit-learn v0.20.3 [58], and the dependence score between each feature and class label is measured by chi-square [57].