# A Novel Ensemble Belief Rule-Based Model for Online Payment Fraud Detection
## 5 Experimental Study
### Comparative Research  
In comparative experiments, we chose several classic classifiers to conduct a comparative study of the ensemble BRB classifier which includes support vector machine, decision tree, K-Nearest Neighbors, logistic regression, bagging tree and boosting tree (Adaboost). We split the original dataset into a training dataset and test dataset with a ratio of 6 to 4. Also, we conducted experiments combining the BRB model with common methods for handling imbalanced classification problems (SMOTE and cost-sensitive learning). The experimental results are shown in Figures 13 and 14, and Tables 6 and 7. In Table 5, \"FS\" means \"Feature Selection\" and \"CS\" means \"Cost Sensitive\". Figures 13 and 14 are confusion matrices used to evaluate the classification performance of the model. In these figures, \"Output class\" represents the model's predicted results, while \"Target Class\" represents the true labels of the samples. \"0\" denotes \"Not Fraud\" and \"1\" denotes \"Fraud\". In Figures 13 and 14, the squares with green backgrounds represent correctly classified examples, while the squares with red backgrounds represent incorrectly classified examples. These two figures provide insight into the detailed classification results of the model. Figure 13: Classification results of ensemble BRB in Dataset #1. As shown in Tables 5 and 6, multiple evaluation metrics were utilized to assess the classification performance of the classifiers in this study. Specifically, the confusion matrix was used to observe the detailed classification outcomes, the F1 score was used to evaluate the overall performance and the recall was used to measure the performance in the minor class. After all, for experiments in this study, the algorithm's performance in the minor class is of greater concern. Therefore, compared to accuracy, the recall and F1 score are more representative. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **F1 Score** & **Accuracy** & **Recall** \\ \hline SVM & 0.8164 & 0.9994 & 0.7435 \\ Decision Tree & 0.7714 & 0.9993 & 0.8272 \\ KNN & 0.7966 & 0.9994 & 0.8010 \\ LR & 0.7245 & 0.9992 & 0.7539 \\ Bagging Tree & 0.8232 & 0.9995 & 0.7068 \\ Boosting Tree & 0.8176 & 0.9995 & 0.7749 \\ BRB & \textbackslash{} & \textbackslash{} \\ BRB + FS & 0.7395 & 0.9991 & 0.7801 \\ BRB + FS + SMOTE & 0.7852 & 0.9992 & 0.8325 \\ BRB + FS + CS learning & 0.8060 & 0.9993 & 0.8482 \\ Our method & 0.8375 & 0.9995 & 0.8978 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Classification results of comparative research in Dataset #1. Figure 14: Classification results of comparative research in Dataset #1. For Dataset #1, based on the data presented in Table 6, it can be observed that the accuracy scores of various classifiers do not exhibit significant differences overall. This is because the dataset has a very high class imbalance, so the performance in the main class plays a decisive role in determining the overall accuracy. Hence, in order to accurately assess the effectiveness of these classifiers, it is important to evaluate their performance specifically in the minor class. Table 5 shows that our method ranks at the top for all three metrics. It achieves the highest F1 score, accuracy and recall, which is a remarkable performance. These results confirm the effectiveness of our method in imbalanced datasets and demonstrate that it is competitive with traditional machine learning classifiers for the fraud detection task. In contrast, some classifiers, notably SVM, are negatively affected by the data imbalance, resulting in low recall. Furthermore, our method offers better interpretability compared to other classifiers. The results show that a standard BRB model is unable to handle the fraud detection task effectively. With as many as 30 attributes, the BRB model faces a combinatorial explosion problem that makes modeling impossible. Therefore, attribute dimensionality reduction is essential.