# A Novel Ensemble Belief Rule-Based Model for Online Payment Fraud Detection
## 5 Experimental Study
### Comparative Research
Additionally, our method demonstrates significant improvements when compared to approaches that combine BRB with SMOTE or cost-sensitive learning. Although combining the BRB model with SMOTE or cost-sensitive learning methods can improve its performance in minority class samples, the enhancement is limited due to inherent drawbacks in these approaches. SMOTE tends to cause overfitting, while cost-sensitive learning methods struggle with determining optimal parameters. For Dataset #2, according to Table 7, although our method did not achieve the best performance across all the metrics, we can see that its performance is close to that of the best-performing bagging tree. Moreover, Dataset #2 is not an imbalanced dataset, and the comparison between these two datasets better demonstrates the effectiveness of our method in real-world imbalanced scenarios. Additionally, testing on multiple datasets demonstrates our model's generalization capability. To further validate the effectiveness of our method on the fraud detection task, we plotted the Receiver Operating Characteristic (ROC) curves for the classification results of various models, as shown in Figures 15 and 16. The ROC curve is an important metric for evaluating the performance of binary classification tasks, as it comprehensively reflects the overall performance of a classifier. The area under the ROC curve (AUC) provides a comprehensive assessment of a classifier's quality. The ROC curve visually illustrates the trade-off between sensitivity and specificity at different threshold settings, aiding in the selection of an appropriate threshold to balance false positives and false negatives. Moreover, the ROC curve is not influenced by the prior probability distribution of the data, making it effective even for imbalanced datasets, rendering it highly suitable for evaluating classification tasks in practical scenarios. These characteristics have led to the widespread application of ROC curves in fields such as machine learning and data mining. \begin{table}
\begin{tabular}{c c c c} \hline
**Model** & **F1 Score** & **Accuracy** & **Recall** \\ \hline SVM & 0.9567 & 0.9579 & 0.9325 \\ Decision Tree & 0.9970 & 0.9970 & 0.9984 \\ KNN & 0.9392 & 0.9334 & 0.8901 \\ LR & 0.9546 & 0.9557 & 0.9334 \\ Bagging Tree & 0.9998 & 0.9998 & 1.0000 \\ Boosting Tree & 0.9989 & 0.9991 & 0.9992 \\ BRB & \(\backslash\) & \(\backslash\) & \(\backslash\) \\ BRB + FS & 0.9556 & 0.9591 & 0.9284 \\ Our method & 0.9950 & 0.9973 & 0.9952 \\ \hline \end{tabular}
\end{table} 
Table 7: Classification results of comparative research in Dataset #2. From these two figures, we can observe that ensemble BRB method achieved the highest AUC value of 0.9788. The bagging tree and boosting tree methods closely followed, with both methods exhibiting nearly identical performance, reaching an AUC of 0.9691. Due to the inherent characteristics of the KNN and decision tree algorithms, their predictions cannot typically be transformed into probability distributions. Consequently, their ROC curves exhibit a two-segment linear form. A larger AUC value indicates that the ensemble BRB model exhibits superior classification capabilities compared to the other models. Figure 16: ROC curve of comparative research in Dataset #1. Figure 15: ROC curve of ensemble BRB in Dataset #1. The above experiments demonstrate that the proposed method exhibits good performance. However, there are also some limitations. First, while the large scale of the dataset used in the experiments reflects the potential of the proposed method in this field, the challenges posed by the massive scale of real-time credit card transaction data in practical applications warrant further investigation. Human in the Loop (HITL) [48] is an approach in machine learning and artificial intelligence that integrates human input into the training, decision-making, or refinement process of algorithms. It leverages human expertise to address model weaknesses, correct errors and improve system performance.