# A Novel Ensemble Belief Rule-Based Model for Online Payment Fraud Detection
## 5 Experimental Study
### Modeling Steps and Classification Results of Ensemble BRB
Excessively random feature selection would degrade the classification performance of the weak classifiers, while overly similar feature selection would fail to maintain the diversity required for effective ensemble learning. As shown in Tables 4 and 5, the performances of selected weak BRB models in the test set vary, but after the model selection process, most of the metric values fall within a stable range. For instance, in Dataset #1, the F1 scores are concentrated between 0.76 and 0.82. By combining the outputs of all the weak BRBs, the ensemble BRB achieves the optimal performance across all the metrics reported in the table, demonstrating the effectiveness of the innovative ensemble method proposed in this paper for BRB models. To further validate the effectiveness of our sampling ratios, we also analyzed the performance of individual weak BRBs classified under different sampling ratios, as well as the overall ensemble performance. As shown in Figures 9 and 10, when the sampling ratio is below 50%, the performance of the weak BRB classifiers is highly variable and generally poor, and the ensemble performance is also suboptimal. As the sampling ratio increases, the \begin{table}
\begin{tabular}{c c c c} \hline
**Model** & **F1 Score** & **Accuracy** & **Recall** \\ \hline Sub\_BRB\_1 & 0.7911 & 0.9993 & 0.7435 \\ Sub\_BRB\_2 & 0.8208 & 0.9994 & 0.8272 \\ Sub\_BRB\_3 & 0.8010 & 0.9993 & 0.8010 \\ Sub\_BRB\_4 & 0.7680 & 0.9992 & 0.7539 \\ Sub\_BRB\_5 & 0.7649 & 0.9993 & 0.7068 \\ Sub\_BRB\_6 & 0.7810 & 0.9993 & 0.7749 \\ Sub\_BRB\_7 & 0.7766 & 0.9992 & 0.8010 \\ Ensemble BRB & 0.8375 & 0.9995 & 0.8978 \\ \hline \end{tabular}
\end{table} 
Table 4: Performance of weak BRBs and ensemble BRB in Dataset #1. \begin{table}
\begin{tabular}{c c c c} \hline
**Model** & **F1 Score** & **Accuracy** & **Recall** \\ \hline Sub\_BRB\_1 & 0.9527 & 0.9633 & 0.9372 \\ Sub\_BRB\_2 & 0.9532 & 0.9673 & 0.9230 \\ Sub\_BRB\_3 & 0.9485 & 0.9688 & 0.9142 \\ Sub\_BRB\_4 & 0.9095 & 0.9230 & 0.8872 \\ Sub\_BRB\_5 & 0.9264 & 0.9441 & 0.9102 \\ Sub\_BRB\_6 & 0.9126 & 0.9301 & 0.9223 \\ Sub\_BRB\_7 & 0.9448 & 0.9724 & 0.9401 \\ Ensemble BRB & 0.9950 & 0.9973 & 0.9952 \\ \hline \end{tabular}
\end{table} 
Table 5: Performance of weak BRBs and ensemble BRB in Dataset #2.  
performance of the weak classifiers gradually improves, stabilizing when the ratio exceeds 70% Additionally, as shown in Figures 11 and 12, although the performance of individual weak classifiers approaches optimal levels when the sampling ratio reaches 80% or 90%, the ensemble model's performance is actually lower than that of the ensemble model with a 60-70% sampling ratio. This is because, at very high sampling ratios, the classifiers lack diversity, making ensemble learning less effective. In contrast, with a 60-70% sampling ratio, the weak classifiers maintain both diversity and individual performance, resulting in the optimal ensemble model at this level. Additionally, ensemble size is a key factor affecting the ensemble performance. As the number of ensemble members increases, the performance of the ensemble model gradually improves. However, after reaching a critical point, additional ensemble members become meaningless and do not further improve the model's performance. In the experiments on both datasets, we selected an ensemble size of seven, as performance improvements became insignificant when exceeding this number. Figure 10: Performance of weak BRBs under different sampling ratio in Dataset #2. Figure 9: Performance of weak BRBs under different sampling ratio in Dataset #1.