# A Novel Ensemble Belief Rule-Based Model for Online Payment Fraud Detection
## 5 Experimental Study
### Evaluation Matrices  
To avoid the impact of data imbalance and comprehensively consider the accuracy and completeness of the model, the experimental results are evaluated in the form of F1 score, precision and recall, which can be described as in Equation (19) to Equation (22).  
\[F1\ score=\frac{2\times Recall\times Precision}{Recall+Precision} \tag{19}\]  
where F1 score is the hybrid indicator of the recall and precision of a classifier; the formulas of accuracy, precision and recall are shown as follows.  
\[Accuracy=\frac{TP+TN}{TP+FP+TN+FN} \tag{20}\]  
\[Precision=\frac{TP}{TP+FP} \tag{21}\]  
\[Recall=\frac{TP}{TP+FN} \tag{22}\]  
where TP (True Positive), FN (False Negative), FP (False Positive) and TN (True Negative) are the four indicators obtained from a confusion matrix, as shown in Table 2. TP is the number of instances where the model correctly predicted the positive class. FN is the number of instances where the model incorrectly predicted the negative class, while the actual class was positive. FP is the number of instances where the model incorrectly predicted the positive class, while the actual class was negative. TN is the number of instances where the model correctly predicted the negative class.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Size** & **Distribution** & **Imbalance Ratio** & **\#Attributes** \\ \hline Dataset \#1 & 284,807 & 284,315/492 & 577.88 & 30 \\ Dataset \#2 & 568,630 & 284,315/284,315 & \(1\) & 29 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of fraud detection datasets.  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **Predicted Positive** & **Predicted Negative** \\ \hline Actual Positive & TP & FN \\ Actual Negative & FP & TN \\ \hline \hline \end{tabular}
\end{table}
Table 2: Definition of confusion matrix.  
In fraud detection, our primary concern is how well the model performs when facing the challenge of extreme class imbalance. Utilizing multiple evaluation metrics allows a more comprehensive and objective assessment of the model's performance. Among these, accuracy is used to measure the overall accuracy of the model. However, due to the significant class imbalance in the dataset, this metric often fails to differentiate between models effectively. Therefore, the F1 score and recall on minority class are more indicative of the model's comprehensive performance.  
In credit card fraud detection, the data used for model construction are crucial, as biases in data may exist. Therefore, it is essential to select a large-scale dataset to cover as many scenarios as possible. Additionally, using multiple datasets can help better validate the model's effectiveness and generalization capability. Furthermore, unintended consequences of model deployment should also be discussed. In real-world scenarios, the number of fraud samples is significantly smaller than that of normal samples, making data imbalance a persistent challenge in fraud detection. The key lies in accurately identifying fraud samples. If the model fails to achieve this, unintended consequences may arise, such as misclassifying normal samples as fraud or overlooking fraud samples. The former leads to wasted resources, while the latter leaves potential risks. Therefore, the ability to accurately detect fraud samples is of paramount importance.