## 3 Method
### Evaluation Criteria  
We used the false positive rate (FPR), false negative rate (FNR), recall, precision, accuracy, and F1-score as indicators to evaluate the model's performance, and the calculation formula is shown in Table 5. To offer exhaustive evaluations, these classification performance metrics have been comprehensively used in related research [1; 2; 3; 4; 5; 6; 10; 11; 12; 13; 14]. The confusion matrix which is the basis for calculating the performance evaluation index is presented in Table 6. The confusion matrix is the most common way to evaluate performance. In Table 6, each row represents an actual value, and each column represents a predicted value. A true positive (TP) and a true negative (TN) indicate a correct classification, which implies that the predicted class and the actual class match. By contrast, false negative (FN) and false positive (FP) indicate incorrect classification, which implies that an actual positive was predicted as negative and an actual negative was classified as positive.  
Accuracy is the ratio of predicting the actual loan execution as loan execution, and the actual loan not execution as loan not execution among all targets. Recall is the ratio predicted by loan execution among the actual loan execution targets, and precision is the ratio of the actual loan execution among the targets predicted by loan execution. The F1-score is commonly used to accurately evaluate model performance for imbalanced data and is calculated as the harmonic average of recall and precision. The accuracy of the prediction model is important because the purpose of this study is to predict the success or failure of outbound telemarketing operations.  
In addition, for efficient telemarketing operations, the selection performance of marketing to target customers is an important factor. Therefore, recall and precision must also be checked. Finally, the data imbalance problem of the training data was solved using the SMOTE technique. However, because the validation and testing data had a class imbalance problem, it was necessary to measure the F1-score, FPR, and FNR. In particular, the FPR metric in this study indicates the probability of placing calls perceived as spam to customers who do not require information on insurance policy loans. In other words, the FPR metric represents the percentage of customers who perceive telemarketing contacts as  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & & \multicolumn{2}{c}{**Predict**} \\ \cline{3-4}  & & **Positive** & **Negative** \\ \hline \multirow{4}{*}{**Actual**} & \multirow{2}{*}{**Positive**} & TP & FN \\  & & (True Positive) & (False Negative) \\ \cline{1-1}  & & FP & TN \\ \cline{1-1}  & & (False Positive) & (True Negative) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Confusion matrix for positive and negative records.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Measures** & **Formulation** \\ \hline False Positive Rate & FP \\ (FPR) & FF+TN \\ False Negative Rate & \\ (FNR) & \(\frac{\text{FN}}{\text{TP}+\text{FN}}\) \\ Recall & TP \\ Precision & \(\frac{\text{TP}}{\text{TP}+\text{FN}}\) \\ Accuracy & \(\frac{\text{TP}+\text{FN}}{\text{TP}+\text{FN}+\text{FN}}\) \\ F1-score & \(2\times\frac{\text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}}\) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Measures of model performance.  
spam. Therefore, FPR is an important metric used to evaluate performance. As described above, the performance of predictive models was evaluated using all six indicators, and in particular, recall, precision, and F1-score were calculated as macro averages. Based on the evaluation metrics of our research, a good prediction model should have a high F1-score and Accuracy and low FPR and FNR.