# A Multi-Modal Profiling Fraud-Detection System for Capturing Suspicious Airline Ticket Activities
## 7 Experimental Results
### Performance Improvements on Transactional Dataset  
In our previous work [5], some tests and improvements were carried out in order to achieve maximum success on the transactional dataset. During these tests, several feature-selection mechanisms, measurement metrics, and train/test set organization methods were explored in order to improve fraud-detection performance for aviation. In this study, oversampling techniques and deep learning approaches were analyzed on the airline ticket sale dataset and the results are presented in this subsection. It has been observed that there are important limitations to increasing the success of a transaction-based fraud-detection system in order to prevent financial loss for airline companies and fulfill the sales experience of airline customers. Despite all the improvement and optimization efforts, the precision, recall, and especially the F1 score metrics could not reach acceptable levels either on the number of transactions or on the cost basis. The success rates after the feature-selection process is applied are given in Figure 2. In addition, the newly proposed cost-based measurement metric results are presented in Figure 3. The first test results using shallow and deep learning algorithms after performing only the necessary cleaning operations on the dataset are given as initial results in these figures. Examining these figures shows us that no significant change was attained for the decision tree algorithm after the feature selection was applied, whereas the naive Bayes algorithm performs greatly after feature selection. The primary cause of the unexpectedly poor performance observed prior to the pre-processing step is the correlation between the provided features. The performance of this basic algorithm was significantly improved by both the feature-selection process and the inclusion of engineered features. Apart from the DT algorithm, we observed a noticeable increase between 2.1% and 48.7% in terms of the quantity-based F1 score. The cost-based evaluation shows that the quantity-cost success balance is preserved and the most successful algorithm is SVM. Based on all these observations, it can be understood that the feature-selection process makes a substantial contribution to the fraud-detection problem in the aviation domain. Even the poor success results obtained via the naive Bayes algorithm increased after feature selection was applied and came to the same success level with other algorithms. In order to observe the effect of the feature-selection process, some tests have been carried out using the CHI\({}^{2}\) and PCA algorithms. The most successful classification algorithm was SVM, whereas the most successful feature-selection method was CHI\({}^{2}\). The number of features that gave the highest success results on average for all algorithms was 55. Also, a significant increase of up to 48.7% was observed in the naive Bayes test results. The details of the test results are given in our previous study [5]. \begin{table}
\begin{tabular}{c c c} \hline
**Dataset Configuration** & **Value** \\ \hline \hline  & Fraud/Legitimate Ratio & 1:5 \\ \hline  & Dataset Time Range & 15 Months \\ \hline BRUS & Number of Fraud & 5512 \\  & Number of Legitimate & 27,578 \\ \hline
**Oversampling \(\times\) 5** & Number of Fraud & 27,560 \\  & Number of Legitimate & 137,890 \\ \hline
**Oversampling \(\times\) 20** & Number of Fraud & 11,240 \\  & Number of Legitimate & 551,560 \\ \hline  & Transactional dataset & 41 \\
**\#Features** & CC Profile & 10 \\  & PNR Profile & 9 \\  & Record Profile & 2 \\ \hline
**Distribution** & Type & Chronological \\  & Train/Test Ratio & 70/30 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: The details of our dataset and train/test setup. was excluded and its results are not included in this section. Comparing the SMOTENC and Random Oversampling algorithms showed that SMOTENC has a higher success rate of 20.6% and 10.5% for the naive Bayes algorithm and MLP algorithm, respectively.