IEEE-CIS Fraud Detection - Top 5% Solution
By: Arun Mohan
Tags: Kaggle Competition, Fraud Detection, Kaggle, Machine Learning, Xgboost

EDA

It's not ideal to explain the analysis of each and every feature here. I will explain some of my interesting observations. You can find the rest of the analysis from my repository.

Let us load the data first

Now let us load the saved files

We have train data of shape (590540, 434) and test data of shape (590540, 434). Let us see how many values are missing in train and test data

We will see columns with more than 90 % values missing.

We can see that columns like id_21,id_22,id_23,id_24,id_25,id_26,id_27,id_01,id_07,id_08 have more than 99 percent missing values. Its better we drop those features.

Is_Fraud (Target feature)

This is the target variable which we need to predict. Let us see how it is distributed.

Observations:

As expected we can see that class is heavily imbalanced. Here 96.5% of transactions are not fraud and the rest 3.5% of transactions are fraudulent. We will choose Area under the ROC curve (AUC) as a metric for our ML problem.

Why accuracy not for imbalanced datasets?

Consider a scenario like this where we have an imbalanced data set. For example, consider the case of credit card fraud detection with 98% percentage of points as non-fraud(1) and the rest 2% points as fraud(1). In such cases even if we predict all the points as non-fraud also we will get 98% accuracy. But actually, it is not the case. So we can’t use accuracy as a metric.

What is AUC ROC?

AUC is the area under the ROC curve. It tells how much the model is capable of distinguishing between classes. Higher the AUC, the better the model is at predicting 0s as 0s and 1s as 1s. The ROC curve is plotted with TPR against the FPR where TPR is on the y-axis and FPR is on the x-axis.

On analysis, one other thing I found was that in most of the cases for a client if one of the transactions is fraud all of his/her earlier transactions are marked as fraud. But this is not true for all the cases. There are also exceptions in it. This makes our prediction harder.

Note: I’m able to identify the clients by creating my own uid. I will explain as we proceed further.

Transaction Date

This indicates a time delta from a given reference DateTime. For example, 86400 can be 86400 seconds from a particular time, for eg: from 2/12/2010 12:05:09 pm. Here we have to check the data obtained in train and test is continuous with time. This is important for the train-test split. That means training data should be from an earlier period and test data should be from a later period of time.

Observations:

There is a slight gap in between, but otherwise, the training set is from an earlier period of time and test data from a later period of time. This can impact train validation split or cross-validation techniques should be used.

Another observation is that there are certain days in which the number of transactions is more than 10000

Now let us see fraudulent transactions on different days of the week as well as among different hours.

fraudulent transaction vs days

fraudulent transaction vs hours

We can see that on the 3rd fay fraudulent transactions are very less, similarly, in the 7th hour, the percentage of fraudulent transactions is high compared to other hours.

Transaction Amt

This is the amount associated with each transaction. Let us see how the transaction Amount is distributed. (I also took a log of the transaction amount which gives a better idea to interpret)

Transaction Amount distribution

Observations:

We can see that there is one point in train data where the amount > 30000. It's better we remove the outlier since it can affect our models(especially distance-based algorithms like logistic regression, knn etc.) in prediction.

Also, Outliers like this can cause an overfitting problem. For instance, tree-based models can put these outliers in leaf nodes, which are noise and not part of a general pattern. Therefore, I decided to remove the values larger than 30,000 in the training set.

It seems that the transactions with ‘LogTransactionAmt’ larger than 5.5 (244 dollars) and smaller than 3.3 (27 dollars) have a higher frequency and probability density of being fraudulent. On the other hand, the ones with ‘LogTransactionAmt’ from 3.3 to 5.5 have a higher chance of being legit.

Product CD

This is the product code associated with each transaction. Let us see how much percent of transactions are fraud among different product codes.

product code

We can see that among transactions related to product code C about 12 percent are fraud transactions. similarly close to 6% of transactions are fraud among product codes.

D Columns

We have D columns from D1 to D15. These represent time deltas such as days between previous transaction and current transaction, days between current and first transaction, etc. Even though they have not given a clear understanding regarding D cols, we got some insights after a good analysis.

We assume D1 as the day since credit card usage has begun. Subtracting this from Transaction day will result in content value per client.

df_train['D1n'] = df_train['D1'] - df_train.TransactionDT/np.float32(24*60*60)

before normalization

after normalization

We did a normalization of all D cols and allowed the model to decide which are important and which are not.

Card1- Card6

These columns represent payment information related to cards. Some interesting observations from these features are:

There is a wide range of values in card1. If we see the distribution there are overlapping's. card1 alone cannot distinguish between fraudulent and nonfraudulent transactions. No missing values are there in card1.

A small percent of data is missing in card2. Similar to card1 a lot of unique values are present in card2 also. card3 has a low correlation with other card columns.

card 4 indicates which type of card do client uses visa, Mastercard, American Express, or discover. Similarly, card6 indicates the type of card- debit or credit.

P-email domain and R-email domain: These are email domains of purchaser and recipient respectively. About 76% of values are missing in the R-email domain. In purchaser email domains most mails are from gmail.com. Among those More fraud transactions comes from domain protonmail.com which is more than 90% which is a serious issue.

addr1 and addr2: These are some address-related information related to the client. Around 11% of data is missing from both addr1 and addr2. There are about 332 unique values in addr1 and 74 in the case of addr2.

dist1 and dist2: This might indicate the distance between transaction location and card owner address. This is just my assumption. We have about 93% of values missing in dist2. There is no correlation between dist1 and dist2. While dist2 is very weakly correlated with all other attributes.

C1-C14: These are information such as counting. C3 is somewhat different from other Cxx features. For c3, from box pot, it is clear that there is no value beyond 3 in train data for fraudulent transactions. For no fraudulent transactions, the value ranges from 0 to 26. Also, C3 is very weakly co-related with other Cxx columns.

M1-M9: More than 50 percent of values are missing among M columns. These features do not clearly distinguish between fraudulent and nonfraudulent transactions.

V1-V399: These are vesta-engineered features. These are masked information that can be ranking, counting, or other entity relations. There exist a strong correlation (>0.9) between many features. If possible It's better we reduce the number of these features as it can reduce the computational complexity of the model.