IEEE-CIS Fraud Detection - Top 5% Solution | by Arun Mohan | Towards Data Science

I know that you or anyone around you have been the victim of some fraud activities. Directly we can’t control people on making easy money through fraud activities. Online transactions are one of the areas where fraud transactions can occur and can lead to loss of money.
In this competition, we will detect online fraudulent transactions using machine learning techniques. Our objective is to implement a solution that comes under the top 5% in Kaggle leaderboard which is not an easy task. IEEE-CIS Fraud Detection is a Kaggle competition organized by the IEEE Computational Intelligence Society. Vesta Corporation who is one of the forerunners in guaranteed e-commerce payment solutions provided the dataset for this competition.
In this blog, I will give a good overview of how I reached out to the top 5% result. I won't be explaining each and every line of code. But rather I will give a good overview of my solution. You can always refer to my code here.
The data is broken into two files identity and transaction, which are joined by TransactionID each for train and test. Thus we have 4 files — train transactions, train identity, test transactions, and test identity. It is important to note that all transaction does not have corresponding identity information.
Let us get familiarity with various columns present:
Transaction data
Transaction id: Id related to the transaction
TransactionDT: timedelta from a given reference datetime (not an actual timestamp)
TransactionAMT: transaction payment amount in USD
ProductCD [Categorical]: product code(the product for each transaction)
card1–6 [Categorical]: payment card related information like card type, country etc
addr1, addr2 [Categorical]: address information
dist1,dist2: some distance information
P_emaildomain [Categorical]: email domain of purchaser.
R_emaildomain [Categorical]: email domain of the recipient.
C1-C14: counting, such as how many addresses are found to be associated with the payment card, etc. The actual meaning is masked.
D1-D15: time delta, such as days between the previous transactions, etc.
M1-M9 [Categorical]: match, such as names on card and address, etc.
Vxxx: Vesta engineered rich features, including ranking, counting, and other entity relations.
Identity data
Transaction id: Id related to the transaction
DeviceType [Categorical]: Type of device used for the transaction
DeviceInfo [Categorical]: More information about device used
id 1–38 [Categorical+numeric]: network connection information,browser information etc (id 12–38 are categorical information)
Note: Actually we don't have access to exact information about columns. Mostly it is because of security purposes as we are handling transaction data.
The main business objective of finding fraud transactions can be following:
If a fraud transaction is found out, the company should immediately block that card.We should be able to predict the probability of fraud transactionWe should not predict fraud transactions as nonfraud. Also the vice versa. So precision and recall should be taken care of.
If a fraud transaction is found out, the company should immediately block that card.
We should be able to predict the probability of fraud transaction
We should not predict fraud transactions as nonfraud. Also the vice versa. So precision and recall should be taken care of.
Do exploratory data analysis on each and every column and understand the impact of each feature.Build a baseline model.Create new features and see whether those features improve the performance of the model. If yes retain the features or else drop those features. I used forward feature selection for that.Iteratively continue this approach until we get a good score.
Do exploratory data analysis on each and every column and understand the impact of each feature.
Build a baseline model.
Create new features and see whether those features improve the performance of the model. If yes retain the features or else drop those features. I used forward feature selection for that.
Iteratively continue this approach until we get a good score.
It's not ideal to explain the analysis of each and every feature here. I will explain some of my interesting observations. You can find the rest of the analysis from my repository.
Let us load the data first
Now let us load the saved files
We have train data of shape (590540, 434) and test data of shape (590540, 434). Let us see how many values are missing in train and test data
We will see columns with more than 90 % values missing.
We can see that columns like id_21,id_22,id_23,id_24,id_25,id_26,id_27,id_01,id_07,id_08 have more than 99 percent missing values. Its better we drop those features.
Is_Fraud (Target feature)
This is the target variable which we need to predict. Let us see how it is distributed.
Observations:
As expected we can see that class is heavily imbalanced. Here 96.5% of transactions are not fraud and the rest 3.5% of transactions are fraudulent. We will choose Area under the ROC curve (AUC) as a metric for our ML problem.
Why accuracy not for imbalanced datasets?
Consider a scenario like this where we have an imbalanced data set. For example, consider the case of credit card fraud detection with 98% percentage of points as non-fraud(1) and the rest 2% points as fraud(1). In such cases even if we predict all the points as non-fraud also we will get 98% accuracy. But actually, it is not the case. So we can’t use accuracy as a metric.
What is AUC ROC?
AUC is the area under the ROC curve. It tells how much the model is capable of distinguishing between classes. Higher the AUC, the better the model is at predicting 0s as 0s and 1s as 1s. The ROC curve is plotted with TPR against the FPR where TPR is on the y-axis and FPR is on the x-axis.
On analysis, one other thing I found was that in most of the cases for a client if one of the transactions is fraud all of his/her earlier transactions are marked as fraud. But this is not true for all the cases. There are also exceptions in it. This makes our prediction harder.
Note: I’m able to identify the clients by creating my own uid. I will explain as we proceed further.
Transaction Date
This indicates a time delta from a given reference DateTime. For example, 86400 can be 86400 seconds from a particular time, for eg: from 2/12/2010 12:05:09 pm. Here we have to check the data obtained in train and test is continuous with time. This is important for the train-test split. That means training data should be from an earlier period and test data should be from a later period of time.
Observations:
There is a slight gap in between, but otherwise, the training set is from an earlier period of time and test data from a later period of time. This can impact train validation split or cross-validation techniques should be used.
Another observation is that there are certain days in which the number of transactions is more than 10000
Now let us see fraudulent transactions on different days of the week as well as among different hours.
We can see that on the 3rd fay fraudulent transactions are very less, similarly, in the 7th hour, the percentage of fraudulent transactions is high compared to other hours.
Transaction Amt
This is the amount associated with each transaction. Let us see how the transaction Amount is distributed. (I also took a log of the transaction amount which gives a better idea to interpret)
Observations:
We can see that there is one point in train data where the amount > 30000. It's better we remove the outlier since it can affect our models(especially distance-based algorithms like logistic regression, knn etc.) in prediction.
Also, Outliers like this can cause an overfitting problem. For instance, tree-based models can put these outliers in leaf nodes, which are noise and not part of a general pattern. Therefore, I decided to remove the values larger than 30,000 in the training set.
It seems that the transactions with ‘LogTransactionAmt’ larger than 5.5 (244 dollars) and smaller than 3.3 (27 dollars) have a higher frequency and probability density of being fraudulent. On the other hand, the ones with ‘LogTransactionAmt’ from 3.3 to 5.5 have a higher chance of being legit.
Product CD
This is the product code associated with each transaction. Let us see how much percent of transactions are fraud among different product codes.
We can see that among transactions related to product code C about 12 percent are fraud transactions. similarly close to 6% of transactions are fraud among product codes.
D Columns
We have D columns from D1 to D15. These represent time deltas such as days between previous transaction and current transaction, days between current and first transaction, etc. Even though they have not given a clear understanding regarding D cols, we got some insights after a good analysis.
We assume D1 as the day since credit card usage has begun. Subtracting this from Transaction day will result in content value per client.
df_train['D1n'] =  df_train['D1'] -   df_train.TransactionDT/np.float32(24*60*60)
We did a normalization of all D cols and allowed the model to decide which are important and which are not.
Card1- Card6
These columns represent payment information related to cards. Some interesting observations from these features are:
There is a wide range of values in card1. If we see the distribution there are overlapping's. card1 alone cannot distinguish between fraudulent and nonfraudulent transactions. No missing values are there in card1.
A small percent of data is missing in card2. Similar to card1 a lot of unique values are present in card2 also. card3 has a low correlation with other card columns.
card 4 indicates which type of card do client uses visa, Mastercard, American Express, or discover. Similarly, card6 indicates the type of card- debit or credit.
P-email domain and R-email domain: These are email domains of purchaser and recipient respectively. About 76% of values are missing in the R-email domain. In purchaser email domains most mails are from gmail.com. Among those More fraud transactions comes from domain protonmail.com which is more than 90% which is a serious issue.
addr1 and addr2: These are some address-related information related to the client. Around 11% of data is missing from both addr1 and addr2. There are about 332 unique values in addr1 and 74 in the case of addr2.
dist1 and dist2: This might indicate the distance between transaction location and card owner address. This is just my assumption. We have about 93% of values missing in dist2. There is no correlation between dist1 and dist2. While dist2 is very weakly correlated with all other attributes.
C1-C14: These are information such as counting. C3 is somewhat different from other Cxx features. For c3, from box pot, it is clear that there is no value beyond 3 in train data for fraudulent transactions. For no fraudulent transactions, the value ranges from 0 to 26. Also, C3 is very weakly co-related with other Cxx columns.
M1-M9: More than 50 percent of values are missing among M columns. These features do not clearly distinguish between fraudulent and nonfraudulent transactions.
V1-V399: These are vesta-engineered features. These are masked information that can be ranking, counting, or other entity relations. There exist a strong correlation (>0.9) between many features. If possible It's better we reduce the number of these features as it can reduce the computational complexity of the model.
As a baseline, we created 3 models. — Logistic regression, Random Forest and XG Boost. We obtained the following results after a small hyperparameter tuning.
Logistic regression: 0.84018(Train AUC), 0.84245(Test AUC)
Random Forest: 0.9030(Train AUC), 0.8600(Test AUC)
Xgboost: 0.994(Train AUC), 0.9234(Test AUC)
As Xgboost gives better performance, we decided to stick to it. We also plotted feature importance to get a basic idea of the influence of various features. For Xgboost we got a Public leaderboard score of 0.9004 and 0.9238 for private LB.
Both columns are unique. One is time-related information and the other is a unique id. Adding this to the model doesn't make much sense. So we removed those features.
From our Xgboost model, after plotting feature importance we came to know that all V cols do not contribute much to the model. Also on analysis, we found that there exists a strong correlation between several ‘Vxx’ columns. So we decided to reduce the number of columns by doing a correlation analysis.
How to reduce columns based on NaN values?
Group columns based on a number of missing values Eg. if there are 4 columns v1,v2,v3, and v4. If v1 and v3 have 56 missing values and v2 have 21 and v4 have 5 missing values, we have 3 groups [‘v1’,’v3'], [‘v2’] and [‘v4’]
In each group, For each column in that group find the correlation with other columns and take only columns with a correlation coefficient > 0.75. Take the largest list with common elements as a subgroup. Each group contains several subgroups. For eg: if we have [[v1,v2],[v6],[v1,v4,v2,v5],[v5,v4]] ,our output will be [[v1,v2,v4,v5],[v6]]. Now from each subgroup choose the column with the most number of unique values. For eg, In subgroup [v1,v2,v4,v5], let v2 have the most unique values. So out output becomes [v2,v6].
The above figure indicates some part of our output. We will do correlation analysis on each group to reduce the number of columns.
Here columns ‘V35’, ‘V40’, ‘V41’, ‘V39’, ‘V38’, ‘V51’, ‘V37’, ‘V52’, ‘V36’, ‘V50’, ‘V48’, ‘V42’,‘V43’, ‘V44’, ‘V46’, ‘V47’, ‘V45’, ‘V49’ 168969 missing values. So they form a group. Now we will plot a correlation matrix.
Next, we will group them based on correlation. We will consider columns with a correlation >0.75 to be in the same group.
[[‘V35’,’V36'],[‘V37’,’V38'],[‘V39’,’V40',’V42',’V43',’V50',’V51',’V52'],[‘V41’],[‘V44’,’V45'],[‘V46’,’V47'],[‘V48’,’V49']]
We will reduce this using the above function and will end up with the following columns:
[‘V36’, ‘V37’, ‘V40’, ‘V41’, ‘V44’, ‘V47’, ‘V48’].
Similarly, we will do for all the groups and finally end up with the following V columns.
['V1', 'V3', 'V4', 'V6', 'V8', 'V11', 'V13', 'V14', 'V17', 'V20','V23', 'V26', 'V27', 'V30', 'V36', 'V37', 'V40', 'V41', 'V44', 'V47', 'V48', 'V54', 'V56', 'V59','V62', 'V65', 'V67', 'V68', 'V70', 'V76', 'V78', 'V80', 'V82', 'V86', 'V88', 'V89', 'V91', 'V96','V98', 'V99', 'V104', 'V107', 'V108', 'V111', 'V115', 'V117', 'V120', 'V121', 'V123', 'V124', 'V127','V129', 'V130', 'V136', 'V138', 'V139', 'V142', 'V147', 'V156', 'V162', 'V165', 'V160', 'V166', 'V178','V176', 'V173', 'V182', 'V187', 'V203', 'V205', 'V207', 'V215', 'V169', 'V171', 'V175', 'V180', 'V185','V188', 'V198', 'V210', 'V209', 'V218', 'V223', 'V224', 'V226', 'V228', 'V229', 'V235', 'V240', 'V258','V257', 'V253', 'V252', 'V260', 'V261', 'V264', 'V266', 'V267', 'V274', 'V277', 'V220', 'V221', 'V234','V238', 'V250', 'V271', 'V294', 'V284', 'V285', 'V286', 'V291','V297', 'V303', 'V305', 'V307', 'V309', 'V310', 'V320', 'V281', 'V283', 'V289', 'V296', 'V301', 'V314', 'V332', 'V325', 'V335', 'V338']
Reducing V cols doesn’t decrease the score on local validation. We got 0.993 AUC on train data and 0.9231 on cv data. Even though it reduced the Private LB score by 0.001, we were able to reduce 211 columns and thus improved the computational complexity of the model. So dropping those features is a better approach.
I also tried to reduced C and M cols in a similar way. But it decreased the score by more than 0.002 AUC. So as of now, I kept it.
From Kaggle's discussions, I came to know that D1 is the day since credit card usage began. Subtracting this from Transaction Day will make it an almost constant value per client. I did the same approach for all the D cols. To check the newly created features are helpful or not I did a forward feature selection.
How to select D cols which are to be modified?
We know that our baseline provides an AUC of 0.923. This will be our initial bese_score
On the first iteration, I added normalized D1 and removed the original D1, and created a model out of it and the test AUC is noted. Now, this is repeated for all Dcols from D1 to D15. Each time AUC is noted and choose the Dcol which improves the AUC from baseline (0.923)more. This AUC becomes our best_score.
On the next iteration, the earlier D col is included and step 1 is repeated for all the remaining columns along with D1.
Iterations are repeated until the AUC is not improved.
Finally, we end up with normalizing ‘D15’, ’D4', ’D2', ’D 11', and ‘D10’. Our local AUC became 0.934.
Next, I created the following features:
day of the week: Day of the week in which transaction occurred
hour: Hour in which transaction occurred
cent: cents associated with the transaction amount
LogTransactionAmt: Log of the transaction amount
P_email_company: company to which P_email_domain belongs. It is obtained from the internet.
Device_corp: I created this column from Device Info. It is basically parent company details obtained from device info.
Hour fraud status: From the graph, we obtained while doing EDA, we know the percentage of fraudulent transactions in each hour. Based on this I created 4 categories very low, low, medium, and high so that each hour falls on any of these categories.
Then I used FFS and see which of the features improve the result. But only the hour column improved the score to 0.9338. The function for forward feature selection looks as follows:
Next, I created some encoding features. Before that let us familiarize ourselves with the following functions.
Based on these I created about 45 features.
Then I did a forward feature selection and selected those features which improved model performance. As a result, I ended up with the following 6 new features
card4_addr1_R_emaildomain', 'card2_FE','card1_FE', 'card1_addr1_R_emaildomain', 'card3_addr1_P_emaildomain', 'card1_addr1','card4_addr1_P_emaildomain_FE'
These improved my local validation score to 0.9634.
Why UID(identifying clients) based features will work?
Suppose we have the following data:
Here if we construct a decision tree, it can look as follows
In this case, we can see that out of 11 points, 8 points are correctly classified. Suppose we can identify individual clients. I will explain to you how I am able to find it. If I am able to find such uid’s our data look as follows.
Let us create a new group aggregate from this uid. Let us construct a decision tree based on that.
We can see that in this case, all 11 points are classified correctly. Even though in real case this won't be as accurate, definitely it can lead to improving the performance of the model. So finding the uid is key in improving our score.
On Analysis I understood that card1, D1, and addr1 will help me to identify the client. So I combined all those 3 columns and used as uid.
Now I created several means, standard deviation, and count aggregations of several features with this uid. I used the following function for making aggregations.
I created about 100 aggregation features based on uid. But after Forward feature selection on different subsets, I end up with 9 new features.
'M9_uid_mean' ,  'M5_uid_mean' ,  'D2_uid_mean' ,  'D15_uid_mean' ,  'C13_uid_mean' ,  'C9_uid_mean' ,  'C1_uid_mean' ,  'C11_uid_mean' ,  'TransactionAmt_uid_std'
This improved my Local cv score to 0.9470. The results from the private leader board and public leader board were 0.9158 and 0.9423 respectively.
From analysis, it was clear that no uid is perfect. So I tried to create a new uid just by combining card1 with addr1. Then I created similar aggregations using new uid and see which of those new features improves the result of Forward feature selection. I end up with the following 4 new features:
'M4_uid2_mean' ,  'M1_uid2_mean' ,  'M7_uid2_mean' ,  'M8_uid2_std' 
This improved my local CV to 0.9481. Also, new private and public LB scores had become 0.917 and 0.940 respectively. Thus UID based features really help us to improve the performance of the model.
These are the two important techniques that literally improved the performance of my algorithm. As you know our model was tuned earlier, but currently, a lot of new features have been introduced. Let us find out the best parameters that fit the model. For that, I used Randomized Search Cv from Sklearn.
After a randomized search, we end up with the following parameters
n_estimators=5000,max_depth=12,learning_rate=0.002,subsample=0.8, colsample_bytree=0.4
First, we created a new column that indicates the month of the transaction.
We will predict test.csv using GroupKFold with months as groups. The training data are the months December 2017, January 2018, February 2018, March 2018, April 2018, and May 2018. We refer to these months as 12, 13, 14, 15, 16, 17. Fold one in GroupKFold will train on months 13 thru 17 and predict month 12. Note that the only purpose of month 12 is to tell XGB when to early_stop we don't actually care about the backward time predictions. The model trained on months 13 thru 17 will also predict test.csv which is forward in time.
Our code looks as follows:
Finally, our local CV score after cross-validation has become 0.951265. New Private and Public LB score has become 0.928873 and 0.954825 respectively. It ranked roughly 226 among 6351.(Top 3.5%).
Our progress throughout the project is as follows:
You can check my entire code at https://github.com/arunm8489/IEE-CIS-Fraud-detection. It's not the end. Still, there is a score of improvement. Some of the approaches I have not tried:
My entire work was around the XgBoost algorithm. You can try other algorithms like Lightgbm, cat boost etc.
I have reduced v cols based on the number of NaN values and Pearsons-correlation coefficient. You can also try PCA in each NaN groups and see whether it improves the performance
I have applied forward feature selection to select important features. I applied it to different subsets. Trying it on different subsets can yield different results.