CNN application on structured data-Automated Feature Extraction | by Sourish Dey | Towards Data Science

Written: 13th Aug 2018 by Sourish Dey
https://www.linkedin.com/in/sourish-dey-03420b1a/
Importance Feature Engineering:
In my previous article, I discussed the importance of the creation of rich features from the limited number of features. Indeed, the real quality of machine learning/deep learning model comes from extensive feature engineering than from the modeling technique itself. While specific machine learning technique may work best for particular business problem/dataset, features are the universal drivers/critical components for any modeling project. Extracting as much information as possible from the available data sets is crucial to creating an effective solution.
In this article I will discuss about a not so popular method of feature engineering in industry(at least for structured data) — generating features from structured data using CNN(yes you heard it correct, Convolutional Neural Network), a family of modern deep learning model, extensively used in the area of computer vision problem. We will also test this method on a small data to create features, hands-on- as if It’s Not Run, It’s Not Done.
Traditionally analysts/data scientists used to create features using a manual process from domain/business knowledge. Often it’s called handcrafted feature engineering. While in data science we can’t deny the importance of domain knowledge, this type of feature engineering has some drawbacks:
1. Tedious: Manual feature engineering can be a tedious process. How many can new features be created from a list of parent variables? For example from the date variable, a data scientist can create 4 new features (month, year, hour, and day in the week) can be created. However, another data scientist can create 5 additional features(weekend indicator, festive season indicator, X-mass month indicator, seasonality index, week no in a month and so on). Is there any relationship/interaction with any other variable? So manual feature engineering is limited both by human time constraints and imagination: we simply cannot conceive of every possible feature that will be useful.
2. Influence of Human Bias: More often than not any human being working on a particular domain/ modeling project, builds up deep bias for some features(especially if it’s created by that analyst earlier!), irrespective of it adds value to the model or not.
Here it comes the power of automated feature engineering. Here no of features can be created is practically infinite and without any human bias. Also, this captures all possible complex non-linear interactions among features. Off course we can apply dimension reduction/feature selection technique at any point in time to get rid of redundant/zero importance features.
Here the business objective is to predict the probability of credit default based on credit card owner’s payment status, balance and payment history monitored for past few vintages(last 6 months from the predicted period). For sake of simplicity let’s ignore the lag period(generally there is a lag between customer information extraction and data upload in the system for analysis). The motivation of this business problem and data preparation is from https://medium.com/@guaisang/credit-default-prediction-with-logistic-regression-b5bd89f2799f.
I have not used cross-sectional components(eg. Gender, Education etc.) in the data and only kept time series variables(balance, payment history etc.) to introduce CNN in this business problem. The goal of this article is to conceptualize and implement CNN on this structured data and generate 100 new features from this data using CNN model. You can get the data and entire code here.
About the data:
The datasets utilizes a binary variable, ‘default payment next month’ (Yes = 1, No = 0), as the response variable. There are 18 features(excluding ID) in this set:
1:6 =X1 the repayment status in the last month of the prediction period; . . .;X6 = the repayment status in the 6th month before prediction period. The measurement scale for the repayment status is: -1 = pay duly; 1 = payment delay for one month; 2 = payment delay for two months; . . .; 8 = payment delay for eight months; 9 = payment delay for nine months and above. 0 means there is no transaction
X7 = amount of bill statement in last month of reference time frame;.... X12 = amount of bill statement in 6th month before the reference time frame;
X13 = amount of bill statement in last month of reference time frame;.... X18 = amount of payment in 6th month before the reference time frame;
Deep Learning methods, specifically CNNs, have seen a lot of success in the domain of image-based data, where the data offers a clearly structured topology in the regular lattice of pixels.Although detailed discussion about convolutional neural network (CNN, or ConvNet) is beyond scope of this article, let’s take a look at what makes CNNs so popular?
Deep Learning methods, specifically CNNs, have seen a lot of success in the domain of image-based data, where the data offers a clearly structured topology in the regular lattice of pixels. Although detailed discussion about CNN or ConvNet is beyond scope of this article, let’s take a look what makes CNNs so popular?
Reduced Parameters(parameter sharing): During convolution operation (layer), each output value (convolved feature in the figure) is not required to connected to every neuron in the previous layer (image in the figure), but only to those, called receptive fields, where the convolution kernel is applied currently. This characteristic of convolution layer is called Local Connectivity.Also same weights are applied over the convolving until the next update of the parameters. This characteristic of convolution layer is called Parameter Sharing.These reduced the number of parameters needed dramatically compared to usual ANN(artificial neural network) structure, where there is a connection between every single pair of input/output neuron.
Shift/Translation In-variance: It means that when the input shifts the output also shifts but stays otherwise unchanged. To be specific to an image, you can recognize an object as an object, even when its appearance varies in some way.
So in nutshell to apply CNN on any data, the data needs to be prepared in a way, so that there are local patterns and the same local patterns are relevant everywhere. To specific our data we have time series horizons of events(balance, payments etc.) for an individual customer, which makes the data full of local patterns for her and hence these parameters can be shared across customers. We just need to prepare the data suitable to feed into a CNN- feature matrix of for individual customers, same like an image frame matrix of d * w * n(d, w and n is the depth, width and no of the channel of an image frame).
To elaborate if we have N customers with m features(here balance/payment etc.) across t time horizons(here month1, month2 etc.) for p different trades(trades are different lines of credit, such as retail cards, mortgage etc.). So simply m features, t time horizons and p trades for individual customers are analogous to width, depth and no of channels of an image feature array. Precisely for our data the dimension for each customer will be m * t * p and for the entire data will be N*m*t*p. Now this makes the originally structured data into image frame type data, ideal to apply CNN to extract complex non-linear local patterns across customers by a convolution operation.
The original data set was consisting of one trade- retail card. To make it more relevant and more challenging I created dummy data set(tagging target variable status as original data) for one additional trades — mortgage. The data is sparser than retail card data and values may not be 100% logical. The final data set(CreditDefault.csv) is kept here.
So we have 30k distinct customer ids and total 60k observations, first 30k observations are of the retail card, next 30k is for the mortgage.
A detailed solution including data preparation code is at https://github.com/nitsourish/CNN-automated-Feature-Extraction/blob/master/CNN_feature_extraction.ipynb. This is the tricky part of this problem is to prepare the data in the format(image like format), so that we can apply CNN.Precisely for our data the dimension for each customer will be 3(no of features) * 6(time horizons) * 2(trades) and there will be 30000 frames of this dimension. So for the entire data it will be an array of will be 30000 * 3(width) * 6(depth) * 2(no of channels). So the data will be perfect to train a Convolutional network.
After preparation of channel specific data, we see the dimension:
shape of channel1(retail)data: (30000, 3, 6, 1)shape of channel2(mortgage)data: (30000, 3, 6, 1)
After merging these two arrays the data is proper to feed in CNN as the input volume to extract complex features with non-linear interaction.
shape of input volume(layer)data: (30000, 3, 6, 2)
Now the data looks like image frame data image like data (30000 image frame of volume 3*6*2). Now we need to map target variable(default payment next month) against customer ID in one hot encoding format.
X = np.concatenate((retail, mort),axis=3)Y = event_id[['default payment next month']].valuesY = to_categorical(Y,2)In our data we have payment default rate is: 28.402671 percent
In the data we have 28.40% payment_default rate. Although event rate is not 50%, we can say it’s fairly balanced data. so no need to do any weight correction
To extract features from CNN model first we need to train the CNN network with last sigmoid/logistic dense layer(here dimension 2) w.r.t. target variable. The objective of the training network is to identify the correct weights for the network by multiple forward and backward iterations, which eventually try to minimize binary cross entropy(misclassification cost). Here I will use keras framework with back-end as TensorFlow(tf). For our business problem AUC is the evaluation metric and we will do the optimization by maximizing AUC value in each epoch(Keras doesn’t have default AUC metric. However we can leverage tf to create customized evaluation function for AUC.)
We built following CNN with 4 conv layers(conv + activation + pooling(optional)) and 2 FC(fully connected layers) before reaching to the last sigmoid output layer.
We trained our model for epochs = 100 utilizing early stopping to prevent overfitting.It took less than 10 mins time to train using my NVIDIA GTX GeForce 1060 GPU with val_auc = 0.8317.
Feature Extraction Methodology
Now by using a pre-trained model, we can directly apply these weights on a data, removing the last sigmoid/logistic layer(in this problem until the dense layer of dimension 100). We can use this to any new data of same business problem to calculate these features.We need to just feed forward the network and it will directly map final weights to calculate features at some intermediate layer, without building the network again.This is a kind of transfer learning.Based on our requirement of no of features we can extract features at any intermediate dense layer with the desired dimension. For this problem, we will extract feature using an intermediate model till ‘feature_dense’ layer of the CNN.
#Preparing Indermediate model-removing last sigmoid layerintermediate_layer_model = Model(inputs=model.input,outputs=model.get_layer('feature_dense').output)intermediate_layer_model.summary()
The output layer dimension is 100, so if we predict using this network on a new data with the same input dimension, we can create 100 new complex features.
#predict to get featured datafeauture_engg_data = intermediate_layer_model.predict(X)feauture_engg_data = pd.DataFrame(feauture_engg_data)print('feauture_engg_data shape:', feauture_engg_data.shape)feauture_engg_data shape: (30000, 100)
We successfully captured 100 complex interactions among raw variables.
· Let’s have some sneak-peak whether these features have any value. Although there are a few better ways, for sake of simplicity let’s capture some bivariate relationship of these new features with the target variable ‘default payment next month’.Although for categorical target variable Correlation is not 100% correct method, we will calculate the correlation of all new values with the target as an approximation of the variables which may be important for the final model.
#Sorting correlation values in decsending ordernew_corrs = sorted(new_corrs, key = lambda x: abs(x[1]), reverse = True)#Let's see top 10 correlation valuesnew_corrs[:10] [('PAY_1', 0.32479372847862253), ('PAY_2', 0.26355120167216467), ('PAY_3', 0.2352525137249163), ('feat_78', -0.22173805212223915), ('PAY_4', 0.21661363684242424), ('PAY_5', 0.20414891387616674), ('feat_86', -0.20047655459374053), ('feat_6', -0.19189993720885604), ('PAY_6', 0.1868663616535449), ('feat_3', -0.17394080015462873)]
We can see 4 newly created variables(feat_78,feat_86,feat_6,feat_3) are in top 10, in terms of correlation(pseudo although) with ‘default payment next month’. Although based on this nothing concretely can be said about the predictive strength of these variable, at least these variable are worthy of further investigation.
We can also look at the KDE(kernel density estimation) plot of the highest correlated variables, in terms of absolute magnitude correlation.
Now the critical ask is for this problem why CNN/deep learning method is only used for feature extraction. Why not use it as a classifier also. Two main reasons is there:
Local pattern: The data we used in this business problem(which is applicable to most of the scenarios where we use structural data), consists cross sectional(static)variables as well, where we will not find any local pattern. So we can’t use CNN on the entire data as classifier. Of course for the data which holds Shift/Translation Invariance property, we can use CNN as end classifier.
Infrastructural challenge: Building in large-scale industrial data and bringing into production requires significant infrastructural support(GPUs, AWS etc.), which may not available to every organization.
Latency in model run: Even with proper infrastructure forward propagation of decently complex deep learning models takes significantly higher time than highly effective classifiers like XGBoost, RF, SVM etc. Indeed some application, which requires blazing fast real-time prediction, the best solution, in that case, is to extract feature with an automated technique like CNN feature extraction and then to add relatively simpler classifier.
Regulatory challenge: Even with the immense focus on machine learning/AI, organizations especially into the domain of banking/insurance/finance have to go through internal/external regulatory processes before implementation of predictive models and it may require painstaking persuasion to make these latest techniques on the production system. On the contrary, if the end classifier is the traditional model, there may not be a challenge on a modeling methodology perspective.
As modern Artificial Intelligence and Deep Learning guru, Andrew Ng accurately said, “...applied machine learning is basically feature engineering.”, the importance of creating the meaningful feature space cannot be overstated and automated feature engineering techniques aim to help the data scientist with the problem of feature creation seamlessly building hundreds or thousands of new features from raw data set. However, the purpose is not to replace data scientist. Rather will allow her to focus on other valuable parts of the machine learning pipeline, such as feature selection, exploring new methodology, delivering robust models into production etc. We just discussed one of the potential methods of automated feature engineering. Surely there are a few other comparable techniques. Love to explore those in future. Meanwhile eager to hear the other applications of CNN on structured data(relational database), in any other business problems.