Data Pre Processing Techniques You Should Know | by Maneesha Rajaratne | Towards Data Science

Hi guys! Welcome Back. Today we will be discussing feature engineering techniques that can help you to score a higher accuracy.
As you know data can be very intimidating for a data scientist. If you have a dataset in your hand, and if you are a data scientist on top of that, then you kind of start thinking of varies stuff you can do to the raw dataset you have in your hand. Well, that’s the nature of a data scientist. So I am still in the learning process of becoming a data scientist. I am trying to fill up my mind with varies data preprocessing techniques because these techniques are very essential to know if you want to play with data.
For this analysis, I will be using credit card transaction dataset provided by Kaggle. I already did an article on Credit Card Fraud Detection using Autoencoders. The link is here: https://medium.com/@manisharajarathna/credit-card-fraud-detection-using-autoencoders-in-h2o-399cbb7ae4f1
It is a data mining technique that transforms raw data into an understandable format. Raw data(real world data) is always incomplete and that data cannot be sent through a model. That would cause certain errors. That is why we need to preprocess data before sending through a model.
Here are the steps I have followed;1. Import libraries2. Read data3. Checking for missing values4. Checking for categorical data5. Standardize the data6. PCA transformation7. Data splitting
As main libraries, I am using Pandas, Numpy and time;Pandas: Use for data manipulation and data analysis.Numpy: a fundamental package for scientific computing with Python.
As for the visualization I am using Matplotlib and Seaborn.For the data preprocessing techniques and algorithms, I used Scikit-learn libraries.
# main librariesimport pandas as pdimport numpy as npimport time# visual librariesfrom matplotlib import pyplot as pltimport seaborn as snsfrom mpl_toolkits.mplot3d import Axes3D plt.style.use('ggplot')# sklearn librariesfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.model_selection import train_test_splitfrom sklearn.preprocessing import normalizefrom sklearn.metrics import confusion_matrix,accuracy_score,precision_score,recall_score,f1_score,matthews_corrcoef,classification_report,roc_curvefrom sklearn.externals import joblibfrom sklearn.preprocessing import StandardScalerfrom sklearn.decomposition import PCA
You can find more details on dataset here: https://www.kaggle.com/mlg-ulb/creditcardfraud
# Read the data in the CSV file using pandasdf = pd.read_csv('../input/creditcard.csv')df.head()
df.shape> (284807, 31)
df.isnull().any().sum()> 0
Since there are no missing values found in the dataset, I didn’t use any missing values handling techniques.
So the dataset is labeled as 0s and 1s.
0 = non fraud
1 = fraud
All = df.shape[0]fraud = df[df['Class'] == 1]nonFraud = df[df['Class'] == 0]x = len(fraud)/Ally = len(nonFraud)/Allprint('frauds :',x*100,'%')print('non frauds :',y*100,'%')
# Let's plot the Transaction class against the Frequencylabels = ['non frauds','fraud']classes = pd.value_counts(df['Class'], sort = True)classes.plot(kind = 'bar', rot=0)plt.title("Transaction class distribution")plt.xticks(range(2), labels)plt.xlabel("Class")plt.ylabel("Frequency")
The only categorical variable we have in this data set is the target variable. Other features are already in numerical format, so no need of converting to categorical data.
I used seaborn distplot() to visualize the distribution of features in the dataset. We have 30 features and target variable in the dataset.
# distribution of Amountamount = [df['Amount'].values]sns.distplot(amount)
# distribution of Timetime = df['Time'].valuessns.distplot(time)
# distribution of anomalous featuresanomalous_features = df.iloc[:,1:29].columnsplt.figure(figsize=(12,28*4))gs = gridspec.GridSpec(28, 1)for i, cn in enumerate(df[anomalous_features]):    ax = plt.subplot(gs[i])    sns.distplot(df[cn][df.Class == 1], bins=50)    sns.distplot(df[cn][df.Class == 0], bins=50)    ax.set_xlabel('')    ax.set_title('histogram of feature: ' + str(cn))plt.show()
In this analysis I will not be dropping any features looking at the distribution of features, because I am still in the learning process of working with data preprocessing in numerous ways.So I would like to experiment step by step on data.
Instead all the features will be transformed to scaled variables.
# heat map of correlation of featurescorrelation_matrix = df.corr()fig = plt.figure(figsize=(12,9))sns.heatmap(correlation_matrix,vmax=0.8,square = True)plt.show()
The dataset is contains only numerical input variables which are the result of a PCA transformation. Features V1, V2, ... V28 are the principal components obtained with PCA, the only features which have not been transformed with PCA are ‘Time’ and ‘Amount’. So PCA is effected by scale so we need to scale the features in the data before applying PCA. For the scaling I am using Scikit-learn’s StandardScaler(). In order to fit to the scaler the data should be reshaped within -1 and 1.
# Standardizing the featuresdf['Vamount'] = StandardScaler().fit_transform(df['Amount'].values.reshape(-1,1))df['Vtime'] = StandardScaler().fit_transform(df['Time'].values.reshape(-1,1))df = df.drop(['Time','Amount'], axis = 1)df.head()
Now all the features are standardize into unit scale (mean = 0 and variance = 1)
PCA (Principal Component Analysis) mainly using to reduce the size of the feature space while retaining as much of the information as possible. In here all the features transformed into 2 features using PCA.
X = df.drop(['Class'], axis = 1)y = df['Class']pca = PCA(n_components=2)principalComponents = pca.fit_transform(X.values)principalDf = pd.DataFrame(data = principalComponents             , columns = ['principal component 1', 'principal component 2'])finalDf = pd.concat([principalDf, y], axis = 1)finalDf.head()
# 2D visualizationfig = plt.figure(figsize = (8,8))ax = fig.add_subplot(1,1,1) ax.set_xlabel('Principal Component 1', fontsize = 15)ax.set_ylabel('Principal Component 2', fontsize = 15)ax.set_title('2 component PCA', fontsize = 20)targets = [0, 1]colors = ['r', 'g']for target, color in zip(targets,colors):    indicesToKeep = finalDf['Class'] == target    ax.scatter(finalDf.loc[indicesToKeep, 'principal component 1']               , finalDf.loc[indicesToKeep, 'principal component 2']               , c = color               , s = 50)ax.legend(targets)ax.grid()
Since the data is highly imbalanced, I am only taking 492 rows from the non_fraud transactions.
# Lets shuffle the data before creating the subsamplesdf = df.sample(frac=1)frauds = df[df['Class'] == 1]non_frauds = df[df['Class'] == 0][:492]new_df = pd.concat([non_frauds, frauds])# Shuffle dataframe rowsnew_df = new_df.sample(frac=1, random_state=42)# Let's plot the Transaction class against the Frequencylabels = ['non frauds','fraud']classes = pd.value_counts(new_df['Class'], sort = True)classes.plot(kind = 'bar', rot=0)plt.title("Transaction class distribution")plt.xticks(range(2), labels)plt.xlabel("Class")plt.ylabel("Frequency")
# prepare the datafeatures = new_df.drop(['Class'], axis = 1)labels = pd.DataFrame(new_df['Class'])feature_array = features.valueslabel_array = labels.values
# splitting the faeture array and label array keeping 80% for the trainnig setsX_train,X_test,y_train,y_test = train_test_split(feature_array,label_array,test_size=0.20)# normalize: Scale input vectors individually to unit norm (vector length).X_train = normalize(X_train)X_test=normalize(X_test)
For the model building I am using K Nearest Neighbors. So we need find an optimal K to get the best out of it.
from wikipedia;
In pattern recognition, the k-nearest neighbors algorithm(k-NN) is a non-parametric method used for classification and regression. In both cases, the input consists of the k closest training examples in the feature space. The output depends on whether k-NN is used for classification or regression.
The KNN algorithm assumes that similar things exist in close proximity. In other words, similar things are near to each other.
“Birds of a feather flock together.”
The training algorithm only stores the data, the calculations are done by prediction algorithm.
neighbours = np.arange(1,25)train_accuracy =np.empty(len(neighbours))test_accuracy = np.empty(len(neighbours))for i,k in enumerate(neighbours):    #Setup a knn classifier with k neighbors    knn=KNeighborsClassifier(n_neighbors=k,algorithm="kd_tree",n_jobs=-1)        #Fit the model    knn.fit(X_train,y_train.ravel())        #Compute accuracy on the training set    train_accuracy[i] = knn.score(X_train, y_train.ravel())        #Compute accuracy on the test set    test_accuracy[i] = knn.score(X_test, y_test.ravel())
We can find the best K value by plotting the accuracy of testing set with the accuracy of training set. The best value for k is the point which gives the maximum testing accuracy.
#Generate plotplt.title('k-NN Varying number of neighbors')plt.plot(neighbours, test_accuracy, label='Testing Accuracy')plt.plot(neighbours, train_accuracy, label='Training accuracy')plt.legend()plt.xlabel('Number of neighbors')plt.ylabel('Accuracy')plt.show()
idx = np.where(test_accuracy == max(test_accuracy))x = neighbours[idx]
I am using Scikit-learn KNeighborsClassifier() to build the model.
#k_nearest_neighbours_classificationknn=KNeighborsClassifier(n_neighbors=x[0],algorithm="kd_tree",n_jobs=-1)knn.fit(X_train,y_train.ravel())
# save the model to diskfilename = 'finalized_model.sav'joblib.dump(knn, filename)# load the model from diskknn = joblib.load(filename)
After we build the model we can use predict() to predict the labels for the testing set.
# predicting labels for testing setknn_predicted_test_labels=knn.predict(X_test)
from pylab import rcParams#plt.figure(figsize=(12, 12))rcParams['figure.figsize'] = 14, 8plt.subplot(222)plt.scatter(X_test[:, 0], X_test[:, 1], c=knn_predicted_test_labels)plt.title(" Number of Blobs")
#scoring knnknn_accuracy_score  = accuracy_score(y_test,knn_predicted_test_labels)knn_precison_score  = precision_score(y_test,knn_predicted_test_labels)knn_recall_score    = recall_score(y_test,knn_predicted_test_labels)knn_f1_score        = f1_score(y_test,knn_predicted_test_labels)knn_MCC             =    matthews_corrcoef(y_test,knn_predicted_test_labels)
A confusion matrix is a table that is often used to describe the performance of a classification model (or “classifier”) on a set of test data for which the true values are known. Scikit-learn provides facility to calculate confusion matrix using the confusion_matrix method.
import seaborn as snsLABELS = ['Normal', 'Fraud']conf_matrix = confusion_matrix(y_test, knn_predicted_test_labels)plt.figure(figsize=(12, 12))sns.heatmap(conf_matrix, xticklabels=LABELS, yticklabels=LABELS, annot=True, fmt="d");plt.title("Confusion matrix")plt.ylabel('True class')plt.xlabel('Predicted class')plt.show()
I tried without standardizing the data to get a better accuracy. But after I learnt this method and applied it, it gave a promising result. I am still doing experiments and still learning about data preprocessing techniques. I only used KNN algorithm for this dataset.Feel free to comment and give an Up vote if you find this kernel helpful.
You can find more about PCA transformation here : https://towardsdatascience.com/pca-using-python-scikit-learn-e653f8989e60