Python Module for easy data pre-processing for beginners | by Benjamin Lau | Towards Data Science

Features in structured data can come in many forms including continuous, categorical, datetime and more. Normally to build a predictive model, we would look at feature selection or feature engineering to gather more insights or input more relevant data into the model. However, in cases where the data has a large number of features, feature engineering can become overwhelming fast. There is a need to process these data quickly and create an initial model to better understand your dataset or the problem at hand. Creating a model before any feature engineering has the benefits of:
Knowing the relative features importance of all your featuresHaving a rough idea of the model that works and those that don'tGives the insight to help with feature engineering/ features selection
Knowing the relative features importance of all your features
Having a rough idea of the model that works and those that don't
Gives the insight to help with feature engineering/ features selection
Introduction to Machine Learning for Coders from fast.ai used this exact approach and implemented what Jeremy Howard referred to as ‘Machine Learning Driven EDA’. Exploratory data analysis was done according to what the model tells you instead of domain knowledge or pre-determine assumption of the problems.
Here I present a simple implementation of data preprocessing with a python module I wrote.
The module can be found in my GitHub at https://github.com/Benlau93/easy_processing
You can download the module and place it in the same directory as your notebook and import using
import easy_processing as ep
To test the module, I will be using it for 2 datasets from kaggle competition. One will be for a regression problem and the other for a classification task. Both use RandomForest model from the sklearn.ensemble library.
Firstly, the House Prices: Advanced Regression Techniques dataset.
import libraries and dataset
import numpy as npimport pandas as pddf = pd.read_csv("train.csv",low_memory=False)X = df.drop(["SalePrice"],axis = 1)y=df["SalePrice"]
We start with handling missing values in our data. proc_miss replace NaN in continuous variables with the median of non-missing values in the respective feature columns
X, missing_val = ep.proc_miss(X)
missing_val that was returned is a dictionary with the column that contains missing values as keys and the median replaced as values
To process categorical variables, proc_cat convert categorical columns given by the user as a list into pandas categorical object. In the event that cat_cols is not given, columns with ‘object’ dtype are converted instead. max_cardi is the maximum cardinality of the features that will be converted to dummies variables. Cardinality just refer to the number of classes in each categorical features.
X, cat_dict = ep.proc_cat(X,max_cardi=5)
cat_dict returned is a dictionary with the categorical columns as keys and the Series.cat.categories as values
With just 2 lines of code, your data is ready to enter into a model. To check if all your features are in numeric format,
Training the model,
from sklearn.ensemble import RandomForestRegressorm = RandomForestRegressor(n_jobs=-1)m.fit(X,y)m.score(X,y)
The score obtained is 0.9711 (R2)
We can now test it on our testing dataset, get an initial prediction, submit to kaggle and get an initial score.
df_test = pd.read_csv("test.csv",low_memory=False)X_test, missing_val = ep.proc_miss(df_test,missing_val)X_test, cat_dict = ep.proc_cat(X_test,cat_dict=cat_dict,max_cardi=5)y_pred = m.predict(X_test)
For test/validation set, proc_miss requires a missing_val parameter so that it replaces the same median in each column of the test set. In cases where missing values are found in a column of the test set that isn't in the training set, the missing_val variables will be updated. The same logic applies to proc_cat where cat_dict is required as a parameter to ensure the encoding is consistent.
y_pred can now be exported into csv format and submit to kaggle. The score obtained for this dataset is 0.15296, which is top 68% in the leaderboard. Now you can look at the relative feature importance, hyperparameter tuning, and feature engineering to further improve your score.
The second dataset I will be using the Prudential Life Insurance Assessment
Loading of libraries and data
import numpy as npimport pandas as pdimport easy_processing as epdf = pd.read_csv("insurance_train.csv",low_memory=False)X = df.drop(["Response"],axis = 1)y=df["Response"]
Going through the exact same steps as before but using RandomForestClassifier instead of RandomForestRegressor
X, missing_val = ep.proc_miss(X)X, cat_dict = ep.proc_cat(X,max_cardi=5)from sklearn.ensemble import RandomForestClassifierm = RandomForestClassifier(n_jobs=-1)m.fit(X,y)m.score(X,y)
The score obtained here was 0.98939054
Processing the test set,
df_test = pd.read_csv("insurance_test.csv",low_memory=False)X_test, missing_val = ep.proc_miss(df_test,missing_val)X_test, cat_dict = ep.proc_cat(X_test,cat_dict=cat_dict,max_cardi=5)y_pred = m.predict(X_test)
After submitting y_pred to kaggle, I attained a score of 0.50516 in public score and 0.51187 in private leaderboard. Not remotely good enough but it is a start.
Disclaimer: fastai has a similar, more optimized and efficient library for this. My implementation is a concise version which can help beginner to go through the whole code and understand each implementation under the hood. All documentation can also be found in the GitHub (https://github.com/Benlau93/easy_processing).
Thanks for reading.