The Ultimate Python Package to Pre-Process Data for Machine Learning
Tags: Machine Learning, Package, Data Science, Data Preprocessing, Python

Selected usage examples

To see the full usage documentation, click here

The examples below are shown using this car rental dataset that contains the following features:

year: Year of the car when it was bought

selling_price: Price at which the car is being sold

km_driven: Number of Kilometres the car is driven

fuel: Fuel type of car (petrol / diesel / CNG / LPG / electric)

seller_type: Tells if a Seller is Individual or a Dealer

transmission: Gear transmission of the car (Automatic/Manual)

owner: Number of previous owners of the car.

Sample of the dataset

Missing Values:

When a feature has a lot of missing values it might become irrelevant to use in the model because it contains very little information. For this sake, ‘NBprocessing’ provides a helpful function that plots the amount and percent of missing values per feature in a table.

As can see, only features with missing values were plotted to the table.

Fill missing values by categories ratio:

One of the hardest parts in dealing with missing values is what should one do when having missing values in a categorical feature. This function will take the categories ratio, without the missing values, and fill them by keeping this ratio. For example the ‘Fuel’ feature in our dataset:

On the right side, we can see the portion of each category, where the missing values are part of the division. On the left side, the division is without the missing value portion. The result that we would like to achieve is keeping the left ratio without having missing values.

As can see from the above, successfully fill the missing values, and keep the categories ratio.

Category Ratio:

This function enables the user to better understand the distribution of the categories in each feature. It plots a table with all the input features and the portion of every category in the feature. To better see imbalanced features it plots 90% and higher categories in red color. If no feature was selected, it will plot all the dataset features.

Outliers:

Outliers are the noise points in the data that are usually very remote from the feature mean. ‘NBprocessing’ provides two helpful functions that enable the user to drop the outliers, after informing how much data will be lost by conducting this action.

The “Hammer” method

When having normally distributed data, 99.7% of it will be in the boundary of 3 standard deviations from the mean. Thus, by removing the remaining 0.3% we will drop the outliers and keep most of the data. However, features are not always normally distributed and a lot have only long head or back tail. Thus, this function provides the user to choose what the percent of data to drop — top and bottom limits.

‘km_driven’ feature distribution before dropping outliers

As can see from the above, the ‘km_driven’ feature has a long head-tail without a back-tail. Thus, we only have outliers in the top boundary.

‘km_driven’ feature distribution after dropping outliers

The “Tweezers” method

In this case, we would like to drop outliers by giving the function the top and bottom value limits.

First, the user provides the top and bottom bounders using a dictionary with the features name as keys. Then, will check how much data is out of the boundaries and will be dropped. If it’s a reasonable amount the user will be able to use the boundaries to drop the missing values.

Only 1.32% will be lost by conducting this action — will proceed.

Summary

NBprocessing is a very powerful package that has a lot to offer to the Data Science ecosystem. It provides tools to Explore, Process, Plot, and Model tabular data without copy and pasted code from one project to the other. It has many more functions that are shown in this Jupyter Notebook.

If you would like to become a contributor and help grow and promote this package — please don’t hesitate to DM me here, on Github or LinkedIn.

Would like to thank Israel Tech Challange (ITC) for providing me the platform to create this package and to Shir Meir Lador for the inspiration.