Data Preprocessing- A significant step in Machine Learning
By: Saurav Anand
Tags: Machine Learning, Data Science, Python

“ Data Preprocessing is not advantageous it is absolutely essential “

Introduction

Data Pre-processing is the most important step in the data mining process. It transforms the raw data into an understandable format. Data in the Real world data is :-

The phrase “Garbage In ,Garbage Out” is particularly applicable to machine learning and data mining.

If there is much irrelevant and redundant data or noisy and unreliable data then it is difficult to understand the pattern and our machine learning algorithm will not fit well to predict the correct estimations.(No Quality data , no correct predictions).

Data Preprocessing in Machine Learning Process

Data Pre-processing Methods

There are various methods which deals with the preparation and transformation of the initial dataset. Methods are divided into following categories :

Data Cleaning :- Fill in the missing values,smooth noisy data,identify or remove outliers and resolve inconsistencies.

Data Integration:-Integration of multiple databases,data cubes, or files.

Data Transformation:-Normalization and Aggregation

Data Reduction:-Obtains reduced representation in volume but produces the same or similar analytical results.

Data Preprocessing steps in Machine Learning:-

Step 1: Importing Libraries

The most important step is to import the libraries required for the program.Libraries contain important classes which contains certain built in methods. These methods can be called and used by the program.The below diagram shows how to import the libraries in python:-

These are the most commonly used libraries in python.

Numpy:- It is the fundamental package for scientific computing and all mathematical operations with Python.

Pandas:-It is used for data manipulation and analysis. Pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.

Matplotlib:- Python 2D plotting library which produces publication quality figures in a variety of hard copy formats and interactive environments across platforms.

Seaborn:- It is another Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics.

Step 2:- Importing the dataset/Creating a Data Frame

In the real world applications , the data set is provided and we need to import it using pandas library . Below we have imported the csv file and reading it.

Now we need to divide the data into X and y respectively . “X” represents the matrix of features containing the independent variables and “y” represents the vector containing the dependent variables.To read the values from column we use the iloc method of pandas along with array indexing and slicing concept.

Step 3:- Handling the Missing Data

Handling of missing data is very important otherwise the interpretation about the data will be wrong and inaccurate. Missing data can be handled in the following ways:-

a) Drop Missing values :- This is the most easiest and fastest way of handling missing values.It reduces the quality of our model as it reduces sample size because it works by deleting all other observations where any of the variable is missing.

Note :-Dropping is only advised to be used if missing values are few (say 0.01–0.5% of our data). Percent is just a rule of thumb.

b) Impute Missing Values with Most Frequent:- In this case the missing values are replaced with the Most Frequent occurred value. Below is the implementation:-

c)Impute Missing Values With Means:- Replacing the missing values with Means.Below is the implementation:-

d) Imputing the missing values using K-nearest neighbor:- In this case, the missing values are replaced by the predictions.

Step 4:-Encoding the categorical data

Sometimes the data present is in the form of text and are in the qualitative format. We need to identify the column of the data which are categorical (True/False,Yes/No or any depicting any class,group etc) . This process is done to encode the data into numerical type from text type.

Consider the following data set of independent variables, here we see the variable “Country” as the categorical data.it is in textual form instead of numerical form.

To convert Categorical variable into Numerical data we can use LabelEncoder() class from preprocessing library.

As you can see above , LabelEncoder() has done it’s job and successfully converted the text values to numerical . But still the problem is not solved . As per the conversion , we have following observation:-

0 refers to France, 1 refers to Germany and 2 refers to Spain respectively.

Before talking about the solution , let’s understand the type of the categorical values :-

There are two types of categorical values :-

Nominal Categorical Values :-These variables cannot be ordered. Example name of the country(above example), movie genres(drama, fiction,comedy etc )

Ordinal Categorical Values:- These variables can be ordered. Example of economic status which is lower, medium and poor.Here we can arrange them

lower <medium<poor

In the above case , the equations in the model will think that Spain has a higher value than Germany and France, and Germany has a higher value than France.

But this is not true these are actually three Categories and there is no relational order between the three.So we are going to use the Dummy variables.

Dummy variable Explanation

A Dummy variable or Indicator Variable is an artificial variable created to represent an attribute with two or more distinct categories/levels.

Instead of having one column here above, we are going to have three column.

Number of Columns =No of Categories

It’s going to be 3 in this case. To accomplish the task, we will import yet another library called OneHotEncoder.

OneHotEncoder takes an important parameter called categorical_features which takes the value of the index of the column of categories.

The code above will select the first column to OneHotEncode the categories.

We will use the fit_transform() with OneHotEncoder along with the toarray()

The data set contains the categories (the first column)is encoded as 0’s and 1’s into first three columns.

Step 5: Splitting the Dataset into Training set and Test Set

In machine learning , we divide the data into training set and test set respectively.

Training set :-It is the subset of your data on which your model will learn how to predict the dependent variable with the independent variables.

Test set:- It is the complementary subset from the training set . on which you evaluate your model to check if it manages to predict correctly the dependent variable with the independent variables.

A general rule of the thumb is to allocate 80% of the dataset to training set and the remaining 20% to test set. It is because if we have only some values in the training set , our model wouldn’t be able to learn any correlation between the independent and dependent variables.

For this task, we will import test_train_split from model_selection library of scikit.

We will be creating four sets of data

X_train is the training part of the matrix of features.

X_test is the test part of the matrix of features.

y_train is the training part of the dependent variable that is associated to X_train here.

y_test is the test part of the dependent variable that is associated to X_train here.

We will assign to them the test_train_split, which takes the parameters — arrays (X and Y), test_size (if we give it the value 0.5, meaning 50%, it would split the dataset into half. Since an ideal choice is to allocate 20% of the dataset to test set, it is usually assigned as 0.2. 0.25 would mean 25%, just saying).

Step 6:- Feature Scaling

Consider the following data set

In this set , the values in the Age and Salary are not in the same scale which is not good for our model. Most of the models are based on the Euclidean distance.

Euclidean Formula

Euclidean _graphical presentation

In the machine learning equations, the square difference with the lower value in comparison to the far greater value will almost be treated as if it does not exist.

Feature Scaling methods :-

Normalization:-It scales features between 0 and 1,retaining there proportional range between each other.

Standardization:-It scales features to have a mean of 0 and standard deviation of 1.

We will be importing the class StandardScaler from the sckit preprocessing library and as usual create an object of that class.

Now we will fit and transform our X_train set

When should we use Standardization and Normalization?

Generally we should use normalize when the data is normally distributed and scale (Standardization) when the data is not normally distributed.However what is commonly done is that the two scaling methods are tested.

Conclusion

Thus we can say that pre-processing in ML is to improve generalization of our model.

For achieving better results from the applied model in Machine Learning projects the format of the data has to be in a proper manner. Some specified Machine Learning model needs information in a specified format which we will discuss in the coming blogs.

Another aspect is that data set should be formatted in such a way that more than one Machine Learning and Deep Learning algorithms are executed in one data set, and best out of them is chosen.

Reference :-