Data Normalization with Python scikit-learn | by Angelica Lo Duca | Towards Data Science

Following the series of publications on data preprocessing, in this tutorial, I deal with Data Normalization in Python scikit-learn . As already said in my previous tutorial, Data Normalization involves adjusting values measured on different scales to a common scale.
Normalization applies only to columns containing numeric values. Five methods of normalization exist:
single feature scaling
min max
z-score
log scaling
clipping
In this tutorial, I use the scikit-learn library to perform normalization, while in my previous tutorial, I dealt with data normalization using the pandas library. I use the same dataset used in my previous tutorial, thus results can be compared. Indeed, we obtain the same results using the two methodologies.
Anyway, in this tutorial I don’t deal with log scaling and clipping, which will be the object of a future tutorial.
The scikit-learn library can be used also to deal with missing values, as explained in my previous post.
All the scikit-learn operations described in this tutorial follow the following steps:
select a preprocessing methodology
fit it through the fit() function
apply it to data through the transform() function.
The scikit-learn library works only with arrays, thus when performing every operation, a dataframe column must be converted to an array. This can be achieved through the numpy.array() function, which receives the dataframe column as input. The fit() function receives as input an array of arrays, each representing a sample of the dataset. Thus the reshape() function could be used to convert a standard array to an array of arrays.
All the code described in this tutorial can be downloaded from my Github repository.
As an example dataset, in this tutorial we consider the dataset provided by the Italian Protezione Civile, related to the number of COVID-19 cases registered since the beginning of the COVID-19 pandemic. The dataset is updated daily and can be downloaded from this link.
First of all, we need to import the Python pandas library and read the dataset through the read_csv() function. Then we can drop all the columns with NaN values. This is done through dropna() function.
import pandas as pddf = pd.read_csv('https://raw.githubusercontent.com/pcm-dpc/COVID-19/master/dati-regioni/dpc-covid19-ita-regioni.csv')df.dropna(axis=1,inplace=True)df.head(10)
Single Feature Scaling converts every value of a column into a number between 0 and 1. The new value is calculated as the current value divided by the max value of the column. This can be done through the MaxAbsScaler class. We apply the scaler to the tamponi column, which must be converted to array and reshaped.
import numpy as npfrom sklearn.preprocessing import MaxAbsScalerX = np.array(df['tamponi']).reshape(-1,1)scaler = MaxAbsScaler()
Now we can fit the scaler and then apply the transformation. We convert it to the original shape by applying the inverse reshape() function and we store the result into a new column of the dataframe df.
scaler.fit(X)X_scaled = scaler.transform(X)df['single feature scaling'] = X_scaled.reshape(1,-1)[0]
The scikit-learn library also provides a function to restore the original values, given the transformation. This function also works for the transformations described later in this article.
scaler.inverse_transform(X_scaled)
which gives the following output:
array([[5.000000e+00],       [0.000000e+00],       [1.000000e+00],       ...,       [5.507300e+05],       [6.654400e+04],       [3.643743e+06]])
Similar to Single Feature Scaling, Min Max converts every value of a column into a number between 0 and 1. The new value is calculated as the difference between the current value and the min value, divided by the range of the column values. In scikit-learn we use the MinMaxScaler class. For example, we can apply the min max method to the column totale_casi.
from sklearn.preprocessing import MinMaxScalerX = np.array(df['totale_casi']).reshape(-1,1)scaler = MinMaxScaler()scaler.fit(X)X_scaled = scaler.transform(X)df['min max'] = X_scaled.reshape(1,-1)[0]
Z-Score converts every value of a column into a number around 0. Typical values obtained by a z-score transformation range from -3 and 3. The new value is calculated as the difference between the current value and the average value, divided by the standard deviation. In scikit-learn we can use the StandardScaler function. For example, we can calculate the z-score of the column deceduti.
from sklearn.preprocessing import StandardScalerX = np.array(df['deceduti']).reshape(-1,1)scaler = StandardScaler()scaler.fit(X)X_scaled = scaler.transform(X)df['z score'] = X_scaled.reshape(1,-1)[0]
In this tutorial, I have illustrated how to normalize a dataset using the preprocessing package of the scikit-learn library. We can achieve the same results by manually applying the formulas, as explained in my previous tutorial.
The main advantages of using scikit-learn are the following:
you don’t need to know the formula in advance, because you have prepackaged classes to perform operations
you can restore the original data through inverse_transform() function.
If you wanted to learn how to perform the other aspects of data preprocessing using the scikit-learn library, stay tuned...
If you wanted to be updated on my research and other activities, you can follow me on Twitter, Youtube and Github.