Data normalization in machine learning
By: Mahbubul Alam
Tags: Artificial Intelligence, Machine Learning, Data Science, Normalization, Transformation

I wrote about cluster analysis in the previous article (Clustering: concepts, tools and algorithms), where I had a short discussion on data normalization. I touched upon how data normalization impacts clustering, and unsupervised algorithms generally.

But I felt that I missed the opportunity to go into more details. But of course, that wasn’t the focus of that article, so today I want to pick up on that.

What is normalization?

Let’s first define what exactly is normalization.

Let’s say we have a dataset containing two variables: time traveled and distance covered. Time is measured in hours (e.g. 5, 10, 25 hours ) and distance in miles (e.g. 500, 800, 1200 miles). Do you see the problem?

One obvious problem of course is that these two variables are measured in two different units — one in hours and the other in miles. The other problem — which is not obvious but if you take a closer look you'll find it — is the distribution of data, which is quite different in these two variables (both within and between variables).

The purpose of normalization is to transform data in a way that they are either dimensionless and/or have similar distributions. This process of normalization is known by other names such as standardization, feature scaling etc. Normalization is an essential step in data pre-processing in any machine learning application and model fitting.

Does normalization help?

Now the question is how (on earth) exactly does this transformation help?

The short answer is — it dramatically improves model accuracy.

Normalization gives equal weights/importance to each variable so that no single variable steers model performance in one direction just because they are bigger numbers.

As an example, clustering algorithms use distance measures to determine if an observation should belong to a certain cluster. “Euclidean distance” is often used to measure those distances. If a variable has significantly higher values, it can dominate distance measures, suppressing other variables with small values.

What tools and techniques are used?

Several methods are applied for normalization, three popular and widely used techniques are as follows:

Rescaling: also known as “min-max normalization”, it is the simplest of all methods and calculated as:

Mean normalization: This method uses the mean of the observations in the transformation process:

Z-score normalization: Also known as standardization, this technic uses Z-score or “standard score”. It is widely used in machine learning algorithms such as SVM and logistic regression:

Here, z is the standard score, µ is the population mean and ϭ is the population standard deviation.

Show an example

Let’s do an experiment, it’s always good to see an algorithm in action. The example is not going to be dramatic, I’m just showing it as an illustration to give an intuition.

Let’s import some libraries — pandas for data wrangling, matplotlib for visualization and preprocessing and KMeans from the sklearn library.

Let’s also import data from a GitHub repo as a csv file. That’s the Iris dataset, already cleaned, so you can import and follow along right away.

I’m going to use two features — petal_length and sepal_length — for clustering of data points.

# importing libraries

import pandas as pd

import matplotlib.pyplot as plt

from sklearn import preprocessing

from sklearn.cluster import KMeans

df=pd.read_csv(" # importing **cleaned** datadf=pd.read_csv(" iris.csv ") # feature selection

df = df[["petal_length", "sepal_length"]]

After importing libraries and data, first we’ll implement the KMeans clustering algorithm without normalization. I’ve annotated each line of code so you know what’s going on.

# inputs (NOT normalized)

X_not_norm = df.values # instantiate model

model = KMeans(n_clusters = 3) # fit predict

y_model = model.fit_predict(X_not_norm) # visualizing clusters

plt.scatter(X_not_norm[:,0], X_not_norm[:,1], c=model.labels_, cmap='viridis') # counts per cluster

print("Value Counts")

print(pd.value_counts(y_model))

The outputs are the count of data points in each cluster and the visualization of those clusters.

Now let’s re-run the model, this time after normalizing the inputs using preprocessing from the sklearn library.

# normalizing inputs

X_norm = preprocessing.scale(df) # instantiate model

model = KMeans(n_clusters = 3) # fit predict

y_model = model.fit_predict(X_norm) print("Value Counts")

print(pd.value_counts(y_model)) # visualize clusters

plt.scatter(X_norm[:,0], X_norm[:,1], c=model.labels_, cmap='viridis')

In the following are the outputs before and after the normalization of data. First, if you compare the value counts there are some changes — for example, counts cluster 0 is reduced by 4 members.

If you closely examine the data points in the left and the right figures you might be able to see which data points shifted from pre-normalized to post-normalized model. These changes are often at the boundaries rather than at either end of the spectrum in the distribution. Again, as I said, it’s not too dramatic, but you get the point.

Differences of clustering before and after normalization (source: Author)

The other side of the coin…..

So far we’ve got the impression that normalization is absolutely a great thing to happen in data science! Not really, it does some good things but creates some bad side effects along the way.

Normalization compresses data within a certain range, reduces the variance and applies equal weights to all features. You lose a lot of important information in the process.

One example is what happens to outliers — normalization lease absolutely no traces of outliers. We perceive outliers as bad guys and we need to get rid of them ASAP. But remember, outliers are real data points, once you lose that just to get a better model, you lose information.

In the process of normalization, the variables lose their units of measurements too. So at the end of modeling, you can’t really tell what are the key differences between the variables.

I hope this was a useful article. If you have comments feel free to write them down below. You can follow me on Medium, Twitter or LinkedIn.