Normalization, its types and Normalization layers
By: Sarthak Khandelwal
Tags: Normalization, Machine Learning, Research, Scikit Learn, TensorFlow

The article moves through understanding Normalization, different types of normalization to the discussion of some Normalization layers.

SATA— Would COVID-19 never end?

RAM— No dear! It would take time for things to be normalized.

In a similar manner, Data Science also employs the word Normalization.

Normalization in laymen terms is basically bringing something to normal or stable state. Well, in this article we would dig a little deeper about normalization, some of its types and different Normalization layers used to normalize the output of hidden layers in a neural network.

In context of machine learning and data science, Normalization is basically defined as the process of bringing the features or explanatory variables to a similar scale. Normalization is very important in terms of stability or accuracy of the learning model. Having features on the same scale also helps us get rid of the curse of dimensionality. Many machine learning practitioners consider normalization and standardization as different terms and many of them consider both as the same. However, I would go with the latter approach as one way or other, both are bringing the features onto a similar scale.

Normalization could be of various types. In this post, we would be considering 4 major types of normalizations —

1. Z-Score Normalization

2. Min-Max Scaling

3. Log Scaling

4. Feature Clipping

Z-Score Normalization:

In this normalization we subtract the feature values with their mean and divide by standard deviation. Due to this, the new mean becomes 0 and standard deviation becomes 1. This is the same as the Standard Normal Distribution.

image: author

We already have Z-score normalization implemented in scikit-learn’s preprocessing API with the name StandardScaler and works very well.

image: author; Visualizing the results of StandardScaler

As seen from above, most of the data is centered around the origin.

Min Max Scaling:

This normalization technique involves scaling the dataset to a range of [0, 1] by subtracting the feature columns with the minimum value of the particular column and dividing it by the difference of the minimum and maximum values of the same column. This normalization technique is useful when the distribution of features is somewhat uniform in nature.

image: author; X=feature column, X_min=minimum value of X, X_max=maximum value of X

Min Max Scaling is also implemented in scikit-learn’s preprocessing API with the name MinMaxScaler.

image: author; Visualizing the result of MinMaxScaler

Log Scaling:

Log Scaling is a slightly less used normalization technique. It is helpful when there is a large imbalance in the distribution of the feature values. Log Scaling changes the distribution and helps in improving the linear performance of the model. But it is not always true with all datasets. Sometimes Log Scaling may lead to sparsity in the dataset or we may loose some important feature information.

image: author

Although scikit-learn does not provide any function for Log Scaling but we can implement the function for Log Scaling very easily.

The resultant plot of the above code is visualized below. As we can see that the Log Scaled plot is more linear than the Original one and helps in improving the linear performance of the model.

image: author; Visualizing the results of Log Scaling

Feature Clipping:

Feature clipping is somewhat naïve normalization technique in which we are free to choose a range to put our feature values in. For example we can set the range to be [-5, 5] and clip the dataset in this range such that all values which are greater than 5 would be clipped at 5 and all the values which are less than -5 would be clipped at -5. Note that the values which are already in this range would be unaffected.

Let us also visualize the above code. We can see that most of the values are clipped at x_min and x_max and hence if we are performing this normalization, we have to be very much specific about the range. This technique is useful when we have extreme outliers so that our model becomes less sensitive to them.

image: author; Visualizing the results of Feature Clipping

So these were some of the types of Normalization techniques. There are also some other techniques implemented in scikit-learn such as RobustScaler, MaxAbsScaler, normalize etc. See the references for more details.

Pheww! You can take a break to digest things until now if you wish.