Understand Data Normalization in Machine Learning | by Zixuan Zhang | Towards Data Science

If you’re new to data science/machine learning, you probably wondered a lot about the nature and effect of the buzzword ‘feature normalization’. If you’ve read any Kaggle kernels, it is very likely that you found feature normalization in the data preprocessing section. So, what is data normalization and why the heck is it so valued by data practitioners?
There are different types of data normalization. Assume you have a dataset X, which has N rows(entries) and D columns(features). X[:,i] represent feature i and X[j,:] represent entry j. We have:
Z Normalization(Standardization):
I used to falsely think this method somehow yields a standard Gaussian result. In fact, standardization does not change the type of distribution:
This transformation sets the mean of data to 0 and the standard deviation to 1. In most cases, standardization is used feature-wise
Min-Max Normalization:
This method rescales the range of the data to [0,1]. In most cases, standardization is used feature-wise as well
Unit Vector Normalization:
Scaling to unit length shrinks/stretches a vector (a row of data can be viewed as a D-dimensional vector) to a unit sphere. When used on the entire dataset, the transformed data can be visualized as a bunch of vectors with different directions on the D-dimensional unit sphere.
Wow, normalization is indeed a broad term and each of them has pros and cons! I ’ll only focus on standardization in this article otherwise this article will go way too long.
In theory, regression is insensitive to standardization since any linear transformation of input data can be counteracted by adjusting model parameters. Let’s say we have a linear model with parameters W and B:
Column-wise change can be achieved by subtracting a matrix M and multiplying with a diagonal matrix T
It’s easy to show that we can counteract this transformation by adjusting W and B
Therefore, standardization of input data should not affect the output or accuracy. Well, if the math says standardization plays little role in regression, why is it still so popular?
a). Standardization improves the numerical stability of your model
If we have a simple one-dimensional data X and use MSE as the loss function, the gradient update using gradient descend is:
X is in the gradient descent formula, which means the value of X determines the update rate. Therefore, a larger X will lead to a greater leap in the gradient landscape. Meanwhile, a larger X leads to smaller W, given Y:
When X is large, the distance between the initial W (which is randomly picked) and the global minimum is very likely to be small. Therefore, the algorithm is more likely to fail when X is larger (learning rate is fixed) because the algorithm makes giant leaps toward the very close target W while baby steps are needed. This overshooting will make your loss oscillate or explode.
Experiment:
I artificially created a dataset using the following code:
X = np.arange(-5,5,0.1)w = 0.5b = 0.2Y = X*w+b+(np.random.rand(X.shape[0],)-0.5)#add some noise to YX.reshape(100,1)Y.reshape(100,1)
A simple linear regression code I used for the experiment:
class linear_regression:    def fit(self, X,Y, lr=0.00001):        epoch = 0        loss_log = []        while (epoch<8000):            pred = self.predict(X)            l = self.loss(pred,Y)# get loss            loss_log.append(l)            delta_w = 2*np.matmul((Y-pred).T, X) # gradient of w            delta_b = sum(2*(Y-pred))            self.w = self.w + lr*delta_w            self.b = self.b + lr*delta_b            epoch = epoch+1        plt.plot(loss_log)        plt.ylim([0.07,0.1])        print (loss_log[-5:-1])    def loss(self, pred,Y):        error = Y-pred        return sum(error**2)/error.shape[0] # MSE    def predict(self, X):        return X*self.w+self.b# n by 1    def __init__(self):# initial parameters        self.w=0.5# fixed initialization for comparison convenience        self.b=0.5
The following table shows the final MSE with respect to different learning rate and feature range.
When X has a larger coefficient(therefore large range), the model needs a smaller learning rate to keep stability.
b). Standardization may speed up the training process
A corollary to the first ‘theorem’ is that if different features have drastically different ranges, the learning rate is determined by the feature with the largest range. This leads to another advantage of standardization: speeds up the training process.
Let’s say we have two features:
The learning rate determined by X2 isn’t ‘great’ for X1 since X1 is okay with much larger learning rate. We are taking baby steps while giant leaps could have been used instead, which leads to longer training time. We can reduce the training time by standardizing these two features.
Experiment:
I artificially created a dataset with two features using the following code:
X1 = np.arange(-5,5,0.1)X2 = np.arange(-5,5,0.1)X = np.column_stack((X1,X2))W = np.array([[0.1],[0.2]])b = 0.1Y = np.matmul(X,W)+b+(np.random.rand(X.shape[0],1) -0.5)# add some noise to labels
With slightly modified linear regression code:
class linear_regression:    def fit(self, X,Y, lr=1e-4):        epoch = 0        loss_log = []        while (epoch<80000):            pred = self.predict(X)            l = self.loss(pred,Y)# get loss            loss_log.append(l)            if l < 0.0792:                print (epoch)                break            delta_w = 2*np.matmul((Y-pred).T, X) # gradient of w            delta_b = sum(2*(Y-pred))            self.w = self.w + lr*delta_w.T            self.b = self.b + lr*delta_b            epoch = epoch+1        plt.plot(loss_log)        plt.ylim([0.07,0.1])        print (loss_log[-5:-1])    def loss(self, pred,Y):        error = Y-pred        return sum(error**2)/error.shape[0] # MSE    def predict(self, X):        return np.matmul(X,self.w)+self.b# n by 1    def __init__(self):        self.w=np.array([[0.5],[0.5]])# for comparison convenience        self.b=0.3
The following table shows the number of iterations needed to reach the global minimum for a given coefficient and learning rate:
We can see that when the data isn’t standardized, the model needs to take more iterations (training time) to complete.
Distance-based clustering algorithms such as K-mean and K-NN are very likely to be affected by standardization. Clustering algorithm needs to calculate distances between entries. The most common distance used is the Euclidean distance:
It is obvious that feature scaling will change the numeric distances between nodes.
Experiment:
Create a dataset using the following code:
def random_2D_data(x,y,size):    x = (np.random.randn(size)/3.5)+x    y = (np.random.randn(size)*3.5)+y    return x,yx1,y1 = random_2D_data(2,20,50)x2,y2 = random_2D_data(2,-20,50)x3,y3 = random_2D_data(-2,20,50)x4,y4 = random_2D_data(-2,-20,50)x = np.concatenate((x1,x2,x3,x4))y = np.concatenate((y1,y2,y3,y4))
Cluster the data as is using K-mean:
However, doing the same thing on the standardized data yields a totally different result:
Great, we have results contradict to each other. Which result should I pick?
Standardization gives ‘equal’ considerations for each feature.
For instance, X has two feature x1 and x2
If you calculate the Euclidean distance directly, node 1 and 2 will be further apart than node 1 and 3. However, node 3 is totally different from 1 while node 2 and 1 are only different in feature 1 (6%) and the share the same feature 2. This is because feature 1 is the ‘VIP’ feature, dominating the result with its large numerical value.
Therefore, if we don’t know which features are ‘diamond’ features and which are ‘coral’ features, it is a good practice to consider them equally using standardization.
Tree-based decision algorithms like decision trees or random forests seek for the best split point in each feature. The split point is determined by the percentage of labels correctly classified using a feature, which is resilient to feature scaling. Therefore, standardization doesn’t have any significant impact on this type of ML models.
Neural networks can counteract standardization in the same way as regressions. Therefore, in theory, data standardization should not affect the performance of a neural network. However, empirical evidence shows that data standardization is beneficial in terms of accuracy [1]. Currently, I don’t yet see the reason but maybe it’s related to gradient descending.
It is easy to understand why standardization improves training time. Large input values saturate activation functions such as sigmoid or ReLu(negative input). These types of activation functions feedback small or no gradient at all in the saturated region and therefore slows down training.
Standardization is beneficial in many cases. It improves the numerical stability of the model and often reduces training time. However, standardization isn’t always great. It can harm the performance of distance-based clustering algorithms by assuming equal importance of features. If there are inherent importance differences between features, it’s generally not a good idea to do standardization.
Reference:
[1]: Shanker M, Hu MY, Hung MS, Effect of Data Standardization on Neural Network Training, https://www.sciencedirect.com/science/article/pii/0305048396000102