Feature Scaling — Effect Of Different Scikit-Learn Scalers: Deep Dive | by Kaushik Choudhury | Towards Data Science

In supervised machine learning, we calculate the value of the output variable by supplying input variable values to an algorithm. Machine learning algorithm relates the input and output variable with a mathematical function.
Output variable value = (2.4* Input Variable 1 )+ (6*Input Variable 2) + 3.5
There are a few specific assumptions behind each of the machine learning algorithms. To build an accurate model, we need to ensure that the input data meets those assumptions. In case, the data fed to machine learning algorithms do not satisfy the assumptions then prediction accuracy of the model is compromised.
Most of the supervised algorithms in sklearn require standard normally distributed input data centred around zero and have variance in the same order. If the value range from 1 to 10 for an input variable and 4000 to 700,000 for the other variable then the second input variable values will dominate and the algorithm will not be able to learn from other features correctly as expected.
In this article, I will illustrate the effect of scaling the input variables with different scalers in scikit-learn and three different regression algorithms.
In the below code, we import the packages we will be using for the analysis. We will create the test data with the help of make_regression
from sklearn.datasets import make_regressionimport numpy as npfrom sklearn.model_selection import train_test_splitfrom  sklearn.preprocessing import *from sklearn.linear_model import*
We will use the sample size of 100 records with three independent (input) variables. Further, we will inject three outliers using the method “np.random.normal”
X, y, coef = make_regression(n_samples=100, n_features=3,noise=2,tail_strength=0.5,coef=True, random_state=0)X[:3] = 1 + 0.9 * np.random.normal(size=(3,3))y[:3] = 1 + 2 * np.random.normal(size=3)
We will print the real coefficients of the sample datasets as a reference and compare with predicted coefficients.
print("The real coefficients are ". coef)
We will train the algorithm with 80 records and reserve the remaining 20 samples unseen by the algorithm earlier for testing the accuracy of the model.
X_train, X_test, y_train, y_test = train_test_split(X, y,           test_size=0.20,random_state=42)
We will study the scaling effect with the scikit-learn StandardScaler, MinMaxScaler, power transformers, RobustScaler and, MaxAbsScaler.
regressors=[StandardScaler(),MinMaxScaler(),PowerTransformer(method='yeo-johnson'),RobustScaler(quantile_range=(25,75)),MaxAbsScaler()]
All the regression model we will be using is mentioned in a list object.
models=[Ridge(alpha=1.0),HuberRegressor(),LinearRegression()]
In the code below, we scale the training and test sample input variable by calling each scaler in succession from the regressor list defined earlier. We will draw a scatter plot of the original first input variable and scaled the first input variable to get an insight on various scaling. We see each of these plots little later in this article.
Further, we fit each of the models with scaled input variables from different scalers and predict the values of dependent variables for test sample dataset.
for regressor in regressors:    X_train_scaled=regressor.fit_transform(X_train)    X_test_scaled=regressor.transform(X_test)    Scaled =plt.scatter(X_train_scaled[:,0],y_train, marker='^', alpha=0.8)    Original=plt.scatter(X_train[:,0],y_train)    plt.legend((Scaled, Original),('Scaled', 'Original'),loc='best',fontsize=13)    plt.xlabel("Feature 1")    plt.ylabel("Train Target")             plt.show()    for model in models:        reg_lin=model.fit(X_train_scaled, y_train)        y_pred=reg_lin.predict(X_test_scaled)           print("The calculated coeffiects with ", model , "and", regressor, reg_lin.coef_)
Finally, the predicted coefficients from the model fit are printed for the comparison with real coefficients.
On first glance itself, we can deduce that same regression estimator predicts different values of the coefficients based on the scalers.Predicted coefficients with MaxAbsScaler and MinMax scaler is quite far from true coefficient values.We can see the importance of appropriate scalers in the prediction accuracy of the model from this example.
As a self-exploration and learning exercise, I will encourage you all to calculate the R2 score and Root Mean Square Error (RMSE) for each of the training and testing set combination and compare it with each other.
Now that we understand the importance of scaling and selecting suitable scalers, we will get into the inner working of each scaler.
Standard Scaler: It is one of the popular scalers used in various real-life machine learning projects. The mean value and standard deviation of each input variable sample set are determined separately. It then subtracts the mean from each data point and divides by the standard deviation to transforms the variables to zero mean and standard deviation of one. It does not bound the values to a specific range, and it can be an issue for a few algorithms.
MinMax Scaler: All the numeric values scaled between 0 and 1 with a MinMax Scaler
Xscaled= (X-Xmin)/(Xmax-Xmin)
MinMax scaling is quite affected by the outliers. If we have one or more extreme outlier in our data set, then the min-max scaler will place the normal values quite closely to accommodate the outliers within the 0 and 1 range. We saw earlier that the predicted coefficients with MinMax scaler are approximately three times the real coefficient. I will recommend not to use MinMax Scaler with outlier dataset.
Robust Scaler- Robust scaler is one of the best-suited scalers for outlier data sets. It scales the data according to the interquartile range. The interquartile range is the middle range where most of the data points exist.
Power Transformer Scaler: Power transformer tries to scale the data like Gaussian. It attempts optimal scaling to stabilize variance and minimize skewness through maximum likelihood estimation. Sometimes, Power transformer fails to scale Gaussian-like results hence it is important to check the plot the scaled data
MaxAbs Scaler: MaxAbsScaler is best suited to scale the sparse data. It scales each feature by dividing it with the largest maximum value in each feature.
For example, if an input variable has the original value [2,-1,0,1] then MaxAbs will scale it as [1,-0.5,0,0.5]. It divided each value with the highest value i.e. 2. It is not advised to use with large outlier dataset.
We have learnt that scaling the input variables with suitable scaler is as vital as selecting the right machine learning algorithm. Few of the scalers are quite sensitive to outlier dataset, and others are robust. Each of the scalers in Scikit-Learn has its strengths and limitations, and we need to be mindful of it while using it.
It also highlights the importance of performing the exploratory data analysis (EDA) initially to identify the presence or absence of outliers and other idiosyncrasies which will guide the selection of appropriate scaler.
In my article, 5 Advanced Visualisation for Exploratory data analysis (EDA) you can learn more about this area.
In case, you would like to learn a structured approach to identify the appropriate independent variables to make accurate predictions then read my article “How to identify the right independent variables for Machine Learning Supervised.
"""Full Code"""from sklearn.datasets import make_regressionimport numpy as npfrom sklearn.model_selection import train_test_splitfrom  sklearn.preprocessing import *from sklearn.linear_model import*import matplotlib.pyplot as pltimport seaborn as snsX, y, coef = make_regression(n_samples=100, n_features=3,noise=2,tail_strength=0.5,coef=True, random_state=0)print("The real coefficients are ", coef)X[:3] = 1 + 0.9 * np.random.normal(size=(3,3))y[:3] = 1 + 2 * np.random.normal(size=3)X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.20,random_state=42)regressors=[StandardScaler(),MinMaxScaler(),PowerTransformer(method='yeo-johnson'),RobustScaler(quantile_range=(25, 75)),MaxAbsScaler()]models=[Ridge(alpha=1.0),HuberRegressor(),LinearRegression()]for regressor in regressors:    X_train_scaled=regressor.fit_transform(X_train)    X_test_scaled=regressor.transform(X_test)    Scaled =plt.scatter(X_train_scaled[:,0],y_train, marker='^', alpha=0.8)    Original=plt.scatter(X_train[:,0],y_train)    plt.legend((Scaled, Original),('Scaled', 'Original'),loc='best',fontsize=13)    plt.xlabel("Feature 1")    plt.ylabel("Train Target")             plt.show()    for model in models:        reg_lin=model.fit(X_train_scaled, y_train)        y_pred=reg_lin.predict(X_test_scaled)           print("The calculated coeffiects with ", model , "and", regressor, reg_lin.coef_)