Feature Scaling — Effectively Choose Input Variables Based on Distributions | by Sushmitha Pulagam | Towards Data Science

We often come across a situation dealing with a variety of numerical variables consisting of different ranges, units, and magnitudes while building an ML model. As a common practice, we will apply Standardization or Normalization techniques for all the features before building a model. However, it is crucial to study the distributions of the data before making a decision on which technique to apply for feature scaling.
In this article, we will go through the difference between Standardization and Normalization along with understanding the distributions of the data. In the end, we will see how to select the strategies based on Gaussian and Non-Gaussian distribution of the features to improve the performance of the Logistic Regression model.
Both these techniques are sometimes used interchangeably but they refer to different approaches.
Standardization: This technique transforms the data to have a mean of zero and a standard deviation to 1.
Normalization: This technique transforms the values in variables between 0 and 1.
We are using the Pima Indian Diabetes dataset and you can find the same [here]
import pandas as pdimport numpy as npdata = pd.read_csv(“Pima Indian Diabetes.csv”)data.head()
From the above, we can see that the numerical variables are varying in different ranges and the Outcome is the target variable. We will perform both the scaling techniques and apply Logistic Regression.
👉 Applying Standardization to all features and modeling.
From the sklearn library, we need to use StandardScaler to implement Standardization.
from sklearn.preprocessing import StandardScalerY = data.OutcomeX = data.drop("Outcome", axis = 1)columns = X.columnsscaler = StandardScaler()X_std = scaler.fit_transform(X)X_std = pd.DataFrame(X_std, columns = columns)X_std.head()
Let us do the train and test the split for the standardized features.
from sklearn.model_selection import train_test_splitx_train, x_test, y_train, y_test = train_test_split(X_std, Y, test_size = 0.15, random_state = 45)
Now we are going to apply Logistic Regression on the standardized dataset.
#Building Logistic Regression model on the Standardized variablesfrom sklearn.linear_model import LogisticRegressionlr_std = LogisticRegression()lr_std.fit(x_train, y_train)y_pred = lr_std.predict(x_test)print('Accuracy of logistic regression on test set with standardized features: {:.2f}'.format(lr_std.score(x_test, y_test)))
From the above, we can see that the accuracy of the model with all the features applying Standardization technique is 72 percent.
👉 Applying Normalization to all features and modeling.
From the sklearn library, we need to use MinMaxScaler to implement Normalization.
from sklearn.preprocessing import MinMaxScalernorm = MinMaxScaler()X_norm = norm.fit_transform(X)X_norm = pd.DataFrame(X_norm, columns = columns)X_norm.head()
Let us do the train and test the split for the normalized features.
# Train and Test split of Normalized featuresfrom sklearn.model_selection import train_test_splitx1_train, x1_test, y1_train, y1_test = train_test_split(X_norm, Y, test_size = 0.15, random_state = 45)
Applying Logistic Regression on the normalized dataset.
#Building Logistic Regression model on the Normalized variablesfrom sklearn.linear_model import LogisticRegressionlr_norm = LogisticRegression()lr_norm.fit(x1_train, y1_train)y_pred = lr_norm.predict(x1_test)print(‘Accuracy of logistic regression on test set with Normalized features: {:.2f}’.format(lr_norm.score(x1_test, y1_test)))
The accuracy of the model when all the features are normalized is 74 percent.
👉 Understanding the distribution of features
Let us plot the histograms of the variables to study the distribution.
# Plotting the histograms of each variablefrom matplotlib import pyplotdata.hist(alpha=0.5, figsize=(20, 10))pyplot.show()
Gaussian Distribution — BMI, BloodPressure, Glucose.
Non-Gaussian Distribution — Age, DiabetesPedigreeFunction, Insulin, Pregnancies, SkinThickness
Finally, we came to an experiment waiting to select the variables and apply both the strategies based on the distributions on the same dataset.
To apply this strategy, we are going to use Column Transformer and Pipeline concepts from sklearn as we need to do the mixed type of techniques by subsetting the columns.
As mentioned above, we are initiating different pipelines for Gaussian and Non-Gaussian features
from sklearn.compose import ColumnTransformerfrom sklearn.pipeline import PipelineStandardize_Var = ['BMI','BloodPressure', 'Glucose']Standardize_transformer = Pipeline(steps=[('standard', StandardScaler())])Normalize_Var = ['Age','DiabetesPedigreeFunction','Insulin','Pregnancies','SkinThickness']Normalize_transformer = Pipeline(steps=[('norm', MinMaxScaler())])
Now, let us build the Logistic Regression model on the data with selective features for Standardization and Normalization.
x2_train, x2_test, y2_train, y2_test = train_test_split(X, Y, test_size=0.2)preprocessor = ColumnTransformer(transformers= [(‘standard’, Standardize_transformer, Standardize_Var), (‘norm’, Normalize_transformer, Normalize_Var)])clf = Pipeline(steps=[(‘preprocessor’, preprocessor), (‘classifier’, LogisticRegression(solver=’lbfgs’))])clf.fit(x2_train, y2_train)print(‘Accuracy after standardizing Gaussian distributed features and normalizing Non-Gaussian features: {:.2f}’.format(clf.score(x2_test, y2_test)))
👉 Final key details
Below is the accuracy details for the different models we have built so far.
Accuracy after standardizing all the features: 0.72
Accuracy after normalizing all the features: 0.74
Accuracy after applying Standardization to Gaussian distribution features and Normalization to Non- Gaussian distribution features: 0.79
We need to perform Feature Scaling when we are dealing with Gradient Descent Based algorithms (Linear and Logistic Regression, Neural Network) and Distance-based algorithms (KNN, K-means, SVM) as these are very sensitive to the range of the data points. This step is not mandatory when dealing with Tree-based algorithms.
The main focus of this article is to explain how the distribution of the data plays an important role in feature scaling and how to select the strategies based on Gaussian and Non-Gaussian distribution to improve the overall accuracy of the model.
You can get the complete code from my GitHub [profile]
Thank you for reading and Happy Learning! 🙂