How to Differentiate Between Scaling, Normalization, and Log Transformations | by Bex T. | Towards Data Science

Learn the underlying difference between Standardization (Scaling), Normalization, and the Log transforms. Develop strong knowledge in when to apply each of them and choose one method over the other.
You have probably come across this in courses or articles:
The features in the dataset should conform to the statistical assumptions of the models.
What does it mean to conform to statistical assumptions? Many models implemented in Sklearn might perform poorly if the numeric features do not more or less follow a standard Gaussian (normal) distribution. Except for tree-based models, the objective function of Sklearn algorithms assumes the features follow a normal distribution.
Actually, using the word assume would be putting things lightly. For models like K-Nearest-Neighbors, feature transformation is a requirement for the algorithm to perform expectedly:
In practice, you may even face scenarios where feature transformations have an even larger effect than a 5% increase in performance.
There are many techniques you can apply to make your features more or less follow a normal distribution. They differ based on the underlying distributions of each feature.
This article will teach you about 3 such techniques: Scaling, normalization, and logarithmic transformers. You will develop a practical understanding of their differences and when to apply them in your own workflow.
Before applying any of the techniques you will be learning, it is important to explore each feature visually. You don’t have to create perfect plots; simple histograms and boxplots with default styles will be enough to identify distributions. Take a look at these histograms from the Diamonds dataset of Seaborn:
Pay close attention to the shape of the distributions and the ranges on the X-axes. From the subplots, we can see that price and carat features follow a skewed distribution. Depth, table, and x may remotely resemble a normal distribution.
Apart from the distributions, the range of values is also an important aspect of features. Boxplots are best at showing this using the 5-figure summary:
>>> diamonds.boxplot()
The above plot is enough to tell us that the features have very different scales. As diamond prices have a larger scale of values, they squished other boxplots into a line. Here is the 5-figure summary for further comparison:
>>> diamonds.describe().T.round(2)
Now that you learned how to identify the shape of the distributions and the difference of each feature’s scale, by the end of the article, we will have made all features have the same scale and more or less follow a normal distribution like so:
One of the default methods of solving situations where one feature has a much larger variance than others is using scaling (also called standardization):
According to the official Sklearn guide on scaling:
Many elements used in the objective function of a learning algorithm assume that all features are centered around zero and have variance in the same order. If a feature has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.
So, to achieve high performances with many models, scaling is often required. Sklearn implements this in the StandardScaler() transformer. It transforms numerical features in the dataset to have a mean of 0 and a variance of 1.
SS accomplishes this by using two operations:
Centering: subtract the mean from each value in the distribution.Scaling: divide each result by the standard deviation.
Centering: subtract the mean from each value in the distribution.
Scaling: divide each result by the standard deviation.
The operations leave the original feature following a normal distribution. Here is how we would do this manually:
We are leaving out price and carat feature because they follow a skewed distribution. More on this later.
Let’s do the same with the StandardScaler():
The same results.
Checking the mean and variances:
Now, let’s pay attention to the effectiveness of the scaling. Depth and x now genuinely look like a Gaussian distribution. However, the features table, y, and z are still squished into the corner of their plots, suggesting the presence of outliers (otherwise, the bulk of the histograms would be in the center). This means the scaling worked better on the depth and x features but not on the others. We will make a note of this for the next sections.
When a feature does not follow a linear distribution, it would be unwise to use the mean and the standard deviation to scale it. For example, here is what would have happened if we scaled the skewed distributions of price and carat:
The fact that the features are still skewed confirms that standardization does not work on them.
To implement non-linear transformations, Sklearn offers a PowerTransformer class (uses logarithmic functions under the hood) that helps minimize skewness and map any distribution to a normal one as close as possible:
>>> diamonds[["carat", "price"]].var()carat    1.000019price    1.000019dtype: float64
The new features look much better than the old skewed ones. So, whenever faced with a skewed distribution, use the PowerTransformer class.
An alternative method to scaling is called normalization. Instead of using the variance and the mean, normalization uses the minimum and the maximum values of the distribution. The following equation is used for each value :
The effect of this transformation is to give the distribution an absolute minimum and a maximum value, usually between 0 and 1. Sklearn provides a similar MinMaxScaler transformer to implement this:
Even though it forces features to follow a normal distribution, the features won’t have unit variance and a mean of 0:
However, there are caveats to using this method. For example, if the maximum value in the training set is smaller than the maximum in the test set, the scaling may lead to unexpected results in predictions. The same is true for the minimum value.
Also, MinMaxScaler does not work well with features with outliers. I leave you to think about why (hint: pay attention to the formula of the MMScaler).
Besides, MinMaxScaler does not change the shape of the distribution at all. After normalization, the values lie within the given range, but the shape of the distribution remains unchanged.
Because of these reasons, StandardScaler is used more often.
In this section, we will try to predict the diamond cuts using a LogisticRegression algorithmm. We will be using the StandardScaler and PowerTransformer inside a pipeline. If you are not familiar with how Sklearn pipelines and ColumnTransformers work, check out this post:
towardsdatascience.com
Let’s get started. First, we will build the feature/target arrays and extract the names of the columns we will be applying the transformers:
Next, we will create a ColumnTransformer object which maps the transformers to the relevant columns:
We will plug this transformer into a Pipeline ending with a LogisticRegression:
Finally, we divide the data into training and test sets and measure the classifier’s performance:
We got a ROC AUC score of 0.83. At this point, you can start hyperparameter-tuning to improve on this.
Whenever you are doing preprocessing, always watch out for data leakage. Since all the transformers we learned today derive metrics from the underlying distributions of the features, it is easy for them to leak data from the test data.
That’s why it is advised to divide the data into train/test sets before doing any preprocessing. Also, never use the fit_transform() method on the test set. The transformers should all be fitted to the training data, and the subsequent transformations should only use transform method:
However, when using these transformers in pipelines, you don’t have to worry about this. Sklearn takes care of data leakage under the hood when fitting and transforming features.
In this article, you learned how to feature engineer your numeric features to conform to the statistical assumptions of many models. Specifically, we learned how to:
Scale data using StandardScaler, a transformer used when we want a feature to follow a normal distribution with mean 0 and unit variance. Used most often with distributions without too many outliers or other generally-shaped distributions.
Log transform data using PowerTransformer, a transformer used when we want a heavily skewed feature to be transformed into a normal distribution as close as possible.
Normalize data using MinMaxScaler, a transformer used when we want the feature values to lie within a certain min and max values. It doesn't work well with many outliers and prone to unexpected behaviors if values go out of the given range in the test set. It is a less popular alternative to scaling.
Sklearn Preprocessing Official Guide
How to Use Normal Distribution Like You Know What You Are Doing
How to think Probabilistically with Discrete Distributions
Logarithmic Scale in Data Science
Automatic Hyperparameter Tuning with Sklearn GridSearchCV and RandomizedSearchCV
11 Times Faster Hyperparameter Tuning with HalvingGridSearch
Intro to Object-Oriented Programming For Data Scientists