Types Of Transformations For Better Normal Distribution | by Tamil Selvan S | Towards Data Science

One of the most common assumptions in statistical analysis is normality. Do you agree?
Hello world, this is my first blog for the Data Science community. In this blog, we are going to see the various types of transformations of data to better fit for normal distribution (Gaussian Distribution).
We know that in the regression analysis the response variable should be normally distributed to get better prediction results.
Most of the data scientists claim they are getting more accurate results when they transform the independent variables too. It means skew correction for the independent variables. Lower the skewness better the result.
Transformation is nothing but taking a mathematical function and applying it to the data.
Let us get started!
Log TransformationSquare-Root TransformationReciprocal TransformationBox-Cox TransformationYeo-Johnson Transformation (Bonus)
Log Transformation
Square-Root Transformation
Reciprocal Transformation
Box-Cox Transformation
Yeo-Johnson Transformation (Bonus)
For better clarity visit my Github repo here
Numerical variables may have high skewed and non-normal distribution (Gaussian Distribution) caused by outliers, highly exponential distributions, etc. Therefore we go for data transformation.
In Log transformation each variable of x will be replaced by log(x) with base 10, base 2, or natural log.
import numpy as nplog_target = np.log1p(df["Target"])
The above plot is the comparison of original and Log transformed data. Here we see the skewness is reduced in the transformed data. (best skew value should be nearly zero)
This transformation will give a moderate effect on distribution. The main advantage of square root transformation is, it can be applied to zero values.
Here the x will replace by the square root(x). It is weaker than the Log Transformation.
sqrt_target = df["Target"]**(1/2)
In this transformation, x will replace by the inverse of x (1/x).
The reciprocal transformation will give little effect on the shape of the distribution. This transformation can be only used for non-zero values.
reciprocal_target = 1/df["Target"]
The skewness for the transformed data is increased.
It is one of my favorite transformation techniques.
Box-cox transformation works pretty well for many data natures. The below image is the mathematical formula for Box-cox transformation.
All the values of lambda vary from -5 to 5 are considered and the best value for the data is selected. The “Best” value is one that results in the best skewness of the distribution. Log transformation will take place when we have lambda is zero.
from scipy.stats import boxcoxbcx_target, lam = boxcox(df["Target"])#lam is the best lambda for the distribution
Here, we noticed that the Box-cox function reduced the skewness and it is almost equal to zero. Worked well ;)
For this transformation, values strictly to be positive.
This is one of the older transformation technique which is very similar to Box-cox transformation but does not require the values to be strictly positive.
This transformation is also having the ability to make the distribution more symmetric.
from scipy.stats import yeojohnsonyf_target, lam = yeojohnson(df["TARGET"])
Yeo-Johnson transformation worked beautifully and better than Box-cox.
In this blog, we have seen the different types of transformations (with one bonus oldest transformation technique) for the better fit of normality. All the time Yeo-Johnson will not be the right choice. For some kind of data, Box-cox will work better so we must go with the best transformation technique.
In the next blog, we will see about uniform distribution. :)
Reach me at LinkedIn ;)
Comments are welcome :)