Identify your Data’s Distribution | by Abhishek Mungoli | Towards Data Science

Every day we come across a variety of Data like Sensor Data, Sales Data, Customer Data, Traffic Data, etc. Further, depending on the use case, we do a variety of processing and try out several algorithms on it. Have you ever wondered these questions about your Data:
What your Data look like?
Is there any confidence you can attach to the values it can take? What is the possibility of an extreme value ‘x’ to occur?
The algorithm you are applying on it may make some assumption about Data distribution, are those assumptions correct?
Can the Data distribution change over time or during special seasons?
Data Distribution is a function that lists out all possible values the Data can take. It can be a continuous or discrete Data distribution. Several known standard Probability Distribution functions provide probabilities of occurrence of different possible outcomes in an experiment. Some well-known probability distributions are Normal, Log-Normal, Beta, Gamma, etc. which have a standard form.
But, the challenge is the real-world Data may not follow any well-known Probability Distributions. In this case, we can approximate the most probable Probability Distribution and check it’s Goodness of fit.
Through this blog post, I intend to highlight the benefits of knowing your Data and approximating its’s Probability distribution. I will also show practical examples of how to do that and measure the Goodness of fit of the fitted distribution to that of the observed. The Dataset used in the blog is Kaggle’s Used Car Dataset. The code used in this blog post can be downloaded from here.
It’s a good practice to know your Data once you start working on it. Many Algorithms, like Linear Regression, assumes variables to follow a particular distribution. The cost of not meeting the assumptions could be high at times.
Knowing the underlying probability distribution, we can find it’s Probability density function. This helps us in attaching confidence intervals to the range of values Data is likely to take. We can also find the probability of extreme value to occur.
Distribution has parameters. With these parameters, we can keep track of how the Distribution has changed over time or during a particular season/event.
The standard probability distributions have well known statistical properties that simplify the job for us. We can explain the Data and its behaviour with just a few parameters.
For this blog, I will be using the Kaggle Used Car Dataset. We will try to approximate the distributions of various variables and also check its Goodness of fit. I have filtered out a few columns. Let’s have a look at the data :
Id: A unique identifier, identifying the vehicle
Price: Selling Price of the vehicle
Year: Year of Manufacturing
Manufacturer: The Brand of the Car
Condition: Car’s condition
Odometer: Odometer reading denotes the distance traveled
The Dataset has 539K records. We will try to identify/approximate the Distribution of ‘price’ and ‘odometer’ variables.
Scipy Library of Python allows estimating the parameters of 200+ distributions. Further, the Goodness of fit can be tested by various metrics like Chi-square statistics, Kolmogorov–Smirnov test, QQ plots, etc.
As per blog of Adam Hayes, A chi-square (χ2) statistic is a test that measures how expectations compare to actual observed data (or model results). The data used in calculating a chi-square statistic must be random, raw, mutually exclusive, drawn from independent variables, and drawn from a large enough sample. For example, the results of tossing a coin 100 times meet these criteria. [[Source Investopedia]]
The formula for chi-square statistics is given as -
where c is the degree of freedom, O is the observed value and E is the expected value.
Article Link
www.investopedia.com
Do check out my blog on the Chi-square statistical test to decide on the winner of a Bet.
medium.com
Commonly known as the K-S Test, it is used to check the Goodness of fit of the observed Distribution with the theoretical Distribution.
Here’s a great article on the topic.
www.tutorialspoint.com
As per Wikipedia, QQ plots is a graphical method for comparing two probability distributions by plotting their quantiles against each other.
Here’s an interesting StatQuest video on QQ plots
With a good enough refresher, we are all set to our primary task of Distribution fitting.
We will try to approximate ‘price’ using the following Distributions and see which performs the best in terms of Chi-square Statistics -
dist_names = [‘weibull_min’,’norm’,’weibull_max’,’beta’,              ‘invgauss’,’uniform’,’gamma’,’expon’,                 ‘lognorm’,’pearson3',’triang’]
For stable results, I removed extreme outliers (1% data on both ends). Fitting the distributions :
Python code using the Scipy Library to fit the Distribution
Fitting different Distributions and checking Goodness of fit based on Chi-square Statistics
The output sorted in order of Goodness of fit looks like:
Top-2 distributions in terms of Goodness of fit are Exponential and Inverse Gaussian Distribution
The parameters of the two distributions are :
Parameters of Exponential Distribution<scipy.stats._continuous_distns.expon_gen object at 0x12abd2390>(loc = -1.19, scale = 1.19)where Lambda = 1/scaleParameters of Inverse Gaussian Distribution<scipy.stats._continuous_distns.invgauss_gen object at 0x12ac90c50>(mu = 0.45, loc = -1.64, scale = 3.61)
The Scipy Library apart of distribution parameters also provides loc and scale parameters. Loc parameters shift the distribution by the appropriate amount and Scale parameters stretches the distribution as required.
docs.scipy.org
Let’s visualise the Results :
fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(9, 5))# Histogram Plot of Observed Dataaxes[0].hist(y)#Exponential Distribution Fittingaxes[1].plot(y,expon.pdf(y_std,-1.19, 1.19))#Inverse-Gaussian Distribution Fittingaxes[2].plot(y,invgauss.pdf(y_std,0.45, -1.64, 3.61))fig.tight_layout()
Both the distribution performs fairly good. Exponential Distribution has a slight edge over Inverse-Gaussian. Chi-square statistics also suggests the same.
Let’s analyse the QQ plot, as well
Both the distributions do fairly well. Exponential is slightly better than Inverse Gaussian in approximating ‘Price’ Data.
We will try to approximate ‘odometer’ using the following Distributions and see which performs the best in terms of Chi-square Statistics -
dist_names = [‘weibull_min’,’norm’,’weibull_max’,’beta’,              ‘invgauss’,’uniform’,’gamma’,’expon’,                 ‘lognorm’,’pearson3',’triang’]
For stable results, I removed extreme outliers (1% data on both ends). Fitting the distributions :
Python code using the Scipy Library to fit the Distribution
Fitting different Distributions and checking Goodness of fit based on Chi-square Statistics
The output sorted in order of Goodness of fit looks like:
Top-2 distributions in terms of Goodness of fit are Beta and Triangular Distribution. However, the difference in Chi-square statistics is significantly high and Beta seems to be a clear winner.
The parameters of the two distributions are :
Parameters of Beta Distribution<scipy.stats._continuous_distns.beta_gen object at 0x12abb72d0>(a = 1.51, b = 2.94, loc = -1.71, scale = 5.02)Parameters of Triangular Distribution<scipy.stats._continuous_distns.triang_gen object at 0x12acf4050>(c = 0.12, loc = -1.79, scale = 4.90)
The Scipy Library apart of distribution parameters also provides loc and scale parameters. Loc parameters shift the distribution by the appropriate amount and Scale parameters stretches the distribution as required.
docs.scipy.org
Let’s visualise the Results :
Beta Distribution performs much better than Triangular distribution. Visually its clear and Chi-square statistics also suggests the same.
Let’s analyse the QQ plot, as well
QQ plot also makes it clear that Beta Distribution approximates the ‘Odometer’ (Distance) Data well.
With this blog post, I covered the advantage of knowing Data prior to analysing or applying algorithms on it. Knowing/Approximating the Data’s probability distribution also helps us to use its statistical properties and attach confidence interval to the values it can take. I also covered how to do that using Python and introduced various metrics to check the Goodness of fit. The Kaggle Dataset used can be downloaded from here. The code of the project can be downloaded from here.
If you have any doubts or queries, do reach out to me. I will be interested to know if you have some interesting problem to solve and feel that approximating the Data’s Probability distribution can be helpful.
My Youtube channel for more content:
www.youtube.com
About the author-:
Abhishek Mungoli is a seasoned Data Scientist with experience in ML field and Computer Science background, spanning over various domains and problem-solving mindset. Excelled in various Machine learning and Optimization problems specific to Retail. Enthusiastic about implementing Machine Learning models at scale and knowledge sharing via blogs, talks, meetups, and papers, etc.
My motive always is to simplify the toughest of the things to its most simplified version. I love problem-solving, data science, product development, and scaling solutions. I love to explore new places and working out in my leisure time. Follow me up at Medium, Linkedin or Instagram and check out my previous posts. I welcome feedback and constructive criticism. Some of my blogs -
5 Mistakes every Data Scientist should avoid
Decomposing Time Series in a simple & intuitive way
How GPU Computing literally saved me at work?
Information Theory & KL Divergence Part I and Part II
Process Wikipedia Using Apache Spark to Create Spicy Hot Datasets
A Semi-Supervised Embedding based Fuzzy Clustering
Compare which Machine Learning Model performs Better