Understanding Distributions using R | by Sowmya Krishnan | Towards Data Science

How do you communicate data-driven findings?
Data Visualizations!
There are various tools to achieve this and this article will be speaking of one such tool — R.
But even before we can start with visualizing data using R, there are certain concepts and terms we need to understand. By the end of this article, I hope you’ll be able to understand:
a) distributions and how to use them to summarize your data set
b) the difference between a histogram and a density plot
c) normal distribution & the use of standard units
d) how to check for normal distribution using quantile plots
e) when and how to use boxplots
Let us introduce a problem here. Pretend there is an overlord from another planet to whom we need to describe the heights of say, a set of students. To achieve this, we will first need to collect data on the students’ sex and heights in inches. One way to explain the dataset to the extraterrestrial overlord would be to simply share the dataset with it.
Another simpler way to showcase this data would be through distributions — the most basic statistical summary of any list. Different forms of distributions are made use of while describing a list of categorical or continuous variables. For instance, in this case the variable ‘sex’ here can be summarized using the prop.table() command to generate a frequency table and help us understand the proportions of the two categories — male and female, in the given dataset.
library(dslabs)data(heights)# make a table of category proportionsprop.table(table(heights$sex))
With a greater number of categories, we can make use of a bar plot to describe the distributions. But with numerical data, plotting distributions can be far more challenging. That is because the individual data points may not be unique. A person may report his/her height to be 68.897 inches while another person may report the same as 68.503 inches. It is clear that these people have converted their heights from 175 and 174 inches respectively.
Representation of such entries requires a distribution function. This is where the concept of ‘Cumulative Distribution Function’ comes into play. The CDF of a random variable X is defined as,
Let us assume that X is a discrete random variable with range R = {x1, x2, x3....} and the range R is bounded from below (i.e. x1). The below given figure shows the general form of the resulting CDF. The CDF is a non-decreasing function and approaches 1 as x becomes large enough.
Coming back to the height’s dataset example, when we construct a CDF for it we need to assess if it answers important questions like — where is the distribution centered around, what range contains majority of the data, etc. although CDF can help answer these kind of questions, but it is a lot more difficult.
In comparison, we now make use of a histogram to better our understanding of the data. One look at the plot and the overlord would be able to infer important information from it.
We can a symmetric distribution around 69 inches and most of the data lying between 63 and 74 inches. With an approximation of this kind, we always loose some kind of information. In this particular case, the implications are negligible.
We can also make use of smooth density plots to visualize this distribution of data. While smoothening out the noise, the plot’s peaks help identify the regions or range over which majority of the values are concentrated.
One major advantage the density plots have over a histogram is that they are much better at determining the shape of the distribution. E.g., a histogram with say, 5 bins will not produce as distinguishable a shape as a 15-bin histogram would. However, there is no such issue while using density plots.
Let us take the summary statistics one step further and calculate the mean and average deviation on this dataset. We are all familiar with what a normal distribution means. When you plot the points from a random collection of data from independent sources, it generates a bell shape curve (or a Gaussian curve). In this graph, the curve’s centre will give you the mean of the dataset.
Following are the built-in functions in R used to generate a normal distribution function:
dnorm() — Used to find the height of the probability distribution at each point for a given mean and standard deviation.
dnorm() — Used to find the height of the probability distribution at each point for a given mean and standard deviation.
x <- seq(-20, 20, by = .1)y <- dnorm(x, mean = 5, sd = 0.5)plot(x,y)
2. pnorm() — Also known as the ‘Cumulatibe distribution Function’(CDF), pnorm is used to find the probability of a normally distributed random number to be less that the value of a given number.
x <- seq(-10,10,by = .2)y <- pnorm(x, mean = 2.5, sd = 2)plot(x,y)
3. qnorm() — This takes the probability value and gives a number whose cumulative value matches the probability value.
x <- seq(0, 1, by = 0.02)y <- qnorm(x, mean = 2, sd = 1)plot(x,y)
4. rnorm() — This function is used to generate random numbers whose distribution is normal.
y <- rnorm(50)hist(y, main = "Normal DIstribution")
Equation for Normal Distribution:
# define x as vector of male heightslibrary(tidyverse)library(dslabs)data(heights)index <- heights$sex=="Male"x <- heights$height[index]# calculate the mean and standard deviation manuallyaverage <- sum(x)/length(x)SD <- sqrt(sum((x - average)^2)/length(x))# built-in mean and sd functions average <- mean(x)SD <- sd(x)c(average = average, SD = SD)
In case of the heights of the dataset, this distribution is centred around the average and most data points are within two standard deviations from the average. We observe this distribution is defined only by two parameters — mean and standard deviations and therefore it implies that if a dataset follows a normal distribution, it can be summarized by these two values.
In R, we make use of the function scale to obtain standard units. Mathematically, standard unit is defined as follows:
It basically tells us of the number of standard deviations an object x (in this case the height) is away from the mean.
# calculate standard units z <- scale(x)# calculate proportion of values within 2 SD of mean mean(abs(z) < 2)
The proportion of values within -2 and +2 turns of the mean, turns out to be around 95%, which is exactly what the normal distribution predicts it to be.
But how do we check if the distributions are well approximated by a normal distribution?
The main concept here is that we define a series of proportion p, and based on that define quantile q, such that the proportion of values in the data below q is p. Therefore, if the quantiles for the data are equal to those of the normal distributions we can conclude that the data is approximated by a normal distribution.
Let us now calculate the sample and theoretical quantiles to check if the data points fall on an identity line or not.
# calculate observed and theoretical quantilesp <- seq(0.05, 0.95, 0.05)observed_quantiles <- quantile(x, p)theoretical_quantiles <- qnorm(p, mean = mean(x), sd = sd(x))# make QQ-plotplot(theoretical_quantiles, observed_quantiles)abline(0,1)
Indeed, the values fall on the identity line implying the distributions are well approximated by a normal distribution.
Note:
a) A special case of quantiles, percentiles are the quantiles obtained while defining p = 0.01, 0.02, 0.03...., 0.99
b) Quartiles are the 25th, 50th and the 75th percentiles and the 50th percentile gives us the values of the median.
What happens in case your dataset does not follow a normal distribution and the two parameters — mean and standard deviation are not enough to summarize the data? Boxplots can be helpful in such cases. Dividing the dataset into three quartiles, the boxplot graph represents the first quartile, third quartile, minimum, maximum and median in a dataset.
Let us use the same “heights” data set to create a basic boxplot for the relation between the sex (male/female) and heights of the students.
data(heights)boxplot(height~sex, data=heights, xlab="Sex",ylab="height")
From the above plot we can infer that the standard deviations for the two groups are almost similar although on an average, mean are found to be taller than women.
If you want to learn further about other/less common distributions in test statistics, please refer to the ‘Distributions’ in the R Stats Package (link given below)