A Measure of Linear Relationship. Pearson’s product-moment correlation... | by Shinichi Okada | Towards Data Science

Table of ContentsIntroduction1. Import necessary libraries2. Finding the regression line3. Pearson’s correlation coefficient4. Finding Pearson’s correlation coefficient5. Degree of freedom and critical values6. The least-squares regression line7. The mean point on the line of best fit8. Real-life example9. Finding r from more than two arrays10. Heat mapConclusion
In this article, I use Python to analyze bivariate data using Pearson’s product-moment correlation coefficient r, drawing scatter diagrams, lines of best fit, and the least-square regression line. Bivariate data involves two types of related data. Pearson’s product-moment correlation coefficient tells you how linearly these two data are related. We can display the collected data on a scatter plot. The horizontal axis is called the x-axis and we call the coordinate of the x-axis the independent variable. The vertical axis is called the y-axis and we call the coordinate of the y-axis the dependent variable.
I downloaded the data from this link.
Let’s create a sample data.
We need to transpose the data so that data is categorized by columns. Pandas transpose does not modify the original data.
We use the Seaborn to create a scatter plot.
linregress returns slope, intercept, r_value and p-value.
The equation of reegression line is y=1.221x+1.151.Pearson's product-moment correlation coefficient is {r_value:.4f}.p-value is {p_value:.4f}.
Pearson’s product-moment correlation coefficient for two sets of population data, x, and y, is:
where cov is the covariance and σx and σy are the population standard deviations of x and y.
where μx is the mean of x and μy is the mean of y.
Pearson’s product-moment correlation coefficient for two sets of sample data, x, and y, is:
where sxy is the covariance and sx and sy are the sample standard deviations of x and y.
Therefore:
towardsdatascience.com
Let’s use the sample data.
0.93050085576319
In Pandas, ddf=0 is for population and ddf=1 is for a sample. Since we use all data we use 0 here.
0.9305008557631897
pearsonr returns Pearson's product-moment correlation coefficient and p-value.
You can find the table of critical values for Pearson correlation here. The degree of freedom (dof) is the number of data points minus 2. In the above example dof=5–2=3. For 95% confidence which is statistically significant, we use the 0.05 Levels of Significance from the table of critical value. This gives us 0.878. If |r|>critical value, then the correlation is probably NOT due to change which means it is statistically significant. In the above case, |0.9928| > 0.878, so it is 95% confident that there is a statistically significant positive relationship between two variables.
The least-squares regression line is called the line of best fit. Since linregress() returns the slope and y-intercept, we use them to make a line of regression.
We find the mean of each data set and plot it on the same graph.
towardsdatascience.com
We are going to the linear relation between the population and the labour force in Australia. We use the historical population from OECD. Let’s use read_csv and find the dimension(shape) and display the first five data using shape and head().
From the historical population data, let’s select Australia. We also need to select Total in the Sex column and TOTAL in the AGE column.
We need to select the Value column. I will show you two different ways to do it. This website tells you how to use loc and iloc. Since the Python index starts from 0, it locates in the 8th column.
In the second method, we use & logical operator. You can read more details about loc and iloc in this article. We select the column where Country is Australia, Set is Total and AGE is TOTAL.
We need one more data from OECD. The data is the ALFS Labour force. We select data where the Country is Australia. head() shows the first 5 data.
We need the Value column which is the 15th index, so we use 14 in iloc. Remember that the Python index starts from 0.
We can find Pearson’s correlation coefficient from df_pop and df_lab. pearsonr returns r and p-value.
We find the gradient/slope and y-intercept using linregress. We create a scatter plot with the Seaborn and Pandas DataFrame.
There is a strong positive correlation between the population and the labor force in Australia.
Sometimes you want to find which pair of data has the strongest linear relationship. You can use corr() to find it.
Please note that Pearson’s correlation coefficient of A and B are 0.952816 which is the same as we found it before. In this case, A and C have the strongest linear relationship.
We can use the Seaborn to draw a heatmap. Since we have a minimum of 0.88 and a maximum of 1.00 in the previous section, we set vmax and vmin accordingly. We use Blues for the color scheme.
By using Python and its libraries, you can find all the necessary data with a few lines of code. Also, you can visualize your data easily.
Get full access to every story on Medium by becoming a member.