Spearman’s Rank Correlation Coefficient Using Ordinal Data | by Shinichi Okada | Towards Data Science

Table of ContentsIntroduction1. Simplified formula2. General formula3. Pandas corr4. Finding mathematically using the formula (1–1)5 Finding mathematically using the formula (1–2)6. Real-life exampleConclusion
Spearman’s rank correlation coefficient, rs shows the correlation between two ordinal data. How one ordinal data changes as the other ordinal changes.
A function between ordered sets is called a monotonic function.
In this article, I explore different methods to find Spearman’s rank correlation coefficient using data with distinct ranks.
Spearman’s rank correlation requires ordinal data. Examples of ordinal data are:
1st, 2nd, 3rd,
Small, Medium, Large, XL,
Strongly agree, Agree, Neutral, Disagree, Strongly Disagree
Very often, Often, Not Often, Not at all
When all ranks are distinct integers, the Spearman correlation coefficient is computed by the following formula.
This formula is an alternative to Pearson’s correlation if the data are ordinal and monotonic and if there are no ties in data.
A value of 1 means the set of data is strictly increasing and the value of -1 means it is strictly decreasing. A value of 0 means that data shows no monotonic behavior. The Spearman’s correlation between two variables will be high when an observation has a similar rank between the two variables.
towardsdatascience.com
The Spearman correlation coefficient is defined as the Pearson correlation coefficient using the rank variables. After raw data, x and y, are converted to ranked data rx and ry, The Spearman correlation coefficient is defined as:
Let’s find out how the correlation of the rank at 5km run relates to the rank of 10km run. All ranks are distinct integers and no same numbers. There are different ways to find the coefficient.
scipy.stats.spearmanr calculates a Spearman correlation coefficient with the associated p-value.Pandas corrUsing the formula (1–1)Using the formula (1–2)
scipy.stats.spearmanr calculates a Spearman correlation coefficient with the associated p-value.
Pandas corr
Using the formula (1–1)
Using the formula (1–2)
Let’s start. We need to import the necessary libraries.
The following is our sample data.
We find rs and p-value using scipy.stats.spearmanr.
correlation=0.428571, p-value=0.337368
The p-value is the level of significance. It determines the probability that the correlation happened by chance.
correlation=0.428571, p-value=0.337368At 5% level of significance, the final rankings is not significant or independent
Pandas corr returns a correlation matrix. We need to select the value from [0,1].
0.42857142857142855
1-(6*(0+4+1+16+1+9+1)/(7*(7**2-1)))=0.4285714285714286
We find the covariance of rx and ry, a standard deviation of σrx,σry.
As we expected we get all the same results.
A value of 1 means the set of data is strictly increasing. This happens when the ranks do not change. You find the p-value 0 in this case.
correlation=1.000000, p-value=0.000000
The value of -1 means it is strictly decreasing. The rank is in a reversal order.
correlation=-1.000000, p-value=0.000000
Non-monotonic behavior gives a value of 0.
correlation=0.000000, p-value=1.000000
towardsdatascience.com
We will use data from BIXI Montréal. BIXI Montréal is a public bicycle sharing system serving Montréal, Quebec, Canada. It is a big file so it takes time to process data. Let’s use shape() and head() to see the dimension and the head.
There are more than seventy-nine thousand lines with six columns. Columns are start_data, start_station_code, end_date, end_station_code, duration_sec and is_member.
We group by end_station_code and sum the duration_sec. We assign n_rides with the number of end_station_code.
Let’s plot the imported data.
Please note that the majority of data are on the lower end. We can use scipy.stats.rankdata to see the data by ranking.
Let’s find Spearman’s Rs.
correlation, pval = spearmanr(by_end_station)print(f'correlation={correlation:.6f}')
The output is:
correlation=0.964881
The number of rides at a bike stop is almost proportional to the number of commuter seconds spend riding it.
We explored finding Spearman’s correlation coefficient using ordinal data. All four different methods worked well. In Part 2, we are going to use a different data set that has identical data. This will need extra caution to find Spearman’s correlation coefficient.
Get full access to every story on Medium by becoming a member.
IQ of a person with the number of hours spent on games
Free university meals and their CGPA scores
Physics and Math ranks
To compare the Spearman correlation and the Pearson correlation, please read A comparison of the Pearson and Spearman correlation here