Ordinal Logistic Regression. An overview and implementation in R | by Akanksha Rawat | Towards Data Science

Can you guess what is the common link in the variables mentioned below:
Job satisfaction level — Dissatisfied, Satisfied, Highly Satisfied
Performance of an individual — Poor, Fair, Excellent
Impact of a regulation on bank’s performance — Positive, Neutral, Negative
The variables are not only categorical but they are also following an order (low to high / high to low).
If we want to predict such multi-class ordered variables then we can use the proportional odds logistic regression technique.
To understand the working of Ordered Logistic Regression, we’ll consider a study from World Values Surveys, which looks at factors that influence people’s perception of the government’s efforts to reduce poverty.
Our objective is to predict an individual’s perception about government’s effort to reduce poverty based on factors like individual’s country, gender, age etc. In the given case study, individual’s perception can take the following three values - Too Little, About Right, Too Much.
For our analysis, we’ll be using data from the World Values Surveys for Australia, Norway, Sweden, and the United States from ‘carData’ package in R.
library(carData)library(MASS)data(WVS) head(WVS)
Poverty is the multi-class ordered dependent variable with categories — ‘Too Little’, ‘About Right’ and ‘Too Much’. We have the following five independent variables
Religion: member of a religion -no or yes
Degree: held a university degree -no or yes
Country: Australia, Norway, Sweden or the USA
Age: age (years)
Gender: male or female
Let’s now analyze the descriptive statistics for this dataset:
summary(WVS)
We can also analyze the distribution of poverty across age, sex and country
ggplot(WVS, aes(x = poverty, y = age, fill = poverty)) +   geom_boxplot(size = .75) +   facet_grid(country ~ gender, margins = FALSE) +   theme(axis.text.x = element_text(angle = 45, hjust = 1, vjust = 1))
We’ll now fit the Proportional Odds Logistic Regression model using polr function from the MASS package.
model_fit <- polr(poverty~religion+degree+country+age+gender, data = WVS, Hess = TRUE)summary(model_fit)
In the output above, we get the information about
Model equation
The regression coefficients with their values, standard errors and t value. There is no significance test by default but we can calculate p-value by comparing t value against the standard normal distribution.
Estimates for two intercepts
Residual deviance and AIC, which are used in comparing the performance of different models
summary_table <- coef(summary(model_fit))pval <- pnorm(abs(summary_table[, "t value"]),lower.tail = FALSE)* 2summary_table <- cbind(summary_table, "p value" = round(pval,3))summary_table
Since the p-value for all the variables <0.05, hence they are statistically significant at 95% CI
In order to interpret this model, we first need to understand the working of the proportional odds model.
Let J be the total number of categories of the dependent variable and M be the number of independent variables (In the given dataset, J=3 and M = 5).
The mathematical formulation of the Proportional Odds Model is given below
Here, j is the level of an ordered category with J levels and i corresponds to independent variables
In our case
j = 1 refers to ‘Too Little’
j = 2 refers to ‘About Right’
j = 3 refers to ‘Too Much’
i = 1 refers to ‘religion’
i = 2 refers to ‘degree’
i = 3 refers to ‘country’
i = 4 refers to ‘age’
i = 5 refers to ‘gender’
Coefficients :
The categorical variables like gender can be interpreted as: a male individual, as opposed to a female individual, is associated with a higher likelihood of having a positive perception about government’s efforts to reduce poverty. The t-value is greater than 2 and therefore is statistically significant at the 5% level.
The continuous variables like age can be interpreted as : with one unit increase in age the log of odds of having a positive perception about government’s efforts to reduce poverty increases by 0.011
Intercepts:
Mathematically, the intercept ‘Too Little | About Right’ corresponds to logit[P(Y ≤ 1)]. It can be interpreted as the log of odds of believing that the government is doing ‘Too Little’ versus believing that the government is doing ‘About Right’ or ‘Too Much’
Similarly, the intercept ‘About Right | Too Much’ corresponds to logit[P(Y ≤ 2)]. It can be interpreted as the log of odds of believing that the government is doing ‘Too Little’ or ‘About Right’ versus believing that the government is doing ‘Too Much’
Let say we want to predict the probability corresponding to each perception for an individual — Test_Person with the following characteristics
Religion : yes
Degree : no
Country : Norway
Age : 30
Gender : male
Mathematical Computation
Mathematical Computation
By using the intercept and slope values from the Model Summary, we can estimate the desired probabilities in the following manner
The probability corresponding to Too Little perception will be calculated as:
logit[P(Y ≤ 1)] = 0.7298 -[(0.17973*1)+(0.14092*0)+(-0.32235*1)+(0.01114*30)+(0.17637*1)]
=> logit[P(Y ≤ 1)] =0.36185
=> P(Y ≤ 1)= exp(0.36185)/(1+exp(0.36185)) = 0.589
In our case, P(Y ≤ 1) = P(Y =1) = 0.589
Similarly, the probability corresponding to About Right perception will be calculated as:
logit[P(Y ≤ 2)] = 2.5325 -[(0.17973*1)+(0.14092*0)+(-0.32235*1)+(0.01114*30)+(0.17637*1)]
=> logit[P(Y ≤ 2)] =2.16455
=> P(Y ≤ 2)= exp(2.16455)/(1+exp(2.16455)) = 0.897
Hence, P(Y = 2) = P(Y ≤ 2) — P(Y ≤ 1) = 0.897 -0.589
=> P(Y = 2) = 0.308
The probability corresponding to Too Much perception will be calculated as:
Thus, P(Y = 3) = 1-P(Y ≤2)
=> P(Y = 3) = 0.103
2. Computation in R
Fortunately, we can bypass the above mathematical calculation by using the predict function in R
new_data <- data.frame("religion"= "yes","degree"="no","country"="Norway","age"=30,"gender"="male")round(predict(model_fit,new_data,type = "p"), 3)
Our model predicts that the individual Test_Person believes that the government’s effort to reduce poverty are Too Little
If you wish to learn more about this concept, I encourage you to go through the following links: