Multinomial Models for Nominal Data | by Yufeng | Towards Data Science

The popular multinomial logistic regression is known as an extension of the binomial logistic regression model, in order to deal with more than two possible discrete outcomes.
However, the multinomial logistic regression is not designed to be a general multi-class classifier but designed specifically for the nominal multinomial data.
To note, nominal data and ordinal data are two major categories of multinomial data. The difference is that there is no order to the categories in nominal multinomial data while there is an order to those in ordinal multinomial data.
For example, if our goal is to distinguish the three classes of plants in the IRIS dataset, we need to treat the plant categories as a nominal outcome because there is no specific order in the classes of plants. However, if we want to predict the score at the end of the first half in an NBA final into three categories, “winning”, “losing”, or “tie”, we need to regard it as an ordinal outcome because the pairwise distances are not the same among the three classes.
Therefore, it’s necessary to figure out what the type of outcome is before the multinomial modeling.
In this post, I am going to briefly talk about the multinomial regression models for the nominal data. I hope it will help you in your real-life work.
As aforementioned, the multinomial logistic regression was specifically designed for the nominal data. The idea is very similar to that of logistic regression on the binary data, which is to link the probability of belonging to one of the categories to the predictors.
For those who struggle to understand the link functions, you can refer to one of my previous posts about it.
Let Yi be a random variable that can fall in one of the classes, 1, 2, ..., J. And we define pij as the ith variable to be in the jth class.
so, we have
the formula above is for the general case, so when J=2, it becomes the case with binary data.
If we have the total number of observations as ni, then the multinomial distribution could be described as below.
In the multinomial logistic regression, the link function is defined as
where
In this way, we link the log odds ratio between the probability to be in class J and that to be in class 1 to the linear combination of the predictors. Now let’s take a look at the real data and coding of the multinomial logistic regression.
The example is from Faraway’s book, which describes the 1996 American National Election Study. The response variable involves three categories, Democrat, Republican, and Independent. The predictors include age, education level, and income group of the respondents.
# R code> head(nes96)##   popul TVnews selfLR ClinLR DoleLR     PID age  educ   income    vote## 1     0      7 extCon extLib    Con  strRep  36    HS $3Kminus    Dole## 2   190      1 sliLib sliLib sliCon weakDem  20  Coll $3Kminus Clinton## 3    31      7    Lib    Lib    Con weakDem  24 BAdeg $3Kminus Clinton## 4    83      4 sliLib    Mod sliCon weakDem  28 BAdeg $3Kminus Clinton## 5   640      7 sliCon    Con    Mod  strDem  68 BAdeg $3Kminus Clinton## 6   110      3 sliLib    Mod    Con weakDem  21  Coll $3Kminus Clinton
After some pre-processing of the dataset, we build the multinomial logistic regression in R as below.
# R codelibrary(nnet)mulmod = multinom(sPID ~ age + educ + nincome, nes96)
where “sPID” is the response variable, “age”, “educ”, and “nincome” corresponds to the age, the education level, and the income of the respondents, respectively. And nes96 is just the dataset name.
We then check the summary of the fitted model.
# R codesummary(mulmod)
which yields,
## Call:## multinom(formula = sPID ~ age + educ + nincome, data = nes96)## ## Coefficients:##             (Intercept)          age     educ.L     educ.Q    educ.C## Independent   -1.197260 0.0001534525 0.06351451 -0.1217038 0.1119542## Republican    -1.642656 0.0081943691 1.19413345 -1.2292869 0.1544575##                  educ^4     educ^5      educ^6    nincome## Independent -0.07657336  0.1360851  0.15427826 0.01623911## Republican  -0.02827297 -0.1221176 -0.03741389 0.01724679## ## Std. Errors:##             (Intercept)         age    educ.L    educ.Q    educ.C    educ^4## Independent   0.3265951 0.005374592 0.4571884 0.4142859 0.3498491 0.2883031## Republican    0.3312877 0.004902668 0.6502670 0.6041924 0.4866432 0.3605620##                educ^5    educ^6     nincome## Independent 0.2494706 0.2171578 0.003108585## Republican  0.2696036 0.2031859 0.002881745## ## Residual Deviance: 1968.333 ## AIC: 2004.333
The estimated coefficients, as well as the standard errors, are reported by the model summary. The output is a little different from those in the linear models.
Only the statistics of “Independent” and “Republican” are listed in the summary because here “Democrat” is used as the baseline category and the regression is modeling the log odds ratio against this baseline category.
You may notice that there is no indicator to show the significance of the predictors in the model. We can evaluate them in two ways.
First, we can use the popular Z-score to evaluate the significance of the preditors.
# R codez = summary(mulmod)$coefficients/summary(mulmod)$standard.errorsp = (1 - pnorm(abs(z), 0, 1))*2
This procedure is easy to implement because it has a simplified assumption of the normal distribution which holds for a rough estimation. Let’s look at the p-values.
##              (Intercept)        age     educ.L     educ.Q    educ.C    educ^4## Independent 2.464850e-04 0.97722232 0.88951010 0.76893540 0.7489628 0.7905471## Republican  7.107723e-07 0.09464069 0.06630235 0.04189165 0.7509449 0.9374990##                educ^5    educ^6      nincome## Independent 0.5854128 0.4774294 1.751402e-07## Republican  0.6505832 0.8539066 2.165977e-09
It seems that income is a strong preditor with no doubt, but not all the education variables are listed as significant. This is also a shortcoming of using Z-distribution to evaluate the coefficients, which makes it hard to interpret.
A more precise way is to build a new model with a reduced number of predictors and then check the change in the likelihood. First, let’s build the model without variable education.
# R codemulmod_noedu = multinom(sPID ~ age + nincome, nes96)
Then we use a chi-squared distribution to evaluate the change in likelihood between the two multinomial models. The p-value could be calculated as below.
# R codepchisq(deviance(mulmod_noedu)-deviance(mulmod),mulmod$edf - mulmod_noedu$edf, lower.tail = F)
which yields,
0.1819634
which indicates that the variable ‘education’ is not significant relative to the full model. If we do the same thing to the variable ‘income’, we will get the following result.
# R codemulmod_noinc = multinom(sPID ~ age + educ, nes96)pchisq(deviance(mulmod_noinc)-deviance(mulmod),mulmod$edf - mulmod_noinc$edf, lower.tail = F)
which yields,
1.267249e-10
These results indicate that “income” is a significant predictor in the full model.
The interpretation of the coefficient should be such: one unit change in some predictor will result in k unit change in the log odds ratio between the target category to the baseline category (“Democrat” in this example). I will not explore the details of this part.
Another very important check on the response variable before the multinomial modeling is whether the response outcome is hierarchical.
Let’s look at an example of the central nervous system disease data from Faraway’s book.
data(cns)head(cns)
yields,
##         Area NoCNS An Sp Other Water      Work## 1    Cardiff  4091  5  9     5   110 NonManual## 2    Newport  1515  1  7     0   100 NonManual## 3    Swansea  2394  9  5     0    95 NonManual## 4 GlamorganE  3163  9 14     3    42 NonManual## 5 GlamorganW  1979  5 10     1    39 NonManual## 6 GlamorganC  4838 11 12     2   161 NonManual
Here, “NoCNS” means no central nervous system disease, “An” means anencephalus, “Sp” means spina bifida, and “Other” means other types of CNS. “Area”, “Water” and “Work” describe the features of the families.
If we want to model the CNS status using the features of the families, we can simply implement the multinomial logistic regression on the four categories.
However, we can see that “An”, “Sp” and “Other” are just subtypes of CNS and the response variable is dominated by healthy individuals (NoCNS). In such a situation, we’d better treat the response variables as hierarchical variables, where “CNS” and “NoCNS” form a binomial distribution and “An”, “Sp”, and “Other” form a multinomial distribution within the “CNS” category.
Let’s compare the codes and results.
Let’s first establish a binomial logistic regression between the “CNS” and “NoCNS”.
# R codecns$CNS = cns$An + cns$Sp + cns$Otherbinom_mod = glm(cbind(CNS,NoCNS) ~ Water + Work, cns, family=binomial)summary(binom_mod)
which yields,
## ## Call:## glm(formula = cbind(CNS, NoCNS) ~ Water + Work, family = binomial, ##     data = cns)## ## Deviance Residuals: ##      Min        1Q    Median        3Q       Max  ## -2.65570  -0.30179  -0.03131   0.57213   1.32998  ## ## Coefficients:##                 Estimate Std. Error z value Pr(>|z|)    ## (Intercept)   -4.4325803  0.0897889 -49.367  < 2e-16 ***## Water         -0.0032644  0.0009684  -3.371 0.000749 ***## WorkNonManual -0.3390577  0.0970943  -3.492 0.000479 ***## ---## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## ## (Dispersion parameter for binomial family taken to be 1)## ##     Null deviance: 41.047  on 15  degrees of freedom## Residual deviance: 12.363  on 13  degrees of freedom## AIC: 102.49## ## Number of Fisher Scoring iterations: 4
We can find that both “Water” and “WorkNonManual” are significant variables in predicting CNS or not.
Then, we perform the multinomial logistic regression on the subtypes of CNS.
# R codecns_mod = multinom(cbind(An,Sp,Other) ~ Water + Work, cns)z = summary(cns_mod)$coefficients/summary(cns_mod)$standard.errorsp= (1 - pnorm(abs(z), 0, 1))*2p
which yields,
##        (Intercept)     Water WorkNonManual## Sp    4.833577e-02 0.5234640     0.5791105## Other 5.935107e-05 0.4511663     0.4052210
We can see that neither “Water” nor “WorkNonManual” is significant to distinguish different subtypes of CNS.
These results suggest that water quality and manual work do have an influence on the new births’ CNS condition, but neither of them explains the subtypes of the CNS.
However, if we throw everything to a four-category full model, the results are different.
# R codefull_mod = multinom(cbind(NoCNS, An, Sp, Other) ~ Water + Work, cns)z = summary(full_mod)$coefficients/summary(full_mod)$standard.errorsp= (1 - pnorm(abs(z), 0, 1))*2p
which shows,
##       (Intercept)       Water WorkNonManual## An              0 0.066695285    0.02338312## Sp              0 0.001835095    0.07077816## Other           0 0.840643368    0.02348335
This time the predictors show some mixed results in interpreting the subtype of CNSs with different p-values. The fact that subtypes of CNS cannot be predicted by the two variables is not disclosed in the full model.
In summary, I introduced the multinomial regression model and its application situations. The most important point is to check the response variables before the modeling.
The multinomial logistic regression can only be applied to the nominal data instead of the ordinal data.The response variables with potential hierarchical structures should be treated carefully. Pooling everything together to a full model is not optimal sometimes.
The multinomial logistic regression can only be applied to the nominal data instead of the ordinal data.
The response variables with potential hierarchical structures should be treated carefully. Pooling everything together to a full model is not optimal sometimes.
www.analyticsvidhya.com
Faraway, Julian J. Extending the linear model with R: generalized linear, mixed effects and nonparametric regression models. CRC press, 2016.