Multinomial Models for Nominal Data
Tags: Logistic Regression, Multinomial, Regression, Data Science, R

The popular multinomial logistic regression is known as an extension of the binomial logistic regression model, in order to deal with more than two possible discrete outcomes.

However, the multinomial logistic regression is not designed to be a general multi-class classifier but designed specifically for the nominal multinomial data.

To note, nominal data and ordinal data are two major categories of multinomial data. The difference is that there is no order to the categories in nominal multinomial data while there is an order to those in ordinal multinomial data.

For example, if our goal is to distinguish the three classes of plants in the IRIS dataset, we need to treat the plant categories as a nominal outcome because there is no specific order in the classes of plants. However, if we want to predict the score at the end of the first half in an NBA final into three categories, “winning”, “losing”, or “tie”, we need to regard it as an ordinal outcome because the pairwise distances are not the same among the three classes.

Example of (a) nominal data and (b) ordinal data

Therefore, it’s necessary to figure out what the type of outcome is before the multinomial modeling.

In this post, I am going to briefly talk about the multinomial regression models for the nominal data. I hope it will help you in your real-life work.

The multinomial logistic regression model

As aforementioned, the multinomial logistic regression was specifically designed for the nominal data. The idea is very similar to that of logistic regression on the binary data, which is to link the probability of belonging to one of the categories to the predictors.

For those who struggle to understand the link functions, you can refer to one of my previous posts about it.

Let Yi be a random variable that can fall in one of the classes, 1, 2, …, J. And we define pij as the ith variable to be in the jth class.

so, we have

the formula above is for the general case, so when J=2, it becomes the case with binary data.

If we have the total number of observations as ni, then the multinomial distribution could be described as below.

In the multinomial logistic regression, the link function is defined as

where

In this way, we link the log odds ratio between the probability to be in class J and that to be in class 1 to the linear combination of the predictors. Now let’s take a look at the real data and coding of the multinomial logistic regression.

The example is from Faraway’s book, which describes the 1996 American National Election Study. The response variable involves three categories, Democrat, Republican, and Independent. The predictors include age, education level, and income group of the respondents.

# R code

> head(nes96)

## popul TVnews selfLR ClinLR DoleLR PID age educ income vote

## 1 0 7 extCon extLib Con strRep 36 HS $3Kminus Dole

## 2 190 1 sliLib sliLib sliCon weakDem 20 Coll $3Kminus Clinton

## 3 31 7 Lib Lib Con weakDem 24 BAdeg $3Kminus Clinton

## 4 83 4 sliLib Mod sliCon weakDem 28 BAdeg $3Kminus Clinton

## 5 640 7 sliCon Con Mod strDem 68 BAdeg $3Kminus Clinton

## 6 110 3 sliLib Mod Con weakDem 21 Coll $3Kminus Clinton

After some pre-processing of the dataset, we build the multinomial logistic regression in R as below.

# R code library(nnet)

mulmod = multinom(sPID ~ age + educ + nincome, nes96)

where “sPID” is the response variable, “age”, “educ”, and “nincome” corresponds to the age, the education level, and the income of the respondents, respectively. And nes96 is just the dataset name.

We then check the summary of the fitted model.

# R code summary(mulmod)

which yields,

## Call:

## multinom(formula = sPID ~ age + educ + nincome, data = nes96)

##

## Coefficients:

## (Intercept) age educ.L educ.Q educ.C

## Independent -1.197260 0.0001534525 0.06351451 -0.1217038 0.1119542

## Republican -1.642656 0.0081943691 1.19413345 -1.2292869 0.1544575

## educ^4 educ^5 educ^6 nincome

## Independent -0.07657336 0.1360851 0.15427826 0.01623911

## Republican -0.02827297 -0.1221176 -0.03741389 0.01724679

##

## Std. Errors:

## (Intercept) age educ.L educ.Q educ.C educ^4

## Independent 0.3265951 0.005374592 0.4571884 0.4142859 0.3498491 0.2883031

## Republican 0.3312877 0.004902668 0.6502670 0.6041924 0.4866432 0.3605620

## educ^5 educ^6 nincome

## Independent 0.2494706 0.2171578 0.003108585

## Republican 0.2696036 0.2031859 0.002881745

##

## Residual Deviance: 1968.333

## AIC: 2004.333

The estimated coefficients, as well as the standard errors, are reported by the model summary. The output is a little different from those in the linear models.

Only the statistics of “Independent” and “Republican” are listed in the summary because here “Democrat” is used as the baseline category and the regression is modeling the log odds ratio against this baseline category.

You may notice that there is no indicator to show the significance of the predictors in the model. We can evaluate them in two ways.

First, we can use the popular Z-score to evaluate the significance of the preditors.

# R code

z = summary(mulmod)$coefficients/summary(mulmod)$standard.errors

p = (1 - pnorm(abs(z), 0, 1))*2

This procedure is easy to implement because it has a simplified assumption of the normal distribution which holds for a rough estimation. Let’s look at the p-values.

## (Intercept) age educ.L educ.Q educ.C educ^4

## Independent 2.464850e-04 0.97722232 0.88951010 0.76893540 0.7489628 0.7905471

## Republican 7.107723e-07 0.09464069 0.06630235 0.04189165 0.7509449 0.9374990

## educ^5 educ^6 nincome

## Independent 0.5854128 0.4774294 1.751402e-07

## Republican 0.6505832 0.8539066 2.165977e-09

It seems that income is a strong preditor with no doubt, but not all the education variables are listed as significant. This is also a shortcoming of using Z-distribution to evaluate the coefficients, which makes it hard to interpret.

A more precise way is to build a new model with a reduced number of predictors and then check the change in the likelihood. First, let’s build the model without variable education.

# R code

mulmod_noedu = multinom(sPID ~ age + nincome, nes96)

Then we use a chi-squared distribution to evaluate the change in likelihood between the two multinomial models. The p-value could be calculated as below.

# R code

pchisq(deviance(mulmod_noedu)-deviance(mulmod),mulmod$edf - mulmod_noedu$edf, lower.tail = F)

which yields,

0.1819634

which indicates that the variable ‘education’ is not significant relative to the full model. If we do the same thing to the variable ‘income’, we will get the following result.

# R code

mulmod_noinc = multinom(sPID ~ age + educ, nes96)

pchisq(deviance(mulmod_noinc)-deviance(mulmod),mulmod$edf - mulmod_noinc$edf, lower.tail = F)

which yields,

1.267249e-10

These results indicate that “income” is a significant predictor in the full model.

The interpretation of the coefficient should be such: one unit change in some predictor will result in k unit change in the log odds ratio between the target category to the baseline category (“Democrat” in this example). I will not explore the details of this part.

Hierarchical or non-hierarchical?

Another very important check on the response variable before the multinomial modeling is whether the response outcome is hierarchical.

Let’s look at an example of the central nervous system disease data from Faraway’s book.

data(cns)

head(cns)

yields,

## Area NoCNS An Sp Other Water Work

## 1 Cardiff 4091 5 9 5 110 NonManual

## 2 Newport 1515 1 7 0 100 NonManual

## 3 Swansea 2394 9 5 0 95 NonManual

## 4 GlamorganE 3163 9 14 3 42 NonManual

## 5 GlamorganW 1979 5 10 1 39 NonManual

## 6 GlamorganC 4838 11 12 2 161 NonManual

Here, “NoCNS” means no central nervous system disease, “An” means anencephalus, “Sp” means spina bifida, and “Other” means other types of CNS. “Area”, “Water” and “Work” describe the features of the families.

If we want to model the CNS status using the features of the families, we can simply implement the multinomial logistic regression on the four categories.

However, we can see that “An”, “Sp” and “Other” are just subtypes of CNS and the response variable is dominated by healthy individuals (NoCNS). In such a situation, we’d better treat the response variables as hierarchical variables, where “CNS” and “NoCNS” form a binomial distribution and “An”, “Sp”, and “Other” form a multinomial distribution within the “CNS” category.

Let’s compare the codes and results.

Let’s first establish a binomial logistic regression between the “CNS” and “NoCNS”.

# R code

cns$CNS = cns$An + cns$Sp + cns$Other

binom_mod = glm(cbind(CNS,NoCNS) ~ Water + Work, cns, family=binomial)

summary(binom_mod)

which yields,

##

## Call:

## glm(formula = cbind(CNS, NoCNS) ~ Water + Work, family = binomial,

## data = cns)

##

## Deviance Residuals:

## Min 1Q Median 3Q Max

## -2.65570 -0.30179 -0.03131 0.57213 1.32998

##

## Coefficients:

## Estimate Std. Error z value Pr(>|z|)

## (Intercept) -4.4325803 0.0897889 -49.367 < 2e-16 ***

## Water -0.0032644 0.0009684 -3.371 0.000749 ***

## WorkNonManual -0.3390577 0.0970943 -3.492 0.000479 ***

## ---

## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

##

## (Dispersion parameter for binomial family taken to be 1)

##

## Null deviance: 41.047 on 15 degrees of freedom

## Residual deviance: 12.363 on 13 degrees of freedom

## AIC: 102.49

##

## Number of Fisher Scoring iterations: 4

We can find that both “Water” and “WorkNonManual” are significant variables in predicting CNS or not.

Then, we perform the multinomial logistic regression on the subtypes of CNS.

# R code

cns_mod = multinom(cbind(An,Sp,Other) ~ Water + Work, cns)

z = summary(cns_mod)$coefficients/summary(cns_mod)$standard.errors

p= (1 - pnorm(abs(z), 0, 1))*2

p

which yields,

## (Intercept) Water WorkNonManual

## Sp 4.833577e-02 0.5234640 0.5791105

## Other 5.935107e-05 0.4511663 0.4052210

We can see that neither “Water” nor “WorkNonManual” is significant to distinguish different subtypes of CNS.

These results suggest that water quality and manual work do have an influence on the new births’ CNS condition, but neither of them explains the subtypes of the CNS.

However, if we throw everything to a four-category full model, the results are different.

# R code

full_mod = multinom(cbind(NoCNS, An, Sp, Other) ~ Water + Work, cns)

z = summary(full_mod)$coefficients/summary(full_mod)$standard.errors

p= (1 - pnorm(abs(z), 0, 1))*2

p

which shows,

## (Intercept) Water WorkNonManual

## An 0 0.066695285 0.02338312

## Sp 0 0.001835095 0.07077816

## Other 0 0.840643368 0.02348335

This time the predictors show some mixed results in interpreting the subtype of CNSs with different p-values. The fact that subtypes of CNS cannot be predicted by the two variables is not disclosed in the full model.

Takeaways

In summary, I introduced the multinomial regression model and its application situations. The most important point is to check the response variables before the modeling.

The multinomial logistic regression can only be applied to the nominal data instead of the ordinal data. The response variables with potential hierarchical structures should be treated carefully. Pooling everything together to a full model is not optimal sometimes.

References

Faraway, Julian J. Extending the linear model with R: generalized linear, mixed effects and nonparametric regression models. CRC press, 2016.

https://archive.ics.uci.edu/ml/datasets/iris