Negative Binomial Regression: A Step by Step Guide
By: Sachin Date
Tags: Poisson, Artificial Intelligence, Machine Learning, Mathematics, Programming

How to do Negative Binomial Regression in Python

We’ll start by importing all the required packages.

import pandas as pd

from patsy import dmatrices

import numpy as np

import statsmodels.api as sm

import matplotlib.pyplot as plt

Next, create a pandas DataFrame for the counts data set.

df = pd.read_csv('nyc_bb_bicyclist_counts.csv', header=0, infer_datetime_format=True, parse_dates=[0], index_col=[0])

We’ll add a few derived regression variables to the X matrix.

ds = df.index.to_series()

df['MONTH'] = ds.dt.month

df['DAY_OF_WEEK'] = ds.dt.dayofweek

df['DAY'] = ds.dt.day

We will not use the Date variable as a regressor since it contains an absolute date value but we don’t need to do anything special to drop Date as it is already consumed as the index of the pandas DataFrame. So it will not be available to us in the X matrix.

Let’s create the training and testing data sets.

mask = np.random.rand(len(df)) < 0.8

df_train = df[mask]

df_test = df[~mask]

print('Training data set length='+str(len(df_train)))

print('Testing data set length='+str(len(df_test)))

STEP 1: We will now configure and fit the Poisson regression model on the training data set.

Setup the regression expression in patsy notation. We are telling patsy that BB_COUNT is our dependent variable and it depends on the regression variables: DAY, DAY_OF_WEEK, MONTH, HIGH_T, LOW_T and PRECIP.

expr = """BB_COUNT ~ DAY + DAY_OF_WEEK + MONTH + HIGH_T + LOW_T + PRECIP"""

Set up the X and y matrices for the training and testing data sets. patsy makes this really simple.

y_train, X_train = dmatrices(expr, df_train, return_type='dataframe')

y_test, X_test = dmatrices(expr, df_test, return_type='dataframe')

Using the statsmodels GLM class, train the Poisson regression model on the training data set.

poisson_training_results = sm.GLM(y_train, X_train, family=sm.families.Poisson()).fit()

This finishes the training of the Poisson regression model. To see outcome of the training, you can print out the training summary.

print(poisson_training_results.summary())

This prints out the following:

Training summary for the Poisson regression model (Image by Author)

Our real interest lies in the vector of fitted rates λ produced by the training. This rate vector is contained in the parameter poisson_training_results.mu.

print(poisson_training_results.mu)

print(len(poisson_training_results.mu))

The following output shows the first few and last few values of fitted λ vector:

[1920.39434028 2544.81549207 2651.79330653 743.45309242 1072.77132837 1892.9428398 2320.70412868 3269.73598361 3313.21764921 2915.25363322 2614.39482509 2594.44594144 2415.29471195 3181.91998369 2154.15471026 2268.25625592 1793.29625903 2535.31903414 2566.70835529 970.82159668

2510.70775659 3016.19901465 2260.265944 3365.04650316 2695.6143122...

...2340.12964253 2568.40001641 2232.26752534 2604.97128321 145.92037793 2060.10442187 2518.70470296]

This completes STEP1: fitting the Poisson regression model.

STEP 2: We will now fit the auxiliary OLS regression model on the data set and use the fitted model to get the value of α.

Import the api package.

import statsmodels.formula.api as smf

Add the λ vector as a new column called ‘BB_LAMBDA’ to the Data Frame of the training data set. Recollect that λ’s dimensions are (n x 1). In our example it will be (161 x 1). Also recollect that the λ vector is available in poisson_training_results.mu :

df_train['BB_LAMBDA'] = poisson_training_results.mu

Next, let’s add a derived column called ‘AUX_OLS_DEP’ to the pandas Data Frame. This new column will store the values of the dependent variable of the OLS regression. It is the left hand side of the OLS regression equation below:

Auxiliary OLS regression to find α for the NB2 model (Image by Author)

df_train['AUX_OLS_DEP'] = df_train.apply(lambda x: ((x['BB_COUNT'] - x['BB_LAMBDA'])**2 - x['BB_LAMBDA']) / x['BB_LAMBDA'], axis=1)

In the above code snippet, the part in bold is the left hand side of the above aux OLSR equation.

Let’s use patsy to form the model specification for the OLSR. We want to tell patsy that AUX_OLS_DEP is the dependent variable and it is explained by BB_LAMBDA (which is the rate vector λ). The ‘-1’ at the end of the expression is patsy syntax for saying: do not to use an intercept of regression; i.e. just fit a straight line passing through the origin, as suggested by Messrs Cameron and Trivedi.

ols_expr = """AUX_OLS_DEP ~ BB_LAMBDA - 1"""

We are now ready to fit an OLSR model.

Configure and fit the OLSR model:

aux_olsr_results = smf.ols(ols_expr, df_train).fit()

Print the regression params:

print(aux_olsr_results.params)

You will see the following single coefficient being printed out corresponding to the single regression variable BB_LAMBDA. This coefficient is the α that we were seeking:

BB_LAMBDA 0.037343

Is α statistically significant?

We now need to answer a very important question. Is this value of α (0.037343) statistically significant? Or can it be considered to be zero for all practical purposes?

Why is it so important to find this out? Recollect that if α is zero, then the following equation:

The NB2 model’s variance function (Image by Author)

…reduces to Variance = mean. This is the variance function of the Poisson regression model.

If the value of α is statistically not significant, then the Negative Binomial regression model cannot do a better job of fitting the training data set than a Poisson regression model.

The OLSResults object contains the t-score of the regression coefficient α. Let’s print it out:

aux_olsr_results.tvalues

This prints out:

BB_LAMBDA 4.814096

From a t-value calculator, we can see that the critical t-value at a 99% confidence level (right-tailed), and degrees of freedom=(172 observations) —( 6 regression variables)— (1 dispersion parameter α)=165 is 2.34916. This is comfortably less than the t-statistic of α which was 4.814096. We conclude that,

α=0.037343 is statistically significantly.

This completes STEP 2: The determination of α.

STEP 3: We supply the value of alpha found in STEP 2 into the statsmodels.genmod.families.family.NegativeBinomial class, and train the NB2 model on the training data set.

This is a one-step operation in statsmodels:

nb2_training_results = sm.GLM(y_train, X_train,family=sm.families.NegativeBinomial(alpha=aux_olsr_results.params[0])).fit()

As before, we’ll print the training summary:

print(nb2_training_results.summary())

Which prints the following summary:

NB2 model’s training summary (Image by Author)

STEP 4: Let’s make some predictions using our trained NB2 model.

Prediction is again a single-step procedure in statsmodels:

nb2_predictions = nb2_training_results.get_prediction(X_test)

Let’s print out the predictions:

predictions_summary_frame = nb2_predictions.summary_frame()

print(predictions_summary_frame)

Following are the first few lines of the output:

First few rows of output from nb2_predictions.summary_frame() (Image by Author)

Let’s also plot the predicted counts versus the actual counts for the test data.

predicted_counts=predictions_summary_frame['mean'] actual_counts = y_test['BB_COUNT'] fig = plt.figure() fig.suptitle('Predicted versus actual bicyclist counts on the Brooklyn bridge') predicted, = plt.plot(X_test.index, predicted_counts, 'go-', label='Predicted counts') actual, = plt.plot(X_test.index, actual_counts, 'ro-', label='Actual counts') plt.legend(handles=[predicted, actual]) plt.show()

Here’s the output:

Predicted versus actual bicyclist counts on the Brooklyn bridge using the NB2 model

Not too bad! The NB2 model seems to be more or less tracking the trend in the bicycle counts. And just as with the Poisson regression model’s performance, in some cases its predictions are way off the actual values.

Here is the complete Python source code for training a Negative Binomial Regression model and testing its predictions:

The final question before us is:

Statistically, has our NB2 regression model done a better job than the Poisson regression model?

Let’s find out.