Understanding Logistic Regression in Python?

Logistic Regression is a statistical technique to predict the binary outcome. It’s not a new thing as it is currently being applied in areas ranging from finance to medicine to criminology and other social sciences.
In this section we are going to develop logistic regression using python, though you can implement same using other languages like R.
We’re going to use below libraries in our example program,
Numpy: To define the numerical array and matrix
Numpy: To define the numerical array and matrix
Pandas: To handle and operate on data
Pandas: To handle and operate on data
Statsmodels: To handle parameter estimation & statistical testing
Statsmodels: To handle parameter estimation & statistical testing
Pylab: To generate plots
Pylab: To generate plots
You can install above libraries using pip by running below command in CLI.
>pip install numpy pandas statsmodels
To test our logistic regression in python, we are going to use the logit regression data provided by UCLA (Institute for digital research and education). You can access the data from below link in csv format: https://stats.idre.ucla.edu/stat/data/binary.csv
I have saved this csv file in my local machine & will read the data from there, you can do either. With this csv file we are going to identify the various factors that may influence admission into graduate school.
We are going to read the data using pandas library (pandas.read_csv):
import pandas as pd
import statsmodels.api as sm
import pylab as pl
import numpy as np
df = pd.read_csv('binary.csv')
#We can read the data directly from the link \
# df = pd.read_csv(‘https://stats.idre.ucla.edu/stat/data/binary.csv’)
print(df.head())
   admit   gre gpa rank
0   0      380 3.61 3
1   1     660 3.67 3
2   1 800 4.00 1
3   1 640 3.19 4
4   0 520 2.93 4
As we can see from above output, one column name is ‘rank’, this may create problem since ‘rank’ is also name of the method in pandas dataframe. To avoid any conflict, i’m changing the name of rank column to ‘prestige’. So let’s change the dataset column name:
df.columns = ["admit", "gre", "gpa", "prestige"]
print(df.columns)
Index(['admit', 'gre', 'gpa', 'prestige'], dtype='object')
In [ ]:
Now everything looks ok, we can now look much deeper what our dataset contains.
Using pandas function describe we’ll get a summarized view of everything.
print(df.describe())
            admit         gre             gpa          prestige
count    400.000000     400.000000     400.000000     400.00000
mean       0.317500     587.700000       3.389900       2.48500
std        0.466087     115.516536       0.380567       0.94446
min        0.000000     220.000000       2.260000       1.00000
25%        0.000000     520.000000       3.130000       2.00000
50%        0.000000     580.000000       3.395000       2.00000
75%        1.000000     660.000000       3.670000       3.00000
max        1.000000     800.000000       4.000000     4.00000
We can get the standard deviation of each column of our data & the frequency table cutting prestige and whether or not someone was admitted.
# take a look at the standard deviation of each column
print(df.std())
admit      0.466087
gre     115.516536
gpa 0.380567
prestige 0.944460
dtype: float64
# frequency table cutting presitge and whether or not someone was admitted
print(pd.crosstab(df['admit'], df['prestige'], rownames = ['admit']))
prestige   1  2  3  4
admit
0         28 97 93 55
1 33 54 28 12
Let’s plot all the columns of the dataset.
# plot all of the columns
df.hist()
pl.show()
Python pandas library provides great flexibility in how we categorical variables are represented.
# dummify rank
dummy_ranks = pd.get_dummies(df['prestige'], prefix='prestige')
print(dummy_ranks.head())
         prestige_1   prestige_2   prestige_3   prestige_4
0                 0            0            1            0
1                 0            0            1            0
2                 1            0            0            0
3                 0            0            0            1
4                 0            0            0            1
# create a clean data frame for the regression
cols_to_keep = ['admit', 'gre', 'gpa']
data = df[cols_to_keep].join(dummy_ranks.ix[:, 'prestige_2':])
     admit  gre  gpa  prestige_2  prestige_3  prestige_4
0        0  380  3.61          0           1           0
1        1  660  3.67          0           1           0
2        1  800  4.00          0           0           0
3        1  640  3.19          0           0           1
4       0 520 2.93 0 0 1
In [ ]:
Now we are going to do logistic regression, which is quite simple. We simply specify the column containing the variable we’re trying to predict followed by the columns that the model should use to make the prediction.
Now we are predicting the admit column based on gre, gpa and prestige dummy variables prestige_2, prestige_3 & prestige_4.
train_cols = data.columns[1:]
# Index([gre, gpa, prestige_2, prestige_3, prestige_4], dtype=object)

logit = sm.Logit(data['admit'], data[train_cols])

# fit the model
result = logit.fit()
Optimization terminated successfully.
Current function value: 0.573147
Iterations 6
Let’s generate the summary output using statsmodels.
print(result.summary2())
                     Results: Logit
===============================================================
Model:              Logit             No. Iterations: 6.0000
Dependent Variable: admit           Pseudo R-squared: 0.083
Date:               2019-03-03 14:16             AIC: 470.5175
No. Observations:   400                          BIC: 494.4663
Df Model:             5               Log-Likelihood: -229.26
Df Residuals:       394                      LL-Null: -249.99
Converged:       1.0000                        Scale: 1.0000
----------------------------------------------------------------
Coef. Std.Err. z P>|z| [0.025 0.975]
----------------------------------------------------------------
gre 0.0023 0.0011 2.0699 0.0385 0.0001 0.0044
gpa 0.8040 0.3318 2.4231 0.0154 0.1537 1.4544
prestige_2 -0.6754 0.3165 -2.1342 0.0328 -1.2958 -0.0551
prestige_3 -1.3402 0.3453 -3.8812 0.0001 -2.0170 -0.6634
prestige_4 -1.5515 0.4178 -3.7131 0.0002 -2.3704 -0.7325
intercept -3.9900 1.1400 -3.5001 0.0005 -6.2242 -1.7557
==============================================================
The above result object also lets us to isolate and inspect parts of the model output.
#look at the confidence interval of each coeffecient
print(result.conf_int())
                  0           1
gre            0.000120   0.004409
gpa            0.153684   1.454391
prestige_2     -1.295751  -0.055135
prestige_3     -2.016992  -0.663416
prestige_4     -2.370399  -0.732529
intercept      -6.224242  -1.755716
From above output, we can see there is an inverse relationship b/w the probability of being admitted and the prestige of a candidate’s undergraduate school.
So the probability of a candidate to being accepted into a graduate program is higher for students who attended a top ranked undergraduate college(prestige_1= True) as opposed to a lower ranked school (prestige_3 or prestige_4).