Introduction to Logistic Regression
By: Bhanu Soni
Tags: Machine Learning, Artificial Intelligence, Logistic Regression, Linear Regression, Regression

Introduction to Logistic Regression

Part 1

Regression is basically a statistical approach to find the relationship between variables.

Regression methods are an important part of any data analysis concerned with describing the relationship between a response variable (output/outcome) and one or more explanatory variables(input). Pretty often the outcome variable is discrete, i.e. taking on two or more possible values.

Before starting studying the logistic regression model it is important to understand the goal of an analysis using this model: “it is the same as that of any other regression model used in statistics, that is, to find the best fitting and most parsimonious, interpretable model to describe the relationship between an outcome (dependent or response) variable and a set of independent (predictor or explanatory) variables’’.

Independent variables are also known as Covariates.

In the linear regression model, the outcome variable is assumed to be continuous.

Everything that distinguishes a logistic regression model from the linear regression model is the outcome variable in logistic regression is binary or dichotomous.

This difference between logistic and linear regression is recognized both in the form of the model and its assumptions. Once this difference is accounted for, the methods applied in an analysis using logistic regression follows the same basic principles used in linear regression.

For instance, we have a dataset of ‘Coronary Heart Disease’, with following columns age in years (AGE), and presence or absence of evidence of significant coronary heart disease (CHD), identifier variable (ID) and an age group variable (AGE_GROUP).

The outcome (dependent/response) variable is CHD, which is coded with a value of “0” to indicate that CHD is absent, or “1” that it is present in the individual. In general, any two values could be used, but working with 0 and 1 is more convenient.

To explore the relationship between AGE and the presence or absence of CHD in this group let’s plot a scatter plot outcome v/s the independent variable.

We will use this scatterplot to find the relationship between the outcome and the independent variable. Interestingly, all data points fall on one of two parallel lines representing the absence of CHD (y = 0) or the presence of CHD (y = 1).

Although this plot does represent the binary nature of the outcome variable pretty clearly, it does not provide a clear picture of the nature of the relationship between CHD and AGE.

However, The main problem here is that the variability in CHD at all ages is

large. This makes it difficult to see any functional relationship between AGE and CHD.

One common method of removing some variation, however preserving

the structure of the relationship between the outcome and the independent variable is to create intervals for the independent variable and compute the mean of the outcome variable within each group.

We can achieve the same with our dataset by grouping age into the categories (Age_Group). Doing so, we have, for each age group, the frequency of occurrence of each outcome, as well as the percent with CHD present.

On plotting to scatter plot we have:

Here, a clearer picture of the relationship begins to develop. It tells that as age increases, the proportion (mean) of individuals with the sign of CHD increases. This plot provides noteworthy insight into the relationship between CHD and AGE, but the functional form for this relationship needs to be drawn.

Here we have 2 important differences :

It concerns the nature of the relationship between the outcome and independent variables. In any regression problem, the key quantity is the mean value of the outcome variable, given the value of the independent variable. This quantity is called the conditional mean and is expressed as “E(Y |x)”,

Where:

Y = outcome variable

x = specific value of the independent variable.

The quantity E(Y |x) is read “the expected value of Y , given the value x”.

In linear regression, we assume this mean may be expressed as a linear equation, such as:

E(Y |x) = β 0 + β 1 x

It is possible for E(Y |x) to take on any value as x ranges between −∞ and +∞.

In our dataset column named “Mean” provides an estimate of E(Y |x). The estimated values plotted in Plot 2 are close enough to the true values of E(Y |x) to provide a reasonable estimation of the functional relationship between CHD and AGE.

The conditional mean must be greater than or equal to zero and less than 1 (i.e., 0 ≤ E(Y |x) ≤ 1). The change in the E(Y |x) per unit change in x becomes progressively smaller as the conditional mean gets closer to zero or one. The curve is said to be S-shaped. The model we use is based on the logistic distribution.

There are 2 reasons for using logistic distribution :

First, it is an extremely flexible and easily used function. Second, its model parameters provide the basis for estimates of effect.

The logistic regression model we use is

By the above equation: Logistic Regression can be defined as:

It is defined by the following expression ‘’exponential of a linear combination of inputs and coefficients divided by one plus the same exponential that form seems quite complex.’’

Logistic regression predicts the probability of the outcome variable being true. It can be considered the classification counterpart of linear regression.

A transformation of E(Y |x) is necessary for our study of logistic regression is the logit transformation.

This transformation has many desirable properties of a linear regression model. The logit, g(x), is linear in its parameters, may be continuous, and may range from −∞ to +∞.

The second difference between the linear and logistic regression models concerns with conditional distribution of the outcome variable. In the linear

regression model, the error term (ε) associated with the equation assumed to follows a normal distribution with mean zero

However, in logistic regression, the quantity ε may assume one of two possible values. If y = 1 then ε = 1 − E(Y |x)with probability E(Y |x), and if y = 0 then ε = −E(Y |x) with probability 1 − E(Y |x).

Thus, ε has a distribution with mean zero and variance equal to

E(Y |x)[1 − E(Y |x)].

That is, the conditional distribution of the outcome variable follows a binomial distribution with probability given by the conditional mean,

E(Y |x).

In nutshell, we can say that in regression analysis when the outcome variable is dichotomous:

1. The model for the conditional mean of the regression equation must be

bounded between zero and one.

2. The binomial, not the normal, distribution describes the distribution of the

errors and is the statistical distribution on which the analysis is based.

3. The principles that guide an analysis using linear regression also guide us in logistic regression.