THE STORY OF LOGISTIC REGRESSION CONTINUES…
By: Nishesh Gogia
Tags: Data Science Interview, Logistic Regression, Interview Questions, Data Science, Machine Learning

In this article I will continue with the Story of Logistic Regression, if you have not read the previous article, here is the link

Let me make things clear and simple, even in this article i won’t be able to complete Logistic Regression because its a broad topic and we are covering it in detail so it is better to plan what we gonna cover in this article.

We will be covering:

Probablistic approach of sigma function. Optimization Theory. We will derive optimization problem of LR by geometry. WEIGHT VECTOR We will deal with Overfitting problem. We will see L1 and L2 regularization in detail.

So lets get started,

Probablistic approach of sigma function

Uptil now we have derived the optimization problem which can handle outliers by tappering off the distance we know why sigma function is used. One of the reason we use sigma function is that it has a great probalistic interpretation.

Lets see now the probablistic interpretation of sigma function.

CASE 1

Lets say we have the same classification problem in which we are classifying positives with negatives so lets imagine a query point which lies on the hyperplane, now intutively there is a 50 -50 chance for the query point to be classify as positve and as negative as well. So what will be the probablity that point is positive, it is 0.5 without any doubt and same for negative.

P(Y=1)=0.5

we have a sigma function which is defined as

sigma=1/(1+e^(-s)) and here for LR,

s=yi*w^T*xi, for the above problem w^T*xi is “0” because point lies on the hyperplane so s=0 which means sigma =0.5 same as probablity(found above)

CASE 2

Lets say we have the same classification problem in which we are classifying positives with negatives so lets imagine a query point which lies at a distance of 100 from hyperplane as shown in the image, now first think as according to our sigmoid function, below is the graph of sigma function.

when the point is at a far distance, lets say w^T*xi(distance from hyperplane) is 100 as i mention above and in the same direction of w, according to the graph its value will be 0.99999 which you can derive from the formula as well means P(Y=1)=0.99999 means there is a 99 percent probablity that Y belongs to positive class. So we can easily claim one result from this that is,

sigma(w^T*xi)=P(y=1) , where P(y=1) is equal to probability when y=1 means the point belongs to positive class.

It is obvious that “sigma(w^T*xi)” values lies from 0 to 1 as probablity value lies from 0 to 1.

Optimization Theory For LOGISTIC REGRESSION

So we need some optimization tools to optimize our LR problem. We can solve the previous optimization problem as well but we want to make it much better.

Uptil now we have seen our optimization problem which is less impacted by outliers.

Now to find the more optimize one, lets understand some optimization theory.

Monotonic Functions

lets say i have two points x1 and x2 and i know x1>x2 so if there is a function g(x1) and there is a function g(x2) and

if g(x1) >g(x2) then

g(x) is a monotonically increasing function like log(x)

As you can see in the graph, if x increases log(x) increases. Log(x) is a motonotically increasing function.

Now lets say we have a optimization problem in which we need to find the best value of x which minimize x^2

Now we can easily give its answer to be “0". As we can see in the graph below that for x=0 , x^2 is 0 which is the minimum value so x* value is 0 from the above picture.

Now lets say i have a function g(x) =log(x) and i want to find the best value of x for which i can get the minimum of g(f(x)) where f(x) is x^2.

so again i have a optimization problem in which we need to find the x which minimizes g(f(x))

g(f(x))= log(x^2).

Now optimization theory says that x* which we found in the first problem( while using only f(x)=x^2) is equal to x’ which we found in the second problem( while using g(f(x)) if g(x) is monotonically increasing.

We can see it by the graph below as well, we found x*=0 so if we see the plot of log(x^2), we can see it is minimum at 0 and this happens because log(x) is a monotonially increasing function.

Lets apply this theory to our LR OPTIMIZATION PROBLEM

WEIGHT VECTOR

Now one think can be very easily misunderstood that is

“w*” vector, firstly this is a vector which we got after solving the optimization problem.

HOW WE SOLVE IT?

We will discuss this is detail in future but for now lets say we got w*(optimal w) somehow, for now i will write w* as w only.

So “w” is a d dimension vector because otherwise w^T*x is not possible as x is also d dimensions.

Here dimensions simply means features so

We can understand a w vector as weights for each feature.

w=<w1, w2, w3…wd>

for features

f=<f1, f2, f3….fd>

a weight for each feature.

Lets see what happens at the testing time.

So lets say you have a query point(xq) and you need to find its class lable(yq) which can be positive or negative.

so if

w^T*xq>0 then yq=+ve

where w is the optimal w which we found after solving the optimization problem.

if

w^T*xq<0 then yq=-ve

OVER FITTING PROBLEM

Now we have a problem, lets see it through.

Uptil now we got our optimized probelm

lets say zi=yi*w^T*xi

now if we analyze we know by nature, exp(-zi) is always greater equal to zero, (1+exp(-zi)) is always greater than zero and log being a monotic function

log(1+exp(-zi))>0

So summation of all the values of log over different zi’s is greater than zero,means if we want to find the minimum value, it must be zero. So basically we need to find out w* for which the whole summation term becomes zero.

Now if we analyze closely we can say log value is “0” only at x=1, log(1)=0, means 1+exp(-zi)=1, means exp(-zi) becomes 0, for which zi has to become infinity.

zi =yi*w^T*xi, we are given with yi and xi, when we are saying zi’s to become infinity that means w has to become infinite and if zi is infinity we are also saying that zi>0 means we are always classifying the all the training points correctly and that means we are overfitting the model.

Now if we Recall we are foregetting one key aspect that w is normal to hyperplane.

w^t*w=1, we have not used this condition anywhere.

so lets use this.

HOW TO DEAL WITH THIS PROBLEM?

The answer is Regularization, there are two types of regularization L1 AND L2, Regularization simply is like to add the other term so that it create stablity and reduce overfitting.

L2 Regularization

if we write our optimization problem like this,

Now if we see we want to find the w which minimizes this whole term, if we want to reduce log term we need to increase w and if we increase w term then the regularization term will be maximum and we won’t get minimum value and there is a no use of regularization term.

Now if we decrease the value of w, then log term will not be 0 and again we wont find the minimum value.

Here lambda is a hyperparameter which can be find by cross validation

So if lamda is “0” means no regularization term means overfitting.

And if lambda is infinity means influence of loss term will be less so there will be no use training data, model will underfit.

L1 REGULARIZATION

L1 REGULARIZATION IS GREAT IF YOU WANT LOW LATENCY SPECIALLY IN INTERNET COMPANIES WHERE USE EXPECT RESULT TO COME IN SECONDS.

It actually creates a the solution of logistic regresstion that is “w” too sparse means if w are less significant, if some features are less significant in contributing for the solution then L1 will make that w zero.

PPPPPP

WHY L1 CREATES SPARCITY IN “W” COMPARED TO L2 REGULARIZATION??

There is another method of regularization and that is Elastic Net. We bascially combined both L1 and L2 and used simply found two hyperparametes by cross validation.

So thats it for this article, thats more or less the story behind LOGISTIC REGRESSION.

Thanks for reading.