Geometric Intuition OF Logistic Regression
By: Himanshu Negi
Tags: Machine Learning, Logistic Regression, Data Science, Intuition, Statistics

What is Logistic Regression:

It is the multi-class classification technique used in Machine Learning to predict the class label i.e Yi,s where Yi belong to some class but not a real value.

Let,s understand it geometrically:

ASSUMPTION : The biggest assumption before solving it is our data is linearly separable or almost linearly separable. ⇒ Now let we have two classes of points i.e positive and negative.

In above picture we have: W ⇒Normal to plane, Pi(𝜋) ⇒ Plane

So the equation of plane will be : W(transpose)X+b=0 where b is scalar

And If we pass the plane through origin : W(transpose)X=0

So now LET we have are given DATASET (Dn) having positive and negative points.

TASK IN LOGISTIC REGRESSION IS : To find W and b to discover a plane such that it separates positive and negative point i.e we have to find (𝜋).

Here, in above image_1 we can find out the distance from any point to plane Pi(𝜋). Also we assumed W is a unit vector and normal to plane.

So now comes to interesting part via seeing diagram in below picture i.e if we calculate:

1⇒ Distance from positive point to plane (𝜋) it will be positive , i.e di=W(transpose) Xi>0 because (W and Xi are on same side).

NOW THE LOGICAL PART COMES: So our classifier says !

⇒ if W(transpose)Xi>0 then Yi=+1 and

⇒ if W(transpose) Xi<0 then Yi=-1

Now lets take different cases:

case1: For positive points:

Let Pi(𝜋) passes through origin.

Now if Yi is positive i.e (+1) and W(transpose) Xi>0 →(So our classifier saying!) Now if

(Yi* W(transpose) Xi)>0 and Yi is positive then W is correctly classifying the positive point.

case2: For negative points:

let Pi(𝜋) passes through origin

If Yi is negaitive i.e (-1) and W(transpose) Xi<0 (our classifier saying ! )

Now if: (Yi*W(transpose) Xi)>0 and Yi is negative

because (-)*(-)=(+)then W is correctly classifying the negative points.

OBSERVATION : From above logical part is: for both positive and negative points!

If (Yi* W(transpose) Xi)>0 then L.R model is correctly classifying the points Xi,s ……

NOW in the same fashion for negative points as follows:

case3: If Yi is positive i.e (+1) and W(transpose) Xi)<0 then L.R is saying it is negative class:

Then (Yi* W(transpose) Xi)<0 i.e miss classified point.

case4: If Yi is negative i.e (-1 )and W(transpose) Xi)>0 then L.R is saying it is positive class:

Then (Yi* W(transpose) Xi)<0 i.e miss classified point.

OBSERVATION : From above logical part is:

For both positive and negative points :

if (Yi* W(transpose) Xi)<0 then L.R model is incorrectly classifying the points Xi,s.

At the end of the day we want our classifier V.good i.e minimize the no. of miss classification or maximize the no. of correctly classified point. i.e we want as many points possible to have :

(Yi* W(transpose) Xi)>0

i.e: W*=arg max(w)(i=1 to n):(Yi* W(transpose) Xi)……..(OPTIMIZATION PROBLEM)

Here in above equation Xi and Yi are fixed comes from (Dtrain), now we have to compute W variable here,SO our optimization problem here is to find W* (optimal w) to maximize:

Now by understanding function we have (Yi* W(transpose) Xi) is signed distance. Now let,s assume by looking below image:

BY OBSERVING FROM ABOVE IMAGE:

Let we have 10 points equal distance from plane Pi-one(𝜋1) and X 100 unit distance from plane Pi-one(𝜋1): ⇒

CASE 1: Now if we choose W1 and Pi-one(𝜋1) as our separator

Then (i=1 to n):(Yi* W1(transpose) Xi)=5+5–100=-90, Here we are trying to maximize the signed distance.

CASE 2:Now if we choose W2 and Pi-two(^2) as our separator

Then (i=1 to n):(Yi* W2(transpose) Xi)=1+2+3+4+5-1-2-3-4–5+1=1

Now as per our objective we will choose Pi-two(𝜋2) as our classifier.But if we think intuitively Pi-two(𝜋2) is terrible classifier i.e only one point is correctly classified.

Now Accuracy of Pi-one(𝜋1)=(10/11)=0.909

and Accuracy of Pi-two(𝜋2)=(6/11) =0.545

Here the accuracy to maximum sum of signed distance Pi-two(𝜋2) is better then Pi-one(𝜋1) as a classifier.

This is happening because of extreme point which is 100 units distance from Pi-one(𝜋1) in the direction of W1, i.e single extreme/outlier point is changing our model. So max-sum of signed distance can be impacted by outlier.

Now we have to modify W*=(arg max(w)(i=1 to n):(Yi* W(transpose) Xi)) by Technique called Squshing.

SQUASHING →

So instead of using simple signed- distance we will use :If signed- distance is small: Use as it is.If signed- distance is large: Make it a small value.So we want a function :When its value is small : increasing linearly.When its value becomes large : Tapper it off.Now, One such function we have is SIGMOID FUNCTION

Now if we apply Sigmoid function to our W* we will have:

W*=(arg max(w)(i=1 to n):1/1+(exp(-(Yi* W(transpose) Xi))).

So this above function will be less impacted by outlier.

Now we know the property of monotonic function:

If x increases then G(x) increases and If x1>x2 then G(x1)>G(x2)

Now if G(x ) is monotonic then…..

1>argmin F(x)=argmin G(F(x)) ………

2>argmaxF(x)=argmax G(F(x))……

NOW — -> We know that log is monotonic function and log(1/x)=-log(x), Now if we imply log on W* then W* will be:

W*=(arg max(w)(i=1 to n):log(1/1+(exp(-(Yi* W(transpose) Xi))))

W*=(arg max(w)(i=1 to n): -log(1+(exp(-(Yi* W(transpose) Xi))))

W*=(arg min(w)(i=1 to n): log(1+(exp(-(Yi* W(transpose) Xi))))……(equation 1)

Now if there will be no 1 in the above equation then:

W*=(arg min(w)(i=1 to n): log((exp(-(Yi* W(transpose) Xi))))

W*=(arg min(w)(i=1 to n): -(Yi* W(transpose) Xi)

W*=(arg max(w)(i=1 to n): (Yi* W(transpose) Xi)……(equation 2) i.e SUM of signed distances

So in optimization problem of Logistic Regression with a small change using exp and log we have sum of signed distance which have huge problem of outlier so that,s why we will not use …(equation 2) for optimization problem.

Other formulation of W* using probabilistic method is as follows which we are writing directly here just to correlate:

W*=arg min(w)(i=1 to n): -{Yi*log(P(i))+(1-Yi)*log(1-P(i))}

if we correlate this equation with……(equation 1),

Then P(i) ⇒ Sigmoid(W(transpose) Xi).

NOW from ……(equation 1) we get W* optimal, W is also called Weight Vector

i.e W belong to Rd =<w1,w2,w3……..wd>,d=no of features/dimentions.

In L.R we have weight associated with every features:

If Wi is positive then Xqi high ⇒Wi*Xqi will high ⇒(i:1 to d (Wi*Xqi)) will high then sigmoid(W(transpose) Xi) also high i.e P(Yq=Positive) will high.

In similar fashion :

If Wi is nagative then Xqi high ⇒Wi*Xqi will low ⇒(i:1 to d (Wi*Xqi)) will low then sigmoid(W(transpose) Xi) also low i.e P(Yq=Positive) will low ⇒ P(Yq=Negative) will high.

L2 REGULARIZER OVERFITTING AND UNDERFITTING:

So our optimization problem was: from equation………..1

W*=(arg min(w)(i=1 to n): log(1+(exp(-(Yi* W(transpose) Xi))))

Let (Yi* W(transpose) Xi)=Zi

then W*=(arg min(w)(i=1 to n): log(1+(exp(-Zi))), Now

SO if we will pick W such that all training point are correctly classified and all Zi → (minus infinity) then such W is the best W, but with the best W we have problem if some points are incorrect or outlier i.e condition of overfitting.

So the solution of above problem is adding the regularizer i.e L2 HERE: Now the whole equation of optimization will be:

Now if we see as Wj →infinity then SQUARE(Wj) →infinity and out System will not let it do because our optimization system is trying to minimize the W*. So by adding [W(transpose)W] it avoid Wj to — -> infinity.NOW our final OPTIMIZATION EQUATION WILL BE:

SOME abbreviation

Thanks for READING!