Logistic Regression — Part 2. Mathematical and Geometrical Intuition…
By: Bidisha Pyne
Tags: Machine Learning, Logistic Regression, Data Science, Mathematics, Geometry

Prerequisite: This is assumed that you have a basic concept of Logistic Regression. If not, please refer to the first part of this topic.

From the first part, we know why Linear Regression is not used for binary classification. And, how Logistic Regression mitigates the problem of Linear Regression. Also, how easily the same can be implemented with the help of sklearn library.

Now, we will dive deeper to understand how Logistic Regression works in a mathematical and geometrical intuitive way.

Let’s try to understand this below graph.

Figure 1. Linear plane separating two data points

Here, we have two categories the red data points and the blue data points. Let’s consider the red data points to be positive points, that is, +1 and other blue data point to be -1 (negative points) and a linear separable plane is classifying these two types of data point.

This linear plane can be represented mathematically with help of any of the following equations:

Different forms of Linear Equation

Here, c, β0 and b are intercepts and m, β1 and w^T are slope. This best fit line (linear separable plane as in Figure 1) is not calculated with least square method rather it is required to find the best slope value (i.e., the value of m, β1 or w^T)

From Figure 1, as our plane passes through the origin, we can say, the intercept value (i.e., the value of c, β0 and b) to be zero, then our equation(y=w^T*x+b) looks like this: y = w^T * x.

Now, let us understand how we measure the distance of a point from a linear plane with the help of linear algebra.

Figure 2. Distance between a data point and the plane

Let’s consider P be a point and AB be the linear plane with slope w^T, then the distance between them is OP where OP = (w^T * x+ b)/ |w|. ||denotes the absolute value. If we consider the absolute value of w as the unit vector, then |w| = 1 and here, the value of b is considered as 0 from Figure 1. Then, this equation becomes, OP = w^T * x. So, the distance of many such data points with the plane can be considered as follows:

Sum of distance between all data points and the plane

From Figure 2, as point P is above the positive plane so OP =(+ve) w^T * x, but as the point N is on the negative side of the slope, so NM =(-ve) w^T*x.

Now, let’s look into the following cases derived from Figure 1.

Figure 3. Distance between red point and positive slope of the plane

Case 1 from the above Figure 3

Consider, y is my red point i.e., y = +1, and the distance with the plane is +ve w^T * x. Hence, the product of two is as below:

Case 2 from the below Figure 4

Figure 4. Distance between blue point and negative slope of the plane

Similarly, y is the blue point i.e., y = -1, and distance with the plane is (-ve) w^T * x. Hence, the product of two is as below:

Let’s define Case 3 with the help this Figure 5

Figure 5. Distance between blue point and positive slope of the plane

Here, again y is the blue point i.e., y = -1, and the distance with the plane is (+ve) w^T * x. Hence, the product of two is as below:

Now, Case 4 for the below Figure 6

Figure 6. Distance between red point and negative slope of plane

Similarly, again y is the red point i.e., y = +1, and the distance with the plane is (-ve) w^T * x. Hence, the product of two is as below:

So, from the above cases, it is clear that positive product implies correct classification as in cases 1 and 2 whereas, the negative product implies incorrect classification as in cases 3 and 4.

From the above cases, this to be noted that we need to maximum product values as in the Figure 7, to get the best fit line, for that we just need to update the slope.

Maximum Product value

So, the main aim of Logistic Regression is to find the maximum value of the product as above. But, not we need to understand the need of sigmoid function, and why do we actually need that. For, that let’s understand the Figure 8.

Figure 8. Linear plane separating data points

For simplicity, lets consider each of red data points (+ve points) and the blue points (-ve points) are equidistant from the linear plane in Figure 8. for say ie, 2, except the outlier of-course. The the product for each red point is 2 (as, y= +1 and w^T*x > 0) and product for blue one is also 2, (as, y=-1 and w^T * x < 0). Now, coming to the outlier, the distance let it be 100, then the product for that is -100 (as, y=-1 and w^T*x > 0). Hence, summation of product all the data points is equal to (2+2+2+2+2+2+2+2+2+2+2+2–100)=-76.

On a second scenario, if we shift the linear plane to something like in Figure 9.

Figure 9. Another Linear plane to find the best fit

Now, with the new linear separator, the product of the distance with blue points are -6, -5, -4, -3, -2, -1 (for say, as the blue points, y=-1 and w^T*x>0) and with red points are 6, 5, 4, 3, 2,1 (for say, as y=+1 and w^T*x > 0). Then, the product for blue outlier point be 1 (for say, as y=-1 and w^T*x <0)

Hence, the product of all the data points sums up to (-6–5–4–3–2–1+6+5+4+3+2+1+1) = 1. As, 1 > -76, then this linear plane as in Figure 9 can be selected as best fit, which is absolutely wrong. Now, here sigmoid function plays a very important role.

Hope, you understood from the above example that wrong linear plane can be selected because of extreme outliers (or, large values). Now, the sigmoid function converts any such number in a range between 0 to 1.

To convert large values within the range 0 to 1

Sigmoid Function

So, from Figure 8, we can see that the outlier have the product of 100, which is a very big number and impacting our best fit line. Hence, we should pass this product through sigmoid function which converts this number to a small number between 0 to 1 and removing the affect of outlier.

Hence, the sigmoid curve (S shaped curve) can be represent as below:

Figure 10. S-shaped sigmoid curve

Note:

Why we consider the data points to be -1 and +1?

Figure 11. Intuitive Understand of +1 and -1

Geometrically, distance between a point (x,y) and straight line (ax+by+c=0) can also be represented as (ax+by+c) /√(a^2 +b^2).

From the Figure 11, the linear equation is represented as

y = m*x+c => -m*x +y-c = 0

Then, distance can be represented as

(-m*x+y-c) /√(m^2 + 1).

Considering, m=1 (slope) , c=0 (intercept) and a red point as (2, 5), if we calculate the distance

= (-(1*2) +5-0) /√(1+1) = +3/√2 (positive value)

This implies that distance between any point and positive side of the slope is always positive, and hence considered +1.

Considering, the same slope and intercept value for blue point as (5, 2), we can calculate the distance as

=(-(1*5) +2–0) /√(1+1) = -3/√2 (negative value) .

This implies that distance between any point and negative side of the slope is always positive, and hence considered -1.

For the same reason, red and blue points can be considered as positive and negative ones respectively.