Understanding Linear Regression. The Workhorse Of Data Science | by Tony Yiu | Towards Data Science

Despite the prevalence of more complicated and fancier models in recent years, linear regression remains hard to beat because of its versatility, robustness, and explainability.
It’s a simple but powerful model that just gets the job done. And if you want to go into a quantitative field, you will need to understand how it works.
Linear regression is basically line fitting. It asks the question — “What is the equation of the line that best fits my data?” Nice and simple.
The equation of a line is:
Y = b0 + b1*X
Y, the target variable, is the thing we are trying to model. We want to understand (a.k.a. explain) its variance. In statistics, variance is a measure of uncertainty. It’s the tendency of a value to whip about its mean.
The graphic to the left illustrates this. The green variable has a high variance and thus a large cone of uncertainty around its mean (the center of the cone). The red variable has a lower variance. Knowing nothing else about the variables, we would feel much more confident in our ability to not be too far off when guessing the value of the red variable.
But in data science and statistics in general, we’re not worried about variance, we’re only worried about variance that cannot be explained. That is, if we can find some other variable (the feature variable, X), that explains much of the variance in Y, then we are good.
With such a feature variable, if someone were to ask me for my prediction of Y, I just need to look up what X is equal to, and I can be reasonably certain of what Y will be. Much of Y’s variance has been explained by incorporating X.
So why does line fitting help us explain the variance? Think about what the line’s equation represents. It defines the relationship between X and Y. By multiplying X by b1 and summing with b0 (b0 + b1*X), we get our prediction of Y based on a given value of X.
Of course, this only works if X and Y are correlated (either positively or negatively). Correlation measures the tendency of two things to move together (or opposite of each other in the case of a negative correlation). For example, people’s height and weight are positively correlated. The taller you are, the more you tend to weigh.
So line fitting allows us to infer the value of something we don’t know, Y, from the value of something that we do know, X, thanks to the correlation between X and Y.
We are explaining the variance of Y using the X variables in our linear regression equation. By attributing (a.k.a. explaining) variance this way, we are in effect reducing it — explained variance is variance that we no longer have to worry about.
The lynchpin assumption is that the X variables are readily measurable, understood, and not themselves a mystery. A common modeling mistake is to forecast something using variables that are themselves not observable in real-time. A model that forecasts the future using data from the future is no good.
Let’s see how this process works. When we start out with just a Y variable, all we know about it is its distribution. In the figure below, all the variance (the green cone) is unexplained and our best guess of the future value of Y is its mean.
Now let’s say we’ve found a feature variable, X, with a positive correlation to Y. The figure below shows how X helps to explain variance. We can segment the observations of Y into two sections based on their X values. For low values of X, Y has an expected value of Mean 1 and a variance approximated by the left red cone. For high values of X, Y has an expected value of Mean 2 and a variance approximated by the right red cone.
Notice how by segmenting this way, the variance (the red cones) has been reduced relative to the original green cone. And our new prediction of Y, either Mean 1 or Mean 2 depending on what X is, is significantly more precise than our previous naive prediction (the mean of Y).
But why stop at two segmentations? If we drew more blue dashed vertical lines, and broke the data into even more buckets, we could explain even more variance and generate even more precise predictions.
And that’s what linear regression does. In effect, it draws tons and tons of vertical lines that break the data into numerous tiny segmentations. This maximizes both the variance explained and the precision around our prediction.
Let’s quickly cover how linear regression finds the line of best fit. While there is an analytical solution, you can think of it as an optimization problem. The linear regression algorithm wants to:
Find the values for b0 and b1that minimizes the sum of squared errors.
Error is the distance between the data (black dots) and the blue line — in the picture below, the red arrows denote error. We square the error because it can be either positive or negative.
Now that we know that error is the difference between the actual observations (Y, the black dots) and our prediction (the blue line), we can write out our optimization problem in more detail:
Find the values for b0 and b1that minimizes the sum of squared errors.Where squared error for observation i= (Yi - (b0 + b1*Xi))^2And the sum of squared error is just the sum of (Yi - (b0 + b1*Xi))^2 for all observations.
So to calculate the sum of squared errors, we take every vertical distance between a black dot and the blue line, square them, and sum them up. Finding the values of b0 and b1 that minimize this sum of squared errors gets us to the line of best fit.
The parameters (b0, b1, etc.), known as betas, that fall out of a regression are important. In our earlier example, we had just a single feature variable. But say we had three feature variables:
Y = b0 + b1*X1 + b2*X2 + b3*X3
b0 is the intercept between the Y axis and the blue line and tells us the expected value of Y when all our feature variables are 0.
b1 tells us how, when holding X2 and X3 constant, changing X1 impacts Y.
b2 tells us how, when holding X1 and X3 constant, changing X2 impacts Y.
b3 tells us how, when holding X1 and X2 constant, changing X3 impacts Y.
A key assumption for the interpretation of the betas is that you can hold the other feature variables constant. In some cases, it might be extremely hard to shift one feature without changing another. In this case, we would want to include these interaction effects (a topic for another post).
Finally, we want to figure out how much of the original variance we were able to explain (since the goal is to explain, and therefore reduce variance).
We can do that by calculating the R2:
R^2 = 1 - (residual sum of squares / total sum of squares)
The residual (a.k.a. error) sum of squares is proportional to the variance of our prediction errors (the vertical distances between the black dots and the blue line). Prediction error is the difference between the actual observation and our model’s prediction of it. It is a measure of unexplained variance.
Total sum of squares is proportional to the total variance (of Y) that we are trying to explain — it’s actually the numerator of the formula for the variance of Y.
The ratio of residual sum of squares to total sum of squares measures the proportion of variance left unexplained after running the linear regression. In other words, it’s the proportion of uncertainty that we could not make vanish with our linear regression.
Since R2 is one minus that ratio, it can be thought of as the proportion of variance explained:
R^2 = 1 - (residual sum of squares / total sum of squares)= 1 - proportion of variance unexplained= proportion of variance explainedSince: prop. of variance unexplained + prop of variance explained = 1
So if after running our regression, we see an R2 of 0.90, that means the X variables we included in our model helped explain 90% of the variance observed in Y. In other words, we successfully explained away much of uncertainty around the variable we are interested in.
In future posts, we will do some examples as well as look at what happens when some of the assumptions that underpin linear regression are violated. Until then, cheers!
If you liked this article and my writing in general, please consider supporting my writing by signing up for Medium via my referral link here. Thanks!