Building a Linear Regression by Hand | by moxú | Towards Data Science

W e employ linear regression to forecast the value of Y based on the value(s) of X. Because we need to know Y, it is a supervised learning approach. Linear regression is classified into two types: basic and multiple. Let’s start with the easy one. The notebook with all the codes is here. All the equations were made with LaTeX.
Before we begin, it is critical to understand that linear regression is a parametric approach.
Parametric method:
1. It assumes the function’s shape in advance.
2. It simplifies the issue of estimating f (X) to estimating a collection of parameters. This assumption simplifies the task since estimating the collection of parameters is easier than with a completely arbitrary function.
3. The most difficult aspect of these approaches is making the proper estimate. We can only make guesses about the correct form of the curve, resulting in erroneous forecasts.
As a result, we must estimate parameters that will allow us to create a line that best approximates the values of Y, whether known or unknown.
This is the function of a simple linear regression. Variations of this equation can be found in statistical literature, but they all have the same substance. It may be W0 and W1, alpha and beta, and so on.
However, I’d want to bring your attention to something. Take note of the sign ≈. When we estimate values, our major goal is to get our predictions close to the true values of Y through X. That is, we aim to minimize the difference between the true and estimated values as much as feasible.
As a result, when dealing with estimations of f(X) = Y, or more specifically, parameters of f(X) where f (X) = Y or Y = f (X), we do not aim to define f (X) perfectly, but rather estimate parameters that best represent f (X). However, because Y = ^f (X) + e, our estimate of f(X) will equal Y when added with e, the residual.
Where Y = ^f (X) + e, then ^f (X) = ^B0 + ^B1X = ^Y. That is, in order to estimate ^Y, we must first estimate the parameters of ^f(X), ^B0 and ^B1. Y = ^Y + e since Y = ^f (X) + e and ^f (X) = ^Y.
With that in mind, the formula we’ll use is:
No more distractions.
BO denotes the intercept, which is the Y value at X = 0, or the Y value at which the linear regression line begins. Here’s an example. We want to estimate the number of sales conversions based on the amount of investment made in a campaign. How much can I sell if I don’t put up any money?
And what about the B1? The slope is represented by the value of B1. It basically determines the trend line. How does this work in practice? Consider the example of calculating height based on weight. Height is measured in centimeters, and weight is measured in kilograms. We discover a slope of 2.5. It indicates that for every kilogram gained, a person’s height increases by 2.5 millimeters.
How do we estimate B0 and B1 given that we know what they are? These parameters can be estimated in a variety of methods. To begin, let’s look at the Ordinary Least Squares (OLS), which is one of the most frequent and straightforward. To estimate B0, we must first estimate B1.
In statistics, ordinary least squares (OLS) is a type of linear least squares method for estimating the unknown parameters in a linear regression model. OLS chooses the parameters of a linear function of a set of explanatory variables by the principle of least squares: minimizing the sum of the squares of the differences between the observed dependent variable (values of the variable being observed) in the given dataset and those predicted by the linear function of the independent variable. — Source: Wikipedia.
Let’s have a look at what this implies. What do we want to achieve with this method? Reduce something. What should be minimized? The sum of the squares of the differences between the observed and estimated data. Let’s put this equation together.
Backwards, we have the difference between the observed and estimated values. The values we estimate as a result of our regression are referred to as estimated values. The observed values are the true values. Therefore,
observedValue - estimatedValue
Now, square the difference between the estimated and observed values. Hence:
square (observedValue - estimatedValue)
Finally, the sum of the squares of the differences between the observed and estimated data.:
sum (square (observedValue - estimatedValue))
Okay, but what is the significance of the sum? observedValue and estimatedValue are valueS. Consider this: we have two columns, Y and Y. The first is the real value, the observed value, and second is the estimated value. As I previously mentioned, we construct a model and utilize X to estimate Y, resulting in Y. Assuming that each column has three values, Y = [5, 9, 2] and Y = [6, 7, 2]. We have the following results using our formula:
(5–6)2(9–7)2(2–2)2
The total is the sum of these three operations. As a result, (5–6)2 + (9–7)2 + (2–2) 2. We have the Residual Sum of Squares (RSS) as a result of these procedures. RSS = 5 is used in this case. We aim to reduce RSS using OLS.
The general formula for RSS is:
How we code this using Python?
OLS is ONE of numerous methods for reducing this RSS. Another approach is, I don’t know, guessing various parameters of B0 and B1, calculating the error of all, and selecting the smallest one, but the smallest mistake discovered may not be so tiny. We’ll finish with an approach like this. When we have a small number of features and columns, OLS is the quickest and easiest approach to use.
We will utilize fictitious data in our article.
In this case, I made two arrays to represent X and Y. I created 1000 values in order to provide a more intriguing perspective. I split X and Y into four separate halves. The training data will contain 80% of the X and Y values (X train and y train), whereas 20% of the data will be utilized for testing (X test and y test).
Let’s discuss about assumptions before we estimate B0 and B1. What are the fundamental assumptions of linear regression? We utilize linear regressions to forecast new values or to make inferences. There are numerous methods for estimating parameters in a regression. And many methods, such as OLS, require specific assumptions in order for their conclusions to be as accurate and impartial as feasible. These premises do not have to be followed exactly in every linear regression!
It is only logical that this requirement be satisfied when employing linear regression for prediction. In essence, linear regression will draw a straight line through the data. If the data does not have a linear connection, the model cannot handle the data’s complexity. How can we put this concept to the test?
There are two forms of confirmation under most assumptions. There is graphical confirmation as well as mathematical confirmation. I’ll cover both whenever feasible. When things get too complicated, I’ll just cover visuals.
So we may plot the X data against the Y data and look for trends. Let’s execute this command and see what happens.
plt.figure(figsize=(10,10))plt.scatter(X_train, y_train)plt.show()
To me, it appears to be quite linear! There’s even a straight line I can see! But what’s the point? I’m not sure about you, but visual confirmation does not completely satisfy me. Of course, your perspective is determined by your time, money, and education. However, we have all the time in the world here, it is free, and we are knowledgeable enough to use alternative methods.
And, other than graphs, how do we assess a linear relationship? The correlation coefficient between X and Y is computed. But what exactly is the correlation?
In statistics, correlation or dependence is any statistical relationship, whether causal or not, between two random variables or bivariate data. In the broadest sense correlation is any statistical association, though it commonly refers to the degree to which a pair of variables are linearly related. Familiar examples of dependent phenomena include the correlation between the height of parents and their offspring, and the correlation between the price of a good and the quantity the consumers are willing to purchase, as it is depicted in the so-called demand curve.. —Source: Wikipedia.
Following this quick explanation, I shall reiterate a statistician’s mantra: correlation does not indicate causation! Before we can get a clear judgment on causality, we need to run a number of statistical tests. But it is a topic for another day!
In our data, we wish to calculate the degree of correlation between X and Y. There are various approaches to this problem. Pearson’s coefficient is the most commonly used. Its general formula is as follows:
That is, the division of two variables’ standard deviations divided by their covariance. Let’s break things down so you can comprehend it better. We’ll look at two sections. The covariance of X with respect to Y, as well as the variance of X and Y. I’ll leave this information, which includes some graphic explanations, for the idea of variance. It’s definitely worth a look. In summary, covariance denotes the degree to which two variables fluctuate in tandem.
How do we compute the covariance of two variables in Python? Let’s use the formula:
That’s all! And what is the standard deviation of the sample? The standard deviation of a random variable is a measure of its dispersion around the population mean. A low standard deviation implies that the data points are frequently near to the mean or anticipated value. A large standard deviation implies that the data points are distributed throughout a broad range of values (Wikipedia). The square root of the variance is also the standard deviation. Let’s use Python to create our standard deviation formula by utilizing the variance formula and calculating its root:
Let’s take the standard deviations first.
As you can see, we utilize the variance formula to find the root of the problem. We may now use our Pearson coefficient. Let’s start with the original formula and create our function from there:
Given that we’ve already gone more than halfway, let’s simplify this formula even further:
covarianceOfXY / StdX * StdY
Let’s see how the outcome with our data turned out.
The Pearson Correlation Coefficient between X and Y is [0.98478235].
And what is the significance of Pearson’s coefficient? Pearson’s coefficient is a number between -1 and 1, with 1 and -1 representing perfect linear correlations and 0 representing a non-existent relationship. A perfect positive correlation is 1 while a perfect negative correlation is -1.
What should the ideal values be? It is conditional. Sometimes a correlation of 0.7 is optimum, and other times 0.9 is insufficient. It will be determined by the nature of your problem. In a multiple linear regression, you may use Pearson’s coefficient to find the most significant factors in your model or to exclude co-dependent variables.
Now that we’ve identified one of the assumptions, let’s get started on estimating the parameters.
B0 and B1 will be estimated using the OLS formula. The following are the equations:
Looking at it that way, it’s a bit difficult, lol. But, in reality, it’s rather easy. Let’s begin with B1:
As previously stated, B1 is the covariance between X and Y divided by X’s variation. B0 is simply the value of Y when X = 0, taking into account the difference between the median of Y and the product of the angular coefficient and the median of X.
Let’s use Python to estimate B0 and B1?
Because we divide our data in training and testing, our X and Y will be our training X and Y. It’s time to estimate!
Intercept: -4.959319577525191,        Slope: [2.00341869].
To put it another way, our line begins at -4.9593 and progresses at a rate of 2.003 for each increment of X. Now we’ll apply these parameters to new data and see how our predictions fare! Now is the time to build our linear regression!
Because we separated our data before, we can now test and evaluate our model!
y_pred = predict_function(b0_coeficient, b1_coeficient, X_test)
After that? Now it’s time to design our return route!
plt.figure(figsize=(8,6))plt.scatter(X_test, y_test,  color='blue')plt.plot(X_test, y_pred, color='red', linewidth=2)plt.title("Linear Regression Demonstration", fontweight="bold", size=15)plt.show()
It appears that our return went well!
Naturally, linear regression lines will never reach all points. Or, at the very least, if they are correct, it will not be a good thing if the model’s intention is to generalize it to new data. This is due to the error that every linear equation has.
In this moment, we will add the final component of our equation, about which we have already spoken. Error.
The error is the difference between the predicted and actual value of Y. The cost of an error might be the result of a variety of factors. Sometimes it’s a single variable that you’re not considering, and other times you’ve assumed a linear relationship that doesn’t exist in reality.
Another reason might be a data shortage. With so limited data, it is possible that your estimates will fail to reveal the true nature of the problem.To begin, there are several paths to take in order to reduce error. You may look for more data, choose a non-parametric or linear model, and add more variables. The strategies differ and will be determined by your assessment. How much does it cost to look for more data? Is it worthwhile to look for further information? How long will it take to test another model? When working professionally, you should ask all of these questions.
Is there a limit? Yes. Call it an irreversible error. The irreversible error is aleatory and, by definition, cannot be measured or observed. We’ll bring you our new equation, which I’ll explain to you.
Y = ^f(X) + e
Sure, we’ve already completed the first stage and estimated the parameters of f(X). To aid comprehension, we are employing a linear regression to determine or predict the average level of heart beats throughout an hour given a certain quantity of Pfizer vaccine applied in mouses.
However, the error has not yet been committed and will not be committed. Why is this so? We’re guessing you came home late last night, tired and with a heavy night’s sleep. On that same night, one of your mouses, Pitu, bridled by Marquinhos. Pitu, like the other rats in the gaiola, became extremely stressed. Fortunately, no one was injured. Pitu tried to tell you about the injustice the next day, but because you don’t talk to animals, you don’t know what happened.
The stress levels of Pitu were altered by what happened and by the fact that you didn’t understand what happened. Stress levels were an important variable in your results that you ignored for N reasons, one of which was that you don’t speak “mouseguese”.
Magdalena, another little mouse, is lonely and has no one to talk to or play with. When you go out, the others have a party, but Magdalena stays in her corner, mulling over her sorrows. You will most likely be unable to identify Magdalene’s sadness because you did not major in mouse psychology. This variable, which you couldn’t measure or even see, will be crucial in evaluating Magdalene’s results!
Did you get my point? There are factors that are just not observable or collectable for your analysis.
You should be wondering, but what error is this, and why do I believe I heard it somewhere? You saw it up there, but it had a different name: RSS. Will we compute the model’s RSS?
print(f'The RSS is {get_rss(y_pred, y_test)}.')# The RSS is 191.16066429.
With this number, we may look for ways to reduce it. But it isn’t the issue today! I’m mentioning the error for a reason. The second premise of Linear Regression.
Why is this so? There are several disputes over why this assumption should be confirmed and what happens when it is not satisfied, even if it is necessary. As these are quite technical issues, I will not address them in this article.
And how can we know if a data distribution is normal? There are two types of graphs: graphic and mathematical. I’ll be using graphics in this section. If we want to examine the behavior of all the leftovers, we can’t use RSS since RSS looks at the sum of the leftovers, but I’m not interested in the sum of the leftovers, but rather in the individual ones.
Remember that registration at the start? We’ll look at the formula Y = ^Y + e. To calculate the residual, just switch Y from left to right, resulting in: Y − ^Y = e.
e = observedValue— estimatedValue
It’s simple, in Python:
residual_lr = get_residual(y_test, y_pred)
As a result, we will have an array with our leftovers. Will we examine their distribution using a histogram?
plt.subplots(figsize=(12, 6))plt.title('Distribution of Residuals')sns.distplot(residual_lr)plt.show()
Well, it appears to be a normal distribution, albeit slightly asymmetric! Is there another way to confirm that this is a normal distribution graphically? Yes, there is! It’s known as the QQPlot graph!
lst = []for i in residual_lr:    for n in i:        lst.append(n)sm.qqplot(np.array(lst), line='45', fit=True)plt.show()
In this code, I extracted the values from an array of arrays and placed them in a list for our QQPlot to validate! There you have it!
What exactly is this stuff I’ve never seen before? You may be curious. I will not spend too much time conceptualizing the QQPlot, but rather interpreting it. In case you’re interested in learning more.
What can we conclude from a QQ plot? We have a red straight line and a blue dispersion of residuals. We may visualize the distribution of our data using the QQ plot. We have a perfect normal distribution if they perfectly follow the red line. The further our data deviates from this straight line, the less of a normal distribution character remains. For a better understanding, you may find an explanation with visualizations here.
Based on this visualization, we concluded that our waste had a typical distribution, despite some points being further away from the center.
There are several statistical tests that may determine if a distribution is normal or not, such as the Shapiro-Wilk test. But for now, we’ll just stick with the visualization! We’re heading to our next stop.
Linear regression models rely heavily on the premise of homoscedasticity (which means “constant variance”). Homoscedasticity defines a situation in which the error term (the “noise” or random disturbance in the relationship between the independent variables and the dependent variable) is the same for all independent variable values.
Heteroscedasticity (the violation of homoscedasticity) occurs when the size of the error term varies across independent variable values. The impact of a breach of the homoscedasticity assumption is proportional to the degree of heteroscedasticity.
What is the significance of this to us? When we use MMQ, we give all X-values the same weight. We have an issue if some X values have a greater influence than others.
Consider an estimate of luxury goods purchases based on household income. We have impoverished families who do not or only buy a few luxury items. If the purchase of luxury goods grew with every increase in household income, we would have a constant variance. But assume that just a portion of the wealthy families buy luxury items, while the others buy relatively little: we have a problem with heteroscedasticity.
Do you remember the B1 formula? The variance of the values of X is calculated in the divisor section. Because the OLS provides the same weight to all variables of X, very big variations can get in the way!
How can we see this? Let’s put Python to use! Let’s start with a visual representation. We will plot our residuals against the anticipated values to observe the variation of the errors.
plt.scatter(residual_lr, y_pred)plt.show()
To be honest, I couldn’t draw any conclusions from this image! There doesn’t appear to be any pattern in the dots. So, shall we put it to the test?
H0 = homocedasticityH1 != homocedasticityalpha = 0.05
Let’s use the statsmodels.stats library for this!
print(sm.diagnostics.het_goldfeldquandt(residual_lr, y_pred))Test: 1.0206718127917291p-value: 0.45956295321355667
We cannot reject the null hypothesis because of alpha < p-value, hence we cannot deny the presence of homoscedasticity!
Whew! Our model confirmed and passed all of the assumptions! You are most likely thinking “Can we leave now? The model is amazing! “. You can relax if you wish to utilize the model to analyze the variables. You can start examining it tomorrow to see if you can get anything out of it. We’re just halfway there if you want to utilize the model to make predictions!
Now it is time to evaluate our model’s generalizability! As you can see, our model did a good job of staring at the test data. But how well? How do we assess our model? Using metrics!
There are a number of measurements for this. One of these is the R-squared, often known as the Coefficient of Determination. R2 is a measure that expresses how much of the variation in the data is explained by the model. In other words, this metric computes the proportion of variance that the regression model might predict, and so shows us how “closer” the actual measures are to our model. It denotes the proportion of the variability in the response variable that may be explained by the predictor or explanatory variable.
Its R-squared value varies from 0 to 1, with 0 being the poorest possible estimate and 1 representing the best possible estimate. It is typically expressed as a percentage. For example, an R2 = 73 % indicates that the model can explain 73% of the volatility in our data, while the remaining 27% is theoretically a residual variance.
Is it possible to have negative R-squares? Yes, when your model is capable of being worse than kicking the mean of Y for all Y values.
What exactly is the R2 formula? Let’s go over our RSS formula again. We will change it based on it. We have the sum of the squares of the residuals in RSS. However, we wish to calculate the total sum of squares of the residuals.
In RSS, we take the difference between each observed and estimated value of Y, or Y, square it, and add the findings. In TSS, we compute the difference between the Y values and the Y squared mean. That is to say:
TSS = sum ( ( valuesX — meanY ) 2)
R2 is basically:
1 — (RSS / TSS)
rss = get_rss(y_pred, y_test)rst = rst_metric(y_test)print(f'The R2 of the model is {get_r2(rss, rst)[0]*100}%.')
The R2 of the model is 97.17569151561773%
That means, our model was able to explain nearly 97.18 % of our data! The model appears to be generalizable to fresh data!
R-squared can alternatively be calculated by squaring Pearson’s Coefficient:
print(0.98478235 ** 2)# 0.969796276871522
Do you want to know what R-squared would be if you just took the average? Let’s have a look together! Let’s make a list the same size as Y, but with the mean of Y instead.
plt.figure(figsize=(8,6))plt.scatter(X_test, y_test,  color='blue')plt.plot(X_test, lst, color='red', linewidth=2)plt.title("Linear Regression Demonstration", size=15)plt.show()
Now, let’s calculate the R-squared:
rst = tss_metric(y_test)rss = get_rss(lst, y_test)print(f'The R2 of the model is {get_r2(rss, rst)[0]*100}%.')
The R2 of the model is 0.0%.
Remember when I said that R-squared could be less than 0? The greater the RSS is above RST, the lower the R-squared will be.
However, the question persists. What’s the other method for estimating a linear regression? Okay, I’ll swiftly approach a reasonable strategy that I had in mind.
The explanation behind this is as follows. Let’s pick 100 intercept and 100 slope values, then test 10,000 linear regressions, compare them, and pick the combination with the smallest RSS. Let’s get started!
Here we run 10,000 regressions. Shall we calculate and get the smallest RSS?
min_index = rss_list.index(np.min(rss_list))print(f'The lowest RSS is: {np.min(rss_list)} with index {min_index}.')# The lowest RSS is: 190.52186974065032 with index 4552.
We get a little smaller RSS when we use this workaround than when we use OLS! Take this index and use it to plot our regression line on the test data.
plt.figure(figsize=(8,6))plt.scatter(X_test, y_test,  color='blue')plt.plot(X_test, linear_reg[min_index], color='red', linewidth=2)plt.title("Linear Regression Demonstration", size=15)plt.show()
And the R-squared?
rst = tss_metric(y_test)rss = get_rss(linear_reg[min_index], y_test)print(f'The R2 of the model is {get_r2(rss, rst)[0]*100}%.')# The R2 of the model is 97.18512940329813%.
As you can see, the difference is negligible, almost non-existent. While using the MMQ method we got 97.175%, here we get 97.185%.
With that, we’ll conclude our article! I attempted to address the most important aspects of creating a linear regression by understanding what lies behind the five or six lines of code that are sufficient to do everything we’ve discussed here.
The remaining questions are: Is it required to check all of this? Is there another way to achieve consistent results without going through all of this? All I want to do now is finish Kaggle’s Titanic linear regression. Is it really necessary for me to do so? This is a heated debate! Some splashes can be found here. But keep in mind that as a data scientist, you must solve the problem and demonstrate your findings to your peers.
When you deal with real data, it won’t be cute like the ones here, but you need to solve the problem!
References:
Homoscedasticidade — CENTRO DE ESTATÍSTICA APLICADA. Disponível em: <https://estatistica.pt/homoscedasticidade/>. Acesso em: 24 set. 2021.
‌Teste para normalidade e homocedasticidade. Disponível em: <https://biostatistics-uem.github.io/Bio/aula8/teste_normalidade_homocedasticidade.html>. Acesso em: 24 set. 2021.
‌HOW WOULD YOU EXPLAIN COVARIANCE TO SOMEONE WHO UNDERSTANDS ONLY THE MEAN. How would you explain covariance to someone who understands only the mean? Disponível em: <https://stats.stackexchange.com/questions/18058/how-would-you-explain-covariance-to-someone-who-understands-only-the-mean>. Acesso em: 24 set. 2021.
‌DAMACENO, L. Entendendo Regressão Linear: as suposições por trás de tudo! Disponível em: <https://medium.com/@lauradamaceno/entendendo-regress%C3%A3o-linear-as-suposi%C3%A7%C3%B5es-por-tr%C3%A1s-de-tudo-d0e29004c7f8>. Acesso em: 24 set. 2021.
‌SHRUTIMECHLEARN. Step by Step Assumptions — Linear Regression. Disponível em: <https://www.kaggle.com/shrutimechlearn/step-by-step-assumptions-linear-regression>. Acesso em: 24 set. 2021.
‌NATHÁLIA TITO. Scikit-learn ou statsmodels? Avaliando meu modelo de regressão. Disponível em: <https://nathaliatito.medium.com/scikit-learn-ou-statsmodels-avaliando-meu-modelo-de-regress%C3%A3o-f4c04b361fa7>. Acesso em: 24 set. 2021.
‌Regressão multilinear usando Python. Disponível em: <https://ichi.pro/pt/regressao-multilinear-usando-python-75578758662189>. Acesso em: 24 set. 2021.
‌SERGIO MIRANDA FREIRE. 19 Regressão Linear | Bioestatística Básica. Disponível em: <http://www.lampada.uerj.br/arquivosdb/_book/regress%C3%A3o-linear.html#eqReta>. Acesso em: 24 set. 2021.