Linear Regression by Hand. Explanation on how to model a linear... | by Roman Studer | Towards Data Science

The basic idea behind linear regression is quite simple. In mathematical terms we want to predict a dependent variable Y using an independent variable X. By assuming that the two variables correlate in a linear fashion we can predict Y with a simple linear formula:
(The wavy equal sign signifies “approximately”). Simply put, as soon as we know a bit about the relationship between the two coefficients, i.e. we have approximated the two coefficients α and β, we can (with some confidence) predict Y. Alpha α represents the intercept (value of y with f(x = 0)) and Beta β is the slope.
With the help of linear regression, we can answer a lot of questions; e.g.
“Is the rise of the sea level connected to rising temperatures?”,
“How expensive will a house with 3 bedrooms be?”
“How many items do we sell if we increase our marketing budget by 20%?”
Let’s take a look at some data! Even though the observations in the plot above deviate quite a bit, a trend is clearly visible. This indicates a positive correlation between X and Y. We can approximate the linear function that models the relationship between the two by utilizing the method of least squares. Let
be a set of observations, represented as dots in the scatterplot. The method of least squares can now be applied to approximate the coefficients α and β.
The method minimizes the sum of squared error for the linear regression line (Which is the cost function for linear regression, to put it in machine learning terms). The error is represented as the orange lines in the next plot. By differentiating the equation into beta, we get the following formula:
x and y with a bar on top represent the sample mean of x and y.
By plugging the approximated β into the linear equation, we get the approximation for α:
Alternatively, you could minimize the cost function through gradient descent. Now that we know the approximated coefficients for our linear regression, we can plot the equation and do a first visual inspection.
The green line in our plot represents the linear equation whose coefficients we just approximated. The orange arrows are the errors (also called the residuals), they tell us how much the observations deviate from our predicted values. The smaller the sum of errors, the better the regression line fits. A sum of 0 would be a perfect fit, meaning all observations are perfectly linear.
The interpretation of the errors/residues is an important step in building a regression model. By taking a look at the residues we can tell how they are distributed (normal with mean 0 is best)and if they are homoscedastic, i.e. whether the variance of the residues is constant.
Residual Sum of Squares (RSS) and Residual Standard Error(RSE)
A residue is a difference between the predicted value y hat (i.e. approximated) and the observed value y, visualized as the orange line in the plot above. The residual sum of squares RSS is defined by the following formula:
With the help of RSS, we can compute the Residual Standard Error (RSE). The RSE is an estimate for the standard deviation of the true regression line. This means that on average the value of y is one RSE away from the true regression line. The true regression line, also known as the population regression line, describes the real relationship between X and Y. We calculate the RSE as followed:
A visual representation of the residues can be helpful. The residues of a good fit meet three criteria:
Residues have a mean of 0Residues are independentResidues follow a normal distribution
Residues have a mean of 0
Residues are independent
Residues follow a normal distribution
With the help of a Tukey-Anscombe-Diagram and a simple Histogram, we can check the residues on the criteria. A Tukey-Anscombe-Diagram is a Scatterplot that visualizes the residues. The green line (y=0) represents the regression line. Now we can see how the residues are arranged around the regression line:
The visual inspection shows that the residues are independent (they don’t follow a pattern, i.e no systematic errors), the mean of the residues is 0 and they follow a normal distribution.
Click here to view some bad Tukey-Anscome-Diagrams (PDF).
One other way to assess the regression model is the R-Squared value. It tells us how well the model fits the data by measuring the proportion of the variability in Y that can be explained by X. For that we need the RSS and Total Sum of Squares (TSS).
where TSS is the total sum of squares calculated by
R Squared can take on values in the interval 0 to 1. A result close to 1 would mean that a lot of variability can be ‘explained’ by the regression. A result closer to 0 can indicate that we have a poor model. This might indicate that the linear model is wrong or that the variance of the error is high.
The easiest way to do a linear regression is by utilizing the sklearn library.
We start by importing the necessary libraries and then define our data (X and Y). Please note that the data could just as easily be from a DataFrame. The next step is to fit our Model by calling the LinearRegression().fit(x,y) method on our data. This returns an object containing our regression model which now can be used to predict y for x.
import numpy as npimport seaborn as snsfrom sklearn.linear_model import LinearRegressionimport sklearn.metricsx = np.array([4, 6, 8, 10, 12, 14, 18, 18, 21]).reshape(-1,1)y = np.array([2, 6, 4, 9, 5, 8, 6, 10, 10]).reshape(-1,1)# fitting the modelreg = LinearRegression().fit(x,y)# predicting values (x_new is an array of values you want to predict from)x_new = xy_new = reg.predict(x_new)# plot regression lineax = sns.scatterplot(x[:,0],y[:,0])sns.lineplot(x = x_new[:,0],y = y_new[:,0], color='green')ax.set(xlabel='X', ylabel='Y')plt.show()# R Squaredprint(f'R Squared: {reg.score(x,y)}')
Which results in the following:
For further information about how sklearns Linear Regression works, visit the documentation.
R already has a built-in function to do linear regression called lm() (lm stands for linear models). We fit the model by plugging in our data for X and Y. summary() returns a nice overview of our model. Look at that: R-Squared is the same as if we calculate it with Python.
# observed datax <- c(4, 6, 8, 10, 12, 14, 18, 18, 21)y <- c(2, 6, 4, 9, 5, 8, 6, 10, 10)# fit modelreg <- lm(y ~ x)# get summary of modelsummary(reg)# OutputCall:lm(formula = y ~ x)Residuals:    Min      1Q  Median      3Q     Max -2.6377 -1.5507  0.3188  1.3623  3.1449Coefficients:            Estimate Std. Error t value Pr(>|t|)  (Intercept)   2.3768     1.6452   1.445   0.1918  x             0.3478     0.1217   2.858   0.0244 *---Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1Residual standard error: 2.022 on 7 degrees of freedomMultiple R-squared:  0.5386, Adjusted R-squared:  0.4727 F-statistic:  8.17 on 1 and 7 DF,  p-value: 0.02439# plotting the regression lineplot(x,y)abline(reg)
James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2013. 103 An Introduction to Statistical Learning. New York, NY: Springer New York. http://link.springer.com/10.1007/978-1-4614-7138-7 (4. Oktober 2020).
„Least Squares“. 2020. Wikipedia. https://en.wikipedia.org/w/index.php?title=Least_squares&oldid=978615667 (4. Oktober 2020).
„sklearn.linear_model.LinearRegression — scikit-learn 0.23.2 documentation“. https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html (5. Oktober 2020).