Simple Linear Regression
By: Sherwyn D Souza
Tags: Machine Learning, Regression, Linear Regression, Data Science, Univariate Analysis

Simple Linear Regression

An introductory ML algorithm to get started

In this article, we hope to provide a concise exploration of one of the most fundamental ML algorithms: Simple Linear Regression.

Linear regression is intuitive but powerful. For example: given a quantity of goods, we want to predict the price at which a company will sell. Data on past sales are represented by Xs on the chart. These contain actual data on how many goods the company sold (x-axis) and at what price (y-axis).

Simple Linear Regression with Quantity and Price

The goal of linear regression is to fit a line to the shape of the data points as close as possible. Then, for any given x (quantity of goods) we can estimate/predict the y value (price) using the equation of the regression line. It’s considered simple or univariate because we are trying to predict y with only one input variable x.

Key Terms and Definitions

Variables

m: Number of training examples in our data.

Number of training examples in our data. x: Our input values.

Our input values. y: Our output values.

Hypothesis

Our hypothesis is the equation of the line: it uses x to estimate y values.

Hypothesis Function

The hypothesis is a linear function, given fixed real constants of θ₀ and θ₁. The θ₀ term represents the y-intercept — the value of y when x is 0. The θ₁ term represents the slope.

A cost function is a measure of how well the line fits the data. We use mean squared error (MSE); a cost function that measures the average difference between predicted and actual values.

Cost Function: Mean Squared Error

MSE is the metric by which we judge if our line is a good fit. For all m pairs of data, we calculate the difference between the expected value h(xᵢ) and the actual value yᵢ, known as the residual. We square the result to get a positive value, because we only care about the magnitude of the difference. Finally, we add the residual from each data pair, and divide by the number of pairs m to get the average error across all pairs. Actually, we divide by 2m, which is half the average error. This makes calculating the derivative easier, and doesn’t affect our comparison of cost functions for different values of θ₀ and θ₁.

Residuals in MSE

The higher our MSE, the worse our estimates will be, because our data points lie further from our prediction line. The lower our MSE, the better our estimates will be, because our prediction is closer to the data points.

Therefore, the goal of linear regression is to find values of θ₀ and θ₁ for the hypothesis which minimize the MSE cost function. Formally, we want to:

Gradient Descent

Gradient descent describes the approach we use to find the values of θ₀ and θ₁ which minimize our cost function. If we graph our cost (a function of θ₀ and θ₁) it will look something like this:

The basic idea is, starting at random coordinates (θ₀, θ₁), we want to take a step, moving closer to the minimum point by descending in a downward direction. We repeat this process until we converge at the minimum — no matter which direction we move in, our cost J will only increase. At this point, we have found coordinates (θ₀, θ₁) that minimize J. Formally:

Remember, J refers to our cost function. θⱼ refers to the current value of either θ₀ or θ₁. The ∂ symbol refers to the partial derivative, and finally α refers to the step size.

At every iteration, for both θ₀ and θ₁ we update their value to move slightly closer to the minimum point. We multiply our step size α by the derivative term, and subtract it from our current value of θⱼ allowing us to gradually move towards the minimum.

Smaller values of a mean we descend the slope slowly. However, larger values of a could mean we overshoot the minimum, and sometimes fail to return a minimized solution. Consider the case where we are very close to the minimum. The derivative term tells us to move towards the minimum point, but if our value of a is very large, we will move too far in the direction of the minimum and might end up climbing far up the slope on the other side.

Gradient Descent and Linear Regression

Putting it all together, the general idea is that we want to train our algorithm using gradient descent to minimize our cost function, and then plug the values of θ₀ and θ₁ into our hypothesis function h(x) = θ₀ + θ₁x. We then use the hypothesis to estimate y given x.

Find θ₀ and θ₁ with Gradient Descent

Expanding gradient descent and solving for the partial derivative, we get the algorithm below. We start at a random (θ₀, θ₁) location, and gradually descend to our minimum cost, updating θ₀ and θ₁ with each iteration. When we converge at the minimum, we return the values of θ₀ and θ₁. These are the optimal values of θ₀ and θ₁ because they minimize our MSE cost function J.

Use Hypothesis to Estimate

Now that we have the optimal values for θ₀ and θ₁, we simply plug them into our hypothesis function h(x) = θ₀ + θ₁x. When making a prediction on a given x value, we simply:

Conclusion

The bottom line is that linear regression is a powerful introductory ML algorithm. We try to fit a line to the data, measuring closeness of fit by minimizing our cost function, and then use the line to estimate values of y given values of x. Here, we explored univariate linear regression, but multivariate linear regression is more frequently used, because most problems depend on multiple input variables. For data that doesn’t have a linear shape, we might need to look to look to higher order polynomials, such as quadratic regression. But univariate linear regression is (more) simple, giving us a good place to start learning.

This article was inspired by Andrew Ng’s Machine Learning Course on Coursera, available here: https://www.coursera.org/learn/machine-learning.