Connect Dots With Linear Regression
By: Aishwarya Mv
Tags: Cost Function, Linear Regression, Gradient Descent, Machine Learning, Ml Unravelled

Linear regression is one of the most fundamental and most simplistic algorithms for Machine Learning. If someone asks you, “What do you know about ML?” and you reply that you know some of Linear Regression, they’ll believe that you know some real ML but don’t use this jargon when you don’t know anything about it!

Linear Regression is fitting a line between the given data points. For example, the times when you drew a line that passes through maximum points in a graph.

Characteristics of Linear Regression

In any linear regression, you can observe the following things:

Dataset or a table of inputs and their corresponding outputs. Prediction Function. Cost Function. Gradient Descent.

This also includes lots of graphs to help you visualise the data!

Dataset

A dataset is the tabulated data given to the computer and helps it to understand the relationship between the inputs and the outputs. In a usual convention, inputs are denoted by x and outputs are denoted by y. Consider this typical dataset:

Can you find the relation between x and y? Yes, you guessed it right! The function that defines them is 3x + 18. You do this easily, because there was a 0 in the dataset. It would take you longer, if it wasn’t in the table. However, this is pretty easy for a computer.

It is important to remember that the dataset is always visualised, for a better understanding of the distribution of the data points and the model’s performance.

Each data point has a subscript such as x₁, x₂, . . . and y₁, y₂, . . . etc.

Each pair (xᵢ, yᵢ) is called a training example where i varies from 1 to m, if the dataset has m examples and corresponds to one row of the data table. For the above dataset, m = 5.

Prediction Function

It is the function which does the work for you. It’s also called the hypothesis function because it creates a hypothesis about the parameters of the line joining the data points. It can be represented as:

h(x) = θ₀ + xθ₁

Did you notice something about the above equation? It is of the same form as the equation of a line, which is, y = mx + c. Hence, we can conclude that θ₁ s the slope of the line and θ₀ is the y-intercept. It is all about adjusting these two parameters to get an accurate line.

Cost Function

Ever played Fruit Ninja? In arcade mode, have you noticed that the more mistakes you commit by slicing bombs, the lesser is your score. The game makes you pay the price of 10 points, every time you err. This is what the cost function does.

The cost function is the function that helps the model to adjust the parameters suitably in the hypothesis by taxing it when it deviates from the right values. Here’s what a typical cost function looks like:

Got intimidated? Don’t worry. Let’s break this down:

J(θ₀, θ₁) is the name of the cost function, which has the variables θ₀ and θ₁.

is the name of the cost function, which has the variables θ₀ and θ₁. 1÷2m is the average taking factor. It’s like combining m values and dividing by m. Then why is it 2m?.

is the average taking factor. It’s like combining m values and dividing by m. Then why is it 2m?. (h(xᵢ) -yᵢ) is the difference between the predicted value by the model, for any input xᵢ and the corresponding actual output value yᵢ .

is the difference between the predicted value by the model, for any input and the corresponding actual output value . The below equation is the sum of differences squared.

The above expression is applied to each row of the dataset and the differences are obtained. These are squared because squaring minimizes small errors, magnifies large errors and also do not have any negative differences as they cause problems while summing up. Lastly, the summation of these squares is performed.

Finally, this sum is divided by 2m to find the average squared error. This method is called the Mean Squared Error or MSE. More the error, more is the value of the cost function. Hence, the aim of the algorithm should be to minimize the cost as much as possible.

To understand the above equation better, let’s consider the dataset given below. Suppose the model generates a hypothesis function h(x) = 3x + 15. Here, θ₀ =15 and θ₁ = 3 . The error table is as shown below:

Hence, the cost function is 4.5 for the current hypothesis of the model.

Gradient Descent

This function is the star of the show! It is the reason why models can learn things. It makes the model descend the gradient of errors to reach a place of perfection. To visualise this, let’s take some hypotheses and plot them with their cost:

Here θ₁= 3 and θ₀ varies from 11 to 25 (Both the parameters can vary but let us keep one constant for simplification). From the graph, we can see that the cost is 0 when θ₀ is 18, which means the prediction is exactly equal to the actual function and the model doesn’t have to pay any price. Hence, 18 is called the global minimum at which the cost function is the least.

Gradient descent estimates the slope of the cost function at different values of θ₀ and θ₁ . In mathematical terms, the slope is the partial derivative of the cost function. The model would reach its best accuracy when the slope is almost or equal to 0. If we take the partial derivative of the cost function, we get,

(The derivative of x² is 2x . By the same logic, the derivative of the squared error gives 2 times the error. That 2 divides 2m to give 1/m. Hence an extra 2 was added to the average taking factor in the cost function).

The gradient descent algorithm looks like this :

This means, for every (θ₀, θ₁) pair, we estimate the partial derivative of the cost, multiply it with and subtract it from the original value. This is the learning rate or the rate of descent, which controls how slowly or quickly the model should descend the slope of errors. Since it is a descent, we subtract from the original value.

How does this function work for different hypotheses that are on either side of the global minimum?

Keeping θ₁ fixed at 3, let us consider 2 different values for θ₀ , 15 and 25. Their corresponding costs are 4.5 and 24.5 respectively.

Case 1 : θ₀= 15.

∂(J(θ₀, θ₁))/∂θ= -3.

On performing gradient descent keeping =1 , we obtain,

θ₀ = 15-(-3) = 18.

Case 2: θ₀= 25.

∂(J(θ₀, θ₁))/∂θ= 7.

On performing gradient descent keeping =1 , we obtain,

θ₀ = 25 -(7) = 18.

Hence, the gradient descent algorithm works for parameters which are both bigger and smaller than the actual value.

Gradient Descent is of 3 types :

Stochastic Gradient Descent : When gradient descent is performed on each data point in the dataset.

: When gradient descent is performed on each data point in the dataset. Mini-Batch Gradient Descent : When gradient descent is performed on the dataset divided into mini-batches.

: When gradient descent is performed on the dataset divided into mini-batches. Batch Gradient Descent: When gradient descent is performed on the dataset divided into batches.

Problems encountered in Linear Regression

Sometimes, the model does not correctly fit the line to the data. Two of the most common problems that occur are Overfitting and Underfitting.

1. Overfitting: In this, the model tries to memorise the outputs of the dataset instead of learning the generalised concept. Hence, it performs well on the data that is present in the dataset but fails to perform well on any new data. This can be understood with an analogy to students in a science test. If students memorize answers, they can write exceptionally well for known questions, but if the same are tweaked or asked differently, they stumble. Overfitting occurs if the model is too complex, data is not enough to learn or if the data has errors.

2. Underfitting: In this, the model is unable to perform well neither on the training data nor the test data. It cannot memorize the outputs and ends up fitting the wrong line. This is similar to students in a math test. There is no use of memorizing the answers, as the questions would be different. If the concept is not understood, nothing can help to get a good score. Small datasets are one of the main causes of underfitting.

To understand this better, watch this video of training a cat to ring the bell (Turn on the subtitles if you can’t read Korean).

Notice at the start when he grabbed its paw and placed it on the bell. This leads to overfitting as the cat forcibly rings the bell but cannot execute the bell command without his help. On the other hand, the cat struggles in the beginning to touch the end of the spoon characterized by underfitting as it is new to that command but eventually understands that it will be treated every time it touches the end of the spoon.

Take your time, analyse the mathematical concepts and take a grasp on Linear Regression. We will be learning Pandas and Scikit Learn to apply linear regression on basic datasets in the upcoming blogs.

Stay Tuned!