Linear Regression with NumPy
By: Anh Tuan Vu
Tags: Linear Regression, Gradient Descent, Numpy, Coursera, Machine Learning

In this article, I do the week 2 programming exercises of the course Machine Learning at Coursera with NumPy. I avoid using frameworks like PyTorch since they are too powerful in comparison with these exercises.

In general, the workflow of a linear regression problem (with one variable or with multiple variables) is presented in the figure below.

Linear Regression workflow

It consists of four main steps:

Load Data: load data from text files: ex1data1.txt and ex1data2.txt .

and . Define functions: define functions to predict outputs, to compute cost, to normalize features, and to carry out the algorithm gradient descent.

Prepare Data: add a column of ones to variables, do normalize features if needed.

Training: initialize weights, learning rate, and a number of iterations (called epochs), then launch gradient descent.

I also do some more steps of visualization to figure out the data and results obtained.

1. Linear Regression with one variable

In this assignment, you need to predict profits for a food truck.

Suppose you are the CEO of a restaurant franchise and considering different cities for opening a new outlet. The chain already has trucks in various cities, and you have data for profits and populations from the cities.

The data set is stored in the file ex1data1.txt and is presented in the figure below. A negative value for profit indicates a loss.

Profits of food truck in cities

1.1. Load data

The numpy method loadtxt() loads data from a text file to a numpy array.

1.2. Define functions

The numpy method matmul() performs a matrix product of two arrays.

1.3. Prepare data

The numpy method column_stack() stacks a sequence of 1-D or 2-D arrays to a single 2-D arrays.

1.4. Training

After 1500 iterations, we will obtain the new weights w

array([-3.63029144, 1.16636235])

The line predicted is shown in the figure below.

Profits of food truck in cities with prediction line

And the figure below figures out the variation of cost.

2. Linear Regression with multiple variables

In this assignment, you need to predict the prices of houses.

Suppose you are selling your house and you want to know what a good market price would be. The file ex1data2.txt contains a data set of housing prices in Portland, Oregon. The first column is the size of the house (in square fit), the second column is the number of bedrooms, and the third column is the house’s price of the.

The figure below presents the data set in 3D space.

Prices of houses in Portland, Oregon

There’s only one point that we need to pay attention to: house sizes are about 1000 times the number of bedrooms. Therefore, we should perform feature normalization before launching gradient descent so that it converges much more quickly.

Create the function normalize_features()

Then normalize variables before put them in gradient descent.

The other steps are completely similar to the ones of the assignment above. We don’t need to rewrite functions predict() , cost_fn() , and gradient_descent() since they work well with matrices multi-columns.

After 400 iterations, we have the surface of prediction in the figure below.

Prices of houses with prediction surface

Intuitively, linear regression prediction with one variable is a line in the surface of inputs and outputs. The prediction of two variables is the surface. The prediction of the three variables is space, and so on.

The gradient descent logs help us figure out the variation of our model’s cost in the figure below.

The complete code source is in the notebook linear regression. If you want to refer to Matlab codes of this programming exercise, they are in the directory w2.