Linear Regression basics in Python | by Philip Wilkinson | Towards Data Science

This year, as Head of Science for the UCL Data Science Society, the society is aiming to present a series of 20 workshops covering topics such as introduction to Python, a Data Scientists toolkit and Machine learning methods, throughout the academic year. For each of these the aim is to create a series of small blogposts that will outline the main points with links to the full workshop for anyone who wishes to follow along. All of these can be found in our GitHub repository which will be updated throughout the year with new workshops and challenges.
The tenth workshop in the series is an introduction to Linear Regression in Python and kicks of our Data Science with Python Workshop series. This specific workshop will cover what is Linear Regression, exploring the data, implementing the model and model evaluation. While the highlights will be presented in this blog post, the full workshop can be found on our GitHub account here.
If you missed any of our previous workshops these can be found here:
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
Linear Regression is a statistical method used for modelling the dependence or relationship between two or more quantities. The aim of this is to be able to either better understand the existing relationships or to be able to predict the behaviour at points for which we currently don’t have data.
For a set of data points x and y, we can write the equation of the line we are aiming to model as:
y(x) = mx + c
where y(x) is the predicted y-value from the model and the gradient (m) and the y-intercept (c ) are called fit parameters. By using the method of linear regression (also called least squares fitting), we can calculate the values for the two parameters and plot the line of best fit to achieve our aims of better understanding the relationship or finding the estimated values of unknown points.
For this, we have to be able to calculate the slope (m) and intercept (c ) to give us the line of best fit for the data. We can do this using the equations:
where x ̄ and y ̄ represent the mean values of the data. This is made simple however by libraries that have already been implemented such as Scikit-Learn and Statsmodels Api that have linear regression functionality built in. We will first shown how to implement these formulas in Python and then how we can perform more complicated analysis with the Scikit Learn library.
The data that we have is a generated dataset set where we focus firstly only on two variables of X and Y to perform a simple Linear Regression. We can plot the data that we have to understand what relationship we may expect to see using the following code:
# Generates data frame from csv filedf = pd.read_csv("RegressionData.csv")# Turning the columns into arraysx = df["x"].valuesy = df["y"].values# Plots the graph from the above dataplt.figure()plt.grid(True)plt.plot(x,y,'r.')
We can see here that the X and Y variables shows a relationship whereby the Y variable seems to change linearly as X changes. This means that we can try to model this using a linear fit and so we can use linear regression.
For this, we can use the equations detailed above to attempt to calculate the gradient and y-intercept from the data as follows:
# calculating the means of the x and y datamean_x = np.mean(x) mean_y = np.mean(y)# calculating the slopeslope = np.sum((y - mean_y)*x) / np.sum((x - mean_x)*x) print ("Gradient:", slope)# calculating the interceptintercept = mean_y - slope*mean_x print ("Intercept", intercept)#out:Gradient: 0.9773554490236186Intercept 0.33323427076670953
Which we can then use to plot the line of best fit onto our data with:
x_max = max(x)plt.figure("") # start a new figureplt.grid(True) # add a grid# generate two points for the fitted linex_points = np.linspace(0, x_max*2., 2) y_points = slope*x_points + interceptplt.plot(x, y,'r.') # plotting data as pointsplt.line = plt.plot(x_points, y_points, 'b-') #plotting the line of best fit# setting limits for the axesplt.xlim(-10,110) plt.ylim(-10,110)
From this we can see the expected linear relationship that we identified in the first plot of the data. We can also see here that the intercept is around 0.3 and that the slope is close to 1 so almost a unit linear relationship.
A more efficient and useful way of doing this however is using Python’s Scikit Learn library which has extensive functionality for a more in depth evaluation of linear regression. While we have already calculated the gradient m and the intercept c we can do this with the Scikit Learn and then use this library to evaluate the model performance.
First of all we we need to prepare the data for the model. To do this we can reshape the model from the current format to arrays and then also split the data into a training and test dataset. The reason that we do this is so we can see how our model fits unseen data rather than allowing it to overfit on all the data we have. We can do this with the following code and visualise what it looks like:
# Independant variable or featuresX = x.reshape(-1,1)# Dependant variable or labelsy = y.reshape(-1,1)# Seperates the data into test and training sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)# Plotting the training and testing splitsplt.scatter(X_train, y_train, label = "Training Data", color = 'r')plt.scatter(X_test, y_test, label = "Testing Data", color = 'b')plt.legend()plt.grid("True")plt.title("Test/Train Split")plt.show()
What this shows is that we have taken the majority of the data (80%) as the data which the model will train on, while we have kept a minority of the data (20%) which can be used to evaluate model performance. We can see here that the testing data, while small, covers a wide range of the overall data and so could be a good test data to see how the model has performed.
Now that we have our data seperated into a test and training dataset, we can implement the linear regression model. To implement Linear Regression through Scikit Learn we begin by defining a regressor variable by calling the LinearRegression function. Then we pass regressor.fit(X_train, y_train) which passes the training data into the regression in order to train it.
# Defining our regressorregressor = LinearRegression()# Train the regressorfit = regressor.fit(X_train, y_train)
As before, our interest is in the gradient and intercept from our data which we can extract by calling the attributes .coef_ and .intercept_ from our trained regressor object. Thus we can get:
# Returns gradient and interceptprint("Gradient:",fit.coef_)print("Intercept:",fit.intercept_)#outGradient: [[0.98952479]]Intercept: [-0.1328508]
We can see here that while the gradient is similar to the one that we got before, the intercept is different. This is mainly because we have split our data into training and test data so that we are not working on the whole dataset.
This means that we can evaluate our model on our test data to see how well it is performing. For this, as before, we want to extract the line of best fit which we can now do using the regressor.predict(X_test) method rather than having to calculate the line as before. This means we can then implement this as:
# Predicted values y_pred = regressor.predict(X_test)# Plot of the data with the line of best fitplt.plot(X_test,y_pred)plt.plot(x,y, "rx")plt.grid(True)
The predictions can then be compared against the actual test results by creating a DataFrame to compare them to the actual points as follows:
# Converts predicted values and test values to a data framedf = pd.DataFrame({"Predicted": y_pred[:,0], "Actual": y_test[:,0]})df
We can then provide a score for our model by passing regressor.score(X_test, y_test) . As we mentioned previously this is data that the regressor hasn’t trained on which means it can provide a good basis for how well the model is performing on the data itself. This is the R2 score which shows the amount of variation in the target variable captured by the model. The best score for this is 1 where greater than 0.8 is often taken as an indicator of a good model fit, while 0 is seen as the worst performance.
For our current model the result is given as:
# Determines a score for our modelscore = regressor.score(X_test, y_test)print(score)#out:0.9763002831471521
Which suggests we have a good model fit.
The example we used above involved only two variables which was relatively easy to implement in our own code but in the real world it is highly unlikely that a single feature or independent variable is measured or used by itself. This makes the implementation and equation more complicated and so we can rely on Scikit Learn more to implement this.
The new equation where we have multiple independent variables takes the form
This means that effectively we have multiple values for the gradient and also have an additional term ε term which is the error. In effect, ε accounts for any potential points that might not be a fit for the linear model.
For this we use generated advertising data and their effect on sales. Here, the independent variables are given the by the TV , Radio and Newspaper columns while Sales is our independent variable. The data for this looks like:
# Converts advertising csv to a data framedf = pd.read_csv("advertising.csv")df
The process for then applying the linear regression algorithm under the Scikit Learn library takes a similar path where we take our X and Y variables, we reshape it, split into training and test data, implement the model, predict the unseen data and then extract the score. This can all be implemented as:
# Independent variablesX = df.drop("Sales",axis=1)# Dependent variabley = df["Sales"].values.reshape(-1,1)# Splitting into test and training dataX_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)# Defining regressorregressor = LinearRegression()# Training our regressorfit = regressor.fit(X_train,y_train)# Predicting valuesy_pred = fit.predict(X_test)# Scoring our regressorfit.score(X_test,y_test)#out:0.9174440089730509
Where we can see our model fits generally well the data that we have with a score of 0.917 and the implementation was relatively simple.
Of course, we can create a more complicated model and evaluate it in a more detailed way which we can do by using other metrics or evaluation methods built into the Scikit Learn library which more information can be found here. Also, beyond Scikit Learn you can also use the Statsmodels Api Library which has a more detailed summary for your regression implementation built in which can be used to evaluate your model, the implementation of which can be found here.
If you want any further information on our society feel free to follow us on our socials:
Facebook: https://www.facebook.com/ucldata
Instagram: https://www.instagram.com/ucl.datasci/
LinkedIn: https://www.linkedin.com/company/ucldata/
And if you want to keep up date with stores from the UCL Data Science Society and other amazing authors, feel free to sign up to medium using my referral code below.
philip-wilkinson.medium.com
Or check out other stories by me: