Advanced Regression with Python | Towards Data Science

In a series of weekly articles, I will cover some important statistics topics with a twist.
The goal is to use Python to help us get intuition on complex concepts, empirically test theoretical proofs, or build algorithms from scratch. In this series, you will find articles covering topics such as random variables, sampling distributions, confidence intervals, significance tests, and more.
At the end of each article, you can find exercises to test your knowledge. The solutions will be shared in the article of the following week.
Articles published so far:
Bernoulli and Binomial Random Variables with Python
From Binomial to Geometric and Poisson Random Variables with Python
Sampling Distribution of a Sample Proportion with Python
Confidence Intervals with Python
Significance Tests with Python
Two-sample Inference for the Difference Between Groups with Python
Inference for Categorical Data
Advanced Regression
Analysis of Variance — ANOVA
As usual, the code is available on my GitHub.
Imagine that you want to predict the salary of a Data Scientist based on the number of years he has been coding. To build this relationship, we can sample 20 random people from the population of Data Scientists and plot the relationship (notice that the salary unit is 10,000€). At the same time, we can calculate the line of best fit.
import pandas as pdimport matplotlib.pyplot as pltimport numpy as npfrom scipy.stats import t, chi2salaries = [[4, 5.5, 3.5, 6, 7, 9, 3.4, 5.3, 4.3, 5.3, 6, 6.2, 6.5, 7, 7.1, 4.3, 5.2, 5, 5.7, 3.2]]yearsCoding = [[2, 5, 1, 4, 5, 10, 2, 2, 4, 3, 4, 6, 5, 6, 7, 4, 3, 4, 4, 3]]n = 20plt.scatter(yearsCoding[0], salaries[0])plt.xlabel('Years Coding')plt.ylabel('Salary');
The line of best fit can be calculated by minimizing the squared distance between the data points and the line. The equation for the regression line is the following:
where b is the slope, and a is the intercept of the regression line. Notice that by calculating this line for our data, we calculate the line of best fit for a single sample of 20 Data Scientists. We could calculate a different line of best fit for a different sample that we took from our population. This happens because we are estimating parameters for a population. If you could get the salaries and number of years coding for all the Data Scientists in the world to build your model, you would be estimating the true population parameters. In that case, we replace a and b with α and β:
Since we are using samples to estimate the population parameters, we can make inferences based on those samples. With that in mind, we know that b will not be exactly equal to β, but can we say that there is a positive linear relationship or a non-zero linear relationship between salaries and years of coding? As a matter of fact, we can do it by defining a confidence interval around this statistic. This way, we will have a good sense of where the true parameter might actually be. In previous articles, we saw how to compute such interval: we subtract/add the critical value t* multiplied by the standard error of b.
In the same way, we could define a hypothesis test for the slope parameter. One states that there is no relationship between the variables as a null hypothesis and, as the alternative, that there is a relationship between the variables (you could also be more specific and define the alternative hypothesis to be only positive or negative).
As in any other inference procedure, we must respect the conditions to ensure valid results. In this case, for the slope of linear regression, we must ensure that:
The data must be randomly generated;
The individual observations should be independent (or approximately independent — remember the 10% rule);
The relationship between the variables must be linear;
For any given x in the population, the distribution of y must be normal;
The variance should be constant for any given x.
We already defined our equation for the regression line; now, we need to define our cost function and a method to update our parameters. For our cost function, we will be using the Mean Squared Error:
To update our parameters, we will be using gradient descent. I will not cover the method in-depth; I only want to give some intuition. The first step is to compute the gradient of the cost function with respect to each parameter:
Then we update the parameters accordingly:
where α represents the learning rate.
def fit_lr(X, y, num_iter=1000, lr=0.01):    n_samples = X.shape[0]    slope = 0    intercept = 0    for _ in range(num_iter):        y_predicted = np.dot(X, slope) + intercept        ds = (1/n_samples) * np.dot(X.T, (y_predicted - y))        di = (1/n_samples) * np.sum(y_predicted - y)        slope -= lr * ds        intercept -= lr * di            return (slope, intercept)slope, intercept = fit_lr(np.array(yearsCoding[0]), np.array(salaries[0]))plt.scatter(yearsCoding[0], salaries[0])plt.xlabel('Years Coding')plt.ylabel('Salary')plt.plot(yearsCoding[0],slope*np.array(yearsCoding[0]) + intercept, color='r');
print('Slope=' + str(slope))print('Intercept=' + str(intercept))Slope=0.7096702503648314Intercept=2.4147010493601404
What would be the salary of a Data Scientist that codes for 15 years?
slope * 15 + intercept13.059754804832611
It would be around 130,000€.
We can treat the slope ^b as a normally distributed random variable with a mean of b and a variance equal to σ2 divided by the sum of squares of X.
As we do not know the population variance, we use the sampling variance to calculate the Standard Error (SE):
We can describe the SE as the standard deviation of the sampling distribution of the slope of the regression line.
SS_xx = np.sum((np.array(yearsCoding[0]) - np.mean(np.array(yearsCoding[0])))**2)SS_xx79.20000000000002SE_b = (np.sqrt(np.sum((np.array(salaries[0])-(np.array(yearsCoding[0])*slope+intercept))**2)/(n-2))) / np.sqrt(SS_xx)SE_b0.08687984291368046
We can finally calculate the 95% confidence interval for the slope of our linear regression.
print('95% Confidence interval=[' + str(np.round(slope - t.ppf(0.975, df=n-2)*SE_b,2)) + ',' + str(np.round(slope + t.ppf(0.975, df=n-2)*SE_b,2)) + ']')95% Confidence interval=[0.53,0.89]
Recall that a 95% confidence interval means that the true slope is contained in our confidence intervals 95% of the time.
We can use this interval to test the following hypothesis at the α= 0.05 level of significance:
What can we conclude? Assuming that H_0 is true, we face the situation where β=0 does not overlap with the 95% interval, which happens with less than 5% probability. Thus, we reject H_0 and accept the suggested H_1. It states that there is some relationship (non-zero) between the number of years coding and the salary of a Data Scientist.
Now, suppose that the relationship between the salary of a Data Scientist and the number of years of coding is not linear. In this case, we see that it is closer to an exponential relationship. What happens if we try to fit a line to these data?
salaries =    [[3.5, 7.5, 2.5, 7, 9, 40, 3.4, 4.3, 5.3, 5.3, 4, 12.2, 8.5, 10, 18.1, 4.3, 5.2, 5, 5.7, 5.2]]yearsCoding = [[2, 5,   1  , 4, 5, 10, 2  , 2  , 4,   3  , 4, 6  , 5  , 6 , 7   , 4   , 3  , 4, 4  , 3  ]]n = 20plt.scatter(yearsCoding[0], salaries[0])plt.xlabel('Years Coding')plt.ylabel('Salary');
slope, intercept = fit_lr(np.array(yearsCoding[0]), np.array(salaries[0]))plt.scatter(yearsCoding[0], salaries[0])plt.xlabel('Years Coding')plt.ylabel('Salary')plt.plot(yearsCoding[0],slope*np.array(yearsCoding[0]) + intercept, color='r');
print('Slope=' + str(slope))print('Intercept=' + str(intercept))Slope=3.35501894004377Intercept=-5.5729005963397205
We see that the line that we have fitted does not explain the data at all. We even get a negative salary for someone that just started coding, and we would heavily underestimate the salary of someone that codes for 10 years. We have two options here. The first is to fit an exponential line to the data. The problem with this approach is that we lose the tools we have been developing to fit and analyze a linear relationship between two variables. On the other hand, we can transform our data before fitting the regression line.
plt.scatter(yearsCoding[0], np.log(salaries[0]))plt.xlabel('Years Coding')plt.ylabel('ln(Salary)');
slope, intercept = fit_lr(np.array(yearsCoding[0]), np.log(np.array(salaries[0])))plt.scatter(yearsCoding[0], np.log(salaries[0]))plt.xlabel('Years Coding')plt.ylabel('ln(Salary)')plt.plot(yearsCoding[0],slope*np.array(yearsCoding[0]) + intercept, color='r');
print('Slope=' + str(slope))print('Intercept=' + str(intercept))Slope=0.3173211894646696Intercept=0.5195321580809141
With the transformation, we see that the line of best fit actually explains the relationship between the variable years of coding and the transformed variable salary. But notice that the relationship between the original variables is not linear but exponential.
What would be the salary of a Data Scientist that codes for 15 years in this scenario?
np.exp(slope * 15+intercept)196.24227641696947
It would be almost 2M€! Compare it with the result from the linear dataset that we used before.
This article covered how to solve a linear regression problem and then use our line of best fit to make inferences about our parameters. In this case, we focus on the slope parameter, using confidence intervals and hypothesis testing to evaluate the type of association between two variables of interest. Finally, we introduced the concept of transforming variables to deal with the fact that the data can have nonlinear patterns.
You will get the solutions in next week’s article.
Márcia collected data on the battery life and price of a random sample of Portable Computers. Based on the data presented below, what is the test statistic for the null hypothesis that the population slope is 0?Rui obtained a random sample of colleagues at work and noticed a positive linear relationship between their ages and the number of kilometers they said they walked yesterday. A 95% confidence interval for the slope of the regression line was (15.4, 155.2). Rui wants to use this interval to test H_0: β=0 vs. H_1: β ≠ 0 at the 5% significance level. Assume that all conditions for inference have been met. What should Rui conclude?
Márcia collected data on the battery life and price of a random sample of Portable Computers. Based on the data presented below, what is the test statistic for the null hypothesis that the population slope is 0?
Rui obtained a random sample of colleagues at work and noticed a positive linear relationship between their ages and the number of kilometers they said they walked yesterday. A 95% confidence interval for the slope of the regression line was (15.4, 155.2). Rui wants to use this interval to test H_0: β=0 vs. H_1: β ≠ 0 at the 5% significance level. Assume that all conditions for inference have been met. What should Rui conclude?
According to a distributor of surfboards, 66% of the boards are common, 25% are uncommon, and 9% are rare. José wondered if the rarity levels of the boards he and his friends owned followed this distribution, so he took a random sample of 500 boards and recorded their rarity levels. The results are presented in the table below. Carry out a goodness-of-fit test to determine if the distribution of rarity levels of surfboards José and his friends own disagrees with the claimed percentages.
According to a distributor of surfboards, 66% of the boards are common, 25% are uncommon, and 9% are rare. José wondered if the rarity levels of the boards he and his friends owned followed this distribution, so he took a random sample of 500 boards and recorded their rarity levels. The results are presented in the table below. Carry out a goodness-of-fit test to determine if the distribution of rarity levels of surfboards José and his friends own disagrees with the claimed percentages.
table = [['Cards', 345, 125, 30]]alpha = 0.05df = pd.DataFrame(table)df.columns = ['Rarity level', 'Common', 'Uncommon', 'Rare']df = df.set_index('Rarity level')df
arr = df.to_numpy()arr = np.concatenate((arr, (np.sum(arr)*np.asarray([0.66, 0.25, 0.09])).reshape(1,-1)))chi_sq_statistic = np.sum((arr[0]-arr[1])**2/arr[1])chi_sq_statistic5.681818181818182print('P-value = ' + str(np.round(1-chi2.cdf(chi_sq_statistic, df =2), 4)))P-value = 0.0584if 1-chi2.cdf(chi_sq_statistic, df =2) < alpha:    print('Reject H_0')else:    print('Fail to reject H_0')Fail to reject H_0