Linear Regression with one or more variables | by Luca Zammataro | Towards Data Science

Linear Regression is a statistical model used in Machine Learning that falls in the “Supervised Learning” class of algorithms, and it applies to the analysis of biomedical data. We use it for predicting continuous-valued outputs, differently from the Logistic Regression, that instead is used for predicting discrete-valued outputs (i.e., classification).
Before starting, I suggest readers following the interesting course in Machine Learning at Coursera by Andrew NG [1]. The course provides an excellent explanation of all the arguments treated in this post.
Let’s start with our example: we have a dataset of patients checked for Systolic Blood Pressure (SBP) and monitored for age and weight, and we want to predict the SBP for a new patient. Implicitly we hypothesize that factors such as weight and age influence the SBP. For our dataset, we will take values of a Table from [2]
The Variable to be explained (SBP) is called the Dependent Variable, or Response Variable, and it matches with our output variable or target vector. Instead, the variables that explain the input (age and weight) are called Independent Variables or Predictor Variables, or Features. If the dependent and independent variables are continuous, as is the case for SBP, age, and weight, then a Correlation coefficient can be calculated as a measure of the strength of the relationship between them. [3]
We say that Linear Regression represents an evolution of the Correlation. The difference between them is: Correlation refers to the strength of the relationship between two or more variables. The Regression, instead, refers to an ensemble of statistical techniques and algorithms for describing the relationship between two or more variables [2].
Linear Regression assumes that the relationship between one or multiple input features and the relative target vector (outputs) is approximatively linear. [4], and it enables the identification and characterization of this relationship. The consequence of this assumption is that, in the Linear Regression model, the input features have an “effect” on the target vector (output), and this effect is constant.
The “effect” is often identified as “coefficient”, “weight” or “parameter”, and more simply, we say that Linear Regression computes a weighted sum of the input features, plus a constant called “bias term” or intercept [5].
To simplify, let’s start with the application of the Linear Regression with one variable.
Starting One Variable is a fundamental step if we want to understand thoroughly how the Linear Regression works. We will use the dataset in Table 1, only extrapolating the feature “Age” and using the “SBP” column as output.
All the code presented in this post is written in Python 2.7, and it’s self-explicative also for the porting to many other languages. For the implementation environment, I suggest the use of Jupyter Notebook.
Linear Algebra calculation will be primarily used, because of one the intrinsic advantages in avoiding, where possible, ‘while’ and ‘for’ loops. To accomplish this goal, we will use NumPy, a robust library of math functions for scientific computing with Python.
Before starting with the description of the Linear Regression model, it’s necessary to take a glance at our data and try to understand whether Linear Regression could apply to the data. The aim is predicting, for example, systolic pressure value, based on the patient age.
First of all, we import all the packages required for the Python code that will be discussed in this post: NumPy, Pandas and matplot. These packages belong to SciPy.org, which is a Python-based ecosystem of open-source software for mathematics, science, and engineering.
Numpy is necessary for the Linear Algebra calculations.
Pandas is an open-source library providing high-performance, data structures, and data analysis tools for Python. A pandas DataFrame is a two-dimensional size-mutable, potentially heterogeneous tabular data structure. It consists of three principal components, the data, rows, and columns, with labeled axes (rows and columns). Follow this link to the GeeksForGeeks portal, for having access to detailed information about the use of the pandas DataFrame.
Matplotlib is fundamental for creating all the plots.
In this step, we will create a dataset with our values. An example of the comma-separated-values format, with a header, is the following:
Age,Weight,SBP60,58,11761,90,12074,96,14557,72,12963,62,13268,79,13066,69,11077,96,16363,96,13654,54,11563,67,11876,99,13260,74,11161,73,11265,85,14779,80,138
Copy and paste the values in a file, and save it as “SBP.csv”
Once we have created the SBP.csv dataset, upload it and create a DataFrame using the Pandas pd object. The Python code for the dataset uploading is the following:
Typing “df” in a Jupyter cell, will display the DataFrame content as following:
The SBP dataset is formed by 3 columns (Age, Weight, and SBP), but we will upload the first and the last columns (Age and SBP); our model will determine the strength of the relationship between Age and SBP. Pandas makes easy accessing to DataFrame variables, that will be copied in an X vector, containing the input, and a y vector, for the output.
Before going ahead, the SBP dataset reported here has to be rescaled and normalized. In Machine Learning, scaling and normalization are required, especially when discrepancies in order of magnitude among the features and the output occurs. As mentioned in the Introduction, we will use Linear Algebra on matrixes to avoid, where possible, while and for loops on variables. Moreover, programming Linear Algebra will result in good readability of the code.
For this purpose, In Python, we can take advantage of NumPy. As mentioned before, NumPy is a library of math functions for scientific calculation and for Linear Algebra. All the operations can be simplified, creating a NumPy object, and using some of the associated methods. The reason why we want to use NumPy is that, even though many operations on matrixes can be done using the regular operators (+, -, *, /), NumPy guarantees a better control over the operations, especially if the matrixes are big. For example, with NumPy, we can multiply arguments element-wise, like the features matrix X and the output vector y:
An in-depth discussion on the use of NumPy goes beyond this post. To implement Scaling and Normalization of features, this code is what we need:
Code 5 implements a Python function called FeatureScalingNormalization(). This function takes X that is the features vector as an argument, and return 3 arguments: 1) the same X vector but scaled and normalized (X_norm), 2) mu, that is the average values of X in training set) and 3) sigma that is the Standard Deviation. Also, we will store mu and sigma because these parameters will be fundamental later. Copy the following code and paste it in a new Jupyter Notebook cell:
Typing ‘X’ in a Notebook cell will display the new values of X:
array([-0.73189052, -0.59728997,  1.15251726, -1.13569219, -0.32808885, 0.34491392,  0.07571281,  1.55631892, -0.32808885, -1.53949386, -0.32808885,  1.42171837, -0.73189052, -0.59728997, -0.05888774, 1.82552004])
The X vector containing the Age values is now normalized.
Now we will add a column of ones to the X vector.
This is the new structure of X:
array([[ 1.        , -0.73189052],       [ 1.        , -0.59728997],       [ 1.        ,  1.15251726],       [ 1.        , -1.13569219],       [ 1.        , -0.32808885],       [ 1.        ,  0.34491392],       [ 1.        ,  0.07571281],       [ 1.        ,  1.55631892],       [ 1.        , -0.32808885],       [ 1.        , -1.53949386],       [ 1.        , -0.32808885],       [ 1.        ,  1.42171837],       [ 1.        , -0.73189052],       [ 1.        , -0.59728997],       [ 1.        , -0.05888774],       [ 1.        ,  1.82552004]])
Plotting our data is a useful practice when we want to have an idea of how they are distributed. Plot the data using the matplotlib scatter method:
Visualizing the data at a glance, we can notice a pattern of increasing relationship, between Age and SBP. This is what we expect since systolic pressure is physiologically connected to age increasing.
The idea underlying the Linear Regression is represented by a function that predicts the output y based on the input feature X.
The predicted output is the h = θ * X term that is equal to a constant called “bias term” or “intercept term” or θ_0 plus a weighted sum of the input features X, where θ_1 represents the weight for X. We will call this function “Hypothesis” , and we will use it to “map” from X (Age) to y (SBP).
Since we are using Linear Algebra, for all the calculation, we can write the Hypothesis model in the vectorized form:
where θ_0 and θ_1 are expressed as vector θ=[θ_0, θ_1], and the Hypothesis is equal to θX.
The best performance in predicting y consists of finding θ values for which the distance between the predicted y value and the actual y value is closer to the minimum.
Let’s try two parameters randomly chosen, for the vector θ, for instance: θ = [140.0, 5.0], and see what happens:
The Hypothesis model in Figure 2, represented by the red line, should predict y (the SBP). It represents our h=θX vector in predicting y for the values of θ = [140.0, 5.0]. But this model, obviously, does not fit our data. As highlighted by the blue lines connecting dots with the red line, the Hypothesis “touches” some of the y values, but the rest of the h vector is far from the minimum. So we are tempted to guess which θ could predict y when setting with different values. We could choose θ “by trial and error” to minimize all the distances between the Hypothesis and y. To accomplish this goal, we can calculate the Cost Function for our model.
The Cost Function can register how much far we are from the minimum of the Hypothesis model and can help us in finding the best θ. The equation describing the Cost Function is the following:
Where m is the length of the X vector (in our case = 16), and i is the index assigned to each item in the dataset. The Equation is composed of three components:
the Hypothesis (h=θX)the SquaredError that is = (h-y) ^2The Cost Function J that is then calculated as J = 1/2m * Sum(SquaredError)
the Hypothesis (h=θX)
the SquaredError that is = (h-y) ^2
The Cost Function J that is then calculated as J = 1/2m * Sum(SquaredError)
Since we use Linear Algebra, the vectorized implementation of Equation 3 is the following:
Intuition I.
In order to simplify the explanation, let’s try to manually calculate the Cost Function for a smaller dataset composed only of the first 3 values of the SBP dataset, and with θ = [120.0, 10.0]. These parameters are chosen randomly, at the moment, because we don’t have to set the best θ for now. We will split the X and y, producing arrays X_1 and y_1:
Also, we have to set m=3, because we have three samples now. Let’s plot the data and the Hypothesis as following:
The vector y corresponding to the first three values (the blue dots) of the SBP is:
y = [117.0, 120.0, 145.0]
Since our θ is = [120, 10.0], the product of h = θ*X_1 will be represented by the following vector, (highlighted by the dots on the red line):
h = θ*X_1 = [112.7, 114.0, 131.5]
The dashed blue lines highlight the distances between actual y_1 values and predicted values. Now, we have all we need, to calculate the Cost Function J. We will apply the Cost Function as described in Solution 1:
...the Cost Function (J) is = 39.3
The Cost Function in Python.
The following Python code implements the Cost Function:
The code implements step by step the Cost Function described in Equation 4 (vectorized). Let’s repeat again:
the Hypothesis (h=θX)the SquaredError that is = (h-y) ^2)the Cost Function J that is = 1/2m * Sum(SquaredError)
the Hypothesis (h=θX)
the SquaredError that is = (h-y) ^2)
the Cost Function J that is = 1/2m * Sum(SquaredError)
Now that we have understood the mechanism underlying the Cost Function calculation, let’s go back to the complete SBP dataset (16 patients). If we want to calculate the Cost Function for the whole SBP dataset, using θ = [140.0; 5.0], we will type :
The function will return J = 138.04, which is the Cost Function calculated for θ = [140.0; 5.0]. This J is not the minimum J that we could find, since that we have manually set θ, without any idea about how to minimize it. The following Intuition II could help us in understanding better the limit of our manual approach.
Intuition II.
The following code generates randomly 10 θ vectors and passes them to the calcCostFunction, producing a table of the relative Cost Functions (J):
The resulting output is:
[Th0 Th1] J[38. 55.] 5100.4688623710845[71. 47.] 2352.6631642080174[28. 76.] 7148.466632549135[73. 75.] 3579.826857778751[79. 47.] 1925.1631642080174[12. 42.] 7320.026790356101[68. 25.] 1992.2131192595837[25. 92.] 8565.015528875269[51. 46.] 3667.1483894376343[13. 62.] 7992.509785763768
The “take-home message” is that trying to handly minimize J, is not the correct way to proceed. After 10 runs on randomly selected θ’s, the behavior of J is unpredictable. Moreover, there is no way to guess J basing on θ. So the question is: How we can choose θ, to find the minimum J? We need an algorithm that can minimize J for us, and this algorithm is the argument of the next Step.
We are interested in finding the minimum of the Cost Function using Gradient Descent, which is an algorithm that can automatize this search. The Gradient Descent calculates the derivative of the Cost Function, updating the vector θ by mean of the parameter α, that is the learning rate. From this moment on, we will refer to the SBP dataset, as the training set. This clarification is essential since Gradient Descent will use the difference between the actual vector y of the dataset and the h vector prediction, to “learn” how to find the minimum J. The algorithm will repeat until it will converge. θ updating has to be simultaneous.
Since we use Linear Algebra, the vectorized implementation is the following:
Note that here we have to transpose X since X is a [16, 2] matrix and Error is a [16, 1] vector.
Gradient Descent implementation.
The following Python code implements the Gradient Descent. We will use the vectorized form of Equation 5:
To run the Gradient Descent, we have to initialize θ, iterations, and α, that together with X and y are the arguments of the gradientDescent function:
The results are collected in the “results” list. This list is composed of the found θ, plus two lists containing the θ and J histories. After 2000 iterations the Gradient Descent has found θ = [128.4, 9.9], and J = 59.7, which is the minimum J. We will use the two lists for plotting the Gradient Descent activity. The following code will plot the training set and h.
The Hypothesis h now fits with our data!
Let’s plot the θ history:
A plot of the θ history is shown in Figure 6. The red curve represents θ_0, the green curve θ_1. After 2000 iterations, θ is = [128.4; 9.9]
Now let’s plot the J history:
After circa 200 iterations the Cost Function falls down, stabilizing around 59.7 after 1500 iterations. The J curve depends on α, that we have set to 0.01.
Now that we have found the best θ, we can predict the Systolic Blood Pressure for a 75 years-old person. The query is a vector, composed of two numbers [1, 75]. The first number corresponds to the feature x_0. To run the prediction, we have to scale and normalize the query vector, using the mu and sigma parameters that we have calculated in Step 5: Feature Scaling and Normalization. The query vector will be [1, 1.29]. Then, we have to multiply the query for the θ vector (θ = [128.4, 9.95]). The following code implements the prediction.
The SBP for a 75 years-old man is: 141.2
Let’s do some experiments with the learning rate α. Changing α will affect the dynamics of J. If α is too little, the Gradient Descent will converge slowly, and we need to train it with more iterations for finding the minimum of J. Contrarily, if α is too big, the Gradient Descent risks to never converge. Interestingly, for values of α around 1.9, the Gradient Descent converges, but for the first 40 iterations, the behavior of θ is turbulent, for successively reaching stability. (Figure 8)
We can create a contour plot, which is a graph containing many concentric tracks. For each track there are various pairs of θ associated with a constant value of J. The θ corresponding to the minimum J, lies at the center (the red dot). The other concentric lines correspond to all the different values of J. The most the distance from the center, the higher the value of the Cost Function J.
Code 20 produces the following plot:
Note that θ = [128.4; 9.9] corresponding to the minimum J (59.7), is the red dot in the center of the graph. The blue dots track the path of the Gradient Descent in converging to the minimum.
We have explained the statistical mechanisms underlying Linear Regression with one variable: the feature Age in the SBP dataset. The major part of the code here proposed works also with multiple variables. The SBP dataset is composed of 2 features (Age and Weight) and one output: SBP. In this step, we will update the code in a way that it can fit with multiple variables. The only adjustments required concern:
The Dataset uploadingThe Feature Scaling and Normalization functionThe code for adding a column of “ones” to the vector XThe Prediction Query.
The Dataset uploading
The Feature Scaling and Normalization function
The code for adding a column of “ones” to the vector X
The Prediction Query.
Dataset uploading
The code for uploading the dataset has to be modified in order to produce a new X vector containing Age and Weight of each patient:
The numpy method used for producing the new X vector is .vstack() because we now want an X vector with two sets of distinct features.
Feature Scaling and Normalization function
The code concerning Feature Scaling and Normalization is modified in vectors mu and sigma. Now the two vectors will accept two parameters each.
Adding a column of “ones” to the vector X
The line for adding “ones” to the X vector is modified as follows:
The Prediction Query
The query for the predictions and the code for the normalization are modified as follows:
The prediction result, in this case, is an SBP =143.47
With these changes, the Python code is ready for Linear Regression with multiple variables. You have to update the code every time you will add new features from your training set!
I hope you find this post useful!
Andrew NG, Machine Learning | Coursera.John Pezzullo, Biostatistics For Dummies, Wiley, ISBN-13: 9781118553985Schneider, A; Hommel, G; Blettner, M. Linear Regression Analysis Part 14 of a Series on Evaluation of Scientific Publications, Dtsch Arztebl Int 2010; 107(44): 776–82; DOI: 10.3238/arztebl.2010.0776Chris Albon, Machine Learning with Python Cookbook, O’Really, ISBN-13: 978–1491989388.Aurélien Géron, Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, O’Reilly, ISBN-13: 978–1491962299.
Andrew NG, Machine Learning | Coursera.
John Pezzullo, Biostatistics For Dummies, Wiley, ISBN-13: 9781118553985
Schneider, A; Hommel, G; Blettner, M. Linear Regression Analysis Part 14 of a Series on Evaluation of Scientific Publications, Dtsch Arztebl Int 2010; 107(44): 776–82; DOI: 10.3238/arztebl.2010.0776
Chris Albon, Machine Learning with Python Cookbook, O’Really, ISBN-13: 978–1491989388.
Aurélien Géron, Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, O’Reilly, ISBN-13: 978–1491962299.