Animations of Multiple Linear Regression with Python | by Tobias Roeschl | Towards Data Science

In this article, we aim to expand our capabilities in visualizing gradient descent to Multiple Linear Regression. This is the follow-up article to “Gradient Descent Animation: 1. Simple linear regression”. Just as we did before, our goal is to set up a model, fit the model to our training data using batch gradient descent while storing the parameter values for each epoch. Afterwards, we can use our stored data to create animations with Python’s celluloid module.
This is the revised version of an article about the same topic I uploaded on July 20th. Key improvements include the cover photo and some of the animations.
Multiple linear regression is an extended version of simple linear regression in which more than one predictor variable X is used to predict a single dependent variable Y. With n predictor variables, this can be mathematically expressed as
with
and b representing the y-intercept (‘bias’) of our regression plane. Our objective is to find the hyperplane which minimizes the mean squared distance between the training data points and that hyperplane. Gradient descent enables us to determine the optimal values for our model parameters θ, consisting of our weights w and our bias term b, to minimize the mean squared error between observed data points y and data points we predicted with our regression model (ŷ). During training, we aim to update our parameter values according to the following formula until we reach convergence:
with ∇J(θ) representing the gradient of our cost function J with respect to our model parameters θ. The learning rate is represented by α. The partial derivatives for each weight and the bias term are the same as in our simple linear regression model. This time, however, we want to set up a (multi-)linear regression model which is flexible to the number of predictor variables and introduce a weight matrix to adjust all weights simultaneously. Additionally, we store our parameter values in arrays directly during the fitting process itself, which is computationally faster than using for-loops and lists, as we did in the last article. I decided to arbitrarily set the initial parameter values for the weight(s) to 3 and to -1 for the bias. In Python, we import some libraries and set up our model:
After that, we intend to fit our model to some arbitrary training data. Although our model can theoretically handle any number of predictor variables, I chose a training dataset with two predictor variables. We intentionally use a particularly small learning rate, α=0.001, to avoid overly large steps in our animations. Since this article focuses on animations rather than statistical inference, we simply ignore the assumptions of linear regression (e.g. absence of multicollinearity, etc.) for now.
x_train = np.array([ # two independent variables
    [1,-2],
    [2,1],
    [4,1],
    [5,-3],
    [6,4],
    [7,5]
])

y_train = np.array([ 
    [14],
    [-12],
    [-31],
    [-21],
    [-51],
    [-37]
])

model=LinearRegression(x_train,y_train, lr=0.001) # set learning rate
model.fit(x_train,y_train, numberOfEpochs=100000) # set number of epochs

# Stored parameter values:
w0=model.AllWeights.T[0]
w1=model.AllWeights.T[1]
b= model.AllBiases
c=model.AllCosts
cl=model.cl

print("Final weights: "+ str(model.w)) # print final model parameters
print("Final bias: "+ str(model.b))
print("Final costs: " + str(model.cost(x_train,y_train)))

Final weights: [[-6.75922888]
 [-2.07075472]]
Final bias: [[7.23420837]]
Final costs: 75.63518594476348

To ensure our fitted parameters converged to their true values, we verify our results with sklearn’s inborn linear regression model:
# cross-check results with sklearn's linear regression model:
import sklearn
from sklearn.linear_model import LinearRegression

reg = LinearRegression().fit(x_train, y_train)
print(reg.coef_)
print(reg.intercept_)

[[-6.75922888 -2.07075472]]
[7.23420837]

Now, we can finally create our first animation. Like before, we want to begin with visualizing values our cost function and parameters take on with respect to the epoch while plotting the corresponding regression plane in 3-D. As described previously, we intend to only plot values for selected epochs, since the largest steps are usually observed at the beginning of gradient descent. After each for-loop, we take snapshots of our plots. Via Camera’s animate-function we can turn snapshots into animations. In order to get the desired regression plane, we introduce a coordinate grid (M1, M2) via numpy.meshgrid and define a function “pred_meshgrid( )” to calculate the respective z-values with respect to the model parameters at a certain epoch. The dashed connecting lines seen in the following animations can be obtained through line plots between training data points and predicted points. By returning the final parameter values (see commented-out code!) we obtained in our animation, we ensure that we roughly visualized model convergence despite not using the full range of parameter values, we stored during the fitting process.
Despite its simplicity, plotting the values parameters take on for each epoch separately is actually the most informative and most realistic way of visualizing gradient descent with more than two model parameters. This is because we can only witness changes in costs and all model parameters involved simultaneously this way.
When creating more sophisticated animations of gradient descent, especially in 3-D, we have to focus on two model parameters while keeping the third parameter ‘fixed’. We are generally more interested in the weights rather than the bias term of our multiple linear regression model. In order to visualize how costs steadily decrease with our adjusted weights (w0, w1), we have to set up a new linear regression model where the bias term b is fixed. This can easily be done by setting the initial value for b to a predefined value b_fixed and remove the part of code where b is being updated in the new model. b_fixed can take on any value. In this case, we just set it to the y-intercept value the former model converged to:
# set b to the y-intercept value we previously observed. 
b_fixed= float(model.b) 

# Define new model with fixed intercept:
class LinearRegression_fixed_b(object):  
    def __init__(self,x,y,b,lr=0.01):
        self.lr=lr
        self.b=np.array([[b]]) # set fixed(!) starting value for b
        self.w=np.ones((1, x.shape[1])).T   *3   # set starting weights to 3.
          
    def cost(self,x,y):     
        pred = x@self.w+self.b 
        e=y-pred    
        return np.mean(e**2)  

    def step(self, x,y):
        pred = x@self.w+self.b
        e=y-pred
        self.w = (self.w.T-self.lr*(np.mean(e*(-2*x), axis=0))).T 
           
    def fit(self, x,y, numberOfEpochs=1000000):
        self.AllWeights= np.zeros((numberOfEpochs, x.shape[1]))
        self.AllBiases= np.zeros(numberOfEpochs)
        self.AllCosts= np.zeros(numberOfEpochs)
        self.cl= np.zeros((numberOfEpochs,len(x)))
        
        for step in range(numberOfEpochs):
            self.AllWeights[step]=self.w.T
            self.AllCosts[step]=self.cost(x,y)
            self.cl[step]=(self.predict(x)).T.flatten()
            self.step(x,y)

    def predict(self, x):
        return (x @ self.w + self.b) 

        
model=LinearRegression_fixed_b(x_train,y_train,b_fixed, lr=0.001) # set learning rate
model.fit(x_train,y_train, numberOfEpochs=100000) # set number of epochs
    
w0=model.AllWeights.T[0]
w1=model.AllWeights.T[1]
c=model.AllCosts
cl=model.cl

print("Final weights (fixed-intercept model): "+ str(model.w))
print("Final bias (fixed-intercept model): "+ str(float(model.b)))
print("Final costs (fixed-intercept model): " + str(float(model.cost(x_train,y_train))))

Final weights (fixed-intercept model): [[-6.75922888]
 [-2.07075472]]
Final bias (fixed-intercept model): 7.234208367512638
Final costs (fixed-intercept model): 75.63518594476345

After accumulating new data with our new model, we yet again create another coordinate grid (N1, N2) for the following animations. This coordinate grid enables us to plot the costs for every possible pair of w0 and w1 within the given range of numbers.
Contour plots allow us to visualize three-dimensional surfaces on two-dimensional planes via contour lines and filled contours (= contourfs). In our case, we want to plot w0 and w1 on the x-axis and y-axis respectively with the costs J as contours. We can label specific contour levels within the graph with Matplotlib’s contour-function. Since we know that our final costs are roughly 76, we can set our last contour level to 80.
When the y-intercept was allowed to vary, we saw significant changes of parameter values and costs with later epochs, that we did not want to miss out in our regression- and parameter plots. However, with b being fixed, most of the ‘action’ appears to be confined to the first 400 epochs. Thus, we limit the epochs we intend to visualize in the following animations to 400, which is less computationally expensive and results in more appealing animations. To confirm this impression, we can compare the final parameter values, the fixed intercept model returned after 100,000 epochs, to the parameter values we obtain after 400 epochs (see commented-out code below!). Since parameter- and cost values generally match up, it is fair to say that we visualized model convergence despite restricting the number of epochs being visualized.
For the last plot, we want to incorporate the same epochs as we did with the contour plot. This time, however, we want to visualize gradient descent in three dimensions using a surface plot.
Theoretically, plotting the trajectory of gradient descent in the x-y-plane - as we did with the contour plot - corresponds to the ‘real’ trajectory of gradient descent. In contrast to the surface plot above, gradient descent actually does not involve moving in the z-direction at all since only parameters are free to vary. For a more in-depth explanation see here. Lastly, I want to point out that creating animations with Celluloid can be very time-consuming. Especially animations involving surface plots can take up to 40 minutes to return the desired result. Nevertheless, I preferred Celluloid over other packages (e.g. matplotlib.animation) due to its simplicity and clarity. As always, creative input and constructive criticism is appreciated!
I hope you found this article informative and useful. Should any questions arise or if you noticed any mistakes, feel free to leave a comment. In the next article of this series, I will dedicate myself to animations of Gradient Descent on the example of Logistic Regression. The complete notebook can be found on my GitHub.
Thank you for your interest!