Simplifying Multivariate Regressions using Matrices
By: Naman Agrawal
Tags: Regression, Mathematics, Econometrics, Statistics, Data Science

What is a Multivariate Regression?

Multivariate regression is a tool used by statisticians to express the relationship between a dependent variable (Yᵢ) and several independent variables (X₁ᵢ, X₂ᵢ, X₃ᵢ, …. Xₖᵢ). In general, multivariate regression models can be represented as:

Yᵢ is called the regressand and Xᵢ‘s are called regressors. εᵢ is the associated error, a term used to quantify the effect of unobserved factors in the regression. The β‘s represent regression coefficients i.e. the parameters which will help us in understanding the relationship between Y and X’s. k represents the number of regressors or dependent variables in our regression. i represents a particular observation associated with the regressor, regressand and the error term. Here, we assume that we have n observations for each variable.

Preliminary points to be noted: i. A variable set, quite simply, refers to a set of variables. So, Y, X, β, and ε are all variable sets. A variable is one element of that variable set. X₁, X₂, X₃, β₀, β₁, β₂, etc. are variables. Y and ε are also variables. They can be considered variable sets with a single variable. Finally, observation refers to the value of a specific variable in that set. X₁₁, X₂₁, Y₁, Y₂, ε₂, etc. are all observations. ii. Notations used in econometrics are often different from those used in statistics. When we represent the regressor as Xᵣₛ, r represents the particular variable and s represents the observed value for that variable. For example, X₅₄ means the 4th observation associated with the 5th regressor.

Dealing with multivariate regressions can become very difficult because of the large amount of data that needs to be fitted. Hence, matrices are often used to make it easier to amalgamate the data and carry out various mathematical operations.

Now, we are ready to represent regressions using matrices!

Let’s start by replacing the variables with vectors of observations:

1) Y:

In matrix notation for Y, we use the rows to represent observations and columns to represent the variables. There is only 1 dependent variable in our regression. So, the number of columns is 1. Suppose there are n observations associated with Yᵢ. Then, the number of rows will be n. So, we must have a (n)x(1) matrix. The matrix can be represented as:

2) β:

We follow a similar process. However, this time we use the rows to denote the variables. Here, β represents the parameters we wish to know. It does not make sense to have an observation for β. It is, quite simply, a set of unobserved quantities, we want to calculate. So, the number of columns, which would otherwise represent the number of observations is 1. The reason we interchange the notations in β and Y will be clear when we frame the final matrix equation.

So, do we have k rows and 1 column? Nope. Notice that we have a β₀ in our model too. So, the number of parameters we want to estimate is k+1. Hence, we will have a matrix with k+1 rows and 1 column i.e. a (k+1)x(1) matrix. The matrix is represented as:

3) X:

We follow the same notation process we used for Y. The columns will represent the variables and the rows will represent the observations associated with each variable. This time, unlike Y, we have many variables. There are a total of k X’s. However, we introduce one more X so that we have one X for every β . Let this X be X₀ᵢ . So, we are essentially replacing our original model with:

To get back our original model, we can always replace X₀ᵢ with unity. Now that we have associated every β with an X, there are k+1 variables and n observations for every variable. This gives us a (n)x(k+1) matrix:

Note: Don’t confuse Xᵣₛ with rᵗʰ row and sᵗʰ column. Remember that in econometrics, r is the variable and s is the observation in such a notation.

4) ε:

Process of developing a vector matrix for ε is almost the same as that of Y. Here, we have only one ε for every set of variables. Thus, we must have a single column matrix. Since we have n observations for ε, we have n rows, giving us a (n)x(1) matrix. The matrix is shown as:

Now, that we have developed matrices for each variable set, we can put them together in a simple equation to get to our regression model. The equation used is:

where Y, X, β, and ε are the matrices discussed above.

For matrix multiplication, it is pivotal that the number of columns of the first matrix is equal to the number of rows of the second matrix as we have ensured with X and β. Hence, the slight detour was seen while deriving the matrix notation for β.

The above equation will help us correctly assign data set values and run multivariate regressions. To better comprehend how the equation works, let’s plug in the matrices:

On further solving:

In summation notation:

Now, if we compare every to the corresponding term in the other matrix, we can verify that in general:

Hence, we have successfully used matrices to simplify multivariate regressions.

Through mathematical manipulations on the matrix equations, we can arrive at Ordinary Least Squares (OLS) estimators. The process will be described in more detail in my next few articles!