Matrix Autoregressive Model for Multidimensional Time Series Forecasting | by Xinyu Chen (陈新宇) | Towards Data Science

Autoregressive (AR) process is a representative and classical model in time series analysis and it has broad applications in nature, science, and economics. AR can describe certain time-varying processes with a linear relationship in univariate time series. Another important counterpart of AR is vector autoregressive (VAR) model, which can describe the coevolution patterns of multivariate time series. In this blog post, we will
introduce a matrix autoregressive (MAR) model for modeling multidimensional time series data,
discuss the optimization scheme and reproduce the MAR model with Numpy in Python.
For multivariate time series, if the variables are dependent, then VAR can explore the coevolution patterns of these variables.
To estimate the coefficient matrix, we can build a least squares problem by using L2 norm.
It would be not difficult to write the Python code for reproducing a VAR model.
import numpy as npdef var(X, pred_step):    N, T = X.shape    temp1 = np.zeros((N, N))    temp2 = np.zeros((N, N))    for t in range(1, T):        temp1 += np.outer(X[:, t], X[:, t - 1])        temp2 += np.outer(X[:, t - 1], X[:, t - 1])    A = temp1 @ np.linalg.inv(temp2)    mat = np.append(X, np.zeros((N, pred_step)), axis = 1)    for s in range(pred_step):        mat[:, T + s] = A @ mat[:, T + s - 1]    return mat[:, - pred_step :]
We can evaluate the VAR code by a toy example.
Write Python code here:
import numpy as npX = np.zeros((2, 10))X[0, :] = np.arange(1, 11)X[1, :] = np.arange(2, 12)pred_step = 2mat_hat = var(X, pred_step)print(mat_hat)
Run the code, and the expected output is
[[11. 12.] [12. 13.]]
As can be seen, these forecasts are completely same as the ground truth data.
In real world, multidimensional time series data are very common. For example, in finance, we can summarize seasonal economic indexes for some countries as a matrix, just like this:
In this matrix, if we take into account time information, then we would get a “tensor”:
Intuitively, the tensor is given as follows,
Instead of vector-form observations in each time t, we have matrix-form observations. Very recently, Chen et al. (2021) proposed a new matrix autoregressive (MAR) framework for building this multidimensional time series data. They took into account a bilinear structure like this:
where the matrix A is of size m-by-m, as the matrix B is of size n-by-n. Both two matrices are played as coefficients in MAR.
This framework has many advantages for building multidimensional time series data:
The framework can maintain the original data representation in the form of matrix.
The framework can reduce the amount of parameters in autoregressive models. For example, if we use VAR to explore such data, we would have (mn)2 parameters in the coefficient matrix. But using MAR, we only have m2+n2. This can avoid the over-parameterization issue in VAR for handling high-dimensional data.
As mentioned above, we can use a bilinear structure to build a MAR model. One question is that how to estimate coefficient matrices. The first impulse is write the optimization problem as a loss function:
where we use Frobenius norm.
If we define the objective as a function here:
Then, we have
It is not hard to obtain the closed-form solution to A by letting the derivative as 0. The closed-form solution to A is given by
If we rewrite the objective as:
Then, we have
Let the derivative as 0. The closed-form solution to B is given by
As mentioned above, we have two closed-form solutions. Since the solution to A involves B, and the solution to B involves A. We can use a classical algorithm for solving such problem, that is Alternating Least Squares (ALS) algorithm. The fundamental idea of this algorithm is updating the least squares solution of each variable in an iterative manner. For example, our case becomes
where l denotes the count of iteration.
By these equations, we can define a Python function here with ALS algorithm:
import numpy as npdef mar(X, pred_step, maxiter = 100):    m, n, T = X.shape    B = np.random.randn(n, n)    for it in range(maxiter):        temp0 = B.T @ B        temp1 = np.zeros((m, m))        temp2 = np.zeros((m, m))        for t in range(1, T):            temp1 += X[:, :, t] @ B @ X[:, :, t - 1].T            temp2 += X[:, :, t - 1] @ temp0 @ X[:, :, t - 1].T        A = temp1 @ np.linalg.inv(temp2)        temp0 = A.T @ A        temp1 = np.zeros((n, n))        temp2 = np.zeros((n, n))        for t in range(1, T):            temp1 += X[:, :, t].T @ A @ X[:, :, t - 1]            temp2 += X[:, :, t - 1].T @ temp0 @ X[:, :, t - 1]        B = temp1 @ np.linalg.inv(temp2)    tensor = np.append(X, np.zeros((m, n, pred_step)), axis = 2)    for s in range(pred_step):        tensor[:, :, T + s] = A @ tensor[:, :, T + s - 1] @ B.T    return tensor[:, :, - pred_step :]
We set up a simple example to test the algorithm. The forecasting task is given as follows,
Write Python code here:
import numpy as npX = np.zeros((2, 2, 10))X[0, 0, :] = np.arange(1, 11)X[0, 1, :] = np.arange(2, 12)X[1, 0, :] = np.arange(3, 13)X[1, 1, :] = np.arange(4, 14)pred_step = 2tensor_hat = mar(X, pred_step)print(tensor_hat[:, :, 0])print()print(tensor_hat[:, :, 1])
Run the code, and the expected output is
[[11. 12.] [13. 14.]][[12. 13.] [14. 15.]]
As can be seen, these forecasts are completely same as the ground truth data.
Rong Chen, Han Xiao, Dan Yang. Autoregressive models for matrix-valued time series. Journal of Econometrics, 2021.
This is a simple story about MAR model. If you are interested in this blog post, we highly recommend you to reproduce these codes and it would be not hard to follow.