How To Model Time Series Data With Linear Regression | by Jiahui Wang | Towards Data Science

Welcome back! This is the 4th post in the column to explore analysing and modeling time series data with Python code. In the previous three posts, we have covered fundamental statistical concepts, analysis of a single time series variable, and analysis of multiple time series variables. From this post onwards, we will make a step further to explore modeling time series data using linear regression.
We all learnt linear regression in school, and the concept of linear regression seems quite simple. Given a scatter plot of the dependent variable y versus the independent variable x, we can find a line that fits the data well. But wait a moment, how can we measure whether a line fits the data well or not? We cannot just visualize the plot and say a certain line fits the data better than the other lines, because different people may make different evaluation decisions. How can we quantify the evaluation?
Ordinary least squares (OLS) is a method to quantify the evaluation of the different regression lines. According to OLS, we should choose the regression line that minimizes the sum of the squares of the differences between the observed dependent variable and the predicted dependent variable.
We can find a line that best fits the observed data according to the evaluation standard of OLS. A general format of the line is:
Here, μi is the residual term that is the part of yi that cannot be explained by xi. We can find this best regression line according to OLS requirement, but are we sure OLS generates the best estimator? One example is when there is an outlier, the ‘best’ regression line calculated according to OLS obviously does not fit the observed data well.
2.1 Gauss-Markov Assumptions for Cross-sectional Data
It turns out that only when certain assumptions are fulfilled, OLS calculates the best linear unbiased estimator (BLUE) that well estimates the population parameters. For cross-sectional data, Gauss-Marcov assumptions have six assumptions that ensure estimators calculated using OLS are BLUE. When any one of the Gauss-Marcov assumptions is violated, the sample parameters calculated using OLS no longer represent population parameters well.
Linearity in parameters. This assumption requires that parameter β is linear. However, there is no requirement for linearity in the independent variable. yi=α + βxi2 +μi and yi=α + βIn(xi) +μi both have linear β.The independent variable x and dependent variable y are both random variables. It is worth mentioning that if x and y are both random variables, the residual term μ will not be autocorrelated.No perfect collinearity between multiple independent variables x1 and x2. If there is perfect collinearity, linear regression results will be random, as it cannot differentiate the contribution of x1 and x2. Typically, when R2 result is good but t test for each independent variable is poor, it indicates collinearity.The residual term μ is endogenous. To be endogenous, μi does not change with xi. It can be expressed as cov(μi, xi)=0. Endogeneity may arise from reverse causality or measurement error in x, which causes cov(μi, xi)!=0.Homoscedasticity in residual term μi. It requires the variance of μi does not change with xi.No autocorrelation of the residual term μi. It can be expressed as cov(μi, μj)=0. Autocorrelation of μi can arise from omitted independent variable, mis-specified regression function, measurement error in the independent variables, and cluster errors.
Linearity in parameters. This assumption requires that parameter β is linear. However, there is no requirement for linearity in the independent variable. yi=α + βxi2 +μi and yi=α + βIn(xi) +μi both have linear β.
The independent variable x and dependent variable y are both random variables. It is worth mentioning that if x and y are both random variables, the residual term μ will not be autocorrelated.
No perfect collinearity between multiple independent variables x1 and x2. If there is perfect collinearity, linear regression results will be random, as it cannot differentiate the contribution of x1 and x2. Typically, when R2 result is good but t test for each independent variable is poor, it indicates collinearity.
The residual term μ is endogenous. To be endogenous, μi does not change with xi. It can be expressed as cov(μi, xi)=0. Endogeneity may arise from reverse causality or measurement error in x, which causes cov(μi, xi)!=0.
Homoscedasticity in residual term μi. It requires the variance of μi does not change with xi.
No autocorrelation of the residual term μi. It can be expressed as cov(μi, μj)=0. Autocorrelation of μi can arise from omitted independent variable, mis-specified regression function, measurement error in the independent variables, and cluster errors.
2.2 Gauss-Markov Assumptions for Time Series Data
Time series data is slightly different from the cross-sectional data. For cross-sectional data, we are getting samples from a population and Gauss-Markov assumptions require the independent variable x and dependent variable y are both random variables. For time series data, we are getting samples from the same process, and we can no longer assume that the independent variable x is random variable. Thus, Gauss-Markov assumptions are stricter for time series data in terms of endogeneity, homoscedasticity, and no autocorrelation. Since x is no longer a random variable, the requirement needs to be fulfilled for all xk at all time points instead of just xi at the time point as the residual term μi.
3.1 Linear Regression in Python
Here, we continue to use the historical AAPL_price and SPY_price obtained from Yahoo finance. We scatter plot AAPL_price against SPY_price first. Then, to find to what extent AAPL_price can be explained by the overall stock market price, we will build linear regression model with SPY_price as the independent variable x and AAPL_price as the dependent variable y.
Linear regression can be easily done with statsmodels library in Python.
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport statsmodels.api as smAAPL_price = pd.read_csv('AAPL.csv',usecols=['Date', 'Close'])SPY_price = pd.read_csv('SPY.csv',usecols=['Date', 'Close'])X = sm.add_constant(SPY_price['Close'])model = sm.OLS(AAPL_price['Close'],X)results = model.fit()plt.scatter(SPY_price['Close'],AAPL_price['Close'],alpha=0.3)y_predict = results.params[0] + results.params[1]*SPY_price['Close']plt.plot(SPY_price['Close'],y_predict, linewidth=3)plt.xlim(240,350)plt.ylim(100,350)plt.xlabel('SPY_price')plt.ylabel('AAPL_price')plt.title('OLS Regression')print(results.summary())
Together with the plot to visualize the OLS linear regression results, we can print a summary table, which looks like this:
Why are we doing these complex hypothesis testing? How can we interpret these hypothesis testing results? We will answer these questions in the following sessions.
3.2 Why Hypothesis Testing on Linear Regression?
Since we are using samples to estimate the population, we need to evaluate how well the population parameters are estimated by the sample parameters. To conduct hypothesis testing on sample parameters, we need to know the sample parameter distribution.
According to the central limit theorem, when the sample size is large enough, the sample distribution of β is normal distribution:
However, we do not know the exact population residual variance (σ2). We can use sample residual variance (σhat2) to estimate population residual variance, but this way sample β distribution is no longer a normal distribution. It becomes t distribution instead:
3.3 How To Interpret OLS Statistical Summary?
Now it is time to come back to the OLS Regression Results table and try to interpret the summary results.
The first session of the summary table has R2 and F-statistic, which measure the overall explainability of the independent variables over the dependent variable.
R2 is the explained sum of squared errors divided by the total sum of squared errors. R2 lies in between 0 and 1, and a larger R2 indicates the dependent variable is better explained by the independent variables. R2 = explained sum of squared errors/total sum of squared errors. With more independent variables, the resulting R2 will be closer to 1, but at the same time, the more independent variables may result in overfitting. Adjusted R2 prefers fewer independent variables by penalizing the excess independent variables.
F statistic tests against the joint effect of the independent variables. A low p-value of F statistic test indicates that the independent variables do not explain the dependent variable well.
The second session of the summary table is the t-statistic which tests against each independent variable. Using the F-statistic and t-statistic together helps to check whether there is collinearity in the independent variables. A good F-statistic and poor t-statistic indicates collinearity.
Durbin-Watson and Jarque-Bera reported in the third session of the summary table measures the stationarity and normality of the residual term, which will be discussed in detail in the following sessions.
The residual term is important. By checking whether the Gauss-Marcov assumptions are fulfilled using the residual term, we can infer the quality of the linear regression.
4.1 Normality test
It is important to test if the residuals are normally distributed. If the residuals are not normally distributed, the residuals should not be used for z test or any other test derived from normal distribution, such as t test, F test and chi2 test.
import pandas as pdimport statsmodels.api as smfrom scipy import statsAAPL_price = pd.read_csv('AAPL.csv',usecols=['Date', 'Close'])SPY_price = pd.read_csv('SPY.csv',usecols=['Date', 'Close'])X = sm.add_constant(SPY_price['Close'])model = sm.OLS(AAPL_price['Close'],X)results = model.fit()residual = AAPL_price['Close']-results.params[0] - results.params[1]*SPY_price['Close']print('p value of Jarque-Bera test is: ', stats.jarque_bera(residual)[1])print('p value of Shapiro-Wilk test is: ', stats.shapiro(residual)[1])print('p value of Kolmogorov-Smirnov test is: ', stats.kstest(residual, 'norm')[1])
Output:
p value of Jarque-Bera test is: 0.0p value of Shapiro-Wilk test is: 9.164991873555915e-20p value of Kolmogorov-Smirnov test is: 1.1324826980654097e-55
If we choose a significance level of 0.05, then all the three normality tests indicate the residual term does not follow normal distribution.
4.2 Homogeneity test
Three commonly used statistical testing for heteroscedasticity are Goldfeld-Quandt, Breusch-Pagan, White test. In the same sequence, more general homogeneity is tested.
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport statsmodels.api as smimport statsmodels.stats.api as smsAAPL_price = pd.read_csv('AAPL.csv',usecols=['Date', 'Close'])SPY_price = pd.read_csv('SPY.csv',usecols=['Date', 'Close'])X = sm.add_constant(SPY_price['Close'])model = sm.OLS(AAPL_price['Close'],X)results = model.fit()residual = AAPL_price['Close']-results.params[0] - results.params[1]*SPY_price['Close']print('p value of Goldfeld–Quandt test is: ', sms.het_goldfeldquandt(results.resid, results.model.exog)[1])print('p value of Breusch–Pagan test is: ', sms.het_breuschpagan(results.resid, results.model.exog)[1])print('p value of White test is: ', sms.het_white(results.resid, results.model.exog)[1])
Output is:
p value of Goldfeld–Quandt test is: 2.3805273535080445e-38p value of Breusch–Pagan test is: 2.599557770260936e-06p value of White test is: 1.0987132773425074e-22
If we choose a significance level of 0.05, then all the three normality tests indicate the residual term does not follow normal distribution.
4.3 Stationarity test
Durbin-Watson test detects autocorrelation of the residual term with lag of 1, while Breusch-Godfrey test detects autocorrelation of the residual term with lag of N, depending on the setting in the test.
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport statsmodels.api as smAAPL_price = pd.read_csv('AAPL.csv',usecols=['Date', 'Close'])SPY_price = pd.read_csv('SPY.csv',usecols=['Date', 'Close'])X = sm.add_constant(SPY_price['Close'])model = sm.OLS(AAPL_price['Close'],X)results = model.fit()import statsmodels.stats.api as smsprint('The Durbin-Watson statistic is: ', sms.durbin_watson(results.resid))print('p value of Breusch-Godfrey test is: ', sms.acorr_breusch_godfrey(results,nlags=1)[3])
Output:
The Durbin-Watson statistic is: 0.06916423461968918p value of Breusch-Godfrey test is: 4.646673126097712e-150
Both Durbin-Watson and Breusch-Godfrey tests indicate there is autocorrelation of the residual term with lag of 1. When Durbin-Watson statistic is 2, there is no autocorrelation. When Durbin-Watson statistic is towards 0, there is positive autocorrelation.
5.1 Violation of Gauss-Marcov Assumptions
When the Gauss-Marcov assumptions are violated, the estimators calculated from the samples are no longer BLUE. The following table shows how violation of Gauss-Marcov assumptions affects the linear regression quality.
5.2 Weighted Least Squares (WLS)
To account for heteroscedastic error, Weighted Least Squares (WLS) can be used. WLS transforms the independent variable and the dependent variable, so that OLS remains BLUE after the transformation.
5.3 Generalized Least Squares (GLS)
To account for both heteroscedastic error and serial correlated error, Generalized Least Squares (GLS) can be used. GLS transforms the independent variable and the dependent variable in a more complex way than WLS, so that OLS remains BLUE after the transformation.
In this post, we learnt that OLS generates good estimators only when Gauss-Marcov assumptions are fulfilled. Thus, after linear regression, it is always important to check the residual terms to ensure the Gauss-Marcov assumptions are not violated. Luckily, using the statsmodels library in Python, many statistical tests are automatically conducted during the linear regression. A simple print of the OLS linear regression summary table enables us to quickly evaluate the quality of the linear regression. If there is violation of the Guass-Marcov assumptions, further solutions of WLS and GLS are also available to transform the independent variable and dependent variable, so that OLS remains BLUE.
Hope you have enjoyed learning time series data modeling using linear regression!