How Good Is My Predictive Model — Regression Analysis
By: Farhad Malik
Tags: Machine Learning, Fintech, Regression Analysis, Mathematics, Data Science

Forecasting is an important concept in econometric and data science. It is also widely used in artificial intelligence and risk management by quantitative analysts, statistical modellers and technologists. Additionally, a large number of studies are carried out to understand and predict variables and their future movements.

Please read disclaimer before proceeding.

Predictive analysis is a black art. There are many right answers.

What will we cover in this article?

This article focuses on the concept of regression analysis. In particular, I have divided the article into 6 parts.

Explanation Of Regression Analysis 3 Steps Of Regression Analysis Understanding Of Residuals Ordinary Least Square Explanation Overview Of Model Criterion

Let’s Start

One of the most famous methodologies to forecast a variable’s behaviour is to use regression analysis. This technique requires formulating a mathematical equation/model that can be used to give us an estimated value which is as close as possible to the actual observed value.

Regression Analysis

Regression analysis process is primarily used to explain relationships between variables and help us build a predictive model. Moreover, it can explain how changes in one variable can be used to explain changes in other variables. Regression analysis could be linear or non-linear.

Regression analysis is all about projecting a dependent variable on a set of one or more predetermined independent variables.

Understanding regression analysis is important when we want to model relationships between variables. It can help us understand how close our calculations are to reality. Regression analysis is a simple yet powerful technique.

Regression Analysis Applications

A large number of applications exist that utilise regression analysis:

Algorithms used in supervised machine learning for fitting a smoothing function to target data points and during fitting a decision boundary between target data points. In finance, regression analysis is used by analysts to understand markets, in particular what traders need to buy/sell to replace high exposure stocks. Regression analysis process helps analysts perform better sensitivity analysis where different values of independent variable are chosen and applied into some form of calculation to see how the values impact dependent variable. For instance, hedge funds can use regression analysis to see changes in interest rates and how bond price will charge. Regression analysis can be used in businesses whereby managers can send employees on training courses and use the regression analysis to see how training impacts productivity.

3 Steps Of Regression Analysis

To help us understand regression analysis, I will explain the concept of regression analysis in 3 easy to understand steps. Furthermore, within each step, I will attempt to explain key terms too:

Step 1: Set a goal

Let’s assume our goal is to understand the relationship between interest rates and EUR/USD exchange rate and how strongly correlated these two variables are.

Our dependent variable is: Interest Rates as it is dependent on EUR/USD exchange rates. Dependent variable is known as regressand in statistics.

Our independent variable is: EUR/USD exchange rates. Independent variable is known as regressor in statistics.

Step 2: Plot a scatter plot

Aim is to understand how interest rates move by changes in EUR To USD exchange rates.

Start by collecting data over a period of time, let’s say daily rate changes for 1 year. This data is known as training data.

Plot a scatter plot chart, where Y axis represents interest rates and X axis represents EUR to USD exchange rates. The scatter plot indicates the relationship between regressand and the regressor(s).

From the data, calculate mean of interest rates and draw a line, denoted as Ỹ (Y hat)

Draw a best fit line between interest rates and EUR to USD exchange rates data points. This could be an iterative task as the best fit line needs to minimise the difference between the line and the actual point.

Step 3: Formulate An Equation To Predict Dependent Variable

From the line, do you see a relationship? May be linear positive or negative relationship or may be non at all?

The best fit line is known as conditional expectation line. From the best fit line, we get three key components:

Intercept — where line crosses y axis — Known as ɑ (ALPHA) Slope — Tell us expected change in y over unit change in x — β (BETA) Distance between actual data point and the best fit data point — ε (RESIDUAL)

SLOPE is also known as Beta

Mathematically this relationship is represented as linear formula:

y at time t = Alpha + Beta at time t * x at time t + residual at time t.

The formula above states that forecast of y is conditional on x and an error term; whereby an error term is the difference between our best fit line and the actual value. It is the random component which our formula cannot predict and it is also known as residual.

The formula above is building foundation for risk management. The concepts are indicating that when the error terms are small then we have a very good regression fit. As a result, variable x can be used to explain variable y.

Example — Interest Rates Vs EUR To USD Exchange Rates

The chart below illustrates best fit line for Interest Rates vs EUR To USD exchange rates:

Y axis = Interest Rates

X axis = EUR To USD Exchange Rates

Y mean = 4.5

Each blue dot = Actual Value of Y

Red Line = Best fit line

Distance between Point On Red Line and Actual Value Of Y = Residual (Also the unexplained difference)

The line above states:

Intercept = 3.7 (When EUR/USD = 0)

Slope = 1.1 (Change in interest rates over change in EUR/USD rates)

Therefore:

Forecasted Value of Interest Rate = 3.7 + 1.1 x (EUR To USD Exchange Rate) + residual

We can now use any EUR to USD exchange rate to find value of interest rate.

Understanding Of Residuals

Residuals are noise terms. They are also known as error terms. Residuals indicate deviation of Y values from each expected value.



Residual tells us how good our model is against the actual value. No one knows what each residual value represents in true population but all statistician know that it exists.

Calculating the real values of intercept, slope and residual terms is complex in nature. Ordinary Least Square can help us formulate a better quality model.

Ordinary Least Square Explained

Finding the best fit line is iterative and complex in nature. We can employ Ordinary Least Square (OLS) methodology and rules to find the best fit line. OLS estimates the population beta and alpha by minimizing the sum of square residuals.

What Is OLS?

OLS is a method for estimating the unknown parameters in a linear regression model. The most suitable regression models are based on Ordinary Least Squares (OLS) Estimation.

There are 6 common OLS assumptions:

Errors are independent of x, have a constant variance and their mean is 0. Errors are uncorrelated with each other. Errors have a normal distribution. Large outliers are not observed in the data Relationship is linear between y and x(s) Model has not omitted any appropriate independent variable.

The best fit line is the one that minimises sum of squared differences between actual and estimated results. Taking average of minimum sum of squared difference is known as Mean Squared Error (MSE).

Smaller the value, better the regression model.

Pitfalls of Breaking OLS Assumptions:

Regression analysis model is based on strict restrictions of OLS. If we break these restrictions (which can happen in real stochastic nature of variables), we can then create downward biased coefficients.



In this part of the article, I will explain pitfalls of linear regression:

Specification error: This happens if we omit a variable that the true model is dependent on and if the omitted variable is correlated with the included variables. It can lead to biased estimated coefficients, which leads to estimation errors. Omitted variable bias is when omitted variable is correlated with the included regressor and is a determinant of the dependent variable. Autocorrelation: OLS assumes that there is no systematic correlation in the residuals. This can lead to biases in the standard error which can cause problems with inference. Efficiency Error: When our estimation ignores all available information. Heteroskedasticity: When residuals have changing variance across observations. In that case, it is best to check residuals, perform diagnostics and construct standard errors or may be provide an alternative specification. Multicollinearity: When regression is subject to multicollinearity then it means that the x variables are highly correlated e.g. using two currencies that are fixed to each other and it can lead to unreliable and unstable Beta. The solution is to remove the variables that are highly correlated. Calculated covariance of the regressors (x), if it is high than the regressors are highly correlated.

The pitfalls can lead to inconsistent, biased and inaccurate results.

How Do We Calculate Mean Squared Error

As the name implies, Mean Squared Error (MSE) is the mean of the residuals (error) squared. To calculate MSE, we need to calculate:

Explained Sum Of Squares (ESS) Sum Of Squared Residuals (SSR) Total Sum of Squares, (TSS)

Explained Sum Of Squares (ESS)

ESS is Explained or Estimated Sum Of Squares

Squaring(Y Estimated — Mean value of Y) and then sum all of the values.

Sum Of Squared Residuals (SSR)

SSR is sum of squared residuals. It is calculated by:

Squaring(Y Estimated — Actual Y Value) And then sum all values.

Total Sum of Squares (TSS)

TSS is simply total sum of all squares. It is calculated as ESS+SSR. This means total sum of squares is sum all squares of (y estimated — residual) + sum of squares of (y estimated — mean y)

Let’s understand the points above in greater detail:

First we need to subtract fitted values of y (y hat) from actual values of y. The difference is known as residual. Some of the values will be positive and the other values will be negative. Therefore square the residuals and then sum them. This is known as sum of squared residuals: SSR From the points of y, we can calculate the mean of y. Calculate the difference between value of best fit line and mean of Y. Square the differences and then sum the values. This final value is known as sum of squared deviations of y from its mean: ESS Add them both and you have explained and unexplained added to give you total sum of squares (TSS).

TSS = ESS + SSR

This is how we can forecast population y from estimated values of alpha, beta and residual.

Let’s measure quality of our regression model: R Squared

Once we know the size of residuals, we can start assessing how good our regression fit is. Regression fitness can be measured by R squared and adjusted R squared.



R-Squared: goodness of fit



Measures explained variation over total variation. Additionally, R squared is also known as coefficient of determination and it measures quality of fit.



Formula to calculate R squared is:

R squared = 1 — (SSR/TSS)

This means R squared = 1 — Variance (residual)/Variance (y)

Higher R squared, the better.

Let’s understand R squared with an example

Let’s say you want to hedge a future investment. You could choose a number of assets to find the asset that is correlated with your portfolio. Choose the asset with highest R squared because the asset that is most closely correlated with the asset you wish to hedge will have the least basis risk.



R squared measures the degree to which size of the errors is smaller than that of the original dependent variable of y.

Values Of R Squared

R squared of 0 means our best fit line is poor.

R squared value of 1 means our regression analysis is excellent and we could explain the relationship accurately.

Let’s understand that a bit:

If our best fit line to explain relationship between interest rates and EUR/USD exchange rate is poor then R square will be very small (close to 0). It will be small because SSR will be very large. As a result, R squared will be 0.

If the model is good then SSR will be 0 and R squared will be 1.

Correlation Coefficient: Square root of R square

Correlation between two variables can be calculated by taking square root of R.

Standard Error Of Regression (SER): SER measures variability of actual and estimated values of Y. It is the standard deviation of the residuals. It is calculated as

[Standard deviation of sample/Sqt(N)]

Now from R square, check if OLS conditions are met:

Is variance of residual constant (plot R squared).

Are residuals normally distributed?

R Squared Cons

R squared Has Its Own Problems:

Problem 1: R squared increases as you add more predictors to your model. For example, if you start adding more variables to predict a model then it will likely improve your model’s predictability and will start producing values close to actual values.

R squared increases as you add more predictors to your model. For example, if you start adding more variables to predict a model then it will likely improve your model’s predictability and will start producing values close to actual values. Problem 2: Too many predictors mean more random noise and can produce misleading R squared.

As, adding more independent variables, k, can increase the value of R, let’s introduce a better measure: Adjusted R Squared

Let’s measure quality of our regression model: Adjusted R Squared

R squared by itself is not good enough as it doesn’t take into account the number of variables that gave us the degree of determination. As a result, adjusted R squared is calculated. Let’s go over adjusted R squared metric.

Adjusted R Squared =

[ SSR / (n — K) ] / [ SST (n — 1) ]

Or:

Adjusted R Squared:

1 — [ [(n-1)/(n-k-1] x [1 — R squared]]

n = number of observations,

k = number of independent variables

Degree of freedom in our model is n— 1, where n is the size of the sample.

Adjusted R square is always lower than the R-squared.

Tip: You must assess residual plots, as R-squared alone cannot determine whether the coefficient estimates and predictions are biased.

Adjusted R-squared only improves if the new predictor improves the model. It compares the explanatory power of regression models that contain different numbers of predictors. It is adjusted for the number of predictors in the model.

Overview Of Model Criterion

Models require parameters. Parameters for a model are selected so that they can maximize the likelihood of a value occurring in a sample.

To accomplish higher probability of a value occurring in a sample, first distribution of the variable is calculated and then parameters are searched that maximize likelihood until the likelihood function fails to increase.

We can get higher R squared by adding variables in the forecasting model. This adds the risk of over-fitting the model. For example, assume we want to calculate MSE for a predictive model. Adding more factors (independent variables) can end up giving us better fit.

A number of model selection criterion are implemented to penalise the models that include higher number of independent variables.

These includes Unbiased Mean squared error (MSE) known as s-squared, Akaike (AIC) and Schwarz (SIC) Criterion.

SIC has the highest penalty factor and it increases at exponential rate with increasing more variables. Therefore SIC is the most consistent selection criterion. These criterion are not covered in this article but it’s worth mentioning that these criterion adjust R squared by taking number of independent variables into account. As a result, addition of more variables does not always increase R squared value.

Note: Higher R squared value implies model is fitted better and our forecasting is accurate.

Once residuals (actual value — predicted value) are computed, they can be plotted against time to determine if the model is serially correlated.

Finally

This article highlighted the concept of regression analysis that is required to build a forecasting model. It also explained how to test goodness of fit of a model once it is implemented.

Hope it helps. Please let me know of your feedback.