Exchange Rate Prediction: Machine Learning with 5 Regression Models | by Andrew Nguyen | Towards Data Science

If you have missed my 1st part on this end-to-end project about Exchange Rate Prediction, feel free to check it out here.
As a recap, this project aims to analyze the historical pattern of exchange rates across different country currencies against USD and therefore to forecast the values in 2020. In order to bring out the best model, I have divided the project into 3 parts which, I hope, will possibly cover what is needed for the prediction:
Part 1: Explanatory Data Analysis (EDA) & Data Visualisation (Bonus: Hypothesis Testing)Part 2: Machine Learning with 4 Regression ModelsPart 3: Machine Learning (cont.) with ARIMA
Part 1: Explanatory Data Analysis (EDA) & Data Visualisation (Bonus: Hypothesis Testing)
Part 2: Machine Learning with 4 Regression Models
Part 3: Machine Learning (cont.) with ARIMA
Today, I will bring you through the 2nd part which deploys Machine Learning with the aim of finding the line that best fits the pattern of exchange rates over the years. That being said, Linear Regression would seemingly be the right model or, at least, the foundation for our analysis. Beyond it, I have also tested out other linear regression models before deciding which to use for the forecasting (part 3):
Linear Regression
Multiple Linear Regression
Polynomial Regression
Ridge Regression (L2 Regularization)
Lasso Regression (L1 Regularization)
Let’s get started!
According to Investopedia, there are 3 common ways to forecast exchange rates: Purchasing Power Parity (PPP), Relative Economic Strength, and Econometric Model. Among the three, I decided to go with the third as it involves a multitude of factors that affect currency movements.
www.investopedia.com
“The factors used in econometric models are typically based on economic theory, but any variable can be added if it is believed to significantly influence the exchange rate.” — Investopedia, by Joseph Nguyen.
This method greatly aligns with the direction that employs “any independent variable” that is influential on the dependent variable — linear regression. Therefore, the factors I wanted to test out are interest rate differential, GDP growth rates, and income growth rate, sequentially yet accumulatively. For this project, I used the exchange rate of AUD/USD for the analysis.
Let’s kick off our Linear Regression model with one independent variable: Interest Rate Differential. If you are keen to explore, please refer to this data source.
ir_df = pd.read_csv("aud usd interest carry trade.csv")ir_df.head()
Above is the table of interest rate differential of AUD/USD from 2017 to 2019. If you want to know more what “long carry” and “short carry” are, check out this article on Investopedia about Carry Trade and another on FXCM about Interest-Rate Carry Trade. In short,
Interest-rate carry trades are a form of arbitrage, in which someone makes use of the difference that exists between two markets in order to turn a profit.
As such, let’s transform this data for the analysis (in this case, I used Long Carry Interest Rate):
x_ir = ir_df['Long Carry'].astype(str)x_ir = x_ir.replace({'%':''}, regex = True)x_ir = x_ir.astype(float)x_ir = np.array(x_ir).reshape(-1,1)aud_usd_fx = df_groupby_aud[(df_groupby_aud['month_year'] >= '2017-01') & (df_groupby_aud['month_year'] <='2019-12')].reset_index(drop=True)aud_usd = aud_usd_fx['AUD_USD']y_fx = aud_usd
Here, x_ir refers to the independent variable that affects our exchange rate. For easier use, I removed the “%” from the numbers with regex replace, and converted them to float. As 2D array data is required in the regression model, we need to convert our 1D x_ir to 2D by using .reshape(-1, 1). And if you recall from the 1st part, we have cleaned up the entire dataset, so it’s now easier for us to pick up the AUD/USD data (y_fx) as the dependent variable.
Please note that I only used a short period of time (from 2017 to 2019, 36 months) in this project for the sake of simplicity.
There you go, let’s run our 1st machine learning model by importing relevant functions from Scikit-learn library!
from sklearn.model_selection import train_test_splitfrom sklearn.linear_model import LinearRegression
As you might have known, in order to test the accuracy of our model, we are advised to split the dataset into training and testing sets. Essentially, we train the model on the training set and then predict the values with the testing set.
x_train, x_test, y_train, y_test = train_test_split(x_ir, y_fx, train_size=0.8, test_size=0.2, random_state=1)model = LinearRegression()model.fit(x_train, y_train)y_predict = model.predict(x_test)print(model.score(x_train, y_train))print(model.score(x_test, y_test))
Here, I split the data into 80% training and 20% test, meaning that the model will learn the pattern by 80% training data, and then predict the y values with 20% testing data. .score() tells us the percentage of a dependent variable (exchange rate) being explained by the independent variable (interest rate differential) or as we know, R-squared.
As you can see, the percentage between training and testing data is pretty much close, meaning that the accuracy of the model is relatively good.
However, from my observation, the size of the dataset is relatively small (n=36 as we only cover 2017 to 2019), I wanted to see whether I’m able to NOT split the data into training and testing sets or not, for the sake of simplicity.
model = LinearRegression()model.fit(x_ir, y_fx)y_fx_predict = model.predict(x_ir)print(model.score(x_ir, y_fx))
The difference in the R-squared between split and no-split is minimal, probably due to the small dataset size. Hence, I will NOT split the data and use the entire dataset.
Let’s see how well our model does in creating the “best-fit” line:
month_year = aud_usd_fx['month_year']month_year = month_year.astype(str)plt.figure(figsize=(12,6))plt.scatter(month_year, y_fx, alpha=0.4)plt.plot(month_year, y_fx_predict)plt.title("Linear Regression: AUD/USD Exchange Rate (1 var: Interest Rate)")plt.xlabel("Month-Year")plt.ylabel("Exchange Rate")plt.xticks(fontsize=4)plt.show()
Combined with the R-squared, the line seems to be able to portray the exchange rate to some extent (R-squared is 72%). Let’s add another variable to see how far our R-squared improves, with Multiple Linear Regression.
As shared from the beginning of the article, we are able to add additional variables to the model as long as they are influential on the dependent variable. This leads us to the second Linear Regression model, which includes more than 1 independent variable.
Let’s add GDP Growth Rate to our dataset, which can be found from this domain.
aus_gdp = pd.read_csv("AUS GDP.csv")usa_gdp = pd.read_csv("USA GDP.csv")aus_gdp = aus_gdp.rename(columns={'GDP': 'AUS_GDP'})aus_usa_gdp = pd.merge(aus_gdp, usa_gdp, on="month_year", how="inner")aus_usa_gdp = aus_usa_gdp.rename(columns={'GDP': 'USA_GDP'})aus_usa_gdp['GDP_diff'] = aus_usa_gdp['AUS_GDP'] - aus_usa_gdp['USA_GDP']aus_usa_gdp_20172019 = aus_usa_gdp[(aus_usa_gdp['month_year'] >= '2017-01') & (aus_usa_gdp['month_year'] <='2019-12')].reset_index(drop=True)gdp_diff = ["%.4f" %num for num in aus_usa_gdp_20172019['GDP_diff']]
After loading the data, I applied EDA to clean up and extracted gdp_diff from the dataframe to a list, so as to combine with x_ir variable for our Multiple Linear Regression.
x_ir_gdp = np.array(list(zip(x_ir, gdp_diff)))x_ir_gdp = x_ir_gdp.astype(np.float)model.fit(x_ir_gdp, y_fx)y_fx_predict_2 = model.predict(x_ir_gdp)print(model.score(x_ir_gdp, y_fx))
After adding another variable to the model, R-squared has improved quite significantly, from 72% to 87.2%. Let’s compare it visually against the first line we created:
Looks good, right! Okay, let’s try adding another independent variable to our model, Consumer Price Index (CPI). Again, the dataset can be found from this domain.
cpi = pd.read_csv("aus usa cpi difference.csv")cpi_diff = cpi['CPI_diff']x_ir_gdp_cpi = np.array(list(zip(x_ir, gdp_diff, cpi_diff)))x_ir_gdp_cpi = x_ir_gdp_cpi.astype(np.float)model.fit(x_ir_gdp_cpi, y_fx)y_fx_predict_4 = model.predict(x_ir_gdp_cpi)print(model.score(x_ir_gdp_cpi, y_fx))
This time, R-squared increased really slightly, from 87.2% to 87.3%. This means CPI does not significantly affect the movement of exchange rate nor help improve our model. Visually, there’s no difference either.
Let’s replace CPI variable with another, Unemployment Rate (UER) and see whether this improves our model or not.
unemployment_rate = pd.read_csv("aus usa unemployment rate.csv")unemployment_rate['uer_diff'] = unemployment_rate['aus_unemployment rate'] - unemployment_rate['usa_unemployment rate']uer_diff_all = unemployment_rate['uer_diff']uer_diff = ["%.4f" % num for num in uer_diff_all]x_ir_gdp_uer = np.array(list(zip(x_ir, gdp_diff, uer_diff)))x_ir_gdp_uer = x_ir_gdp_uer.astype(np.float)model.fit(x_ir_gdp_uer, y_fx)y_fx_predict_3 = model.predict(x_ir_gdp_uer)print(model.score(x_ir_gdp_uer, y_fx))
Okay, R-squared has improved a bit better, from 87.2% to 88.9%. Not too bad, statistically and visually. You can continue to add some more variables to the model, but be careful with overfitting!
For me, instead of adding more independent variables, I decided to test out another regression model to find a better-fit line: Polynomial Regression.
What is Polynomial Regression? It’s still one kind of Linear Regression but a much more “adaptable” version.
Traditionally, Linear Regression aims to find the best linear line that fits all your data points, which might not always be the case. Therefore, Polynomial Regression acts as another form that draws the line from coordinates (x, y) with y modeled as the nth degree polynomial.
Let’s import the function from Sciki-learn library and build our model:
from sklearn.preprocessing import PolynomialFeaturespoly = PolynomialFeatures(degree=4)x_poly = poly.fit_transform(x_ir_gdp_uer)model_poly = LinearRegression()model_poly.fit(x_poly, y_fx)y_pred = model_poly.predict(x_poly)print(model_poly.score(x_poly, y_fx))
The only difference in the function between Linear and Polynomial Regression is the degree parameter. Before getting to the result, let me briefly explain: .fit_transform() is to convert your independent variables from linearly to polynomially. The rest of the function stays the same.
Here, I randomly selected the degree of 4 and R-squared is nearly perfect: 99.9%! You may want to see it visually?
The black line represents the fit that Polynomial Regression found with the degree of 4. How amazing? But the question is: “Which degree is the best, to neither overfit nor underfit the data?”
We know that we aim to find R-squared as high as possible, as it well explains the fact that the dependent variable (exchange rate) is affected by the independent variable(s). Apart from it, there is another metric that is able to evaluate the accuracy of our model: Mean Squared Error (MSE).
To find the best degree, I decided to test a range of degrees, from 1 to 10 on our model separately with a for loop and visualize it:
from sklearn.metrics import mean_squared_errorr_squared_list = []mse_list = []a = range(1,10,1)for i in a:    poly = PolynomialFeatures(degree=i)    x_poly = poly.fit_transform(x_ir_gdp_uer)        model_poly = LinearRegression()    model_poly.fit(x_poly, y_fx)    y_pred = model_poly.predict(x_poly)        r_squared = model_poly.score(x_poly, y_fx)    mse = mean_squared_error(y_pred, y_fx)        r_squared_list.append(r_squared)    mse_list.append(mse)
As you can see in both charts, as the line increases from degrees of 1 to 4, the R-Squared and MSE improve significantly (the higher the better for R-Squared, the lower the better for MSE). However, it peaks at degree of 4 and no further improvement is seen since the degree of 5. This means that degree of 4 brings out the best accuracy score for our model! So let’s stick with 4 then!
Yay! We finally got the line that best fits the exchange rate over the years, thanks to Polynomial Regression!
These final two are my latest experiment in the “ecosystem” of linear regression models as I have heard about somewhere before. Oh yes, it regards L1 & L2 Regularisation! As such, I wanted to test out these two models in my project as well to see how they perform against the above models.
However, before jumping into them, let’s quickly point out the origin of these models, Regularisation, and the roles they play against what we might commonly know about Linear Regression.
It all started with Overfitting. In a nutshell, overfitting occurs when a machine learning model is too tailored to a certain dataset (especially with too many independent variables/features) and hence unable to generalise to other datasets. Hence, regularisation was introduced to prevent this phenomenon. Essentially, what regularisation does is:
Reduce the complexity of the model while maintaining the number of independent variables/features
Technically, reduce the magnitude of the coefficients as penalty term to the loss function
If you want to deep dive into how it works, mathematically, check out this article on Towards Data Science for more details.
Ridge Regression and Lasso Regression were developed from this concept of Regularisation. The only difference between the two is:
Ridge Regression uses L2 Regularisation technique, which shrinks the coefficients to zero, but NOT absolute zero
Lasso Regression uses L1 Regularisation technique, which shrinks the coefficients to absolute zero
This is because Ridge keeps all variables/features while Lasso only keeps the important ones that matter, hence voted as the variable/feature selection technique when our dataset has TOO MANY variables/features.
Okay, back to our dataset!
Even though our model does not have too many variables (in this case, 3), I still wanted to see how Ridge and Lasso Regression perform on my dataset.
from sklearn.linear_model import Ridge, Lassoridge = Ridge(alpha=0.001)ridge.fit(x_ir_gdp_uer, y_fx)y_fx_ridge = ridge.predict(x_ir_gdp_uer)print(ridge.score(x_ir_gdp_uer, y_fx))lasso = Lasso(alpha=0.001)lasso.fit(x_ir_gdp_uer, y_fx)y_fx_lasso = lasso.predict(x_ir_gdp_uer)print(lasso.score(x_ir_gdp_uer, y_fx))
R-Squared of Ridge is 88% while of Lasso is 87%. Not too bad when we compare these numbers with our Multiple Linear Regression (88.9%). This proves that overfitting does not occur to our model!
WOW! That’s a wrap for the 2nd part of this project! Too much to absorb, isn’t it? But I do hope you have found this article informative and actionable.
In short, we have found the line that best fits our dataset, greatly explaining the pattern of exchange rates over the years! What’s next?
Again, do look out for the final part of my project which will cover how I forecasted the exchange rates of 2020 using the regression line we have just discovered. In the meantime, give me a clap if you find this helpful and feel free to check out my Github here for the complete repository:
Github: https://github.com/andrewnguyen07LinkedIn: www.linkedin.com/in/andrewnguyen07