Macroeconomic predictions.. Can we create an AI model to predict…
By: Uki D. Lucas
Tags: Machine Learning, Julialang, AI, Macroeconomics, Stock Market

Can we create AI models to predict macro-economic trends?

Not too long ago, the prediction of the weather was a hit or a miss. The joke was that the TV weatherman was always carrying an umbrella on what was predicted to be supposedly a nice day. Today, we can model the weather patterns very accurately for several days into the future.

Similarly, given enough relevant data, including market indicators, human behavioral patterns, and current events, including the “black swan” events, we should be able to model the macro-economics.

In the words of someone very famous, the idea is to be “less wrong”. Being consistently better than an average, even by a tiny percent in predicting the future would have a big benefit.

For years, I have been discussing with my friends the concept of AI modeling based on thousands of macroeconomic data streams, and the opportunity it could give to the architect of such a system. Now, I feel that the data, the software and the hardware is within the reach of that goal.

It is very common to hear the business or the political pundits speak of what amounts to mono-thematic bias based on the very limited ability of a human mind to grasp only a few dimensions of a given topic. Here, I am trying to have AI combine thousands of complex data streams (think of each as a different spreadsheet) into a consistent and verifiable oracle.

At the start, I employed easily accessible market indications, data that influences the immediate macroeconomic changes such as manufacturing, unemployment, new housing permits, inventories, etc.

I made heavy use of averages as I realize that certain events build up to a change over periods of time, sometimes days, sometimes weeks, months, or even a year.

Of course, I realize that historical data is not enough.

I am interested in human behavior as well. Pundit and influencer opinions, even if wrong, definitely affect human behavior. We know for a fact that markets, or rather large groups of people, behave illogically. However, there are patterns in there in how an average Joe listens to the presidential speech, watches few pundits argue, reads an article or two, talks to friends at work, watches few videos titled “Imminent crash in [insert any year], hears that someone got laid off, and finally starts selling their stocks a few days later making a colossal investment mistake.

Such modeling is difficult, but it is achievable. We are already regularly deducting a human sentiment of Google searches, YouTube videos on the newest Apple or Tesla innovations, Twitter blasts, and blog posts. It is all known and proven technology.

This project is not meant to have a quick win, I realize that every hedge fund is pouring millions in similar research. Rather, I want it to be a gradual improvement learning process, a labor of love, a means of recurrent income towards my retirement, I still have about 20 years to solve this complex problem.

The plan is that every few days I add new data sources and continue building the models until I get good at predicting something. Yes, I have used the word “something” on purpose. I am not sure if I will be able to predict the particular price of the S&P 500, but I am sure I will find patterns in some niche, and having a wizard-level insight might open doors, a path less taken.

This article, and the accompanying open-source code, is not meant to teach you about the market, nor to teach you how to code, but rather to share my progress and to receive positive feedback and cooperation.

Project Objectives

to gather and analyze as many market indicators as possible

to learn patterns and interactions in the data

to predict market trends 5, 30, or 90 days ahead

to predict a particular stock price

to produce a re-usable code and document the process

What are Market Indicators?

Market Indicators are collections of data points (think spreadsheets) reflecting the historical performance of a particular area of interest, for example:

“S&P 500” index shows how major 500 stocks as a whole are performing

shows how major 500 stocks as a whole are performing “ISM Manufacturing” index shows how well the manufacturing industry is doing

shows how well the manufacturing industry is doing “GDP” (Gross Domestic Product) index shows how the country is doing

shows how the country is doing etc., etc., etc.

There are hundreds, if not thousands, of such indicators. Each country, state, county, and community have sets of data that reflect some trend.

Why not using the spreadsheets?

The spreadsheets and their graphing capabilities are the bread and butter of an individual person’s market analysis. However, spreadsheets are good when comparing only a few indicators, when the data starts multiplying the inputs are overwhelming. Please remember that an average person can remember about 7 numbers, now try to imagine daily data for the last few decades coming for a few hundred indications, multiplied by a couple of hundred or regions, and add to it sentiments from hundreds of sources. We can easily talk about millions, if not trillions of inputs.

Today’s desktop computers can process many “tera” operations per second. Tera is this much = 1,000,000,000,000. I think the point is clear.

Machine Learning approach

The overwhelming advantage of machine learning, or Artificial Intelligence (AI), is apparent when we try to find subtle patterns in thousands of indicators.

The human brain (using spreadsheets) fails to grasp the wealth of the information presented. Machine learning, on the other hand, can easily detect the patterns in massive datasets and derive a conclusion.

Why Julia?

Julia is modern, fast, elegant, multitasking, and does the math extremely well, of course, we are talking about Julia programming language

unlike C/C++, it is a pleasure to read and write Julia

it is designed in MIT specifically for scientific computing and machine learning

similarly to C, it is extremely fast

similar to Python, it is very easy to learn

it is designed for parallelism

it is designed for distributed computing (in case ~10,000,000,000,000 operations per second is not enough)

Read the source, Luke

I am making changes to the project’s code on a daily basis.

The approach and the code would surely diverge from any examples I would be able to provide here, hence the pun on Obi-Wan Kenobi’s “Remember, Luke, use the source.”, in other words, study the actual code.

The rest of the project is documented at:

Gain Access to Expert View — Subscribe to DDI Intel