Time-Series Data Analysis & Machine Learning Algorithm for Stock Trading
By: Sarit Maitra
Tags: Data Analysis, Machine Learning, Feature Engineering, Algorithmic Trading

DATA WRANGLING & PIPELINE FOR DOWNSTREAM ANALYTICS

Time-Series Data Analysis & Machine Learning Algorithm for Stock Trading

A case study with technical analysis, feature selection, accuracy score & bias-variance trade-off

Image by Sarit Maitra

A lot of interesting work has been conducted in the area of applied ML Algorithms to analyze price patterns and predict stock prices. Most stock traders nowadays use Smart Trading Systems to predict prices based on various situations and conditions. An intelligent trader would predict the stock price and buy a stock before the price rises, or sell it before its value declines. Though it is quite hard to replace the expertise of an experienced trader, however, an investment in a machine learning based prediction algorithm is directly result into high profits.

Moreover, stock market is a financially volatile market and it is important to have a very precise prediction of a future trend. Because of the financial crisis and scoring profits, it is necessary to have a secure prediction of the stock prices. Predicting a non-linear signal requires advanced ML algorithms. Here. we shall work with classification ML algorithms with the help of feature selection to examine the success rate of making a profit.

We have focused on technical analysis here in this article. However, ML can also play a major role in evaluating and forecasting the performance of the company and other similar parameters helpful in fundamental analysis. In fact, many successful automated stock prediction and recommendation systems use some sort of a hybrid analysis model combining both fundamental and technical Analysis.

This article is written in 2 parts, where Part-1 is this part which covers data collection, data wrangling, feature creation and important feature identification which involves multi-collinearity and variance inflation factor. Part -2 covers different ML algorithms, hyper-parameter optimization, accuracy metrics, confusion metrics and bias-variance trade-off.

Let us pull the data and inspect the available features.

Data comprise of Brent Crude Oil continuous futures since 2000 with 11 variables. Individual futures contracts trade for very short periods of time, and are hence unsuitable for long-horizon analysis. Continuous futures contracts solve this problem by chaining together a series of individual futures contracts, to provide a long-term price history that is suitable for trading, behavioral and strategy analysis.

We have here few variables with missing values and are not needed for our future analysis. So, let us drop these variables form the data frame. The other missing values in the data are interpolated using forward-fill (‘ffill ()’)method to propagate last valid observation forward.

The data frame has the index set to dates and the columns are:

Open: The price of the first trade on the given trading day.

High: The highest price at which a stock traded for the given trading day.

Low: The lowest price at which a stock traded for the given trading day.

Settle: This is the the closing price after adjustments and popularly known as Adjusted Closing price. Adjusted prices are adjusted for stock dividends, cash dividends and splits, which creates a more accurate return calculation.

Volume: The number of shares traded for the given trading day.

Daily return

Adjusted closing price which is Settle here, gives us all the information we need to keep an eye on the stock. We can use un-adjusted closing prices to calculate returns, but adjusted closing prices save us some time and effort. The mathematics behind daily returns calculations works like this:

For example, the 5 May 2020 price was $30.97 and the 6 May price was $29.72, the daily return is [(29.72/30.97)-1] * 100 = -(4.03)%.

We have used simple python function to derive at average daily return and volatility for all the rows.

However, there is a catch in daily return calculation:

As an example, if we made 25% one day and lost 20% the next, our geometric average daily return is (1.25*0.8)^ 0.5–1 = 0. Therefore, 0% mean daily return which is arithmetic return and which does not always mean that stock is not making money.

Volatility

Traditionally, risk is always calculated as volatility. Daily volatility is computed by finding out the square root of the variance or std deviation of daily stock price. Volatility not only measures risk, but affects the expectation of long-term (multi-period) returns. Traditional risk frameworks that rely on std deviation generally assume that returns conform to a normal bell-shaped distribution which shows that about 2/3rd of the time (68.3%), returns should fall within one standard deviation (+/-); and 95% of the time, returns should fall within two standard deviations.

Volatility normally erodes returns. We are aware of random walk theory that, std deviation increases in proportion to the square root of time. Considering that let us understand the arithmetic behind this:

we start with $100 and then gain 10% to get $110.

Then we lose 10%, which takes us to $99 ($110 x 90% = $99).

Then we gain 10% again, to net $108.90 ($99 x 110% = $108.9).

Finally, we lose 10% to net $98.01.

It may be counter-intuitive, but our principal is slowly eroding even though our average gain is 0%.

Probability of +/-(1%); +/-(3%); +/-%(5) change in price

We wanted to check the probability of change in price which also gives an indication of high volatility. Here we see that over 96% probability that price may change between +/- 5%.

Data wrangling

Let us add few external data-set to examine if we can find any correlation with Brent Crude oil. This has been done for illustration purpose.

In real-life business analysis, we may need to explore relevant variables which might have an impact on crude oil pricing e.g. weather, US economy, international economy, US dollar exchange rate, geopolitical events, supply and demand statistics etc. and there could be many more.

We have added-

USA real gross domestic product USA real disposable personal income Civilian unemployment rate.

Resampling

These time series data are available on monthly frequency format. We need to summarize or aggregate time series data by a new time period. Therefore, we need to up-sample time series data to a higher frequency and interpolate the new observations. The below database queries are quite similar to SQL queries. Here, I have chained together with a ffill() and then bfill() the remaining NaN values.

Correlation

Correlation Matrix is basically a covariance matrix. It is a matrix in which n-m position defines the correlation between the nth and mth parameter of the given data-set.

It is quite clear from the above values (<0.5) that, none of the additional variables are correlated with actual price column (‘Settle’). So, we drop the idea of adding these variables and will focus of adding different features that we can create from original price & date columns. However, point to be noted here that, correlation can only directly measure linear relationships between its inputs.

Let’s define our problem statement.

Problem statement

As we are aware that, the more of a share is transacted, more it is valuable; and on the other hand, if a share is put into transaction in a low volume, it is not so important for some traders and by default, it’s value decreases. Therefore, depending on the power to predict future values, the above anticipation of the market can generate profit or losses. A list of algorithms are available to develop a model; however, due to non-linear nature of the stock market signals, some methods have yet to give promising results; others have not reacted as well on the stock market exchange.

Technical Analysis

When applying Machine Learning to Stock Data, we are more interested in doing a Technical Analysis to see if our algorithm can accurately learn the underlying patterns in the stock time series.

Feature engineering

We create an ‘outcome’ binary variable, 1 if the trading session was positive (‘Settle > Open’), or else 0. Intuitively, based on the efficient market hypothesis, the price of the stock yesterday is going to have the most impact on the price of the stock today. Thus as we go along the time-line, data-points which are nearer to today’s price point are going to have a greater impact on today’s price.

We examine the number of 0,1 values to check if our dateset is balanced. as we can see that our binary numbers are almost evenly distributed.

Moving Average (MA)

The average of the past n values till today.

Rate of Change (ROC)

The ratio of the current price to the price n quotes earlier. n is generally 5 to 10 days.

Relative Strength Index (RSI)

Measures the relative size of recent upward trends against the size of downward trends within the specified time interval (usually 9–14 days).

Relative strength index (RSI) = 100–100 / (1 + RS), where RS = (Average gain /Average loss) over the last 14 days

Williams %R

It is a type of momentum indicator that moves between 0 and -100 and measures overbought and oversold levels. The Williams %R used to find entry and exit points in the market.

Ease of Movement (EVM)

It is a volume-based oscillator which indicates the ease with which the prices rise or fall taking into account the volume of the security.

We write the below function to clean the data set of nan, Inf, and missing cells (for skewed datasets).

Feature selection as part of a pipeline

Feature selection is usually used as a pre-processing step before doing the actual learning.

Multi-collinearity

Multi-collinearity is a state of very high inter-correlations among the independent variables. It is therefore a type of disturbance in the data, and if present in the data, the statistical inferences made about the data may not be reliable.

Variance inflation factor (VIF)

VIF measures the impact of multi-collinearity among the X’s in a ML model on the precision of estimation. It expresses the degree to which multi-collinearity among the predictors degrades the precision of an estimate. VIF is computed as (1/(1-R2 )) for each of the k -1 independent variable equations.

Here we see that, out of all the variables (26) used, only 11 are enough to explain the variance in the model. No classifier can compensate for the use of irrelevant or inaccurately measured features. The only way to improve performance is to find better features. This is the reason it is important to choose mathematically correct features that are also separable.

Using variance to reduce feature could be troublesome because we don’t have any absolute scale on which to pick a threshold for a good variance. If we want to be more objective, a grid-search mechanisms is advisable to choose a good variance threshold.

We will test different machine learning algorithms to determine the best model.

Conclusion

Here, we have done experiment with Brent Crude Oil Continuous Futures daily data. The data set was from year 2000 to till data. We have performed necessary data cleaning and data wrangling by transforming and mapping data into a format with the intent of making it more appropriate and valuable for downstream analytics purposes. we have done technical analysis and feature engineering to see if our algorithm can accurately learn the underlying patterns in the stock time series. Necessary cleaning was conducted to perform collinearity statistics. Collinearity statistics helped us to determine the inter-correlations among the independent variables and reduce the number of relevant variables for predictive analytics. We have used selected features based either on their stand-alone characteristics (e.g. variance) or on their relationship with the target (using correlation or information).

In the next part (Part-2), we shall cover the classification algorithm and determine the accuracy of the developed model.

I can be reached here.

Notice: The programs described here are experimental and should be used with caution. All such use at your own risk.

References: