LSTM Network for Regression to Predict Future Prices
By: Sarit Maitra
Tags: Predictive Analytics, Timeseries Forecasting, Neural Networks, Arificialinterlligence

Prediction of stock price is quite a challenging because of highly volatile nature of time series combined with stochastic movement with non-linear . Here, the problem we have in hand is a price prediction issue and we’re trying to predict a numerical value defined in a range (from 9000 to 12500 approx). This problem fits the Regression Analysis framework. We shall be using neural network architecture to try to solve the problem here.

We’ll build a Deep Neural Network here that does some forecasting for us and use it to predict future price. Let us load the hourly frequency data.

Data loading

We have a total of 2001 data points representing Bitcoin in USD . We’re interested in predicting the closing price for future dates.

When we see a time-series, we always want to know if the value of the current time step affects the next one.

plt.figure(figsize = (15,5))

plt.plot(btc.close)

plt.title('BTC Close price (Hourly frequency)')

plt.xlabel ('Date_time')

plt.ylabel ('Price (US$')

plt.show()

We shall use LSTM network here which has the ability to capture long-term dependencies in a sequence (e.g. dependency between today‘s price and that 2 weeks ago). Moreover, uni-variate series is being used here considering only the close price from the series.

Let us normalize the price data using MinMax scaler.

Data normalization

# Feature Scaling Normalization

scaler = MinMaxScaler()

# min-max normalization and scale the features in the 0-1 range.

close_price = btc['close'].values.reshape(-1, 1)

# The scaler expects the data to be shaped as (x, y)

scaled_close = scaler.fit_transform(close_price)

scaled_close = scaled_close[~np.isnan(scaled_close)]

# removing NaNs (if any)

scaled_close = scaled_close.reshape(-1, 1)

# reshaping data after removing NaNs

Data processing for LSTM

LSTMs require 3-D data shape; therefore, we need to split the data into the shape of : [batch_size, sequence_length, n_features]. We also want to save some data for testing.

Let’s build some sequences. Sequences work like walk forward validation approach, where initial sequence length will be defined and subsequently will be shifting one position to the right to create another sequence. This way the process is repeated until all possible positions are used.

SEQ_LEN = 100

# creating a sequence of 100 hours at position 0.

def to_sequences(data, seq_len):

d = []

for index in range(len(data) - seq_len):

d.append(data[index: index + seq_len])

return np.array(d)

def preprocess(data_raw, seq_len, train_split):

data = to_sequences(data_raw, seq_len)

num_train = int(train_split * data.shape[0])

X_train = data[:num_train, :-1, :]

y_train = data[:num_train, -1, :] X_test = data[num_train:, :-1, :]

y_test = data[num_train:, -1, :] return X_train, y_train, X_test, y_test """Walk forward validation:

Initial SEQ_LEN is defined above, so, walk forward will be shifting one position to the right and create another sequence.

The process is repeated until all possible positions are used.""" X_train, y_train, X_test, y_test = preprocess(scaled_close, SEQ_LEN, train_split = 0.95)

# 5% of the data saved for testing. print(X_train.shape, X_test.shape) """Our model will use 1805 sequences representing 99 hours of Bitcoin price changes each for training. We shall be predicting the price for 96 hours in the future"""

Building LSTM model

We will create a 3 layer LSTM Network using Dropout with a rate of 20% to control over-fitting during training.

DROPOUT = 0.2

# 20% Dropout is used to control over-fitting during training

WINDOW_SIZE = SEQ_LEN - 1

model = keras.Sequential() # Input layer

model.add(Bidirectional(LSTM(WINDOW_SIZE, return_sequences=True), input_shape=(WINDOW_SIZE, X_train.shape[-1]))) """Bidirectional RNNs allows to train on the sequence data in forward and backward direction.""" model.add(Dropout(rate=DROPOUT)) # 1st Hidden layer

model.add(Bidirectional(LSTM((WINDOW_SIZE * 2), return_sequences = True)))

model.add(Dropout(rate=DROPOUT)) # 2nd Hidden layer

model.add(Bidirectional(LSTM(WINDOW_SIZE, return_sequences=False))) # output layer

model.add(Dense(units=1)) model.add(Activation('linear')) """Output layer has a single neuron (predicted Bitcoin price). We use Linear activation function which activation is proportional to the input.""" BATCH_SIZE = 64

model.compile(loss='mean_squared_error', optimizer='adam')

history = model.fit(X_train, y_train, epochs=50, batch_size=BATCH_SIZE, shuffle=False, validation_split=0.1)

# shuffle not advisable during training of Time Series

We can use callbacks option during training to prevent our model from over-fitting too; I have not used call-back, but same can be applied during model fitting phase.

Model Evaluation

A simple way to understand the training process is view the training and validation loss.

# history for loss

plt.figure(figsize = (10,5))

plt.plot(history.history['loss'])

plt.plot(history.history['val_loss'])

plt.title('model loss')

plt.ylabel('loss')

plt.xlabel('epoch')

plt.legend(['train', 'test'], loc='upper left')

plt.show()

Here, we can see some improvement in both the training error and on the validation error.

Testing

We have some additional data left for testing purpose. Let’s get the predictions from the model using those data to validate the goodness of fit of our model.

# prediction on test data

y_pred = model.predict(X_test)

# invert the test to original values

y_test_inverse = DataFrame(scaler.inverse_transform(y_test))

# assigning datetime

y_test_inverse.index = btc.index[-len(y_test):]

print('Test data:',)

print(y_test_inverse.tail(3)); print(); # invert the prediction to understandable values

y_pred_inverse = DataFrame(scaler.inverse_transform(y_pred))

# assigning datetime

y_pred_inverse.index = y_test_inverse.index

print('Prediction data:',)

print(y_pred_inverse.tail(3))

Accuracy metrics

print(f'MAE {mean_absolute_error(y_test, y_pred)}')

print(f'MSE {mean_squared_error(y_test, y_pred)}')

print(f'RMSE {np.sqrt(mean_squared_error(y_test, y_pred))}')

print(f'R2 {r2_score(y_test, y_pred)}')

RMSE allows us to penalize points further from the mean. Though the error scores are low but looking at R2 score, there is definitely room for improvement here. The model can be tuned to get better output.

Performance visualization

plt.figure(figsize = (15,5))

plt.plot(y_test_inverse)

plt.plot(y_pred_inverse)

plt.title('Actual vs Prediction plot (Price prediction model)')

plt.ylabel('price')

plt.xlabel('date')

plt.legend(['actual', 'prediction'], loc='upper left')

plt.show()

Looks like our basic and brief model has been able to capture the general pattern of the data. However, it failed to capture stochastic movements, which may be a good sign in terms of we can say that, it generalizes well.

Conclusion & future scope

Predicting stock market returns is a challenging task due to consistently changing stock values which are dependent on multiple parameters which form complex patterns.

Future direction could be:

analyzing the correlation between different cryptocurrencies and how would that affect the performance of our model. adding features using technical analysis to check the model performance. adding features from fundamental analysis to check how those affect the mode. adding sentiment analysis from social networking e.g. twitter and new report to check model performance. GRU network can also be tried with different activation e.g. ‘softsign” to check the performance.

Multivariate analysis of this nature require a great amount of effort and time on feature engineering, data analysis, model training etc.

Connect me here.

References:

Vijh, M., Chandola, D., Tikkiwal, V. A., & Kumar, A. (2020). Stock Closing Price Prediction using Machine Learning Techniques. Procedia Computer Science, 167, 599–606. Vanelin Valkov (2015). Hackers guide to machine learning

Note: The programs described here are experimental and should be used with caution for any commercial purpose. All such use at your own risk.