Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 4 (March 2018) 260–264___________________________________________________________________________________________________________Forecasting Real Time Series Data using Deep Belief Net and Reinforcement Learning
Takaomi Hirata1, Takashi Kuremoto2, Masanao Obayashi2, Shingo Mabu2, Kunikazu Kobayashi3
1

Graduate School of Science and Engineering, Yamaguchi University, Tokiwadai 2-16-1,
Ube, Yamaguchi 755-8611, Japan
2
Graduate School of Science and Technology for Innovation, Yamaguchi University, Tokiwadai 2-16-1,
Ube, Yamaguchi 755-8611, Japan3

School of Information Science and Technology, Aichi Prefectural University, 1522-3 Ibaragabasama,Nagakute, Aichi 480-1198, Japan
E-mail: {v003we, wu, m.obayas, mabu}@yamaguchi-u.ac.jp, kobayashi@ist.aichi-pu.ac.jp

Abstract
Hinton’s deep auto-encoder (DAE) with multiple restricted Boltzmann machines (RBMs) is trained by theunsupervised learning of RBMs and fine-tuned by the supervised learning with error-backpropagation (BP).
Kuremoto et al. proposed a deep belief network (DBN) with RBMs as a time series predictor, and used the same
training methods as DAE. Recently, Hirata et al. proposed to fine-tune the DBN with a reinforcement learning (RL)
algorithm named “Stochastic Gradient Ascent (SGA)” proposed by Kimura & Kobayashi and showed the priority
to the conventional training method by a benchmark time series data CATS. In this paper, DBN with SGA isinvested its effectiveness for real time series data. Experiments using atmospheric CO2 concentration, sunspot
number, and Darwin sea level pressures were reported.
Keywords: deep learning, restricted Boltzmann machine, stochastic gradient ascent, reinforcement learning, errorbackpropagation

algorithm adjusting the output of an agent or system(learner) utilizing the rewards/punishment from theenvironment. , has been studied for decades [3] [4].
Recently, RL is also introduced into deep neuralnetworks (DNNs)[5]-[7]. In [5], a deep Q-network(DQN) is proposed and applied to game (namedATARI) control and reached human level. In [6], acomputer software named AlphaGo, using a DNN andRL, won the world champion of the game Go. In [7], weadopted a policy gradient RL algorithm [8] [9] into adeep belief net (DBN) proposed by Kuremoto et al.
[10]-[13] instead of the conventional fine-tuning method
BP. And using a benchmark data CATS which was usedby time series forecasting competition with ANNs [14][15], the DBN with RL showed the highest prediction
precision comparing to all conventional methods

1. Introduction
Deep learning (DL) is the novel kernel technique ofartificial intelligence (AI) developed rapidly innowadays. As the training method of artificial neuralnetworks (ANNs), in 2006, DL firstly is introduced byHinton’s deep auto-encoder (DAE) [1], which has
multiple stacked restricted Boltzmann machines(RBMs). The learning process of DAE is divided into
two phases: firstly, pretraining, which is a kind of
unsupervised learning using the gradient of network
energy of RBMs, and secondly fine-tuning using thesupervised learning: error-backpropagation (BP) [2].
To deal with the adaptive behavior acquisition
problem in unknown environment, reinforcement
learning (RL), which is a kind of machine learning

Copyright © 2018, the Authors. Published by Atlantis Press.
This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
260

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 4 (March 2018) 260–264___________________________________________________________________________________________________________2.1. DBN with RBMs and MLP

utilized in the competition and the conventional DBN
with BP learning [12] [13].
In this paper, we concentrate to investigate theeffectiveness of the DBN with RL for real time series
forecasting. Three kinds of real time series data which
are weekly average of CO2 concentration in atmosphereat Hawaii, monthly average of sea level pressures atDarwin, the number of sunspot monthly provided byAalto University [16] were used in the forecastingexperiments, and the prediction precision was comparedto the conventional BP learning method. And as theresults, DBN with RL showed the higher performance
than the conventional DBN with BP in the process offine-tuning of the network for these real data forecasting.

As a neural predictor model, a DBN composed bymultiple RBMs and a MLP is shown in Fig. 1 [7] [12][13]. The visible layer of RBM 1 (1st Layer) are input
with raw data of time series data (omitted in the figure).
The hidden layer of RBM L+1 are used as the input
layer of the MLP, where l = 1, 2, …, L. The output of
MLP is with one unit in the case of DBN using BP
learning, and it has two units which are parameters ofGaussian distribution function used in the case of SGA
learning method [4] [7].
2.2. BP learning for DBN
Let E is the mean squared error (MSE) between theFig. 1. A structure of DBN composed by RBMs and a MLP

2. DBN with BP
output of DBN and the teacher signal, the weight ofconnections w between layers of RBMs and MLP, andthe bias of RBMs b are modified as following.

In [10] [11], Kuremoto et al. firstly applied Hinton &Slakhutdinov’s deep belief net (DBN) with restricted
Boltzmann machines (RBMs) to the field of time series
forecasting. In [12] and [13], Kuremoto, Hirata, et al.
constructed a DBN with RBMs and a multi-layer
perceptron (MLP) to improve the previous time series
predictor with RBMs. In [7], Hirata et al. adopted a RL
method named “stochastic gradient ascent” (SGA)
proposed by Kimura & Kobayashi [8] [9] into DBN
instead of the BP learning used in the fine-tuning of thenetwork. In this section, the structure of DBN andlearning methods are introduced.

newwijlold wijl  Eold

wijl

old

 wijl   (j261Ewijl  2wijl 1 )  (1  h lj )  vil
old

old

(1)

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 4 (March 2018) 260–264___________________________________________________________________________________________________________blj

newold blj   b lj

old

E
blj

Kobayashi suggested to adopt the inertia of CE andnamed the learning method as “SGA” and applied it tocontrol problem in the continuous space [8].

old  (jE
b lj 2b lj1 )  (1  h lj )oldoldD i (t )  e i (t )   D i (t  1)

(2)

(7)

 i (t )  ( rt  b ) D i (t )where a is the learning rate, i, j ∊{1, 2, …} indicate thenumber of units in layers of RBM and MLP, l is thenumber of layers of DBN.

In [4], Kuremoto et al. proposed to adopt SGA fortime series forecasting where reward was given asfollowing.
ˆ 2rt  1 if ( yt  yt )  Th  1 elseData Series

Total size

(8)

Testing size

CO2
Sea level pressure

2225
1400

225
400

DBN with BP(The number of units indifferent layers)
15-17-17-1
16-18-18-1

Sun spot number

3078

578

20-20-17-18-1

DBN with SGA(The number of units
in different layers)
20-18-7-2
16-20-8-7-2
19-19-20-10-2(9)Table 1 Sizes of time series data and structures of prediction networks

2.3. SGA learning for DBN

The SGA algorithm and the learning rule forthe weight of DBN’s layers and parameters of
the stochastic policy (Gaussian distributionfunction) were introduced in [7].
SGA is anwhere yt , yˆt ,.Th are the teach signal, the output ofnetwork, threshold, respectively.
In [7], Hirata et al. suggested to use a variable
threshold according to the change of prediction error
instead of the fixed threshold in Eq. (9).

improved RL algorithm of William’s “REINFORCE”,i.e., Reward Increment = Nonnegative Factor × Offset
Reinforcement × Characteristic Eligiblilty:wi   i (r  bi )eiei  ln g ( , w , x )wiTh   t   MSE(3)where  , MSE are constant and mean squared error ofprediction of DBN.

(4)g ( , w , x )  Pr( y   | w , x )(5)y  f (  wij x j )

(6)

(10)

3. Prediction Experiments and Results
We predicted three types of natural phenomenon timeseries data given by Aalto University [16].
 CO2: Atmospheric CO2 from continuous air
samples Weekly averages atmospheric CO2
concentration derived from continuous air samples,
Hawaii, 2225 values
 Sea level pressures: Monthly values of the Darwin
Sea Level Pressure series, 1882-1998, 1300 valuesjwhere  , w, x ,  i , r , bi , ei , y are the stochastic output, theconnection weight to unit i , the input to unit i, learning
rate, reward, bias, and Characteristic Eligibility (CE),the output of network, respectively [9]. Kimura &262

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 4 (March 2018) 260–264___________________________________________________________________________________________________________ Sunspot Number: Monthly averages of sunspot
numbers from 1749 through the present, 3078 values
Table 2 Pdiction MSE of real time series data

CO2Sea level pressure
Sun spot numberDBN withBP
0.26710.9902733.51

DBN with
SGA
0.20470.9003364.05

Fig. 2. Prediction result of CO2 data

In Table 1, the number of samples and structures ofdifferent DBNs were listed. To decide the number ofRBMs, and the number of units on different layers ofRBMs and MLP, random search (RS) [17] was used inthe experiments. As an optimization method, RS used
random values of parameter vector spaces to find thelower forecasting error (MSE).

Fig.5. Changes of evaluation function in random search(CO2, DBN with SGA)Fig.6. Changes in learning MSE (CO2, DBN with SGA)
Fig. 3. Prediction result of Sea level pressure data

In Fig. 2 to Fig. 4, the one-ahead prediction resultsof DBN with BP and DBN with SGA were shown. In
Table 2, the comparison of forecasting precision (MSE)of these different learning methods for DBN was given.
The DBN with SGA showed its priority to the DBN
with BP in all cases of real time series data.
The change of evaluation function in the case ofDBN with SGA for CO2 forecasting is shown in Fig. 5
as a sample. In Fig. 6, the change of MSE in SGA
learning process is shown. The vibration of MSE in Fig.
6 needs to be reduced by optimizing the value oflearning rate and reward, it remains as the future work.

Fig.4. Prediction result of Sun spot number data263

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 4 (March 2018) 260–264___________________________________________________________________________________________________________4. Conclusion

10. T. Kuremoto, S. Kimura, K. Kobayashi, M. Obayashi,Time series forecasting using restricted Boltzmann
machines, Proceedings of The 8th International
Conference on Intelligent Computing (ICIC 2012), 2012,
pp.17–22.
11. T. Kuremoto, S. Kimura, K. Kobayashi, M. Obayashi,Time series forecasting using a deep belief network withrestricted Boltzmann machines, Neurocomputing,Vol.137, No.5, 2014, pp.47–56.
12. T. Kuremoto, T. Hirata, M. Obayashi, S. Mabu, K.
Kobayashi, Forecast Chaotic Time Series Data by DBNs,
Proceedings of The 7th International Congress on Imageand Signal Processing (CISP 2014), 2014, pp. 1304-1309.
13. T. Hirata, T. Kuremoto, M. Obayashi, S. Mabu, Time
Series Prediction Using DBN and ARIMA, Proceedingsof The International Conference on Computer
Application Technologies (CCATS 2015), 2015，pp.2429.
14. A. Lendasse, E. Oja, O. Simula, M. Verleysen, Time
Series Prediction Competition: The CATS Benchmark,
Proceedings of The International Joint Conference on
Neural Networks (IJCNN'04), 2004, pp1615-1620.
15. A. Lendasse, E. Oja, O. Simula, M. Verleysen, Time
Series Prediction Competition: The CATS Benchmark,
Neurocomputing, Vol. 70, 2007, pp.2325-2329.
16. AML research group, The Aalto University School ofScience: http://research.ics.aalto.fi/eiml/datasets.shtml
17. J. Bergstra, Y. Bengio, Random Search for HyperParameter Optimization, Journal of Machine Learning
Research, Vol. 13, 2012, pp.281-305.

In this paper, a reinforcement learning (RL)
method “stochastic gradient ascent (SGA)” forfine-tuning of a deep belief net (DBN) withmultiple restricted Boltzmann machines(RBMs) and a multi-layer perceptron (MLP)
was compared to the conventional learning
method error backpropagation (BP) of DBN
for real time series data forecasting. Different fromthe supervised learning method which uses learning
error exhaustively, RL used a reward function whichallowed a range of errors between the output of themodel and the teach signal and it raised the forecasting
precision for the real time series data as shown as resultsof experiments.
Acknowledgements
This work was supported by JSPS KAKENHI Grant No.
26330254 and No.25330287.
References
1. G.E. Hinton, R.R. Salakhutdinov, Reducing thedimensionality of data with neural networks, Science,
Vol.313, 2006, pp. 504-507.
2. D. E. Rumelhart, G. E. Hinton, & R.J. Williams,
Learning representation by back-propagating errors,Nature, Vol. 232, No. 9, 1986, pp. 533-536.
3. R.S. Sutton, A.G. Barto: Reinforcement Learning: AnIntroduction, 1998, The MIT Press
4. T. Kuremoto, M. Obayashi, M. Kobayashi, NeuralForecasting Systems. In Reinforcement Learning, Theoryand Applications, pp. 1-20, 2008, INTECH.
5. V. Mnih, et al., Human-level control through deep
reinforcement learning, Nature, Vol. 518, 2015, pp. 529533
6. D. Silver, et al., Mastering the game of Go with deep
Neural networks and tree search, Nature, 529, 2016, pp.
484-489
7. T. Hirata, T. Kuremoto, M. Obayashi, S. Mabu, K.
Kobayashi, Deep Belief Network Using ReinforcementLearning and Its Applications to Time Series Forecasting,
Proceedings of Neural Information Processing: The 23rdInternational Conference (ICONIP 2016), Vol.3, 2016,
pp.30-37
8. H. Kimura, S. Kobayashi, Reinforcement learning forcontinuous action using stochastic gradient ascent,
Proceedings of The 5th Intelligent Autonomous Systems,
1998, pp. 288-295.
9. R.J. Williams: Simple statistical gradient-following forconnectionist reinforcement learning, Machine Learning,
Vol. 8, 1992, pp. 229-256

264